Racket Mode users: Want to try "racket-hash-lang-mode"?

Show & Tell
1

I have a long-running topic branch, hash-lang, on which I've been exploring how to use #lang supplied colors and indent.

Instead of this dragging on for N more years, I'd really like to merge it soon. I've been working on it a lot the last couple months.

Why so long

My main concerns have been around: performance; the need for a live back end racket process just to edit a file; how to fit in with the rest of the Emacs ecosystem that only knows about Emacs "char-syntax" not tokens (e.g. still using things like paredit when the lang uses s-expressions); configuring other Emacs details based on the module language; coordinating with @robby to add a lang info key about comments; contemplating how much "plainer" the token coloring is vs. "classic" racket-mode; miscellaneous other questions and anxieties.

The latest approach

Where I've ended up is that the "classic" Emacs-implemented racket-mode will remain. There's a new, alternative major mode, racket-hash-lang-mode, for edit buffers. People can use either mode for various kinds of files as they prefer. Can even switch between major modes in a live buffer.

I wrote an overview, "Which major mode to use" for my Guide docs. (After merge, that would be live on racket-mode.com.)

If you want to try

A few brave people have been trying it from time to time, and reporting issues.

I do have two open issues I can't reproduce, which are my last speed bumps to merging.

Although I'm OK merging it with an "Experimental!" label, I'd love to get a few more testers, first.

Would you like to try?

If you use something like straight.el for packages, you already know what to do.

Or if you've installed Racket Mode from MELPA, then the instructions are something like:

  1. M-x package-delete <RET> racket-mode.
  2. git clone https://github.com/greghendershott/racket-mode.git
  3. cd there and git checkout hash-lang
  4. In your Emacs init file, add that dir to your load-path. If you use use-package, change the :ensure t line to :load-path <dir>. Otherwise (add-to-list 'load-path <dir>).
  5. Restart Emacs

(Someday to revert, just undo step 4, and package-install racket-mode again.)

If you have trouble, ask here or open an issue.

7 Likes
2

Thanks for this!

I think I've got the branch installed properly, but when I try M-x racket-hash-lang-mode, all of the buffers I've tried so far just loose all syntax coloring. Other things, like "macro expand file" and "run in REPL and switch to REPL", continue working.

I'm not sure how to get more detail for debugging.

In my init.el I have:

(use-package racket-mode
  :mode "\\.zuo\\'")

But on the machine where I am trying this, I install almost all of my Emacs packages with Guix, for which I've used this definition:

(define-public emacs-racket-mode-next
  (let ((commit "65ebeb378f4be1bde38894a9f14bf1a83a37a5d3")
        (revision "8"))
    (package
      (inherit (@ (gnu packages emacs-xyz) emacs-racket-mode))
      (version (git-version "0.0.2" revision commit))
      (source
       (origin
         (method git-fetch)
         (uri (git-reference
               (url "https://github.com/greghendershott/racket-mode")
               (commit commit)))
         (sha256
          (base32 "0ssfi9r6qq9yd977qad69m8ybcql6jfhxfx5hqy8pvj5as0ahk0l"))
         (file-name (git-file-name "emacs-racket-mode" version)))))))
3

A configuration tip:

In your Emcas init file (~/.emacs.d or ~/.emacs.d/init.el) add the following:

  (require 'racket-hash-lang)

The require might be unnecessary when installed from MELPA due to autoloads processing, but you'll need it working from source.

You may also want what I'm using to dog-food racket-hash-lang-mode lately to edit .rkt, .scrbl, and .rhm files, all three:

  (add-to-list 'auto-mode-alist '("\\.rkt\\'" . racket-hash-lang-mode))
  (add-to-list 'auto-mode-alist '("\\.scrbl\\'" . racket-hash-lang-mode))
  (add-to-list 'auto-mode-alist '("\\.rhm\\'" . racket-hash-lang-mode))

This makes the files open initially in racket-hash-lang-mode. Once open, you can M-x racket-mode and M-x racket-hash-lang-mode to switch between them.

Another note: If the Racket Mode back end isn't yet running, when you open a buffer using racket-hash-lang-mode: Initially font-lock will be disabled and the file will be read-only, until the back end finishes starting. There's a message about this in the header line. This is some of the stuff I've had to work through, w.r.t. buffers needing some back end state that is running the color-lexer etc.

4

When you open a .zuo or other file, look at the buffer's mode line at the bottom. If it says "Racket", that's racket-mode. If it says #lang, that's racket-hash-lang-mode. Which do you see?

Regardless, if you manually do M-x racket-hash-lang-mode, you should see #lang in the mode line. Do you?

When I open main.zuo from the Racket source, I do get syntax coloring. However racket-xp-mode (which I also have enabled) shows me an error message:

standard-module-name-resolver: collection not found
  for module path: zuo/lang/reader
  collection: "zuo/lang"
  in collection directories:
   /home/greg/.racket/development/collects
   /home/greg/src/racket-lang/racket/collects/
   ... [189 additional linked and package directories]

So maybe in my case I'm getting some default color-lexer behavior. Whereas you're getting whatever the zuo lang provides -- and maybe there's a problem there. I guess I'll try to install whatever I have missing and see if I can reproduce.

5

I have added this for now, though I haven't noticed a difference from having done so, and I would expect the compilation that Guix's emacs-build-system does to be equivalent to MELPA.

I would expect there to be issues with #lang zuo files because there isn't an implementation of Zuo in Racket currently. (I know of @samth's GitHub - samth/zuo, and I've experimented with one, but this is in fact the reason why I've been using racket-mode for Zuo files, whereas I mostly use DrRacket for Racket.) I haven't tried racket-hash-lang-mode with Zuo yet.

For a concrete example, I tried editing this file:

#lang info
(define pkg-name "roman-numeral")
(define collection "roman-numeral")
(define deps '(("base" #:version "6.4")
               "rackunit-lib"))
(define build-deps '("scribble-lib" "racket-doc"))
(define scribblings '(("scribblings/roman-numeral.scrbl" ())))
(define pkg-desc "Utilities for parsing and writing Roman numerals")
(define version "0.1")
(define pkg-authors '(philip))

Initially I see Racket, as expected, and it changes to #lang after I do M-x racket-hash-lang-mode.

What's happening for me is that, with the first file I try after restarting Emacs:

  1. The message about the backend starting appears and all colors go away; then
  2. The message disappears (and the backend appears to be running), but the colors do not come back.

If I then open another file in the same Emacs session—to be concrete, the adjacent main.rkt—I initially have colors from racket-mode, but doing M-x racket-hash-lang-mode makes all the colors go away, with nothing about the backend this time.

In both cases, these messages fly by quickly:

You can run the command ‘racket-hash-lang-mode’ with M-x r-h-m
Showing all blocks ... done

Hope this is useful! And I'm happy to move to a GitHub issue if you prefer.

6

Thanks for the information!

It sounds like you're experiencing racket-hash-lang-mode not highlighting buffer with font-lock · Issue #642 · greghendershott/racket-mode · GitHub.

Could you maybe please chime in there?

  1. They mentioned that indent does work for them. It would be interesting to know for you, too.

  2. If you move someplace you expect color -- say within a string value -- and press C-u C-x = it will open a buffer with properties about the char at point. If you could copy that for me to see, that would be useful.

Thank you!!

1 Like
7

I switched to using racket-hash-lang-mode and I also did notice a problem with syntax highlighting, although not sure if it is a defect: the source code changes are highlighted only if the buffer contains a full valid Racket program, which is not the case while you type the Racket code.

It seems that updates happen too slowly. For example, I was able to type all the code from "(define saved-border..." onwards and hash-lang mode did not update the highlighting until the function items-valid? was completed and all errors were fixed.

824×389 37.7 KB

As a separate issue, while the racket source was invalid, adding comments using Alt-; (comment-dwim) added the comment incorrectly. A single ";;" was added right after the code, instead of a right-aligned single ';', also pressig Alt-; again on a line with a comment added a new comment at the end of the comment, which is not how the function is supposed to behave.

Alex.

8

There are two levels of coloring happening.

A lang can supply a "color-lexer". On the one hand a lexer is fast, and only cares about errors like mismatched string quotes (not, say, mismatched parens). On the other hand, a lexer only knows about tokens -- strings, numbers, comments, etc.

So the basic coloring, similar to what you'd see in DrRacket, is just tokens. It is fairly plain.

In your example screenshot I do see numbers colored, so I think that's working?

Because I prefer more "garish" colors, I added the ability for racket-xp-mode to contribute more colors. Basically, each end of an arrow you'd see in DrRacket (binding or use) times each kind of arrow (lexical, imported, imported from module-language) = six faces. You can customize these faces, but the defaults emphasize binding (definition or require) sites, as well as uses of imports. This is roughly "as colorful" as classic racket-mode, but not exactly the same. This is a relatively recent experiment, last couple weeks, and may need some fine-tuning. I think it works OK-ish for racket and scribble files I've dog-fooded; @samth has pointed out to me it's not ideal for rhombus import aliases.

Some people would call this "semantic" highlighting, as opposed to the fast lexer "syntax" highlighting.

Anyway, the racket-xp-mode check-syntax takes longer -- so the colors it contributes show up later. Also an expansion error can mean it doesn't understand enough to contribute any colors.

Although I tried to explain some of this in the doc strings, and in the "guide" explanation I linked to above, maybe it's still not clear. Even if it's clear, it still may feel surprising and not satisfactory? Also maybe I'm misunderstanding and you're experiencing something else?

9

Yes, your explanation is clear now, and it seems that it is intended behavior.

When I used hash-lang mode yesterday for a few hours, I had large sections of the buffer partially highlighted, and this created a very strange effect. Perhaps this is a side effect of how I write code, where my program remains invalid for large amounts of time while I write it, but I felt unproductive while using hash-lang mode, so this approach is not really for me.

Is the "Alt-;" problem I reported also a side effect of this?

Alex.

10

Sorry I glossed over that. I'll investigate. (The ability of a lang to supply info about comment delimiters, and having a few langs actually do that, is something I added in some recent PRs.)

I'm not sure if racket-hash-lang-mode is for me, either (for s-expression langs like racket). But I'm making myself use it for some real work, for awhile. Having racket-xp-mode add more, "semantic" colors was a carrot.

I think my experience has been better because: (a) I use paredit-mode so I rarely have expansion errors from even temporarily mismatched delimiters. (b) I've grown to use a racket-xp-mode as a kind of pair programming buddy; I tend to pause and see what it thinks, let it help me catch typos or mistakes.

Even if you don't prefer it, it's a kind of success or at least progress (to me) that you can decide that -- that you're not experiencing fundamental bugs (insert dancing bear adage).

1 Like
11

Thanks to everyone who had time to try this, and give feedback.

Today I updated the hash-lang branch with some more commits, which overhaul racket-repl-mode. Some source comments:

;; Traditionally a REPL's output is a hopeless mix of things dumped into
;; stdout and stderr. This forces clients to use unreliable regexps in
;; an attempt to "demux" and recover the original pieces.
;;
;; Instead we want structured output -- distinctly separated:
;;  - current-output-port
;;  - current-error-port
;;  - current-print values
;;    - strings
;;    - image files
;;  - prompts
;;  - structured errors from error-display-handler
;;  - various messages from the back end

In addition, there's no longer a TCP connection to do I/O for each REPL. Instead the I/O is done a structured data over the main command/response/notification channel. This simplifies configuration, especially when working with remote back ends.

This change fixes a few problems unrelated to working with hash langs, but they weren't (IMHO) urgent to fix. Colorizing the entire REPL using classic racket-mode rules, as if it were all racket-lang values, was never really correct, but mostly harmless. But it's pretty bad when the REPL is for a lang that's nothing like racket-lang.

For awhile on the hash-lang branch, I replaced the "too hot bowl" (format everything as racket-lang) with a "too cold bowl" (treat everything plainly). Now it's a warm bowl: current-print values and interaction inputs after a REPL prompt get treated as whatever lang is currently using the REPL. Everything else gets distinct, user-customizable colors (e.g. one for stdout, one for stderr, etc.)

I think this is a good change even when using classic racket-mode. As well as being pretty much necessary for racket-hash-lang-mode, which is why it's on the hash-lang branch.

If anyone has time to (re)try this hash-lang branch, please let me know and thanks in advance!

I'm still aiming to merge this to the main branch in the near future.

3 Likes
12

I’m enjoying the hash-lang branch so far! For the context, I use it primarily for Rhombus.

13

I merged this to the main branch.

Because it was close to 250 commits spanning several years, sometimes wandering among different approaches, I squashed it down to one mega commit.

(I left the hash-lang branch un-squashed in case the history turns out to be helpful. But if you were using this via straight.el or similar, please point back to the main branch master, now.)

I'm grateful to everyone who had time to try it, and supply such high quality bug reports. Thank you!

I think it got enough use that it won't introduce any super serious problems (but we'll see). Anyway I believe it reached the point where, if there's more to learn, that will come from even more people using it. Ergo the merge.

5 Likes
14

p.s. The online documentation is refreshed. This is a "User Guide" style section, followed by copies of the doc strings available within Emacs for the commands, customization variables, and faces.

p.p.s. I rebased the pdb branch onto master, so if you feel under-stimulated using a boring main branch again :stuck_out_tongue: you're welcome to work off pdb.

Seriously you can try racket-pdb-mode in combination with racket-hash-lang-mode now, as well as racket-mode.

(These two branches -- hash-lang and pdb -- have been long-running in parallel for a very long time. That was another big motivation to merge the former, rather than continuing to juggle both. Whew.)

15

If anyone uses Doom Emacs, I would be very glad if you could add a +hash-lang mode to support this new major mode to the Racket Mode plugin!

(I myself couldn't get Racket Mode to work with Doom Emacs even until now... Help me, you're my only hope!)

1 Like
16

Despite your super high quality bug report I haven't been able to figure this out so far. :disappointed: I'm willing to keep trying if/when you have time to pursue with me again. It's probably related to one of the minor modes shown as enabled in your report. We could try disabling some to narrow it down.

17

I created a PR to support racket-hash-lang-mode in Doom Emacs: https://github.com/doomemacs/doomemacs/pull/7543.

Currently the Doom Emacs maintainer appears to be on vacation, so it might take a while to get this merged.

2 Likes