Example of a lang lexer that returns a `dont-stop` struct?

Questions & Answers
1

A lang's module lexer can return a dont-stop struct.

I would like to test my handling of that. But so far (with e.g. langs racket, rhombus, and scribble/manual) I'm never seeing a dont-stop.

  1. What's an example of such a #lang?
  2. What's some minimal example program in that #lang eliciting the dont-stop value?
2 Likes
2

I don't know whether this question is still relevant, but here's a small example using the shrubbery lexer:

(require shrubbery/syntax-color)
(define in (open-input-string "@/{}")) ;; extra "/" would change "@" treatment
(shrubbery-lexer in 0 #f)
;; last result is `(dont-stop '#s(pending-backup-mode 1 initial))

The shrubbery lexer produces backup and dont-stop information somewhat defensively, I think, so it's not the best example of when backup and dont-stop are needed. But in the case above, I'm pretty sure dont-stop and backup really are needed.

1 Like
3

Thanks for the example.

I'd need to reload my brain both wrt the color-lexer details as well as rhombus, so meanwhile if I may ask a dumb surface-level question:

In something like

#lang rhombus
@/{}

would you expect the @ to be tokenized as "at", / as "operator", and {} as "parenthesis", and colored as such in an editor?

Whereas in

#lang rhombus
@//{}

that's an s-expression comment, with @// as "comment", and {} as "parenthesis" with a comment flag (e.g. draw in paren color but dim/faded)?

(That's what I'm seeing on my hash-lang branch and using its racket-hash-lang-mode in Emacs.

Of course, even if that's the desired end-user behavior, that doesn't necessarily mean I'm handling dont-stop optimally. IIRC I was also erring on the side of being defensive. So, someday when I have time to reload my brain, I'll dig into this again.)

p.s. Token types at and operator seem new since I last looked at the color-lexer docs. Last I looked, this seemed documented as a fixed set. Maybe now it's changed to an open-ended set, and the user just maps these in their tool -- if so, that makes sense, and I can handle that accordingly.

4

I was recalling the docs for racket-lexer. But other lexers can return a symbol from a different set -- and in fact it can now be a hash-table to allow expressing token type as well as other attributes like "comment?".

So it's open; always has been.

1 Like
5

Just to confirm, shrubbery @/{} should be tokenized as "at" then "operator" then "parenthesis", while @//{} is indeed "comment" plus "parentheses" with a comment flag. And, yes, my understanding is that the set of type symbols is open, while some symbols are specifically recognized by color:text%.