Color lexer backup distances and `dont-stop`

Questions & Answers
3

It sounds to me like you understand @LiberalArtist .

  1. I would say that dont-stop is useful when you want to color some region in two (or more) different colors but you cannot parse them independently. So, you parse them both then return the first token with a dont-stop and then return the second one. It is shoehorning the ability to return lists of tokens into a flat-style API. (Hope this makes sense.)

  2. I think that the backup distance is a separate thing from the dont-stop result. It may be that you can use one to get the same effect as the other, tho, I'm not sure. Maybe if you have a specific example I can answer this one better. (@mflatt added the backup distance to support the scribble lexer, tho, so he may have a better answer here.)

  3. Sure. The main thing is to keep tokens relatively small; they don't have to be tiny (for example, strings are single tokens in #lang racket and then can, in principle, span a giant file) but the size of the token controls the interactivity when editing the files in DrRacket. Smaller and more tokens means more opportunities for DrRacket to respond to the user's editing events, as DrRacket won't interrupt the colorer in the middle of computing a single token.

I think @greghendershott 's advice is good, tho. Processing an entire line at once and putting the processing result into the mode and then returning a sequence of tokens via dont-stop until the end of the line seems like a (relatively?) easy way to get something going.

hth!