Example of a lang lexer that returns a non-zero backup distance?

Questions & Answers
1

A lang's module lexer can return a backup distance.

I would like to test my handling of that. But so far (with e.g. langs racket, rhombus, and scribble/manual) I'm always seeing a backup distance of zero.

  1. What's an example of such a #lang?
  2. What's some minimal example program in that #lang eliciting the non-zero backup?
2 Likes
2

Doing some old-fashioned searching, it looks like rhombus is one #lang that can do this.

I think a comment there provides a clue to an example test case. Assuming I can cash that in for a working example, I'll mark this self-answered. (Although if anyone knows any, I'd welcome more examples, as well as more #langs + examples.)

3

The comment I was referring to:

;; If we have "@/{" and we add a "/" after the existing one,
;; we'll need to back up more:

However when I try something like:

#lang rhombus
@/{@/{/text} text}

I still don't see the lexer return any non-zero backup amounts.

So I'm probably just being dense and I'll keep looking, but I may need help with an example after all.

4

Another example is the at-exp lexer, which is "scribble-lexer.rkt" in "syntax-color-lib". A backup is needed when | might be a fancy opener like |<<{ or might be | followed by an S-expression and a closing |.

(Sorry that I missed this question earlier!)

1 Like
5

You really should see some non-0 backups with that example. If I put (log-error ">> ~s" backup) just after lex/status within lex-all of rhombus-prototype/shrubbery/lex.rkt and then run racket -l shrubbery/parse with your input, I see backups that are 1 instead of 0.

1 Like
6

Thank you! My goal was to test I was using the non-zero backup correctly --- but it would probably also help if I actually get it correctly in the first place. So that's useful to to know, and I'll fix that problem instead/first. :slightly_smiling_face:

7

When I log all the tokens from the lexer I do see this, too.

I should have started there -- simple case.

Instead I had tried to jump ahead to testing an update: I had been logging only the use of tokens to adjust the starting position. That's fine but the test needs to supply a position corresponding to such a non-zero backup token. I made a mistake with that, ergo saw nothing.

1 Like