# Example of a lang lexer that returns a non-zero backup distance?

**URL:** <https://racket.discourse.group/t/example-of-a-lang-lexer-that-returns-a-non-zero-backup-distance/437>\
**Category:** Questions & Answers\
**Created:** [December 17, 2021, 4:48pm UTC](https://racket.discourse.group/t/example-of-a-lang-lexer-that-returns-a-non-zero-backup-distance/437 "2021-12-17T16:48:54Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![greghendershott](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/greghendershott/32/98_2.png) [@greghendershott](https://racket.discourse.group/u/greghendershott)\
**Post date:** [December 17, 2021, 4:48pm UTC](https://racket.discourse.group/t/example-of-a-lang-lexer-that-returns-a-non-zero-backup-distance/437/1 "2021-12-17T16:48:54Z")

</div>

A lang's [module lexer can return a backup distance](https://docs.racket-lang.org/syntax-color/index.html#%28def._%28%28lib._syntax-color%2Fmodule-lexer..rkt%29._module-lexer%29%29).

I would like to test my handling of that. But so far (with e.g. langs `racket`, `rhombus`, and `scribble/manual`) I'm always seeing a backup distance of zero.

1. What's an example of such a `#lang`?
2. What's some minimal example program in that `#lang` eliciting the non-zero backup?

---

<div class="post-metadata">

**Author:** ![greghendershott](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/greghendershott/32/98_2.png) [@greghendershott](https://racket.discourse.group/u/greghendershott)\
**Post date:** [December 23, 2021, 4:26pm UTC](https://racket.discourse.group/t/example-of-a-lang-lexer-that-returns-a-non-zero-backup-distance/437/2 "2021-12-23T16:26:26Z")

</div>

Doing some old-fashioned searching, [it looks like `rhombus` is one `#lang` that can do this](https://github.com/mflatt/shrubbery-rhombus-0/blob/master/shrubbery/lex.rkt#L376-L396).

I think a comment there provides a clue to an example test case. Assuming I can cash that in for a working example, I'll mark this self-answered. (Although if anyone knows any, I'd welcome more examples, as well as more `#lang`s + examples.)

---

<div class="post-metadata">

**Author:** ![greghendershott](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/greghendershott/32/98_2.png) [@greghendershott](https://racket.discourse.group/u/greghendershott)\
**Post date:** [December 23, 2021, 4:40pm UTC](https://racket.discourse.group/t/example-of-a-lang-lexer-that-returns-a-non-zero-backup-distance/437/3 "2021-12-23T16:40:29Z")

</div>

The comment I was referring to:

> ```scheme
> ;; If we have "@/{" and we add a "/" after the existing one,
> ;; we'll need to back up more:
> 
> ```

However when I try something like:

```scheme
#lang rhombus
@/{@/{/text} text}

```

I still don't see the lexer return any non-zero `backup` amounts.

So I'm probably just being dense and I'll keep looking, but I may need help with an example after all.

---

<div class="post-metadata">

**Author:** ![mflatt](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/mflatt/32/6_2.png) [@mflatt](https://racket.discourse.group/u/mflatt)\
**Post date:** [December 23, 2021, 4:40pm UTC](https://racket.discourse.group/t/example-of-a-lang-lexer-that-returns-a-non-zero-backup-distance/437/4 "2021-12-23T16:40:48Z")

</div>

Another example is the at-exp lexer, which is "scribble-lexer.rkt" in "syntax-color-lib". A backup is needed when `|` might be a fancy opener like `|<<{` or might be `|` followed by an S-expression and a closing `|`.

(Sorry that I missed this question earlier!)

---

<div class="post-metadata">

**Author:** ![mflatt](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/mflatt/32/6_2.png) [@mflatt](https://racket.discourse.group/u/mflatt)\
**Post date:** [December 23, 2021, 4:45pm UTC](https://racket.discourse.group/t/example-of-a-lang-lexer-that-returns-a-non-zero-backup-distance/437/5 "2021-12-23T16:45:15Z")

</div>

> [@greghendershott](#):
>
> ```scheme
> #lang rhombus
> @/{@/{/text} text}
> 
> ```

You really should see some non-0 backups with that example. If I put `(log-error ">> ~s" backup)` just after `lex/status` within `lex-all` of `rhombus-prototype/shrubbery/lex.rkt` and then run `racket -l shrubbery/parse` with your input, I see backups that are 1 instead of 0.

---

<div class="post-metadata">

**Author:** ![greghendershott](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/greghendershott/32/98_2.png) [@greghendershott](https://racket.discourse.group/u/greghendershott)\
**Post date:** [December 23, 2021, 4:48pm UTC](https://racket.discourse.group/t/example-of-a-lang-lexer-that-returns-a-non-zero-backup-distance/437/6 "2021-12-23T16:48:56Z")

</div>

Thank you! My goal was to test I was _using_ the non-zero backup correctly --- but it would probably also help if I actually get it correctly in the first place. So that's useful to to know, and I'll fix that problem instead/first. 🙂

---

<div class="post-metadata">

**Author:** ![greghendershott](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/greghendershott/32/98_2.png) [@greghendershott](https://racket.discourse.group/u/greghendershott)\
**Post date:** [December 23, 2021, 5:06pm UTC](https://racket.discourse.group/t/example-of-a-lang-lexer-that-returns-a-non-zero-backup-distance/437/7 "2021-12-23T17:06:56Z")

</div>

When I log all the tokens from the lexer I do see this, too.

I should have started there -- simple case.

Instead I had tried to jump ahead to testing an update: I had been logging only the use of tokens to adjust the starting position. That's fine but the test needs to supply a position corresponding to such a non-zero backup token. I made a mistake with that, ergo saw nothing.