Racketeer: VS Code extension

[eric]

Racketeer is a new VS Code extension for Racket. It lets you debug, run, test, navigate and package Racket in one place, and all you need to install is Racket 9.1 or newer.

• Debug with breakpoints on lines and on the calls within a line, stepping across files and chosen packages, and live variables.
• See results beside your code: text, images, picts and plots as your program prints them.
• Test from VS Code's Testing view: run, debug or measure coverage of a file, or of a single RackUnit case.
• Work interactively in an Interactions window, with pictures inline and errors that link to their source.
• Navigate with hover documentation, completion, go to definition and live diagnostics, from the bundled racket-langserver.
• Find and rename across a project's module files, including ones that are not open.
• Expand and build with the macro stepper, package management, executables and distributions.

Racketeer's language features come from racket-langserver, which it bundles. That covers hover documentation, completion, go to definition and live diagnostics, and it is the work of that project's developers, so my thanks to them. The bundled copy carries a few local patches, and I am offering changes back upstream.

It is tested on Linux, Windows and macOS. If you have Magic Racket installed, disable it and reload the window, because Racketeer's language tools stay off while it is active.

• Install: search for Racketeer in VS Code's Extensions view, or go to the Marketplace page.
• Source and issues: GitHub - lamestllama/racketeer · GitHub

It is new, so I would really like to hear what works and what does not, especially on larger programs. Please raise problems on the issue page.

[greghendershott]

Quick glance, it seems you wrote new code to annotate/instrument code for step-debugging -- instead of the gui-debugger used in DrRacket (a slight tweak of which I use for Racket Mode).

This isn't necessarily a bad thing! Instead I'm curious why, can you say more please? Were there limitations you wanted to avoid? Or did you want the challenge of writing it out from scratch? Or did you just not know about the existing annotator code?

Again this isn't intended as a criticism -- I'm always curious and open to learning new things from other people working on tools for Racket. Thanks!

[bakgatviooldoos]

@eric

• Find and rename across a project's module files

Chef's kiss. Will definitely give this a spin. Thanks for sharing!

[eric]

Thanks for asking, and no offence taken. The main reason is that its generated code holds no debugger state. The gui-debugger annotator inserts its callbacks as 3D syntax, so annotated code can't be saved in a compiled file, and every debug run expands and annotates each selected module again. Mine is compiled to disk once and reused until a source or something it depends on changes, which makes restarts orders of magnitude quicker on large modules. I haven't looked at whether your tweak changes that.

Otherwise the core is the same idea as the annotator's: rewrite the fully expanded code, wrap expressions in before and after hooks that can skip or replace values, leave tail positions without an after, and reach live variables through closures in continuation marks. I haven't measured overall speed against it.

[bakgatviooldoos]

Hoowee! That test suite is a beast; took 1h 51m on my admittedly geriatric machine. These were the headless tests, not the second batch using Python 3 and friends.

What did the process look like for putting it all together, @eric? Was it an amalgam of existing tests, or did you have to come up with a significant amount yourself?

[damien_mattei]

i just installed your package in VS code, and seems to works but i did not test it on a real racket code instead i use one of my code in curly-infix:
the code start with a #reader SRFI-105 and it is okay to display in VS code but with no more that i already had with Magic Racket.
When i try 'debug and run' the code is well parsed by reader but do not execute. But i admit i'm pushing the things a bit far ahead expecting this would work too with personal readers.

[eric]

I have a #lang DSL which I want to eventually use it on so hopefully soon. Have you got a link to some of your code?

[JustinZed]

Can you publish this to Open VSX Registry (https://open-vsx.org/) so that people with VSCodium and other open source forks can use this?

[bakgatviooldoos]

I am using CodeOSS and it seems to work via the VSIX from the repo, for what it is worth.

Edit: file indexing kills my PC, though .

[damien_mattei]

Examples are in the Scheme+ project repository, here is one link to an example.

[eric]

@bakgatviooldoos
To limit what Racketeer's workspace index reads, list the folders to skip in the user settings or the project's .vscode/settings.json:

{
  "racket-langserver.workspaceAnalysis.excludedDirectories": [
    ".git", "compiled", "node_modules", ".test-output", "dist", "build",
    "vendor"
  ]
}

This list replaces the default, so keep the first six names. vendor is an example. Each entry is a whole folder name that matches at any depth, so globs and paths such as src/old do not work.

Other settings under racket-langserver.workspaceAnalysis:

• maxFiles: files found in each pass (default 1000)
• maxFileBytes: largest file analysed (default 1 MiB)
• timeoutSeconds: time allowed per file (default 20)
• enabled: set to false to stop analysing unopened files, which also turns off rename

Reaching maxFiles or timeoutSeconds makes Find All References report partial results. Rename leaves excluded folders unchanged.

[eric]

Now also available at Open VSX Registry

[NorahBautista]

Thanks for sharing this extension! The workspace analysis settings are especially useful, particularly the ability to exclude specific directories and control file size and timeout limits. I also appreciate the clarification that reaching the analysis limits can result in partial reference results.

Making Racketeer available on the Open VSX Registry is a great addition for developers who use alternative VS Code distributions. Great work on improving the Racket development experience!

[eric]

Thanks for trying it, and sorry it did not work for you. I looked into it in detail with SRFI-105-for-Racket 15.1 and Scheme+ 18.1 on Racket 9.1, and found the cause.

Source locations. The reader's read-syntax converts the whole parsed program in one call:

(datum->syntax #f (literal-read-syntax src in) (vector src 1 0 1 0))

datum->syntax gives every syntax object in the program that one location, line 1, column 0, with span 0. A source-level debugger depends on each expression's own location. So:

• breakpoints after line 1 never hit;
• every step and every stack frame reports 1:0;
• some locals cannot be shown, because their bindings are indistinguishable.

Go to definition and references are affected the same way.

The language itself is fine. The same Scheme+ forms (def, :=, $+>, for-basic) debug correctly when read by the plain Racket reader. A test reader that keeps each form's position makes curly-infix programs debug correctly with no change to the debugger.

Watch and console expressions (my side). Racketeer reads watch and console input with the Racket reader, so {x + 2} fails. Your reader already installs an interaction reader that handles it, and Racketeer will use the language's own interaction reader instead.

What I am doing. I am adding proper support for SRFI-105 and Scheme+ to Racketeer: debugging, editor features and the macro stepper. That includes a version of the reader that keeps source locations and reads exactly the same programs otherwise. Once it works end to end, I will offer you the reader change as a pull request. It should also help error messages, coverage and any other tool that uses source locations.

If you have programs that fail for you, I would be glad to use them as test cases.

[eric]

Thanks for the report. Racketeer 0.8.55 reworks the workspace index and should fix this; on a 53-file project a full pass is an order of magnitude fastrer. If it still fails for you, please tell me your platform and what the "Racket Language Server" output shows.

[damien_mattei]

Yes i already know this. The problem is finding the exact line between all the passes (infix,annotation,etc...) Each scheme implementation has it own way to deal with source location. So this will be special Racket code ,not portable scheme.

Okay..... we opened the "Pandora box" and i understand the analysis of your message is AI agentic generated. My code Scheme+/annot/SRFI-1* date back from another era when there was no AI agent ,so it is 95% hand written code.

But i have no skill,knowledge how Racket deals exactly with source location in Racket code, for this reason i never did it myself, it is not done, i knew it is missing and should be done in future.

What i can do : i have submit this analysis to AI agent and here it is , i read it and corrected the ambiguous or wrong part, i validate the understanding of my parser stages :

First However, I would prefer that we make the source location changes ourselves, on our side, rather than receive them as a pull request. We know
this code very well, and the change goes far beyond the reader's entry point:

"The reader does not work on syntax objects. It reads the source into plain S-expressions, and several passes then rewrite them as plain data
before anything becomes syntax:

• infix expressions are converted to prefix, by two different precedence engines, which creates forms that do not exist in the source;
• quoted data and pragmas are processed, and type annotations are removed;
• when type annotation is on (the ANNOTATION-ON pragma), a further pass specializes the code according to the types, for example (- 1.0 (** z 2)) becomes (fl- 1.0 (fl* z z)).

Only at the very end is the whole program converted with a single datum->syntax call. So keeping locations means that each form must carry
the position of its original text through all of these passes. A reader that keeps locations only for programs without type annotation would
miss the main use of Scheme+, since that mode is where the performance comes from.

There are also two practical constraints. The reader is bootstrapped: it is written in Scheme+ and compiled by itself, so the installed files
are generated. And SRFI-110 is a separate reader that needs the same work."

I completely validate the analysis above , of course it is almost all my hand written code anlyzed by IA. But i will not do the source location myself and i think it is better that i deal with my code on my side.

Eric what i propose is to come back with the source location done (if it is possible) , then you can use and test it (perheaps it will then works out of the box? with your code) .

update: replaced annotation by location in the last sentence

[damien_mattei]

the version of scheme+ with source location is now online Eric.(and also an enhancement introducing ℤ* and ℕ* types for numbers but it has no link with the source location feature)

Source location is a great enhancement as it displays now errors where they are even in a curly infix annotated code.

as a consequence it is now working with your Racketeer code in VS code, see the screenshot where i create voluntary an error of an unbound variable, when i ran the code here is the error and the exact source code line is now displayed

and hightlighted :

note: it also works now with Magic Racket too

Thank you for your work and help Eric, without you i would perhaps have not added 'source location' so early.

I let you test more if you want as i rarely use VS code , more Emacs , and in general ran Racket code in DrRacket.

[eric]

Good work adding the source code location to Scheme+ . Currently I am creating an adapter layer because some languages will require tweaking of the debugger if you want debug displays in the language rather than bare Racket.

The reason that Magic Racket and Racketeer both display the same thing in the editor is because they use the same language server GitHub - jeapostrophe/racket-langserver · GitHub. The only difference in that direction is that Racketeer adds an LSP indexer to it to allow search and replace across a project rather than only across open files.

I consider Racketeer's main contributions to be it's DAP server source code debugger (with instrumentation that is cached on disk allowing repeated debug sessions to be much faster than Dr Racket) and its close integration with VSCode native display of unit tests , macro stepping and use of WXME documents.