A Multi-Language-Oriented Macro System by Michael Ballantyne at the (fourteenth RacketCon) is now available on the Racket YouTube channel at https://youtu.be/KHxZ3mT9BSo
3 Likes
This seems very cool. In the related paper "Macros for Domain-Specific Languages", the first examples are the PEG DSL.
It seems like (require peg) gives a different grammar than the paper (documented here). That documentation is incorrect in its syntax for name binding, lacks features from the paper, gives contract violations rather than relevant syntax errors, and goes into an infinite loop on arith-expr-leftrec rather than rejecting left-recursive non-terminals as described in the paper, so is there a different package that should be used while reading the paper?
1 Like
Yeah, the package you link is an unrelated project. The PEG DSL developed in the paper is available here. Runnable versions of all the code in the paper are packaged up as an artifact archived here or available in source form here.
You might also be interested in the syntax-spec version of the PEG DSL which is adapted from the implementation in that previous paper. It's checked in here. And finally, the more recent paper on syntax-spec.
Thanks for reading, and I'd love to hear your thoughts on any of it.
3 Likes
I definitely want to try to work my way through your research as I have the time. You put your finger on an issue which I've toyed with ever since I first learned about macros: you don't always want to expand in terms of the host language's primitives.
When you look at the excellent papers, blog posts, Inside Racket videos, and patient mailing list posts from those who have produced major extensions within Racket, like Robby Findler, @samth, Alexis King, Stephen Chang, etc., it seems like the basics of producing a DSL is pretty straightforward, but the nitty gritty details still bloom into a level of complexity which isn't quite contained with a perfect abstraction. These people have pioneered different approaches. Their artifacts and strategies are fascinating. Each seems like a hard won increment of progress toward the ultimate goal of finding the best way for systems to cooperate at the level of syntax.
The mechanisms for cooperation which exist inside Racket reflect these different approaches, and they aren't perfectly synthesized and abstracted yet, so studying these systems can lead to the feeling that if you have a practical goal to achieve, it might be easier to implement your own expander, just so you know exactly what is happening. The beauty of Dybvig (et. al.'s) psyntax expander, is that it was the perfect midpoint of source code you could print, read, and run, on that spectrum that ranges from a very abstract model which might fit in the appendix of a paper, to a colossal system which you could never read in full. If you could completely understand psyntax, then you could adapt or clone it for your own language, and I imagine MzScheme must have started as something like that. Connecting to this thought, it is interesting that you describe part of the work in MfDSLs as providing the tools to implement your own expander for a DSL.
Anyway, the PLT research program just seems to keep on making progress figuring out how to take the principal of syntactic abstraction up one level from intra- to inter- language cooperation. I love how PLT has stayed focused on these questions over the decades even as the topic moves in and out of the fashion!
2 Likes
I haven't got to read much of your functional pearl yet, but I started looking at the state machine example from the docs. I've got as far as 1.2.1 Simple binding, and as expected we see the error message:
————— run main.rkt —————
main.rkt:30:14: on: not bound as event-var
at: x
in: (on (event x) (goto green))
main.rkt:30:14
Nice error "for free". An improvement would be if it mentioned Also in this section, there is a bit:green, since that is the unbound reference.
#:binding (scope (bind arg) body)
which seems to be at the wrong level of nesting in the example. Maybe it should be part of transition-spec. Also body would be unbound, I think. I left that out for now. Maybe these issues become more clear later.
Very nice work.
Oh, actually maybe I misinterpreted that error, since I was expecting an error with green. My understanding now is that this section is dealing with scoping arg:event-var.
My best guess, under my feeble 30 minutes of playing with syntax-spec is that the form should look like this:
(nonterminal transition-spec
(on (event-name:id arg:event-var ...)
action:action-spec
...
((~datum goto) next-state:id))
#:binding (scope (bind arg) action))
So body -> action and fix the nesting.
edit 3
So I see in the newer version of the docs in the repo, that you have fixed this to:
(nonterminal transition-spec
(on (event-name:id arg:event-var ...)
action:action-spec
...
((~datum goto) next-state:id))
#:binding (scope (bind arg) ... action ...))
Adding the ellipsis (versus my guess), which makes sense, although this doesn't seem to work, at least in my version of syntax-spec.
I recently released the v3 version of the package to the package server and docs server: syntax-spec: A Metalanguage for Hosted DSLs
It has some changes to the syntax as compared to that in the paper like the newly required ellipses, documented here: 1.3 Release Notes
I also fixed some bugs in the tutorial as part of the v3 release, so if you use that version of the package and docs you'll likely have a better time.
Thanks again for giving it a try!
3 Likes
Version v3 is correctly giving nice error messages:
————— run main.rkt —————
main.rkt:41:13: on: not bound as state-name
at: green
in: (on (event x) (goto green))
main.rkt:41:13
I can't say I'm in love with the syntax for scopes and binding yet, but this work is very impressive. I'm still at the beginning of the first example, and it already looks like a giant step forward.
2 Likes
For some reason, I'm encountering a scope error in the state machine example. Here is my version:
https://gist.github.com/acarrico/89590bcbf82556440404f2e1ec8b336d
For some reason the select-item argument item referenced on line 115 is not correctly scoped, even though we have #:binding (scope (bind arg) ... (import body) ...)) on line 30.
I've checked my source against the more full featured DSL in "~/.racket/8.13/pkgs/syntax-spec-v3/test/dsls/state-machine-for-tutorial.rkt", and I can't see my mistake. I know that I've made an error, since the example runs fine using that version of the machine DSL.
I can also see the scope issue stepping backward from the error in the macro stepper, but otherwise, expanding a DSL gives way too much detail for me to get a clue with the macro stepper.
I suspect that a solution to this issue will help me build understanding of the binding spec tree defined by syntax-spec before "compiling" the forms. I'm beginning to see that this stands in for the scopes added by core syntax (lambda, etc) in a normal macro expander, although I'm still hazy about this.
Incidentally, my version of the machine DSL works fine if you don't refer to the event argument, as in line 116 (vs. 115).
1 Like
Wrapping my vending machine in (module+ test ...) solves the issue. Can anyone explain what is going on?
I must be the perfect example tester: I seem to fall into every trap!
3 Likes
Hey, sorry you ran into this! I'm pretty sure I ran into the same issue but it turns out to be a bug with the Racket expander, so I couldn't immediately fix it.
The ticket for the bug as it appears in syntax-spec is here: Unbound error in definition RHS, probably involving use-site scopes? · Issue #71 · michaelballantyne/syntax-spec · GitHub
And the Racket expander ticket is here: quote-syntax pruning and syntax-local-get-shadower don't include certain use-site scopes · Issue #5212 · racket/racket · GitHub
I should really add the workaround of using module+ test to the tutorial for now though!
Matthew came up with a fix for the expander bug! It'll be in the v8.17 release though, which isn't for a while. In the meantime, I've added a note on the workaround to the tutorial, so hopefully nobody else will run into the problem. Thanks for the report.
3 Likes