Confused about setup for reader and expander for simple DSL

Questions & Answers
1

I'm working on the Beautiful Racket "wires" DSL without looking at his solution, or using the various br libraries -- I want to get things working using standard Racket and not rely on BR's syntactic sugar / setup.

I have a macro that correctly expands the DSL code into working Racket code. But I'm struggling with the boilerplate of making the wires code work with #lang wires and getting the reader and expander working.

I'm trying to get the simplest, most straightforward way of taking some wires code with #lang wires at the top and executing it.

I installed wires as a package using raco, but in main.rkt, I'm struggling with read-syntax and various module-begin stuff.

I have this right now:

#lang racket

(require racket/port)

(define (read-syntax path port)
  (define src-lines (port->lines port))
  (define src-datums
    (for/list ([line src-lines ])
      (read (open-input-string (format "(wires-operator ~a)" line)))))
  (define module-datum `(module wires "main.rkt"
                          ,@src-datums))
  (datum->syntax #f module-datum))

(provide read-syntax)


(provide (rename-out [wires-expander %#module-begin]))
(define-syntax-rule (wires-expander EXPR ...)
  #'(#%module-begin
     (displayln "wires expander ")
     EXPR ...))

And in the REPL, if I do (read-syntax "wires-test.rkt" (open-input-file "wires-test.rkt") ), I get something that looks good:

#<syntax (module wires "main.rkt"
           (wires-operator)
           (wires-operator)
           (wires-operator // I am also allowing comments!)
           (wires-operator)
           (wires-operator x AND y -> d)
           (wires-operator x OR y -> e)
           (wires-operator x LSHIFT 2 -> f)
           (wires-operator y RSHIFT 2 -> g)
           (wires-operator NOT x -> h)
           (wires-operator NOT y -> i)
           (wires-operator 123 -> x)
           (wires-operator 456 -> y))>

(Here wires-operator is my macro, and it works the want I want for those inputs.)

But if I try to execute that code from the command line, I get:

$ racket wires-test.rkt
module: no #%module-begin binding in the module's language
in: (module wires "main.rkt" (wires-operator ...

I'm confused about all these modules, module-begins, and so on.

  1. What exactly does read-syntax need to put in its output? What kind of module should it output?
  2. How do I provide the expander for that, so Racket can take the read-syntax output, run it to convert it into valid Racket code, and execute that? I thought I was doing that with my provide.
2

TL;DR: you probably don’t want a relative module path in the initial import section of your generated module form. Instead, try (module <any-name-here> wires ,@src-datums).

3

I blindly tried that, and I get the same behavior in the REPL (it works and produces the expected syntax object) but now on the command line I get:

default-load-handler: expected a `module' declaration, but found something else
  file: ..../wires-test.rkt

that's mysterious -- the returned syntax object, AFAICS, is nothing but a module declaration! It's effectively just what I have above.

4

I do see one tiny typo - in my "require provide out", I transposed the # and % for module-begin. But fixing that doesn't change anything, so it does seem like Racket isn't connecting the output from read-syntax to the module-begin thing, which, to my understanding, is what will actually run the macro that, well, expands the code into actual Racket code.

5

wires/main.rkt needs to include a reader submodule that connects read-syntax to the #lang, like this code:

In BR "THE READER" section, the (module reader ... is not specific to #lang br/quicklang, but also for any #lang setup.

The syntax/module-reader language in 17.3.3 in Racket Guide is helpful for writing such reader submodules. That is, rather than (module reader racket (provide (rename-out [my-read-syntax read-syntax])) ...), it's likely more convenient to write (module reader syntax/module-reader ...)

1 Like
6

I copied the literal stuff from there, and that example works. (So does the "dollar" stuff in that part of the documentation.)

When I have

(module reader syntax/module-reader

which you say is more convenient, I get an unbound identifier for port->lines, even if I require racket/port inside the module.

What's more convenient about syntax/module-reader?

7

I've made some progress, but now I'm running into this strange error: my reader seems to work, but with this code for the expander:

(provide (rename-out [wires-expander #%module-begin]))
(define-syntax-rule (wires-expander EXPR ...)
  #'(#%module-begin
     (displayln "wires expander ")
     EXPR ...))

when I run my test file, I get:

main.rkt:28:2: module: expansion of #%module-begin is not a #%plain-module-begin form
  at: (t-quote-syntax (#%module-begin (displayln "wires expander ") (wires-operator) ... (wires-operator x LSHIFT 2 -> f)...

The documentation for plain-module-begin isn't helpful. What do I need to do here to get that to expand correctly?

8

Not @shhyou, but from my limited personal experience, when I want to mess around with the reader, it is very convenient to have a setup like the following:

main.rkt ;; for testing the system
syntax.rkt ;; whatever transformers I need to use during expansion
reader.rkt ;; the linking up I think you're busy with here

In particular, syntax/module-reader can be used like so:

;; reader.rkt
#lang s-exp syntax/module-reader
racket/base
#:read my-read
#:read-syntax my-read-syntax
 
... rest of your definitions ...

which can then be used like so:

;; main.rkt
#lang reader "reader.rkt"
(require
  "syntax.rkt") ;; for argument's sake

This all comes from the same set of example material you are looking at, I think, if you are referencing the dollar guides.

17.3.3 Using #lang s-exp syntax/module-reader

I hope that helps a bit, I found it quite useful once I got the basic parts working, although I must admit I am hazy on the precise details.

9

I copied your setup -- and it works, mostly! Now I get an error about port->lines being an unbound identifier. Any ideas about that?

With a different setup, it works...mostly.

Now my problem is this mysterious one: when I run my test file, I get:

wires-operator: bad syntax          
  in: (wires-operator x AND y -> d)   
  context...:             
   /usr/share/racket/collects/syntax/wrap-modbeg.rkt:46:4

This is weird, because handling that syntax is exactly what the wires-operator macro does! In the REPL, it works perfectly:

syntax.rkt> (wires-operator 123 -> x)
syntax.rkt> (wires-operator 456 -> y)
syntax.rkt> (wires-operator x AND y -> d)
syntax.rkt> (d)
72

My syntax.rkt starts with:

#lang racket

(define-syntax wires-operator
                 ;; literals:
  (syntax-rules (AND OR LSHIFT RSHIFT NOT -> //)
    [(wires-operator lhs AND rhs -> dest)
     (define (dest) (wires-and (lhs) (rhs)))]
  ....other cases and support functions...

How does that work in the REPL but not when run as a script? Why is it not expanding the macro there? It seems like the reader is doing what I expect?

10

I couldn't say after messing about a bit.

However, I feel I may have led you astray somewhat, now that I am looking at the way you are approaching the story.

The syntax/module-reader seems to abstract more heavily over certain things, and the fact that I specifically use an example with the s-exp "meta-lang" was probably not helpful. Long-story short, I am guessing that defining a module-wide transformer as you are trying is not going to work that way (but I speak under correction).

I got the following to work, more or less, after almost killing my poor computer due to something causing an endless stream of newlines.

This is taking again from the docs under the #lang reader section.

;; wires-reader.rkt

#lang racket/base

(provide
  (rename-out
   [my-read        read]
   [my-read-syntax read-syntax]))

(require
  racket/port
  racket/string)

(define (my-read in)
  (syntax->datum
   (my-read-syntax #false in)))

(define (operator line)
  (open-input-string (format "(wires-operator ~a)" line)))
 
(define (my-read-syntax src port)
  (define src-datums
    (for/list ([line (in-list (port->lines port))]
               #:when (non-empty-string? line))
      (read (operator line))))
  (define module-datum
    `(module wires racket/base
       (require "wires-syntax.rkt")
       ,@src-datums))
  (datum->syntax
   #false module-datum))

;; wires-syntax.rkt

#lang racket/base

(provide
  wires-operator)

(define-syntax-rule (wires-operator ex ...)
  '(ex ...))

;; main.rkt

#lang reader "wires-reader.rkt"

x AND y -> d

'(x AND y -> d)

I am learning new things, too!

11

Yikes, don't kill your computer!

Thank you for the code -- it works almost exactly how I'd like it to. I've posted my current code: GitHub - dandrake/wires-racket-dsl: a Racket-based domain-specific language for solving the "wires" programming puzzle

From here, my goals are:

  • clean up the code and make it as simple and obvious as possible, so that I can use this as a template for other simple DSLs;
  • get it working so that I can do #lang wires instead of specifying the reader. This is mostly aesthetic, but it would be nice to figure out how to get this to work.
1 Like
12

That sounds like a great idea! I agree about making the #lang "just work".

As some interesting reading in this vein--making DSLs simple--see #langs that Fit In Your Head – Terminally Undead, by @countvajhula.

For what it's worth, regarding the "finicky, brittle" parts, to paraphrase a New Girl reference: "I'm not fully convinced I know how to [code]. I think I've just memorized a bunch of [patterns]!"

It tends to become solid and robust as soon as you've become used to it.

13

I now have #lang wire working! See the first #lang wires working implementation! commit in my github repo.

I have a bunch of feedback for the Racket team about all this, which I'll put in another post.

For the moment, though, I'm very grateful to everyone here for your patience and help -- @bakgatviooldoos , @shhyou , @benknoble .

That looks good. I skimmed it and will read it more carefully later!

1 Like
14

As another nice reference, I was browsing @soegaard's Racket Stories
aggregator, and came across this guide, How to Hash-Lang, by @wilbowma, although I have not yet read through its entirety.

15

Based on the first paragraphs of How to Hash Lang, that's exactly what I want. I'll read it and see if it delivers on what I think it's promising...