Factor out parts of syntax templates

Hi,

I have a couple macros which produce very similar syntaxes, with some changes. I would like to factor out parts of those templates to reduce code duplication.

Essentially, I would like to do something like this:

(define-for-syntax (helper stx-fragment)
  #'(list stx-fragment))

(define-syntax (main-macro stx)
  (syntax-parse stx
    [(_ stuff:expr ...)
     (helper stuff ...)]))

which doesn't work for two main reasons (I guess):

  1. stuff is used outside a template,
  2. stx-fragment will be literally part of the result, instead of being replaced with the value supplied in the argument.

I suppose I could work around (2) using something like

(with-syntax ([stx-frag stx-fragment])
  #'(list stx-frag))

which is clunky, but I have no idea how to deal with (1).

Is there a way to factor out common parts of syntax templates?

1 Like

How about making your helper a macro?

2 Likes

I'm not sure if this is what you're looking for or not, but have you seen define-template-metafunction from 1.11 Experimental ? I make heavy use of it in struct-plus-plus if you want to see real-word examples.

3 Likes

Are you familiar with define-syntax-class?

If you define a syntax class named helper, then you could use that to the right of a : in various patterns, e.g. stuff:helper. Then you can use stuff.x, where x is one of various attributes you supply from helper.

The syntax-parse intro has some examples.

3 Likes

Using quasi-syntax is probably what you are looking for.

#lang racket

(require (for-syntax syntax/parse))

(define-for-syntax (helper stx-fragment)
  #`(list #,@stx-fragment))

(define-syntax main-macro
  (syntax-parser
    [(_ stuff ...)
     (helper #'(stuff ...))]))

(main-macro 'a 'b 'c 'd)
4 Likes

Thank you very much for your replies! I'll try each of your suggestions over the following days and will report back.

Aaah, you're right! I thought that doing define in a begin-for-syntax was exactly doing define-syntax in begin-for-syntax, but I now see that I was wrong. It is even obvious now that you helped me state my confusion :slight_smile:

No, and it looks interesting! I'll try it out, first and foremost to learn more about Racket macros.

Oh, I thought about syntax classes, but I had the impression that they were an overkill for my needs. Now that you evoke them, I realize that they may quite relevant.

I was indeed looking at quasi-syntax and splicing quasi-syntax, but I couldn't figure out how to arrange the whole thing. Thanks a lot for providing a code example, it's very helpful!

6 Likes

I tried all four solutions and I learned a lot of new things about Racket macros :slight_smile: I decided to go with quasisyntax because it seems more basic and easier to comprehend, and because my actual use case is simple enough.

I put here the solutions for the record.

Helper macros

(require (for-syntax syntax/parse))

(begin-for-syntax
  (require racket (for-syntax syntax/parse))
  (define-syntax (helper stx)
    (syntax-parse stx
      [(_ stuff ...)
       #'#'(list stuff ...)])))

(define-syntax (main-macro stx)
  (syntax-parse stx
    [(_ stuff:expr ...)
     (helper stuff ...)]))

(main-macro 1 2 3)

This worked nicely and is rather intuitive. Note that I used a double #' in helper, which yields a macro expanding to a syntax template.

Template metafunctions

(require (for-syntax syntax/parse))

(begin-for-syntax
  (require syntax/parse/experimental/template)

  (define-template-metafunction (helper stx)
    (syntax-parse stx
      [(_ stuff ...)
       #'(list stuff ...)])))

(define-syntax (main-macro stx)
  (syntax-parse stx
    [(_ stuff:expr ...)
     #'(helper stuff ...)]))

(main-macro 1 2 3)

define-template-metafunction is really powerful! However, I couldn't easily make it work with Typed Racket, because some type annotations were missing, and I didn't investigate further.

Syntax classes

(require (for-syntax syntax/parse))

(begin-for-syntax
  (define-syntax-class helper
    (pattern (stuff:expr ...))))

(define-syntax (main-macro stx)
  (syntax-parse stx
    [(_ a:helper)
     #'(list a.stuff ...)]))

(main-macro (1 2 3))

In this case, the syntax class allows capturing different parts which can then be assembled into the final template. Syntax classes are very useful, but I don't think I can make meaningful use of them in my concrete situation.

Quasisyntax and quasisyntax-splicing

(require (for-syntax syntax/parse))

(define-for-syntax (helper stx-fragment)
  #`(list #,@stx-fragment))

(define-syntax (main-macro stx)
  (syntax-parse stx
    [(_ stuff:expr ...)
     (helper #'(stuff ...))]))

(main-macro 1 2 3)

This is just a copy of @SamPhillips's answer which I put here for the record. I find that this version is very useful to somebody who is still learning about macros, because syntax transformations are explicitly visible.

4 Likes

@scolobb thanks for collecting those answers in that way - very useful! I tend to get define-syntax and phases mixed up myself, and your example gave me an opportunity to concretize some concepts.

I think we can think about it this way:

Each phase of execution has its own set of bindings that are available for use during evaluation of that phase. These are called "value bindings," and they refer to ordinary data like numbers, strings, and functions. In order to define bindings for the runtime phase (phase 0) we simply define them. For the syntax phase, we wrap these same definitions in a (begin-for-syntax ...), and if we wanted to define in a preceding phase to that, we can simply nest another (begin-for-syntax ...), or use (require (for meta ...)). All of these result in normal (value) bindings in a particular phase.

Okay, now, we know that "macros are just functions operating at compile time to transform your source code." So that should mean that we can define a function in the syntax phase (phase 1), and it would be a macro, right?

(begin-for-syntax
  (require syntax/parse)

  (define (helper stx)
    (syntax-parse stx
	  [(_ e ...) #'(list e ...)])))

(helper 1 2 3) ;=> error helper undefined

This doesn't work because although we have a syntax-phase function that transforms syntax, i.e. a "macro," our naive use of helper is treated by the compiler as a runtime binding -- which it looks for in the bindings for the runtime phase and doesn't find. Added to that, all we have here is a function defined at phase 1 and an application to be run at phase 0 -- there's nothing here linking these two things. We need a way to tell the compiler that helper refers to a value in the preceding phase, and furthermore, that the value of this binding is expected to be a function that should be used to transform the code for use in the current phase.

I think this could probably be done in many different ways, but we already have one way to associate identifiers with values at this point -- bindings. So Racket introduces a separate category of bindings which can be used in one phase (phase n) to refer to values in the preceding phase (phase n+1 -- note the numbers increase going backwards in time). These cross-phase bindings are called "transformer" bindings. The way to define value bindings is to use define, and the way to define transformer bindings is to use define-syntax. Using these forms in a particular phase creates value or transformer bindings for that phase. Modifying the earlier code to use define-syntax instead:

(begin-for-syntax
  (require syntax/parse))

(define-syntax (helper stx)
  (syntax-parse stx
    [(_ e ...) #'(list e ...)]))

(helper 1 2 3) ;=> '(1 2 3)

So (define-syntax (mac stx) ...) appears to be essentially (but not quite - see below) equivalent to:

(begin-for-syntax (define (mac stx) ...))
(define-syntax mac mac) ; creating a transformer binding in phase n to refer to a value binding in phase n+1

We can test this theory:

(begin-for-syntax
  (require syntax/parse)

  (define (helper stx)
    (syntax-parse stx
      [(_ e ...) #'(list e ...)])))

(define-syntax helper helper)

(helper 1 2 3) ;=> '(1 2 3)

and also:

(begin-for-syntax
  (require syntax/parse)

  (define (helper stx)
    (syntax-parse stx
      [(_ e ...) #'(list e ...)])))

(define-syntax (main-macro stx)
  (helper stx))

(main-macro 1 2 3) ;=> '(1 2 3)

But this isn't a complete description because if we just do (define-syntax ...) without having already defined a syntax-phase binding via (begin-for-syntax (define ...)), then we have a transformer binding in phase n without the corresponding value binding being accessible in phase n+1. For this case though, we can use syntax-local-value.

I'm not sure how canonical the above descriptions are but they appear to be consistent with the docs on Transformer Bindings, and at least for the moment, they make me feel like I understand what's going on here :). Hope it helps!

Also, re: @capfredf 's suggestion to use a macro, in your code example, what you've done is define a macro in the syntax phase, that is, it is a transformer binding at phase 1, and therefore it is used in phase 2 to expand the code that will be run in phase 1. That's cool! Sometimes that kind of thing is necessary. But I think what @capfredf was getting at was simpler than this - in each phase, macro expansion is invoked repeatedly until the code produced reaches steady state. So macros can expand to other macros, and if that happens then expansion will continue to expand those macros in the same phase. E.g. if you define three macros mac, mac2, and mac3, then mac can expand to a use of mac2 which can expand to a use of mac3 which can expand to pure Racket, and all of these would happen as part of Phase 1 expansion. So you could do:

(begin-for-syntax
  (require racket syntax/parse))

(define-syntax (helper stx)
  (syntax-parse stx
    [(_ stuff ...)
     #'(list stuff ...)]))

(define-syntax (main-macro stx)
  (syntax-parse stx
    [(_ stuff:expr ...)
     #'(helper stuff ...)]))

(main-macro 1 2 3) ;=> '(1 2 3)

Finally, since (define-syntax (mac stx) (syntax-parse stx ...)) is such a common pattern, there's a macro for that: define-syntax-parser that could save you some typing.

2 Likes

Thank you very much @countvajhula for your detailed explanations. I took my time to read them attentively multiple times and I found them really illuminating. In particular, the ideas of value bindings in different phases, and define-syntax as a bridge between two consecutive phases are very helpful to understand subtle things.

Thank you as well for your suggestion about not defining the helper macro for syntax, but rather in the same phase as the macro which uses it, as well as for the pointer to define-syntax-parser, which is a very natural thing to have, but which I have never seen before. I'll update my code with this new approach.

1 Like

Something else to be aware of: when using syntax classes, you can use attributes to move some of the "assembly logic" of your macro into the syntax class. Also, you can use splicing syntax classes to make a single syntax class match a sequence of terms instead of a single term. Putting that together:

(begin-for-syntax
  (define-splicing-syntax-class helper
    #:attributes (as-list-expression)
    (pattern (~seq stuff:expr ...)
      #:with as-list-expression #'(list stuff ...)))

(define-syntax (main-macro stx)
  (syntax-parse stx
    [(_ stuff-sequence:helper)
     #'stuff-sequence.as-list-exprssion]))

(main-macro 1 2 3) ;=> (list 1 2 3)

Also, you can use define-syntax-parse-rule (from the syntax/parse/define module) as a shortcut for (define-syntax (macro stx) (syntax-parse stx [pattern #'template])) like so:

(require syntax/parse/define)

(define-syntax-parse-rule (main-macro stuff-sequence:helper)
  stuff-sequence.as-list-expression)

For these reasons, syntax classes are the first tool I recommend reaching for when you're trying to make a macro simpler by moving some of its logic into helpers. And if you make a habit of doing that, almost all of your macros can be written with define-syntax-parse-rule.

5 Likes

Thank you @notjack! I did look at splicing syntax classes, but I didn't understand that I needed #:attributes, ~seq, #:with, etc. I'll take some more time to read the documentation about syntax classes to get a better hang of it.

Also, thanks for the suggestion to use define-syntax-parse-rule. I think a lot of my macros can be shortened using this syntax.

2 Likes

I would also suggest looking at existing code bases for examples of how to use them and the pros/cons of each approach. Again, I make heavy use of them in struct-plus-plus (struct-plus-plus/main.rkt at master · dstorrs/struct-plus-plus · GitHub) and a lot of what I use was learned from Alexis King's struct-update module (struct-update/main.rkt at master · lexi-lambda/struct-update · GitHub) There's a lot more code and varieties of code in SPP but SU does things that SPP doesn't. I haven't dug into the code for the various languages such as rackjure but there might be useful things in there as well.

EDIT: Oh, I misread the date on that last message as May 31. Looks like I'm necro'ing -- sorry for that, but hopefully the content is still useful.

1 Like