Syntax Classes and Module Boundaries

Questions & Answers
1

Hi, Racket Discourse.

I am trying to split up some syntax-classes across different modules, to allow for more easily composable bits and bobs.

The syntax-classes create temporary identifiers which are used as aliases for pattern variables in the eventual match expressions.

When I require the classes from their home, the syntax all seems to look copacetic when quoted, but when unquoted, the temporary identifiers seem to be unbound (no #%top).

I can kind of guess why this might be, in the sense that these identifiers come from a place which is not the call-site, with a different context. However, it's not clear to me how to "settle" them in the requiring module.

I found a previous discussion from which I would guess that datum->syntax is what I am looking for, but as I imply, there are a couple of places this could be injected, to my mind.

Do I parameterize the local syntax classes by the macro's stx and use that to convert to fresh syntax, or is it possible to delay until the macro's body? I am guessing the use of #:with in the ax-system class might also complicate this (although I am fuzzy on the exact difference between #:with and #:attr except that that the latter may contain non-syntax values), because the unbound identifiers would surface there already?

The three modules in question look like this:

;; objects.rkt for the structs
#lang racket/base

(require
  (only-in
   racket/format ~a))

(provide
  (struct-out exp)
  (struct-out log)
  (struct-out neg))

(struct mark [sum]
  #:transparent
  #:guard
  (lambda (sum _)
    (unless (list? sum) (error 'mark "expected a list?, found: ~a" sum))
    sum))

(define (format-term term)
  (if (list? term) (format-mark term "×" "{~a}") term))

(define (format-mark sum none some)
  (define terms (map format-term sum))
  (if (null? sum) none (format some (apply ~a #:separator " " terms))))

(define ((mark-writer-maker none some) self port mode)
  (define sum (mark-sum self))
  (fprintf port (format-mark (mark-sum self) none some)))

(struct exp mark []
  #:transparent
  #:property prop:custom-print-quotable 'always
  #:methods gen:custom-write
  [(define write-proc (mark-writer-maker "○" "(~a)"))])

(struct log mark []
  #:transparent
  #:property prop:custom-print-quotable 'always
  #:methods gen:custom-write
  [(define write-proc (mark-writer-maker "□" "[~a]"))])

(struct neg mark []
  #:transparent
  #:property prop:custom-print-quotable 'always
  #:methods gen:custom-write
  [(define write-proc (mark-writer-maker "◇" "⟨~a⟩"))])

;; grammar.rkt for the syntax-classes
#lang racket/base
;; I might be using `for-template` wrong here
(require
  (for-template
   (rename-in
    "objects.rkt"
    [log <log>]
    [exp <exp>]
    [neg <neg>])
   (only-in
    racket/match ==))
  racket/base
  syntax/parse
  racket/syntax)

#|

sum  :=  (term ...) | var | var*
term :=   exp | log | neg | sum
exp  :=  `sum ; sum is not var*
log  :=  ,sum ; sum is not var*
neg  :=  'sum ; sum is not var*
var  :=   identifier
var* := ,@identifier

|#

(provide
  forms
  name terms
  var-name)
  
(define-literal-set forms
  #:datum-literals (○ □ ◇ • × =)
  (unquote-splicing quasiquote unquote quote))
  
(define form? (literal-set->predicate forms))
(define-syntax-class
  name
  (pattern foo:id
    #:when (not (form? #'foo))))

(define (head? posn) (eq? 'head posn))
(define (body? posn) (eq? 'body posn))

(define-syntax-class
  (any aux?)
  #:attributes
  ([ex 0]
   [ns 0])
  #:datum-literals (_)

  (pattern ,@_
    #:fail-when aux?
    (format "splice boundary error: expected to be inside of a form")
      
    #:attr ex #',@_
    #:attr ns #'(_))
    
  (pattern _
    #:attr ex #',_
    #:attr ns #'(_)))

(define (var-name x)
  (syntax-property x 'self))

(define (push var)
  (define mask (generate-temporary var))
  (syntax-property mask 'self var))

(define-syntax-class
  (var posn)
  #:attributes
  ([ex 0]
   [ns 0])
  
  (pattern var:name
    #:when (head? posn)
    #:attr __ (push #'var)
    #:attr ex #',__
    #:attr ns #'(__))

  (pattern var:name
    #:when (body? posn)
    #:attr ex #',var
    #:attr ns #'(var)))

(define-syntax-class
  (var* posn aux?)
  #:attributes
  ([ex 0]
   [ns 0])
  #:literal-sets (forms)
  
  (pattern ,@var:name
    #:fail-when aux?
    (format "splice boundary error: expected to be inside of a form")
      
    #:when (head? posn)
    #:attr __ (push #'var)
    #:attr ex #',@__
    #:attr ns #'(__))

  (pattern ,@var:name
    #:fail-when aux?
    (format "splice boundary error: expected to be inside of a form")
    
    #:when (body? posn)
    #:attr ex #',@var
    #:attr ns #'(var)))
  
(define-syntax-class
  (sum posn [aux? #false])
  #:attributes
  ([ex 0]
   [ns 0]
   [ls 0])
  #:literal-sets (forms)
  #:local-conventions
  ([any  (any  aux?)]
   [var  (var  posn)]
   [var* (var* posn aux?)]
   [term (term posn)])
    
  (pattern any
    #:attr ex #'any.ex
    #:attr ns #'any.ns
    #:attr ls #false)

  (pattern var
    #:attr ex #'var.ex
    #:attr ns #'var.ns
    #:attr ls #false)
    
  (pattern var*
    #:attr ex #'var*.ex
    #:attr ns #'var*.ns
    #:attr ls #false)
    
  (pattern (term ...)
    #:attr ex #'(term.ex ...)
    #:attr ns #'((~@ . term.ns) ...)
    #:attr ls #true))
  
(define-syntax-class
  (exp posn)
  #:attributes
  ([ex 0]
   [ns 0])
  #:literal-sets (forms)
  #:local-conventions
  ([sum (sum posn #true)])

  (pattern ○
    #:attr ex (if (head? posn) #',(<exp> '()) #',(<exp> null))
    #:attr ns #'())

  (pattern `sum
    #:attr ex (if (attribute sum.ls) #',(<exp> `sum.ex) #',(<exp> (~@ . sum.ns)))
    #:attr ns #'sum.ns))
  
(define-syntax-class
  (log posn)
  #:attributes
  ([ex 0]
   [ns 0])
  #:literal-sets (forms)
  #:local-conventions
  ([sum (sum posn #true)])

  (pattern □
    #:attr ex (if (head? posn) #',(<log> '()) #',(<log> null))
    #:attr ns #'())
    
  (pattern ,sum
    #:attr ex (if (attribute sum.ls) #',(<log> `sum.ex) #',(<log> (~@ . sum.ns)))
    #:attr ns #'sum.ns))

(define-syntax-class
  (neg posn)
  #:attributes
  ([ex 0]
   [ns 0])
  #:literal-sets (forms)
  #:local-conventions
  ([sum (sum posn #true)])

  (pattern ◇
    #:attr ex (if (head? posn) #',(<neg> '()) #',(<neg> null))
    #:attr ns #'())
  
  (pattern 'sum
    #:attr ex (if (attribute sum.ls) #',(<neg> `sum.ex) #',(<neg> (~@ . sum.ns)))
    #:attr ns #'sum.ns))

(define-syntax-class
  (term posn)
  #:attributes
  ([ex 0]
   [ns 0])
  #:literal-sets (forms)
  #:local-conventions
  ([sum (sum posn)]
   [exp (exp posn)]
   [log (log posn)]
   [neg (neg posn)])

  (pattern ×
    #:attr ex (if (head? posn) #',(== '()) #',@null)
    #:attr ns #'())

  ; π·i
  (pattern •
    #:attr ex #',(<log> `(,(<neg> `(,(<exp> '())))))
    #:attr ns #'())
    
  (pattern ex:nat
    #:attr ns #'())
    
  (pattern (~or* sum exp log neg)
    #:attr ex #'(~? sum.ex
                (~? exp.ex
                (~? log.ex
                    neg.ex)))

    #:attr ns #'(~? sum.ns
                (~? exp.ns
                (~? log.ns
                    neg.ns)))))

(define-splicing-syntax-class
  (terms posn)
  #:attributes
  ([ex 0]
   [ns 0])
  #:local-conventions
  ([term (term posn)])
    
  (pattern {~seq term ...}
    #:attr ex #'`(term.ex ...)
    #:attr ns #'((~@ . term.ns) ...)))

;; algebra.rkt for some macros built on the syntax-classes
#lang racket/base

(require
  (only-in
   racket/set set-count list->set)
  (only-in
   racket/match match-lambda)
  (for-syntax
   "grammar.rkt"
   racket/base
   syntax/stx
   syntax/parse
   racket/syntax
   (only-in
    racket/list group-by remove-duplicates)))

(define (same? . xs)
  (= 1 (set-count (list->set xs))))

(begin-for-syntax
  (define (derive-cases clauses)
    (define cls (stx->list clauses))
    (for/list ([bd (in-list cls)])
      (map (lambda (cl) #`(#,cl #,bd)) (remove bd cls))))
  
  (define (masked-vars names)
    (filter var-name (stx->list names)))
  
  (define (group-names names)
    (define grouped (group-by var-name (masked-vars names) free-identifier=?))
    (define samemap (map (lambda (g) (if (null? (cdr g)) #'#true #`(same? . #,g))) grouped))
    (define rootmap (map (lambda (g) #`(#,(var-name (car g)) #,(car g))) grouped))
    #`(#,samemap #,rootmap))
  
  (define (group-names* names**)
    (map (lambda (names*) (map group-names (stx->list names*))) (stx->list names**)))

  (define-syntax-class
    clause
    #:attributes
    ([ex 0]
     [as 0])
    
    (pattern [as:name foo:expr ...]
      #:attr ex #'(foo ...)))

  (define-splicing-syntax-class
    ax-system
    #:attributes
    ([ex 0]
     [ns 0]
     [as 0])
    #:local-conventions
    ([hd (terms 'head)]
     [bd (terms 'body)])

    (pattern {~seq lhs:clause rhs:clause ...+}
      #:with (({(hd) (bd)} ...) ...) (derive-cases #'(lhs.ex rhs.ex ...))
      #:with (({same root} ...) ...) (group-names* #'((hd.ns ...) ...))
      
      #:attr ex
      #'((match-lambda
           [hd.ex
            #:when (and . same)
            (let root bd.ex)]
           ...
           [_
            #false])
         ...)
      #:attr ns #'((~@ . bd.ns) ... ...)
      #:attr as #'(lhs.as rhs.as ...)))

  (define (unique-names names)
    (remove-duplicates (stx->list names) free-identifier=?))

  (define ((format-rule stx ax) as)
    (format-id stx "~a.~a" ax as #:subs? #true))

  (define (format-rules stx ax as)
    (map (format-rule stx ax) (stx->list as))))

(define-syntax (define-axiom stx)
  (syntax-parse stx
    [(_ name:id :ax-system)
     #:with (vars ...) (unique-names #'ns)
     #:with (rule ...) (format-rules stx #'name #'as)
     #:with (body ...) #'ex
     #'(define-values (rule ...)
         (let ([vars null] ...)
           (values body ...)))]))

(define-axiom uniform
  [self A] [wrap (A)])

(define-axiom perturb
  [enfold₀ `(,A)]
  [enfold₁ ,(`A)]
  [clarify    A])

(define-axiom reflect
  [create ,@A 'A] [cancel ()])

(define-axiom arrange
  [gather
   `(,@A ,(B ,@C))]
  [spread
   `(,@A ,(B)) `(,@A ,C)])

As one can see, when the macro's body for define-axiom is quoted, for example, the syntax seems good:

'
(define-values
  (uniform.self uniform.wrap)
  (let ((A null))
    (values
     (match-lambda (`((,A1)) #:when (and #t) (let ((A A1)) `(,A))) (_ #f))
     (match-lambda (`(,A2) #:when (and #t) (let ((A A2)) `((,A)))) (_ #f)))))

Unquoted, however:

A1: unbound identifier;
 also, no #%top syntax transformer is bound in: A1

Besides which, before I go and Frankenstein this bad-boy, is this method of generating the temporary identifiers even advisable to begin with?

2

Your syntax classes in grammar.rkt compute syntax objects that are intended to be used as match patterns. For example, one variant of the var syntax class has (if I spell out one of the reader macros):

#:attr __ (push #'var)  ;; push generates a temp, adds stxprop
#:attr ex #'(unquote __)  ;; used within a match quasi-pattern

The problem is that when the ex syntax is inserted into a match quasi-pattern, the match expression is at phase -1 relative to the grammar.rkt module. But grammar.rkt has no binding for unquote at phase -1 (ie, for-template), so the match expression doesn't recognize it as an escape from the quasiquote pattern; it just treats it as part of the quoted term to match. So that means the match pattern doesn't bind the generated variable, so it's not there for the reference in the root let-clauses.

The fix is to add a (require (for-template racket/base)) to grammar.rkt. You should also double-check your syntax classes and their use of literals. For example, any has a pattern involving unquote-splicing but it does not have a #:literal-sets (forms) declaration.

Other notes:

  1. I figured out what was going on by running the macro stepper on the first define-axiom example. I put it in a separate module that required algebra.rkt first, to minimize the expanded code. Then I had to adjust the macro hiding policy a few times to get it to show the expansion of the match form and see that it wasn't binding A1 as a variable.

  2. I recommend avoiding reader macros for terms like (unquote _) when you refer to them in macros, in patterns or templates. (If you happen to use them in a compile-time expression, though, I would use the reader macro form then.) I find it easier to read that way.

1 Like
3

Hi, @ryanc.

Jissie, man, thanks for the lucid explanation! :racket_heart:

Good catch on the missing #:literal-sets--completely missed that even before the splitting of the original file.

Thank you for the stylistic notes, always nice to hear advice from the masters. I have not yet used the macro stepper beyond playing around, so I will take this as a sign to graduate from playing to tinkering.

In this particular case, I think your recommendation about reader macros is even more salient, given that I am "overloading" the meaning of these in the syntax of the larger macro-system.