Macro that expands to (module .. (provide (all-defined-out)) ..)

[LiberalArtist]

@ndykman

This confirmed that, indeed, the all-provided-out part of the syntax was scoped by the defpkg module and was not in the scope that module* would create. I was stuck on how to fix that.

What the solution does is to force the (all-provided-out) list to be scoped by the expanded syntax; which means that it scoped to the module created by module*. Honestly, I probably never would have gotten there. so I am glad somebody had an answer.

I didn’t figure it out, either, until Matthew explained it to me.

Now that I know the answer, I can see that the docs do specify this behavior—but the fact that it’s a recurring question suggests that an example or further explanation might be in order.

(The scope “on the parentheses” pattern comes up in other contexts, too, e.g. with submod, but I’m not sure where to potentially put documentation on the pattern in general.)

---

@wluker

From playing with it, it appears that it doesn't matter what the syntax object is as long as it is new. So #'this-syntax could be #'anything I guess.

If you were referring to my suggestion above:

@LiberalArtist

you might want to use the scope “on the parentheses” of define-package, i.e. stx or this-syntax.

the this-syntax I was referring to is an expression form. You wouldn’t generally quote it with #', or it would just be equivalent to #'cons or something, an identifier with the lexical context from the use of the #' form. Used as an expression, this-syntax refers to the syntax object currently being parsed.

Here’s an illustration of the difference:

#lang racket
(module demo racket
  (provide demo)
  (require syntax/parse/define)
  (define-syntax-parser demo
    [_
     #`(values
        (list "quoted" (quote-syntax #,#'this-syntax))
        (list "as expression" (quote-syntax #,this-syntax)))]))
(require 'demo)
(demo arbitrary sub-expressions here)
#| ; in DrRacket, the syntax objects will be printed as snips you can
   ; interactively inspect, not just plaintext
'("quoted" #<syntax:demo.rkt:8:41 this-syntax>)
'("as expression" <syntax:demo.rkt:11:0 (demo arbitrary sub-expressio...>)
|#

---

@wluker

I have a follow up question on how to go the opposite direction. The example below now uses (module ..) instead of module*, so now the issue is that the (define ..) sent into the define-package macro doesn't have the correct scope set for the (module ..). define is not defined.

Matthias’ solution directly answers the question you posed, e- ;; <- how to rescope it to be within (module ..), but I would suggest a slightly different approach:

#lang racket/base

(module defpkg racket/base
  (require (for-syntax racket/base
                       syntax/parse
                       racket/syntax))
  (provide (all-defined-out))
  
  (define-syntax (define-package stx)
    (syntax-parse stx
      [(_ name:id e:expr ...)
       #:with (R/B e+ ...) ((make-syntax-introducer)
                            #`(#,(datum->syntax stx 'racket/base)
                               e ...))
       #`(module name R/B
           (provide #,(datum->syntax #'R/B (list #'all-defined-out)))
           e+ ...)])))

(require 'defpkg)
(define-package pkg2
  (define all-defined-out
    "doing this breaks the other approach")
  (define b 2))

(let ()
  (local-require 'pkg2)
  (displayln b))

The core idea in this variant is using (datum->syntax stx 'racket/base) to bring the bindings from the synthesized module’s initial import into scope for the input sub-forms, rather than changing the input sub-forms to use the scope of the define-package implementation. (Note that Matthias’ version could have used #'cons just as well as #'this-syntax, and it would not work with un-#'ed this-syntax or stx.) It requires knowing the fairly obscure fact (I can’t find a reference in the docs to cite for it ATM) that module uses the scope on the term in the language position this way.

Using syntax->datum on complex sub-forms is generally a warning sign that you should look for a better solution. It is very difficult to use correctly and robustly, because syntax->datum throws away all of the scope, source-location information, and properties. (Matthias’ comment about the other arguments to datum->syntax alludes to some of this.) For example, consider a macro that expands to define-package with a body that contains both user code and macro-introduced terms: using syntax->datum on the body will throw out the macro-introduction scope, breaking hygiene for the downstream macro.

The other change, which was a workaround for a problem that surprised me along the way, is the use of (make-syntax-introducer) to add an additional fresh scope to racket/base and the body forms (and, indirectly, “on the parentheses” of (all-defined-out)) that is not on the identifiers provide and all-defined-out. I had anticipated the need to deal with potential shadowing of provide and all-defined-out: what surprised me was that, without (make-syntax-introducer) (e.g. replacing it with values), I got an ambiguous binding error for provide. I haven’t yet worked out why.

[the full ambigous-binding error message]

``` provide: identifier's binding is ambiguous context...: matching binding...: #(provide # 0) #(64982 module) #(129967/2 module defpkg) matching binding...: #(provide # 0) #(65179 module) #(130361/2 module pkg2) in: provide```

I’d also initially tried to use syntax-local-introduce instead of datum->syntax, and I haven’t yet thought through why that didn’t work as I’d hoped.

I think it’s worth noting that macro-introduced submodules are a fairly advanced corner of macrology, and there are some rough edges. It’s been a few years since I did much with it, and not all of the pitfalls are fresh in my mind.