I'm attempting to create a macro that expands to a submodule, but I can't seem to figure out how to use the (provide (all-defined-out)) form indroduced by the macro. Here is an example:
#lang racket/base
(module defpkg racket/base
(require (for-syntax racket/base
syntax/parse
racket/syntax))
(provide (all-defined-out))
(define-syntax (define-package stx)
(syntax-parse stx
[(_ name:id e:expr ...+)
#:with register! (format-id #'name "~a" 'register!)
#'(module* name #f
(provide (all-defined-out)) ;; <- doesn't work!
(define (register!)
(displayln "registered!"))
;; for racket bindings
e ...)])))
(module pkgs racket/base
(require (submod ".." defpkg))
(provide (all-defined-out))
;; a is provided from module 'pkgs
(define a 1)
;; sub module* directly
(module* pkg0 #f
(provide (all-defined-out))
;; b is provided from (submod 'pkgs pkg0)
(define b (+ a 1)))
;; sub module* through macro
(define-package pkg1
;(provide (all-defined-out)) ;; <- will work if uncommented
;; c is provided from (submod 'pkgs pkg1)
(define c (+ a 2)))
)
(let ()
;; a from 'pkgs
(local-require 'pkgs)
(displayln a)
;; b from (submod 'pkgs pkg0)
(local-require (submod 'pkgs pkg0))
(displayln b)
;; c and register! from (submod 'pkgs pkg1)
(local-require (submod 'pkgs pkg1))
(displayln c)
(register!))
I'm assuming that the provide form introduced in (define-package ..) is in the wrong scope, but I have been unable to figure it out how to fix it. Does anyone have suggestions?
To explain a bit more, as you guessed all-defined-out doesn’t literally provide “all defined,” because macros should be able to generate module-level definitions for their internal purposes with the protection of hygiene. What it actually provides, as the docs explain in more detail, is everything that defined and “accessible from the lexical context of the (all-defined-out) form.”
But what is “the lexical context of the (all-defined-out) form”? It can’t be the context of the identifier all-defined-out, because a macro like your define-package that generates a use of (all-defined-out) should be able to work even if all-defined-out isn’t in scope where define-package is used. Some macros have a sub-form like name that might make sense to use for the relevant scope, but sometimes there either isn’t a subform (as with all-defined-out) or the desired scoping behavior has more dimensions to keep distinct. So, often, the scope “on the parentheses” of the form as a whole.
Incidentally, I used #'name because you were already giving that scope to register!, but you might consider making register! a syntax parameter, instead, in which case you might want to use the scope “on the parentheses” of define-package, i.e. stx or this-syntax. I’m guessing you don’t mean for every package to export register!.
Just to following on to @LiberalArtist excellent response, one way to figure this out is to use the macro stepper and select the [menu]->Stepper->View Syntax Properties. This confirmed that, indeed, the all-provided-out part of the syntax was scoped by the defpkg module and was not in the scope that module* would create. I was stuck on how to fix that.
What the solution does is to force the (all-provided-out) list to be scoped by the expanded syntax; which means that it scoped to the module created by module*. Honestly, I probably never would have gotten there. so I am glad somebody had an answer.
This is an example of how tricky it can be in syntax patterns to see what is actually defined in the scope of the created syntax versus the environment in which the syntax transformer is being executed in.
Since no good deed goes unpunished... I have a follow up question on how to go the opposite direction. The example below now uses (module ..) instead of module*, so now the issue is that the (define ..) sent into the define-package macro doesn't have the correct scope set for the (module ..). define is not defined.
#lang racket/base
(module defpkg racket/base
(require (for-syntax racket/base
syntax/parse
racket/syntax))
(provide (all-defined-out))
(define-syntax (define-package stx)
(syntax-parse stx
[(_ name:id e:expr ...)
#:with (e+ ...) (for/list ([e- (syntax-e #'(e ...))])
e- ;; <- how to rescope it to be within (module ..)
#;(datum->syntax #f (syntax->datum e-)))
#'(module name racket/base
(provide (all-defined-out))
e+ ...)])))
(require 'defpkg)
(define-package pkg2
(define b 2))
(let ()
(local-require 'pkg2)
(displayln b))
(datum->syntax #f (syntax->datum #'e)) was my best guess, but obviously wrong. Any ideas on how I should rescope e ... so that it points to the racket/base within (module name racket/base e ...)? I played around with some of the namespace utilities, but failed there as well.
From playing with it, it appears that it doesn't matter what the syntax object is as long as it is new. So #'this-syntax could be #'anything I guess.
Trying to make sense of this... datum->syntax is a macro so #'this-syntax will not get evaluated until (module ..) is being expanded, so the scope will be within the new module. Does this sound right?
I could have used #’stx too. datum->syntax is a function, and it transfers the lexical context of the given syntax to the newly created syntax object. To get DrRacket’s arrows and such you also want to supply optional arguments.
Now that I know the answer, I can see that the docs do specify this behavior—but the fact that it’s a recurring question suggests that an example or further explanation might be in order.
(The scope “on the parentheses” pattern comes up in other contexts, too, e.g. with submod, but I’m not sure where to potentially put documentation on the pattern in general.)
If you were referring to my suggestion above:
the this-syntax I was referring to is an expression form. You wouldn’t generally quote it with #', or it would just be equivalent to #'cons or something, an identifier with the lexical context from the use of the #' form. Used as an expression, this-syntax refers to the syntax object currently being parsed.
Here’s an illustration of the difference:
#lang racket
(module demo racket
(provide demo)
(require syntax/parse/define)
(define-syntax-parser demo
[_
#`(values
(list "quoted" (quote-syntax #,#'this-syntax))
(list "as expression" (quote-syntax #,this-syntax)))]))
(require 'demo)
(demo arbitrary sub-expressions here)
#| ; in DrRacket, the syntax objects will be printed as snips you can
; interactively inspect, not just plaintext
'("quoted" #<syntax:demo.rkt:8:41 this-syntax>)
'("as expression" <syntax:demo.rkt:11:0 (demo arbitrary sub-expressio...>)
|#
Matthias’ solution directly answers the question you posed, e- ;; <- how to rescope it to be within (module ..), but I would suggest a slightly different approach:
The core idea in this variant is using (datum->syntax stx 'racket/base) to bring the bindings from the synthesized module’s initial import into scope for the input sub-forms, rather than changing the input sub-forms to use the scope of the define-package implementation. (Note that Matthias’ version could have used #'cons just as well as #'this-syntax, and it would not work with un-#'ed this-syntax or stx.) It requires knowing the fairly obscure fact (I can’t find a reference in the docs to cite for it ATM) that module uses the scope on the term in the language position this way.
Using syntax->datum on complex sub-forms is generally a warning sign that you should look for a better solution. It is very difficult to use correctly and robustly, because syntax->datum throws away all of the scope, source-location information, and properties. (Matthias’ comment about the other arguments to datum->syntax alludes to some of this.) For example, consider a macro that expands to define-package with a body that contains both user code and macro-introduced terms: using syntax->datum on the body will throw out the macro-introduction scope, breaking hygiene for the downstream macro.
The other change, which was a workaround for a problem that surprised me along the way, is the use of (make-syntax-introducer) to add an additional fresh scope to racket/base and the body forms (and, indirectly, “on the parentheses” of (all-defined-out)) that is not on the identifiers provide and all-defined-out. I had anticipated the need to deal with potential shadowing of provide and all-defined-out: what surprised me was that, without (make-syntax-introducer) (e.g. replacing it with values), I got an ambiguous binding error for provide. I haven’t yet worked out why.
I’d also initially tried to use syntax-local-introduce instead of datum->syntax, and I haven’t yet thought through why that didn’t work as I’d hoped.
I think it’s worth noting that macro-introduced submodules are a fairly advanced corner of macrology, and there are some rough edges. It’s been a few years since I did much with it, and not all of the pitfalls are fresh in my mind.