Let's make sense of macro*-generated `define-runtime-path`

Questions & Answers
1

This is one for the macrology experts—I'm hoping to cement my understanding of this behavior beyond "it works and I don't need to touch it" because I may indeed need to modify it some day!

PS On "macro*-generated" in the title: That's a Kleene star in the sense that one macro expands to another, which expands to a use of define-runtime-path that I want to appear as if the original source. I'm pretty sure all my problems are from using helper macros here. Would plain old functions work better here? What are the tradeoffs[1]? (Am I even right about the nesting of helpers being the issue?)

TL;DR Did I get this fix right?

Recently, I removed some extraneous synthesized identifiers in macros, which required tweaking a macro that used those for lexical and source information. The original use of those IDs is from commit 4b5848b (include original location in ability card, 2023-03-20), which I don't completely understand.

Did I diagnose the problem correctly wrt capturing the correct source/context for define-runtime-path by needing something from the original inputs? Is there a simpler fix?

Reconstructing the original reason for synthesizing define-runtime-path this way

I was able to track down the original conversation that led to this code; here's a summary.

I started with code like this:

(define-syntax-parse-rule (make-dbs ({~literal provide} info-db ability-db)
                                    ;; stuff
                                    ;; ...
                                    ;; ...
                                    )
  ;; stuff
  #:with runtime-path-lib (datum->syntax #'info-db 'racket/runtime-path #'info-db)
  #:with here (datum->syntax #'info-db 'here #'info-db)
  #:with runtime-path-define (datum->syntax #'info-db (list 'define-runtime-path #'here ".") #'info-db)
  (begin
    (provide info-db ability-db)
    (require ;;stuff ...
      runtime-path-lib)
    runtime-path-define
    (define-values (original-info-db original-ability-db)
      #;(use of "here" in here somewhere))
      ;; more stuff
      ;; ...
      ))

where info-db is an identifier synthesized from the source/context of the original macro (a #%module-begin). The user #'(shu hung) helped me, suggesting manipulating only the here context. For me, that appears to have produced runtime-paths based on the macro implementation or module expander, not the input file.

We eventually got to the version with (syntax-e #'(define-runtime-path here ".")), where #'(shu hung) notes:

The lexical information has to be attached to the entire syntax of (define-runtime-path ...) . […]

Here here is [sic] and define-runtime-path all have the lexical scope of the macro definition

so it is intended that here is visible to make-db -generated code and that define-runtime-path is referring to (require racket/runtime-path in the module defining make-db

In sum, it seems that the original needs to make sure

  • define-runtime-path has the make-dbs macro scopes in order to access the correct binding from racket/runtime-path (and not need to embed that require in the expansion);
  • here also has the make-dbs scope (why? I think because it used in the macro?);
  • the whole (define-runtime-path …) form also needs the scopes from the module to which I want the runtime path to be relative—originally, that came from #'info-db.

What broke?

I realized that a macro-generated (provide id) (at least from a module-expander's #%module-begin) needs no special manipulation to make the program (require mod) id work, in part thanks to reading DSLs in Racket: You Want It How, Now?[2]. When making info-db not synthesized via format-id, I noticed that the runtime paths stopped being embedded correctly—Frosthaven Manager displays "AoE not found" or similar at runtime[3].

I don't recall all the experiments I tried, but I remember using this-syntax in make-dbs and #'(imports ...) without success. Scrolling back through debug output in my terminal, I see runtime-paths from the application root (PWD of the application runtime) and from <root>/syntax (which contains the make-dbs implementation).

Finally, I decided to send the callers' this-syntax through to give make-dbs something with the correct scopes to attach to. This feels like a bit of a hack, and a simpler option would be appreciated.

I ended up writing:

However, as a result, the input #'info-db to make-dbs is no longer
original in the sense of coming from the original module's syntax: this
is a problem because it means that the define-runtime-path form gets the wrong context, causing the AoE modules to not be found at runtime.

Perhaps there is a way to use imports, infos, or actions as appropriate
sources, but using #'(imports ...) as the context and source for
datum->syntax also failed. Instead, pass an original syntax that can be
used for this information (and document it). Note that this requires
quasisyntax in order to embed this-syntax from syntax-parse.

To return to the original questions:

  • Did I diagnose the problem correctly?
    • What is meant by "form" in "The enclosing path for a define-runtime-path is determined as follows from the define-runtime-path syntactic form:"? Is it the entire S-expression (define-runtime-path …)?
    • Why does here also need make-dbs scopes? I think because it is used in the macro body.
  • Is there a simpler fix to make sure all the needs from earlier stay met?
  • Should I have just been using the macro stepper all along?

  1. Presumably, one tradeoff is having to return syntax containing (require racket/runtime-path) instead of relying on hygiene to make that work in the helper module. ↩︎

  2. It's code has some flaws, but it did demonstrate unadorned macro-generated provides, where I though synthesizing IDs was required. ↩︎

  3. Evaluating AoE references at compile-time is ongoing work. ↩︎