Would you mind elaborating on this example?
The thing Racket (and specifically struct) does to enable this is exactly the thing you are trying to get rid of.
In more detail: macros require that some bindings that can be referred to at phase 0 have an implementation that runs at phase 1 (that's what macros are). Right now, Racket implements this connection with define-syntax. You could get rid of that approach, and instead just say we're going to have definitions at different phases, and have a separate way to say "treat this phase 1 definition as a macro". That's the proposal you're making. But then you can't declare a binding that works at phase 0 (is a macro) but also you can go from that binding to the phase 1 value it's bound to -- that's the connection we just got rid of.
In other words, we could have a system where struct expands into
(define-for-syntax point-static-info (static-info blah #'make-point))
(declare-macro point point-static-info)
and then match would have an operation get-phase1-value which takes the identifier #'point and produces the value stored in point-static-info. And then we'd have a system that allowed match to work just as it does today.
But this is exactly the system we have -- declare-macro is define-syntax and get-phase1-value is syntax-local-value.
The two kinds of binding are inherent in the concept of macros -- we can adjust how they're presented but we can't make the distinction go away.
2 Likes
Thank you for clarifying. I'll think about it further and see if these explanations address my questions.
Obviously, it appears that this is a necessary dichotomy. But some strand of my intuition is tugging at me and saying, we shouldn't have things like these:
syntax-local-value(why not just an ordinary variable reference with a sensible (i.e. our current) model of hygiene and scope?)phase1-eval(outside of actual expansion, why not just ordinary evaluation at phase 0?)syntax-local-apply-transformer(why not just rely on macros as ordinary first-class functions?)
I realize that these are very dumb questions and there are probably good reasons for all of these, many of which you've enumerated, and which, over time, I hope to consider in depth and understand better.
If I revisit this at some point, I'll share more of my ignorance for general entertainment and the remote possibility of revealing some worthwhile alternative implementations, or at least (at last), my comprehension!
Thanks again for indulging me 
I might be entirely on the wrong track here but what you might be asking might boil down to "why do we even have compilation as an explicit idea in our language designs?". That is, why don't we just manipulate everything dynamically? I'd say that this has been explored (check out smalltalk for a great example of the genre). My sense of the conclusion is that it is important to know things about program before you run it. The most obvious reason is that you can run the program more efficiently but there are other reasons having to do with software maintenance too. Sorry if this is off the mark.
I'm not dismissing the idea of phases, though, which I feel is essentially about (or one way -- a particularly good way! -- to formally think about) compilation. So, I don't think I'm saying that we should manipulate everything dynamically!
I'm keen on the idea of functions operating at phase 1 which transform source syntax at phase 0 (and recursively to higher phases, etc.). I am just wondering if we could avoid having special concepts and infrastructure to support the rich macro tech that we have, and whether, perhaps, the root cause of that is the choice of transformers vs ordinary values as being two kinds of binding rather than being just bindings (of an undifferentiated kind) employed in different ways (i.e. as macros transforming syntax in phase 1 vs as ordinary functions operating at phase 0) that are indicated to the expander (a distinct program) through some other mechanism, such as, perhaps, a module-level directive or just a top level declaration or even a separate configuration file (I'm not advocating this -- just enumerating it as a possibility!).
It may be that such an alternative design, even if it's possible, would entail its own costs that would perhaps exceed any gains, and maybe the existing way we are doing it is, after all, the most convenient way to do it -- and maybe this answer is already affirmed in this thread by Sam and others (I still haven't read Sam's latest response in detail!). To the extent I have an end goal in raising this discussion, it would be to understand such a possible alternative design, and whether there are macros we could write in today's design that would achieve abstracting it and presenting the alternative design on the surface (which could, for example, be then employed by #langs that may prefer that alternative, assuming it exists!).
This is just an attempt to clarify what I may be trying to say -- which, to be clear, could very well be nonsense
(P.S. I'd like to reflect on this and have something more tangible to point to, but I am not able to dedicate time to research this at this very moment -- if others want to continue discussing, I'd be glad to follow along and catch up when I have a chance to.)
1 Like
syntax-local-value: You need an API that takes an identifier because that's what a macro has to work with. If you want to get the static information associated with something specified in the input, you're starting with an identifier that's a value rather than a name that you can just refer to in the macro definition.phase1-eval: this is useful for testing code that's supposed to run at expansion time. Think of it like a mock (in the TDD sense) but for the expansion time API.syntax-local-apply-transformer: this is not a fundamental part of the API, the basic primitive is just applying functions that are values, as you say. But to get hygiene right, as well as managing other metadata like the'originsyntax property, there are a number of things you need to do if you're using a first class function like a macro. You can do those yourself, but this packages them up into an easy-to-use API.
4 Likes
Wait, what is the name of this paper? I don't see a Sam Tobin-Hochstadt paper at Scheme Workshop 2009. Is that a version of 'The Design and Implementation of Typed Scheme'?
It's this paper: https://www2.ccs.neu.edu/racket/pubs/scheme2007-ctf.pdf
Sorry I was unable to respond last week (my account was on hold until today). I did (re-?)read Sam's paper last week. It is a good one. It documents grappling with these issues in the post syntax-case / pre syntax-parse world. I wish this and the other papers of the era became the basis of a textbook!
I sympathize with the OP. Scheme was all about the simplicity of lexical scoping, and it was so nice to get rid of the separate namespace for procedures retained in Lisp! How did bindings get so complicated in the macro+module system?
The first hygienic expander many of us studied was psyntax (Dybvig et. al). It was a syntax-case preprocessor, and didn't have a notion of phases, so I feel like the phase 0 / phase 1 connector explanation, while very valuable in the context of Racket's module system, is somewhat retconed.
If I can cast my mind back to the 1990s, the three types of bindings in the compile time environment (value, syntax, pattern variable) probably evolved because, practically, macro applications had to be distinguished from procedure applications, pattern variables needed to be associated with an ellipsis nesting count, and syntax errors would be more reliable if these two roles were distinguished from value bindings.
In that era, dealing with hygiene and algorithmic complexity were top priority. Sorting out the eval-when mess came later. I didn't follow R7RS, but the discussions linked above seem to indicate that Racket research solving the eval-when problem was somewhat disregarded in the Scheme community.
I don't know if the OP's dream is valid or not, but at one point I experimented with defining and evaluating everything in the compile time environment (phase 1), and just lowering results to phase 0 when exporting an application/runtime/data from the compiler.
To make this work with sophisticated macros and modules, it seems like the compile time environment and its values would need to be pure to avoid the issue of ordering module instantiation (in some sense Racket modules are parameterized due to side effects, and thus they must be instantiated more than once). You can probably see how this solution could not have come in the first era of Scheme/Racket macro+modules since modern research into types and effect systems would be required to do the magic achieved though side-effects in today's Racket expander.
2 Likes
For the record, I must admit that even though I recently read this thread, including @samth's remarks, I still fell into the trap of thinking that a struct's name would be bound to its descriptor at phase 1 rather than being a transformer binding in this parallel thread:
"Obviously" a phase 1 binding wouldn't work since the name would not show up in the match form, etc. The worse part is that I was even using the correct language ("transformer binding") while making this mental mistake. My remarks don't even make sense because it wouldn't be possible to create both a phase 0 binding and a transformer binding for the same identifier. It is no wonder that @samth needed me to clarify.
In this example, the fact that the transformer binding's value acts as both a procedure and a structure may create a garden path to the notion that there must be two bindings, leading to the notion that they must be at different phase levels. Either that or I just need an object lesson every time I step back into the world of macro practice after an extended period of time.