Some time ago I noticed the paper by Ballantyne et. al. re: making your DSL "macro extensible," with the idea to implement it for Qi. The paper was way over my head so I've had it on the backburner to eventually spend more time looking into*.
But in the meantime, I had an idea for a lightweight way to do it. With this way, you can use Racket's macro system to extend the Qi language, and your forms would have exactly the same status as the built-in Qi forms, and will interoperate with the rest of Qi just like anything else. There is just one caveat. The name of your macro must begin with qi:, as this is how Qi recognizes it as an extension.
During the macro expansion phase, Qi will simply "pass through" such forms, unchanged, with the assumption that they will be expanded into something usable as flows by a third party extension.
So in other words, Qi now has macros, and they are just plain old Racket macros following a special naming convention. Read more about how to use them here.
Special thanks to Sam @samth for the tip on implementing this as a parameterized syntax class, and also to Greg @greghendershott for passing on a tip long ago that helped me grok macros, and which informs the above scheme - that "macros are expanded outside in."
Enjoy,
(*) I would still love help with this, if you are familiar with the proposals from this paper. Here's where I'd like to take this in the long term, which isn't really possible with the "lightweight" solution above. Any advice (ideally in the form of issue comments so it's in context when the work eventually gets underway) would be appreciated 
6 Likes
This reminds me of the dream I had in Experiment with defines: define unity. Looking forward to see it come to fruition.
As an example, I would love a more performant/expressive way to handle the following: I have a flow that returns a mix of numbers and #f. I want to pass the numbers to min, but there may not be any numbers (in which case min barfs). The current solution is something like
(~> … (pass number?) (if (~> count zero?) #f min))
The perf is not great: I don't need to count the (possibly large) number of values when I only need to know if there's at least one.
I suppose now I can implement this test "at least one value" as a qi macro.
1 Like
Yeah, you certainly could do it with a macro now @benknoble , though on the face of it I'm not sure how we would ensure "at least one" without first converting the entire value flow into a list - which may run into a similar performance issue, unless you can think of a clever workaround. Btw, as you can imagine, this is a general issue to address wrt Qi's implementation, since there are many cases where we would prefer to avoid a superfluous list conversion, yet the way Racket functions receive an arbitrary number of values is as a list argument, necessitating receiving the arguments as a list, only to convert them back into values again on the way out. This may also be related to the min issue you ran into, regarding which,
... my preference would be for variadic functions like min to provide a sane return value when provided no arguments, and in this particular case, for it to return no values (after all, the minimum of no values is none - it makes perfect sense). This is a bit unusual as it doesn't really match existing conventions but it goes back to what I was getting at with revisiting "Racket's core values" in the RacketCon talk, and as it happens, I believe it is consistent with so-called "Buddhist/sunyata logic" with respect to the idea of logical existence and voidness, so the general inclination here is not without longstanding precedent. I'm planning on eventually putting together a detailed RFC for that, and cases like this one are great empirical data to include for consideration. Maybe I ought to start a draft of it and solicit community contributions / examples to get the ball rolling... but I guess until such a point, if you run into more of these, do send them my way as I'm eager to collect examples for this eventual RFC
One thing I've wondered about is whether core changes are really required for these things or whether it would be possible to use continuations in the implementation to pass multiple values across in ways that can be controlled at the Qi level (I have very little experience with continuations) while avoiding list conversions. That wouldn't be as good as changes in the core of course (which I think can be backwards compatible, as these changes would handle formerly error-resultant cases to produce new, legitimate logical behavior. They wouldn't modify existing legitimate behavior to produce different behavior), since core changes would affect the behavior of all functions like min, which we couldn't do much about at the Qi level. But if possible, it would be a really great fallback.
I haven't had a chance to look at define unity in detail, but I'll take a closer look soon!
1 Like
You've addressed all of the things I said, and all of things I left unsaid. Much appreciated
If I think of more examples where values are awkward, I'll let you know. Thank you for this wonderful effort and the ideas that motivate it.
1 Like
I went back and looked at the implementation of count and it turns out that it was doing something fancy and was slow. I improved it to be closer to the metal and now it's 5X faster. I also added a similar implementation of a new form, live?, to check if there are any values flowing. Now your example could be written as:
(~> … (pass number?) (if live? min #f))
and avoids counting the input as you suggested (although it does still convert the values to a list). Thanks again for pointing this out! I hope this performs well enough for what you need.
1 Like
Brilliant! I'll be curious to see the implementation change
P.S. More concrete work is coming soon on those demo issues.
1 Like
It was just this:
(flow (~> (>< 1) +))
Fun, but unfortunately not as fast as this:
(λ args (length args))
2 Likes