Announcing Qi 5: Flowing with Lists
Qi is an embeddable language for easily expressing flow-oriented computations in your programs. And now, you can use it seamlessly with list operations.
Friends,
It gives me great pleasure to announce that, after a year of work, Qi 5 is out in the wild! Upgrade now via:
raco pkg update qi
This is the "Snow Leopard" of Qi releases --- that is, few flashy new features, but lots of improvements under the hood. The biggest change is in the architecture of the compiler, and specifically of deforestation, in order to support optimized list operations.
Now, you may recall that Qi added support for this standard optimization technique (which avoids constructing intermediate representations in sequences of map, filter, foldl, …) in the last release, and it resulted in a huge speedup! Mission accomplished, right?
Not really. In fact, the mission had only just begun 
. Read on to learn more about our journey since then.
"Good Fences Make Good Neighbors"
For starters, we were uneasy about the fact that Qi achieved the aforementioned speedup by matching and transforming use of Racket syntax, not Qi syntax. This transgression upon the host language's turf felt wrong. In formal terms, it conflated Qi's invariants with those of the host language.
With this release, we've enriched the Qi core language so that it is capable of natively expressing deforestation, forging a clear separation between Qi and Racket.
"Good fences make good neighbors," they say, and that's certainly true of languages!
Producers, Transformers and Consumers
Formerly, while Qi's deforested runtime was robust and awesome (thanks to the amazing work of Michael Ballantyne and Dominik Pantůček!), it was kind of its own special thing in the compiler, with optimized runtimes existing in a one-to-one correspondence with surface operations like map.
This release starts to develop this runtime as a stratification across two levels --- analogous to expansion and compilation --- where the surface level includes the standard list operations, and the core level is composed of primitive streams called producers, transformers, and consumers. With these foundations, we hope to be able to extend deforestation to any functional list operation, including custom ones defined by you!
How do we intend to do that, you ask?
Deep macros
Ordinary macros allow you to specify the expansion of newly defined forms to existing forms in the host language.
Qi now includes "deep" macros that allow you to specify the behavior of a newly defined form at many levels, including both its expansion to Qi and, in addition, a Racket runtime. In the future, we plan to allow specifying compilation parameters for leveraging the deforested runtime, too, using the same approach.
That sounds cool, but does it work?
Introducing qi/list
Well, let's see. This release introduces qi/list, a new collection modeled after racket/list that includes operations like map, filter, and foldl, replacing the need in Qi for the Racket forms of the same name. There are 15 such deforestable operations currently included, with lots more planned!
In an analogous way to how many built-in Racket features are actually implemented using the same macro system that's provided to users, all of the operations in qi/list are implemented using the "deep macro" approach, thus showing that the deforested runtime could eventually be extended to user operations in the same way.
A big difference from racket/list is that these forms exhibit Qi syntax in higher-order positions, so you can do things like this today:
(~> (my_list) (map (~> sqr add1)) (filter (and positive? odd?)))
... where formerly, you might have done:
(~>> (my_list) (map (λ (v) (add1 (sqr v)))) (filter (λ (v) (and (positive? v) (odd? v)))))
Qi is a DSL purpose-built for describing functions. It's the right language for the job, and now, we get to use it here without having to jump through any hoops!
"Zip"
A little while back, Jens Axel posted a simple coding problem as a challenge on Discord. It looked like a problem that would have an elegant solution in Qi, but we were surprised to find that it was awkward! It turned out that there was a missing feature: a values-oriented version of the familiar zip operation from functional languages.
We recently added this feature, and along the way, realized that the natural way to implement it was as a generalization of Qi's "Pink Floyd" operators, △ ("separate") and ▽ ("collect").
(~> ('(a b c) '(1 2 3)) (△ ▽)) ;=> '(a 1) '(b 2) '(c 3)
With Lots More in Store!
There are so many interesting and fun avenues of future exploration, including:
- #lang experiments to design a top-level language exhibiting seamless flow-orientation.
- Exploring Qi as a natural language for working with continuations.
- Compiling Qi to different backends like futures or Big Data primitives.
- And even a possible community compiler competition (!).
You can read all about these plans in the notes from the recent release party.
Thank You
Many people contributed in ways large and small to this release. As Qi follows Attribution Based Economics, in principle, these contributions are recognized by formalized social processes. But as we're still waiting on some promised new ABE tools to be written (get on it, Sid!), these are very out of date. We will get around to updating them, in due time! Until then, these folks made this release possible:
- Dominik Pantůček developed the extensible runtime for deforestation in terms of producers, transformers, and consumers, and defined a super convenient (but very secret, internal) compiler "registry" for easily adding new compiler passes.
- Michael Ballantyne wrote the proof of concept "deep macro" extension scheme, showing how the compiler could be extended. He also contributed to the foundations for a theory of optimization and an understanding of effects in Qi.
- Matthias Felleisen provided behind-the-scenes support as Michael's advisor.
- Ben Knoble reviewed every change and kept us honest on a lot of finer points, both on the technical side as well as on development practices. With hawk-like attention to squirrely little details, he's even managed to inspire at least some of us (e.g., me) to use e.g., i.e., em dashes, en dashes, and … properly
- Stephen De Gabrielle helped out with organizing meetings, maintaining community calendars, and getting the word out on events.
- Sam Tobin-Hochstadt and Jacqueline Firth shared some insights that are likely to improve Qi's performance and also make it more compatible with Typed Racket (thanks also to Onorio Catenacci for motivating this!).
- Sam Phillips for being the godfather of
qi/list, and for other good ideas. - Hendrik Boom first identified the "Schrodinger's Probe" phenomenon, which helped us make progress on Qi's theory of optimization.
- Jens Axel Søgaard inspired the
zip-like generalization of△and▽. - Ben Knoble tested Qi on Frosthaven Manager to ensure there were no unplanned regressions.
- Ben Knoble developed curlique, a way to embed Qi as a top level language feature. This inspired our subsequent language experiments.
- Bogdan Popa suggested ways to dramatically improve build times in development workflows, saving us a lot of time.
- Greg Hendershott and Sage Gerard shared some insights to help iron out Qi's release and compatibility practices.
- Noah Ma wrote a book (!) on category theory in programming, and always inspires us to understand and generalize Qi's flow abstraction using category theory.
- Nia Angle contributed insightful "drive by" opinions.
- Sid Kasivajhula (that's me) coordinated things, provided design direction, and implemented a few things here and there
- Jay McCarthy came up with a fun name for Qi developers …
So try Qi 5! Follow Qi development at the project repo, and stop by at any Qi meetup (on Fridays) if you'd like to participate or even just hang out while we work on cool and fun things together.
~ Sid and the Qiwis 