Qi Accelerated - Qi 4 release announcement
Qi is a functional, flow-oriented language that's simple, expressive and easy to embed anywhere in Racket programs. And now, it's also blazing fast!
Friends,
It gives me great pleasure to announce that, after more than a year of work, we released Qi 4 on Friday! Upgrade now:
$ raco pkg update qi
If you missed last Friday's release event, fear not, it's covered in detail in the notes here:
This is the biggest release we've done yet, featuring major contributions by many community members, and boy, do we have some good stuff for ya! Grab some popcorn 
If you are unfamiliar, Qi is a flow-oriented language emphasizing the functional style while being simple and fun to use, and easy to embed anywhere in Racket programs.
And now, with this latest release, Qi is also blazing fast! Check out these benchmarks:
What is this data telling us?
On functional computations involving standard higher order functions like map, filter, and foldl / foldr, Qi achieves something like a 3x speedup over equivalent code written in plain Racket! It does this by employing the stream fusion / deforestation optimization (the same one used in Haskell's GHC) which traverses input collections just once, and avoids constructing intermediate representations on the way to the final result.
Of course, as Qi compiles to Racket, it cannot truly exceed Racket performance, and Racket provides many specialized and optimized ways of performing the same computations, such as for forms together with lazily constructed sequences like in-list. What we are talking about here is performance of code that Qi considers idiomatic. Qi emphasizes functional programming and the use of higher order functions, and it is this style that we seek to enable by making it perform as well as more declarative or imperative styles that are otherwise faster in Racket.
As those benchmarks show, Qi's performance on many of these tasks is almost on par with the fastest ways that Racket offers to do these computations.
This is an incredible result and it wouldn't have been possible without the contributions of many in the community. I want to especially recognize Michael Ballantyne who supplied the initial implementation of stream fusion that achieved "ignition," Vincent St-Amour for writing a very clear survey of the subject that we consulted frequently, and Dominik PantůÄek for generalizing the implementation into the robust production version we have today.
This release proves that Qi can add useful optimizations to make idiomatic code performant. But it's only the beginning. There are many parts of the language that we'd like to make faster, and optimizations that we've identified to pursue, and I am sure that there are many folks in the community who may have ideas on optimizations that would be natural for Qi. We aim to keep Qi development as accessible as possible and hope to leverage the immense talent and interest here to ensure that we all have the best tools and the best languages. If you'd like to participate in Qi development, please follow updates on the source repo.
By the way, the compiler effort is somewhat unique in that we have the entire project chronicled from start to finish in detailed meeting notes, so this is another way to keep tabs on our progress:
Other highlights of this release:
- The code is now effectively at 100% test coverage (well, technically 99% ... we will get there )
- The wiki contains 79 entries containing developer documentation
- The Qi SDK got an upgrade and includes ways to generate quick local, nonlocal, competitive, and regression reports on benchmarks
- Qi now supports native bindings! You can bind intermediate values in a flow using
as:(~> (3) (as v) (gen v)). Of course, in most cases you won't need bindings, but they can aid clarity in some cases.
In addition to those already mentioned, these folks helped make this release possible:
- Ben Knoble, who kept us honest on normalization rules (i.e. rewriting many different versions of source code to a common and simple representative expression for subsequent optimization), ensuring that they don't change the semantics of the language.
- Michael Ballantyne, whose research enables the stratified DSL architecture that allows Qi to have an optimizing compiler (via Syntax Spec -- a next-generation language workbench library for Racket currently in preview).
- Matthias Felleisen, for behind-the-scenes support as Michael's advisor.
- Dominik PantůÄek, who is working on a new, rigorous and flexible benchmarking suite which we hope can be made available for general purpose benchmarking in the not too distant future (i.e. not specifically for Qi!). It accounts for ambient factors like garbage collection and computes reliable statistics on the generated data -- this how those neat charts above were generated!
- Sam Phillips, who opened the floodgates to deforestation of
racket/listAPIs. - Noah Ma, Siyuan Chen, and also Ben who helped us test the new version on existing Qi codebases like Qi-Cat, Qi-Circuit, and Frosthaven Manager.
- Jair Trejo, who originally suggested a link between Qi's values-oriented computations and Clojure's transducers, which is likely to inform further work as we aim to make Qi's deforestation generic.
- Stephen De Gabrielle, who helped with logistics for community organizing, meetings, etc.
- Sam Tobin-Hochstadt and my friend Alan for designing Qi's new logo
, and also, Sam along with Gustavo Massaccesi, for suggesting ways to test the compiler that helped ensure that it's operating as intended and which saved us a lot of time in identifying the sources of bugs during development.
I've surely missed many people here, but luckily, Qi follows Attribution Based Economics (ABE), a much more robust way to recognize contributions and the people behind them. Here is the full list of people and agencies that have been recognized as contributors to Qi so far (it will soon be updated to account for the compiler work).
Thanks for reading, and enjoy Qi 4!