Hello everyone,
I am new to Racket and was wondering if and how it is possible to have a decent experience (emacs/geiser) in terms of interactive development similar to common lisp. Is there something that allows to recompile just specific parts of the code without re-compiling everything? Like just a function/class definition etc...? Is it possible to have resumable exceptions?
I think this conflicts with the philosophy of the Racket module system. I know what you are talking about, you can do this in elisp, for example, and in common lisp, but things can get out of sync, so it is a bit of a hack. Is compiling whole modules too slow?
One thing I can suggest is to put the stuff you would normally do in the repl into a test or main submodule. This will recreate your state. Resumable exceptions would be nice, but again, that Lisp Machine type of hacking conflicts with the whole idea of the Racket module system--it will actually recompile modules multiple times for different contexts just to avoid leaking the kind of state you desire (see the paper on the module system for details).
Obligatory links:
Disclaimer: I haven't tried the library myself.
What do you mean (or what is) "the whole idea of the Racket module system" ?
Geiser supports Racket via the geiser-racket Emacs package. Have you tried it?
A Racket module can be instantiated more than one time at more than one phase, so what exactly would it mean to change a definition in the module from your editor without reloading? That is what I was getting at.
Back in the day, manually selecting a package to rebuild in CL, or sending a definition from the editor landed you in a potentially inconsistent state. You had to think through what dependent packages you might have to manually reload, based on what macros and eval-when clauses (primitive version of Racket phases) and state was stale. For general development, it done mainly because memory was measured in megabytes, and CPUs in megahertz. The blog post above indicates that Racket made a deliberate choice to avoid this situation.
There are still good reasons to use hot reloading when you need to preserve the environment (to avoid restarting emacs, to avoid losing video game test state, to avoid rebooting the operating system, etc.). In Racket, I think you have to design your program around the issue rather than leaning on a debugger. In addition to the links above, there is discussion in:
What is a scenario for this? Why would I want to instantiate a module more than once? and in multiple phases?
Sorry I am trying to understand.
Ok but somebody linked some articles and racket-reloadable, so are those working hacks to make racket more image-based?
In any modular system, if the modules are parameterized, then you will probably want to instantiate them more than once. In the case of Racket, modules are parameterized by phase level. The syntax at phase n is implemented at phase n+1. The syntax of phase n+1 is implemented at phase n+2. Hopefully that is accurate.
Apparently. I haven't looked into dynamic module reloading in Racket. I have used Racket to dynamically load C libraries. You never get a sane hot loading runtime system for free. Somebody has to specify the protocols and implement it. That could be the Posix environment and your C debugger, the CL metaobject protocol, the emacs/lisp system of symbol properties and hooks, or something you cooked up to remote debug a game console, hitting reload in your user agent/browser. People have implemented all kinds of dynamic systems.
Racket is all about the flexibility, so people can build all kinds of things with it. At the same time, Racket is all about the modularity; it should be possible to write code in a way and not worry about what a particular language form might mean later after someone recompiles it. To that end, I would suggest that Racket is kind of "all in" on the idea of separate compilation; a Racket "module" is, by default, sealed, except when its author explicitly provides a means to extend it.
This is definitely different from classic Lisp, classic Smalltalk, and the "Workbook" philosophy of Jupyter and similar systems. I would argue that the basic question is whether it's more important to you to be able to develop reliable software that is free from defects and will work reliably in the future, or whether it's more important to be able to get results quickly, and avoid recomputing potentially very expensive pieces of data. Both are totally legitimate work modes!
Wow, that's a lot of me mouthing off... I must be trying to avoid real work. Anyhow, apologies for anything I've gotten wrong, or any toes I've stepped on.
Which paper?
I can't find a link in the Racket Reference or Guide.
Matthew Flatt, "Composable and Compilable Macros, You Want it When?"
But, read all the PLT papers, and the old Scheme and Lisp ones! Also the paper @samth recently mentioned:
"Advanced Macrology and the Implementation of Typed Scheme", by Ryan Culpepper, Sam Tobin-Hochstadt, and Matthew Flatt has section "3.2 Modules, or You Want it When, Again?"
It would be great to have a full featured debugger and an image based environment, etc., but it would be a lot of work, just like anything else :).
I think part of the issue with hot patching would be just figuring out the desired semantics. If you actually did a bunch of work to track versions of code and user interaction and data to rebuild the environment correctly, in the end it might look a lot like you just kept your data in source files and hit F5! That is why I mentioned that in CL it was mostly a time-saver to reload just one package.
In reality, It seems like version control, unit testing, and databases have turned out to be more important than debuggers and images.
If I could magically improve the debugging experience for Racket, I think it would be some kind of debug mode for dynamic failures like contract violations and rackunit failures, like maybe automatically rerunning the code with some kind of tracing enabled. Maybe @ungsams has something really cool in mind?
This is definitely different from classic Lisp, classic Smalltalk, and the "Workbook" philosophy of Jupyter and similar systems. I would argue that the basic question is whether it's more important to you to be able to develop reliable software that is free from defects and will work reliably in the future, or whether it's more important to be able to get results quickly, and avoid recomputing potentially very expensive pieces of data. Both are totally legitimate work modes!
I'd like to clarify a misunderstanding which I feel is common in the Racket community w.r.t. the benefits of image-based development, and the drawbacks of lacking it. Of course, I don't mean to accuse you of stepping on toes or anything; this is IMO common across said community, and since I randomly happened on this thread while browsing just now it seemed a good opportunity to try to clear it up.
The benefits of being able to reach into an existing module and change its components in a running program (i.e. image-based development) aren't just that you get to avoid recompiling code, but let's go into that first for completeness. The development benefits are discussed at length in many other places (one example), so I will assume they're already part of the common ground here.
The other thing I want to point out is that this flexibility also extends to the language itself. Not only can programmers redefine code while a program is executing, but the code can be structured to take advantage of the compiler's metaprogramming abilities to rewrite itself; redefining forms at runtime, generating new packages, etc. In "normal" code this is not an advantage, as usually you have no need for self-rewriting code. However, the more you work on fundamental frameworks (e.g. writing DSLs with actually-complex behaviors, or full-fledged general-purpose languages), or alternatively the more esoteric your domain becomes, the more often you find need for code that manipulates other code at runtime, not just at compile time with macros (assuming you don't go for Greenspun's Tenth Rule to get around this constraint, ofc).
For a personal example, in my spare time I'm currently working on learning symbolic AI and researching UX for programming languages that allow direct manipulation of computation graphs. From the development-benefits side it would be very difficult to experiment with e.g. genetic programming if I needed to recompile the package every time I want to change something totally-unrelated to the cached domain-state. From the code-rewriting side, it would be similarly troublesome to wrangle with file-manipulation, a custom module system outside Racket's own, and/or custom DSLs and function-tables to modify variable-values and properties from other modules; this is a prerequisite to the genre of compilation / type-propagation I'm aiming to implement.
(Note: I'm sure someone somewhere has done at least one of these examples in practice, but even if solving said example would eliminate the more abstract point I'm making, proof-of-concept does not necessarily translate to proof-of-practicality)
Aside from the above benefits of avoiding recompilation, the other benefit of image based development is that you get to change the behavior of a module you don't own, rather than being beholden to the designs (and mistakes!) of the original developer. There are two general ways this can be useful:
One way redefining systems you don't own is useful is if you're dealing with circumstances the original authors may not have predicted and are too un-responsive (or otherwise out-of-contact) to request a fix from. In that case, one option for image-based languages is to download the library module directly from the source, modify constructs which are in the API contract (and so have well-defined behaviors), and then use it as normal, including with any other libraries you use but don't own that invoke the initial library. This is something I have done multiple times in Common Lisp programs. The other main option in this circumstance is to instead maintain a full-fledged own fork of the library, which means you need to take up the friction of forking it, publishing it, and ensuring that anyone using your library also uses your fork of the library you needed to modify.
Building off of the above, if we decide not to support modifying behavior inside existing modules, then that indirectly prohibits most non-mathematical libraries from being relied on independently of their maintainers. Users can't fix a library's flaws or gaps themselves unless they take up ownership and publishing of it, so as soon as a library's maintainer steps back from the job, the userbase needs to either immediately find another maintainer to take it up, or seek out an alternative even if it's technologically inferior. This dilemma remains even if the maintainer has a set time at which they promised they'll be coming back; it's not like users don't have to write code in the meanwhile!
Lacking the ability to reach into other people's modules also increases the friction to picking up maintainership of an open-source library. You can't play around with fixes and deploy them in your own code (incrementally working towards a place where you're confident enough to own the library and upstream your fixes there), so instead you have to either commit to maintainership from the start (and know for a fact that you have both the domain knowledge and bandwidth to handle it) or give up on using the library. This works against the ideal that a language ecosystem should be stable over time and only minimally require major-version-upgrades or library-switches in order to maintain availability.
Aside from the above, another way redefining modules you don't own is useful is in writing extensions. This is essentially the same process as fixes; you have a situation that the original module authors didn't anticipate, and there's no way to address it without either accessing internal variables/architecture of the existing module or to redefine the module entirely on your own (or, again, to not use the library, which is of course its own can of worms). Having these kinds of limitations on the ability of others to extend an existing package with custom functionality is a significant constraint on their ability to write their own code without the explicit support of the library authors they depend on; we're replacing dependence on the code level with significantly increased dependence on the human level, which is far more problematic!
(And no, the fact that you are to some degree already dependent on library authors to not do something ridiculous in the code-as-written does not amortize the new dependency of needing them to actively maintain their libraries in order to safely use them in production, nor do I expect that the slight reduction in the former from black-boxed modules outweighs the costs of introducing the latter)
Stepping back a bit, the lack of even optional support for image-based programming is honestly the primary (though not only!) reason I haven't dug deeper into Racket than necessary to understand some of the interesting libraries in this space (e.g. Turnstile+). I have other concerns (for instance, pretty bad introspection compared to e.g. Common Lisp and a lack of an inbuilt effect system for error/signal handling), but they're mostly things that could be fixed either by creating libraries or by working with the developers to improve the basic language runtime.
However, this particular gap is a barrier to both my development style and parts of my domains of interest, and the maintainers don't seem to have any intention of filling it. I'm not willing to invest in getting a proper intuition for a language that doesn't fit either my niche or my normal use-cases unless it becomes required by an employer / community or gets such a larger ecosystem than other Lisps as to be worthwhile despite the flaws in the runtime itself.
From the other side of things, to my understanding (as a novice Racketeer and somewhat-experienced Common Lisper) anything Racket does that image-based Lisps don't do can be implemented with macros and reader-macros, so there's no hard barrier on possible programs to worry about from using a runtime with image support instead. If there was some inherent benefit from the lack of image-based development that couldn't be gotten any other way, then it may be worthwhile to see if that benefit is worth the price. However, the only communication I've seen so far is that it "increases predictability" by making sure other developers can't design their own systems the way they want, which sounds to me like a lack of a proper optional-typing system around runtime modification side-effects (or a culture that gives a similar guarantee of modifications happening only if necessary, until such a typing system could be implemented), rather than a reasonable justification for intentionally constraining the possibility space of developer programs.
(When it's necessary for a design of any sort to constrain power-user behavior, the correct solution is nearly always to instead constrain the defaults while still leaving escape hatches. This maintains the desired default-state, while still addressing domains/problems where the unusual design pattern is genuinely necessary to either solve the problem at all, or to solve it within investment constraints)
If some experienced Racketeers read this, please let me know your thoughts, and any misrepresentations w.r.t. things Racket can do that I may not have encountered yet! Lisps in general are a gift to the programming world in comparison to most other languages I've had to work with, and I'd love to see if there's a way to improve Racket to resolve the issues mentioned above.
I have read it. I think I am experienced. I am not sure what kind of response you’d like. As far as I am concerned the design space for languages is vast. Just focusing on Lisp families you see a spectrum that ranges from 2Lisp, 3Lisp, the 1970s Lisps, Common Lisp, a range of Scheme implementations (it’s essentially a family of its own), Racket, and Clojure variants. Each member has chosen a different point, some more dynamic some more static. We, the Racket team, moved away from the original Scheme world to become more static for a range of reasons (some pedagogic, some implementation oriented, and probably others we didn’t explicitly state). Each member of this vast family has a desire to explore a particular direction. Racket has served some of its users extremely well, and others had to move on because it was too dynamic or too static.
I see your effort of writing down arguments we have been aware of for some decades, and we have tried to respond to those over time. Sadly I am not in a position to understand where you’re going with your write up, and that prevents me from giving an extensive answer.
I see your effort of writing down arguments we have been aware of for some decades, and we have tried to respond to those over time.
Most of the time when I have seen image-based programming discussed in the Racket context, the only one of the above points that has been addressed is the increased development efficiency (as being true but not significant enough to justify a language re-write).
Could you point me to some resources so I could get an idea on the thinking that's been done already on the other issues that arise from this gap?
On a separate note, I suspect I'm not the only one who's not aware of any responses for the other points. This may be indicative of improvement opportunities in how those responses are communicated to people considering ramping-up on Racket.
where you're going with your write-up
I'm working under the assumption that Racket has among its goals the four following ones:
Given the above priorities, this feature gap is a limitation in Racket's ability to fully fulfill its mission.
giving an extensive answer
I had a few questions which were either explicitly or implicitly called out in the above post. If you don't have anything else to bring up, could you answer those (assuming you have the required context)?
#lang reader system) be implemented via macros and reader macros in other Lisps? (And for completeness, are there any constraints on this level which the community is working to fix and could benefit from input from those experienced with other languages who may end up reading this thread?)Is there something that allows to recompile just specific parts of the code without re-compiling everything?
Here is a proof-of-concept.
Put the following in a file "defun.rkt".
Whenever you expect you will need to redefine a function, use defun instead of define.
Then in the repl, you can use defun to redefine the function.
Other functions referring to the function will pick up the new definition.
defun.rkt
#lang racket
(provide defun)
(define globals (make-hasheq))
(define-syntax ref
(syntax-rules ()
[(_ref id)
(hash-ref globals 'id)]))
(define-syntax defun
(syntax-rules ()
[(_defun (id arg ...) body0 body ...)
(begin
(define (id arg ...)
((ref id) arg ...))
(hash-set! globals 'id (λ (arg ...) body0 body ...)))]
[(_defun (id arg ... . rest-args) body0 body ...)
(begin
(define (id arg ... . rest-args)
(apply (ref id) (list* arg ... rest-args)))
(hash-set! globals 'id (λ (arg ... . rest-args) body0 body ...)))]))
An Example:
example.rkt
#lang racket
(provide fact foo)
(define (fact n)
42)
(define (foo n)
(fact n))
Running this gives us a repl.
> (fact 5)
42
> (foo 5)
42
> (defun (fact n) (if (= n 0) n (* n (fact (- n 1)))))
> (fact 5)
120
> (foo 5)
120
> (defun (fact n) 7)
(foo 5)
7
This proof of concept just uses a global "namespace" for all functions defined with defun.
The same technique can be used if one needs "packages" in Common Lisp sense.
A variation of the same idea.
In Chez Scheme symbols have property lists (as in Common Lisp).
This version stores the current function value in the symbol property list.
This might be fast than the previous version.
#lang racket
(require ffi/unsafe/vm)
(define putprop (vm-primitive 'putprop))
(define getprop (vm-primitive 'getprop))
(define-syntax ref
(syntax-rules ()
[(_ref id)
(getprop 'id 'globals)]))
(define-syntax defun
(syntax-rules ()
[(_defun (id arg ...) body0 body ...)
(begin
(define (id arg ...)
((ref id) arg ...))
(putprop 'id 'globals (λ (arg ...) body0 body ...)))]
[(_defun (id arg ... . rest-args) body0 body ...)
(begin
(define (id arg ... . rest-args)
(apply (ref id) (list* arg ... rest-args)))
(putprop 'id 'globals (λ (arg ... . rest-args) body0 body ...)))]))
