Hi @ShalokShalom thanks for posting the video. The title looks interesting but I've not been able to view it yet - it would be helpful if you could say what in the video you found relevant to this discussion. Posting some text in addition to a link will also make it less likely the spam filter will categorise your reply as spam.
Stephen
Here are some concise answers to your assumptions and questions:
Racket is a programmable programming language, meaning its macro system empowers programmers to tailor the language to special purposes. That’s what #lang is about. Another way of thinking about it is that this aspect of Racket is about coarse-grained DSL building and interactions among such DSLs.
Example 1: The construction of the teaching languages for HtDP is the first major example. These languages are widely used, and they inspired similar efforts.
Example 2: Typed Racket is a highly influential example. Its type-to-contracts compiler illustrates what we mean by making interactions between tailored Rackets safe.
Example 3: Rhombus, an ugly-syntax language for parenthophobes with the same powers as Racket when it comes to “tailoring”.
Racket also supports fine-grained DSL construction and interaction. The mechanisms (“macros that work”, “syntax spec”) support the construction of extensions of the expression and definition grammars.
Example 1: The Redex language is a domain-specific language for modeling the type systems and semantics of all kinds of programming languages.
Example 2: The syntax-spec system has just been used to teach undergraduate students how to construct such fine-grained examples,
ranging from logic-langauge extensions to contracts-for-students extensions and flight-planning ones. The system is still under development.
Racket is a plain programming language, more static than Lisp and Scheme (as of when we started) to ensure
— soundness (aka safety)
— compiler correctness reasoning (for an AOT compiler; the story might differe if it had a JIT compiler).
For this, we want a stable (not arbitrarily modifiable) ecosystem of libraries.
This last point implies that Racket is not a vehicle for exploring all possible points in the design space of languages. By moving away from, say, “everything is an S-expression” to somewhat more static and opaque structures (properties checkable before things go really bad), we have definitely restricted the explorable space to the HUGE space of notLisp/notSmalltalk.
In this light, the one question of yours that deserves a special answer is “are there any fundamental benefits of Racket's structure and runtime which cannot (unlike hygenic macros or the #lang reader system) be implemented via macros and reader macros in other Lisps?” Perhaps I fail to understand but the benefits of Racket’s macro systems and #lang approach to “language tailoring” are huge. If someone can implement “everything but those” in Lisp, I’d call this a half-baked quarter-interesting non-Racket. — Matthew, Robby, Sam, John, Jay and I (the core team) have written over 100 papers illustrating this point. I frankly don’t know how to condense them more into a reasonable message than this paragraph and the above basic points about what we think Racket is for.
While it is definitely true that “image-based development” in Racket is far different than many Lisp systems, I think it’s worth noting that there is a non-zero amount of support.
To quote one of @elibarzilay’s comments on “The DrScheme repl isn't the one in Emacs” (where “DrScheme” → “DrRacket” and “MzScheme” → “command-line racket”):
An incremental REPL is gone from PLT Scheme in a similarly superficial way: DrScheme does not support it — but it’s still part of the language, and you can still fire up MzScheme and talk to it like any other REPL.
Among the low-level mechanisms that exist are the compile-enforce-module-constants parameter and support for module redeclarations, e.g. via enter!, dynamic-rerequire, or explicit operations on namespaces.
Environments like XREPL integrate the low-level support into REPL meta-commands, and, as I wrote above:
Beyond the REPL, one tool for principled use of these features is the reloadable package, notably used in the implementation of https://pkgs.racket-lang.org.
Even DrRacket uses such features, e.g. to implement the “Reload #lang Extensions” menu item.
The other thing I want to point out is that this flexibility also extends to the language itself. Not only can programmers redefine code while a program is executing, but the code can be structured to take advantage of the compiler's metaprogramming abilities to rewrite itself; redefining forms at runtime, generating new packages, etc. In "normal" code this is not an advantage, as usually you have no need for self-rewriting code. However, the more you work on fundamental frameworks (e.g. writing DSLs with actually-complex behaviors, or full-fledged general-purpose languages), or alternatively the more esoteric your domain becomes, the more often you find need for code that manipulates other code at runtime, not just at compile time with macros (assuming you don't go for Greenspun's Tenth Rule to get around this constraint, ofc).
[…]
Aside from the above benefits of avoiding recompilation, the other benefit of image based development is that you get to change the behavior of a module you don't own, rather than being beholden to the designs (and mistakes!) of the original developer. There are two general ways this can be useful:
My (biased, less-than-nuanced) perspective is that Racket generally
takes the stance that, actually, compile-time macros are sufficient;
if you're building an inherently (extremely) dynamic system, you'll
probably want to wire things up to enable that kind of dynamism.
Racket's module boundaries are, well, important. Both for pedagogy
(organization of large systems) and safety (?)—students probably don't
need to mistakenly break DrRacket because they redefined something in
its core.
One way redefining systems you don't own is useful is if you're dealing with circumstances the original authors may not have predicted and are too un-responsive (or otherwise out-of-contact) to request a fix from. In that case, one option for image-based languages is to download the library module directly from the source, modify constructs which are in the API contract (and so have well-defined behaviors), and then use it as normal, including with any other libraries you use but don't own that invoke the initial library. This is something I have done multiple times in Common Lisp programs. The other main option in this circumstance is to instead maintain a full-fledged own fork of the library, which means you need to take up the friction of forking it, publishing it, and ensuring that anyone using your library also uses your fork of the library you needed to modify.
The burden is not always so high: I once put up my own version of a
dependency
so I could work around a
bug
that took the author some time to respond to (and later reverted
it.
Thus it is possible to replace a library in targeted ways somewhat
like you described doing for Common Lisp because Racket's package
sources are flexible.
No, I know it's not the same kind of "I write some code to reach into
your code and monkey patch everything"—I thought the Rails folks swore
that off, though?
Anyway, maybe this addresses your later question about "bus factor."
Working in the open and supporting package sources that are highly
flexible seems to mitigate a lot of possible issues.
Building off of the above, if we decide not to support modifying behavior inside existing modules, then that indirectly prohibits most non-mathematical libraries from being relied on independently of their maintainers. Users can't fix a library's flaws or gaps themselves unless they take up ownership and publishing of it, so as soon as a library's maintainer steps back from the job, the userbase needs to either immediately find another maintainer to take it up, or seek out an alternative even if it's technologically inferior. This dilemma remains even if the maintainer has a set time at which they promised they'll be coming back; it's not like users don't have to write code in the meanwhile!
Lacking the ability to reach into other people's modules also increases the friction to picking up maintainership of an open-source library. You can't play around with fixes and deploy them in your own code (incrementally working towards a place where you're confident enough to own the library and upstream your fixes there), so instead you have to either commit to maintainership from the start (and know for a fact that you have both the domain knowledge and bandwidth to handle it) or give up on using the library. This works against the ideal that a language ecosystem should be stable over time and only minimally require major-version-upgrades or library-switches in order to maintain availability.
Maybe it's me: I just don't see how being able to reach into all other
code and change it for your system promotes stability.
I suspect that I would prefer if you let me know if my package was
buggy in such a way (including being closed to the modifications you
wanted to make), so that it could be fixed for more than just you. Or,
if I disagreed, so that you knew my package and your goals were
incompatible.
Sometimes a package doesn't do what you need. That's ok. If you have
to fix it in your own, open source version, fine. I don't see how that
commits you to maintainership beyond the sort described
here.
[…]
However, this particular gap is a barrier to both my development style and parts of my domains of interest, and the maintainers don't seem to have any intention of filling it. I'm not willing to invest in getting a proper intuition for a language that doesn't fit either my niche or my normal use-cases unless it becomes required by an employer / community or gets such a larger ecosystem than other Lisps as to be worthwhile despite the flaws in the runtime itself.
Neat. Sorry to hear it's not for you. If you'll excuse that this might
sound snarky (I'm at least 50% genuinely curious): why jump into a
thread with this much text, then?
From the other side of things, to my understanding (as a novice Racketeer and somewhat-experienced Common Lisper) anything Racket does that image-based Lisps don't do can be implemented with macros and reader-macros,
That's a curious take. Maybe it's true in a Turing tarpit sense? My
days with Common Lisp (few, granted) didn't show me anything like what
Racket routinely does with language-oriented programming… and frankly
the extreme dynamism of the whole system was a lot to hold in my
brain.
I've found I tend to prefer Racket's "mostly less dynamic" point on
the spectrum—as I said earlier, when I need that much dynamism, I'll
pay the cost of building it. (Perhaps some libraries will
help.)
[…]
If some experienced Racketeers read this, please let me know your thoughts, and any misrepresentations w.r.t. things Racket can do that I may not have encountered yet! Lisps in general are a gift to the programming world in comparison to most other languages I've had to work with, and I'd love to see if there's a way to improve Racket to resolve the issues mentioned above.
I can buy that Racket could benefit from adding this capability in a
more structured way, but I have to imagine it would grow out of the
racket-reloadable library linked above as yet another facility for
writing and structuring some programs that need it (like structs,
classes, units, macros, etc.) rather than be so baked in that you can
change Racket itself as it runs.
Others have said things I might have thought, better, so I'll leave it here.
I think I'm mostly in alignment with @benknoble . I appreciate the thought and work you've put into your viewpoint, I think it's coherent and sensible, but I also think it's very different from mine. Specifically, I'm imagining putting together a PR when I discover a problem with a piece of software: if I'm going to tell you about a problem that I'm having with your code, it seems to me vital to be able to say that we're talking about a single piece of code with specific dependencies, not a (forgive me) monkey-patched intertangled updated version of it. You may point out that image-based programming gives me an easy way to capture my current state so that the developer of the code can examine it, but this will require you as developer to spend a probably-large amount of time figuring out what's going on with my environment, and possibly discover that the problem is that I modified your code in some way that I shouldn't have.
Am I misrepresenting your position?
I do wish that Racket had more native support for interactive workflows (somewhat similar to @swapneils I think), as it's particularly handy for long-running programs where there's a long setup time to arrive at the current state (e.g. data analysis, game engine, etc). You'd prefer to make a change anywhere in the system to tweak something without starting over and rebuilding all that state.
There are things like reloadable, but you have to mark a region of code to be reloaded in advance. The ideal state is the ability to change any part of the system without marking it ahead of time.
It's fairly clear to me Racket seems unlikely to gain this ability at this stage of its evolution though, so I'm left to search for other languages for these situations.
I'm totally spitballing here, but it seems like you might be willing to accept a line drawn between library code and target code here, no? That is, the ability to edit parts of your own code, but not other existing libraries? What about a setup that just marked all of your own code as reloadable? (Please note that I say this without any idea what the reloadable "thing" is...)
Or (still spitballing), what if the major change was the ability to freeze and reconstitute large pieces of data? It seems like this is the primary use case for something like this. I realize that in a world where functions are values, this proposal is arguably equivalent to just having image-based programming, but I suspect that you could get a substantial fraction of the desired functionality with only the abiilty to freeze and reconstitute things with easy mappings to (say) s-expressions.
Just wanted to mention Bogdan Popa's remote debugger.
The talk explores how Smalltalk's interactive environment and tools can inspire functional programming, emphasizing code exploration, visualization, and beginner-friendly abstractions.
Pharo ( a modern Smalltalk dialect) has put a high emphasis on discoverability and argues, the difference between a really skilled and a beginner functional programmer lies in how good they
can use the tooling to work around the pain points.
He discusses the implications of code visualization on library adoption and maintenance, highlighting the importance of having concrete metrics for code quality.
He stresses that teaching beginners should extend beyond pure, simple syntax by including strategies for managing and understanding medium-sized codebases. 
Then he speaks about the concept of domain-specific debuggers, and their role in improving the programming experience and they are proposing an alternative approach to managing memory by identifying and analyzing instances of classes.
Honestly, if you like to save time and get an overview as quickly as possible, I think you get the best impression by jumping directly into the IDE.
While it is definitely true that “image-based development” in Racket is far different than many Lisp systems, I think it’s worth noting that there is a non-zero amount of support.
Thank you for the links, these seem useful for when I do have to work with Racket. Thus far my Racket experience is on the command line side since I mainly ran experiments to better understand code in papers.
safety (?)—students probably don't need to mistakenly break DrRacket because they redefined something in its core.
I can see the need for this use-case, as with students their freedom to use their own problem-solving approaches genuinely isn't worth the costs complexity poses on their ability to learn.
On the other hand, as I understand it Schemes (including Racket) are already used in restricted forms when educating students, rather than providing access to the full language. Restricting access to image-orientation by e.g. forcing whole-file compilation when compiling any form in files with a specific #lang doesn't seem impossible, or even difficult if the image-oriented features themselves were well-designed in the context of Racket as a whole.
Neat. Sorry to hear it's not for you. If you'll excuse that this might
sound snarky (I'm at least 50% genuinely curious): why jump into a
thread with this much text, then?
"My" isn't really the important point there, just a data-source identifier 
More relevant is that it excludes some existing development styles and domains of interest.
In terms of motivation, while I'm not particularly attached to any Lisp in particular (or even Lisp for that matter), I am interested in the problem of "design a general-purpose language for both research and production purposes", both for personal use and for better solving some gaps in the overall software ecosystem. The above-mentioned points are among the constraints against Racket specifically solving that problem (through impairing its ability to morph itself into whatever a particular problem/developer needs), and unlike any other such constraints are ones I personally can reliably speak to.
If some aspect of "reasonably ergonomic generality (or mutability into the same) for all human programming problems" explicitly contradicts one of Racket's goals, that would be a clear reason to exclude it from consideration for changes towards fulfilling that goal. Barring such a fundamental contradiction, there's little reason not to engage and see if the community/maintainers are aligned to either my own proposals or to other solutions which would move it closer to the above generality.
On a separate note, opening conversation on somewhat contentious topics from a perspective of genuinely trying to understand whichever viewpoints you aren't already familiar with (in this case, "why does Racket not support image-oriented programming without rolling your own runtime") is a good way to improve both your own understanding and the publicly available documentation for others researching the same topic.
Specifically, I'm imagining putting together a PR when I discover a problem with a piece of software: if I'm going to tell you about a problem that I'm having with your code, it seems to me vital to be able to say that we're talking about a single piece of code with specific dependencies, not a (forgive me) monkey-patched intertangled updated version of it.
I view "code" in general (i.e. the below isn't specifically talking about Racket) as being on an axis between two different categories of use-cases, which I will label arbitrarily below for the sake of having short names to refer to them by.
One class of use-case is static runtimes, which have a specific known behavior that they should execute and then exit (or loop). These should indeed be approached the way you describe, with static code that resolves to a specific and predictable set of steady-states with specific and predictable transitions. Not only does this help the PR use-case, but it also improves predictability in actual usage of the program.
The other class of use-case is dynamic runtimes. From an academic perspective I've only seen this approach used in AI, specifically in "online learning" systems, but outside academia this viewpoint on programs is applicable to a variety of real-world application domains.
Dynamic runtimes are very useful when you want to put an application somewhere in a real-world process, don't know exactly what behavior it should have, and want to maintain availability of its intermediate versions even as you improve it (or as it improves itself) towards better fulfilling its role in that process.
I've mentioned above that some development styles require support for this use-case, meaning that supporting it in turn supports those specific development styles, and the programmers and problems for which they particularly shine.
In practice, it also seems that sufficiently large software applications naturally transition to the dynamic runtime format, with e.g. "microservices" and "continuous deployment" systems providing a very clunky approach to segregated redefinition of portions of the larger system, without compromising the functionality of the other components or the system as a whole. Better support for dynamic runtimes on the language level both improves the expected quality of solutions for this "large application" use-case, and allows smaller teams or individual developers to better scale their software development by using the language's in-built functionalities for what would otherwise require significant additional infrastructure and coding-discipline.
The discussion of benefits from image-based programming above is from the implicit position that supporting use-cases for dynamic runtimes doesn't mean you have to use the language that way; i.e. you can have either static, or static+dynamic, but the only-dynamic case (where the runtime's dynamicity inherently prohibits implementing static programs with predictable behaviors) is avoidable without sacrificing either dynamicity or orthogonal features. Given this position, supporting the dynamic use-case has no bearing on the static use-case you present.
The above position could be disproved if there's meaningful constraints on what features dynamic runtimes can support. I wouldn't expect such constraints, however, since even e.g. proof assistants are implemented mostly-interactively despite effectively requiring whole-program proof-search to maintain the accuracy of their domain-models. Most likely the worst-case cost is higher compile times for certain features (like complex type systems), which can always be ameliorated with optional file-level batch-compilation without sacrificing dynamicity.
Taking the above and applying it to Racket, for Racket to fulfill the "generality" goal mentioned in a different reply above, it ideally should be modifiable into a language supporting dynamic runtime behavior in both the final application and the development of said language itself. Currently this requires reimplementing significant parts of Racket in a compatible manner and redoing that work for every language-creation problem in this category, which is cumbersome and an effective blocker to such usage.
That is, the ability to edit parts of your own code, but not other existing libraries?
Or (still spitballing), what if the major change was the ability to freeze and reconstitute large pieces of data?
I'm not certain exactly what you have in mind in terms of limits on this freezing/reconstitution process?
In terms of clarifying the specific points you seem to be engaging with, image-based programming aims towards erasing the line between "static code" and "running code" (Smalltalk is a good example and the origin of the concept, as @ShalokShalom notes). This barrier can be added back afterwards if the developer wants it, but having the runtime enforce it (barring specific code/compilation options to toggle that enforcement) is a decision with costs to programmer dev-X and breadth-of-language-application, with apparently no benefits that cannot be achieved through other means.
This last point implies that Racket is not a vehicle for exploring all possible points in the design space of languages.
Thank you, given this fact many of my points regarding Racket's conformance to its own goals are indeed irrelevant; though I still feel Racket's 'meta-goal' of supporting PL research could benefit from it becoming a vehicle for exploring all possible points in this design space instead of just some of them.
hygenic macros or the
#langreader system
Apologies, my statement was meant to imply that both of these features seemingly can be implemented in e.g. CL, albeit with some difficulty and with constraints on generality-of-application which would not be meaningfully harmful to their usability. Please see just below for a more nuanced take on the topic.
My days with Common Lisp (few, granted) didn't show me anything like what
Racket routinely does with language-oriented programming…
While I suspect there isn't a way to implement Racket's specific macro system globally in e.g. a CL environment without a fragile parser for environment-related data to attach to the raw s-expr, from what I've read of the docs I haven't found any apparent end goals of Racket's features which on-the-surface seem unimplementable in a language with both s-expr- and reader- macros, especially if you relax the globality constraint to "this property is global within the scope of this library's usage".
I suspect the lower prevalence of such activities in the CL ecosystem is mainly a factor of different community interests; the CL community leans heavily towards either production software or solo hobby projects, rather than PL researchers.
Building off of the above, there's also the point that CL users haven't invested in libraries constraining CL's redefinition or dynamic binding capabilities, since from an application perspective there's very little benefit to this. On the other hand, I suspect (as not-a-professional-PL-researcher) there are few domains of modern PL research where you could easily write papers about languages utilizing image-oriented features, given the incentive to make your results legible via (preferably-mathematical) guarantees to readers/reviewers, and that in fact they would make it harder to derive such static guarantees. As such, if you lack libraries constraining against use of these features, there's a (weak) incentive for PL researchers to move to languages constraining against them instead.
The only case I can imagine where a group has economic incentives specifically to research programming languages in CL's runtime instead of Racket's (rather than being indifferent, library ecosystem and standard-library PL-research-support notwithstanding) is when they're also application developers, where the runtime's dynamicity is an advantage for the language they're developing. Coalton's development by Rigetti computing (a CL child-language with HM typing and a relatively-fluid interface with the host Common Lisp runtime) is one example of this circumstance.
However, unless I get the bandwidth and muse-interest to read through the papers and actually attempt a port, I'm admittedly not qualified to speak further on whether non-image-orientation has a "moat" here.
While the specific implementations were clearly based on how Racket is designed, perhaps the maintainers could chime in if there's some inherent technical barriers to solving the use-cases hygenic macros / #lang parsers address in an image-oriented runtime?
Well, perhaps for certain projects you can find a good place to draw such a line, but I can imagine situations where it moves over time (e.g. from "just my own code" to "my own code plus this one library").
reloadable essentially does what you're describing here I think...? It has you draw the kind of line you are talking about between a "permanent" part and a "reloadable" part of the program. State that you want to preserve across reloads has to be in the permanent side, so you would have to carefully tune where this line falls for your particular project.
That's a lot more fiddly than environments like Common Lisp, Julia, Clojure (along with Java and others on the JVM), etc. which allow editing a function's behaviour while retaining surrounding state without needing to draw any boundaries in advance. Now, I am sure attempting bring this form of "surgical" function replacement to Racket would be quite a complex task, especially once you work through the complexities of modules, macros, etc. which are all baked quite deeply into the guts of Racket's core.
Perhaps, but I assume you'd still be drawing lines in advance around specific variables or structs to be reloaded...? I fear there's still lurking complexities in this direction, but I'd be curious to see more if someone tries to go down such a path.
On the one hand: I want a single source of truth with transparent change history. I'm firmly in the "static", "source is truth", "reproducible results from source" camp.
I'm with @benknoble: If a third party library needs a fix/change, I'd rather fork or copypasta than monkey patch.
In developing Racket Mode for Emacs, I program roughly equally in Racket and Emacs Lisp. I've experienced the convenience of monkey-patching... as well as the inconvenience of needing to restart Emacs after some series of accumulated opaque state hacks.
On the other hand: Most of my Emacs monkey-patching consists of just using eval-defun to redefine a function to be instrumented for debugging. (Which is relatively easy to remember to undo later.)
And that's an interesting point, IMHO: "Running your program in a step debugger" means "rewriting your program to be step-debug-able". I mean, that's true even when a native debugger replaces an opcode.
In other words, instrumenting (debugging, tracing, etc.) is a special case of temporary monkey-patching.
This story could be better for Racket, including for long-running programs. Status quo, you have to stop and specially re-run your program, to instrument it -- because a special evaluator really does rewrite your program to be debug-able or trace-able or whatever, and runs that. (And the instrumented result is too slow for that to be the default.)
And if that story were improved, you could imagine one kind of "instrumentation" could be any arbitrary redefinition (preferably with some change/undo history preserved during the session). Which might be a useful improvement for many people, if not as much as OP would want.
A simple set! + compile-enforce-module-constants may suffice to redefine within a limited scope, but I think the desired thing would be closer to changing the called site in RAM, for all scopes/users?
Does anyone have a reference to how the JVM supports this? I've not heard Java programs described in this way, for example, and I'm now curious. (I assume this is different than the dynamic/reflective things Spring does, for example?)
The JVM allows loading a new class at runtime, even if it is the same name as an existing class, but it will be a new type. Once you have that facility, you can implement things like racket-reloadable but you can't change the implementation of existing objects in the heap, for example (except by using the debugging API).
The JVM allows redefining classes at run time, though this is subject to some restrictions in typical JVMs. With the commonly used OpenJDK JVM, you can't add new functions, change signatures, etc. But you can change a method body and existing object state is preserved. There are other development-focused JVMs (e.g. JetBrains) that allow essentially any changes (first prototyped in a 2010 research project).
In terms of overall workflow, this ability is usually accessed via the JVM debugging protocol. You make changes in your editor, save and build an updated class file, and the new class bytes are sent to the JVM via your debugging connection.
That here is exactly what we were talking about an interactive development experience.
It's implemented in Smalltalk (Pharo) and currently has no Racket implementation.
I had been told by a guy who implements Elixir on that platform,
This project is a good template to understand what an implementation would look like.
Currently, it supports Java, C#, Ruby/Rails, Python, TypeScript, JavaScript, React, COBOL, GraphQL, and Gemstone.
It provides a dedicated environment for defining support for other languages.
Further, there are the general moose tools inside of GT, so you can analyze many different languages:
