Hi @ShalokShalom thanks for posting the video. The title looks interesting but I've not been able to view it yet - it would be helpful if you could say what in the video you found relevant to this discussion. Posting some text in addition to a link will also make it less likely the spam filter will categorise your reply as spam.
Stephen
Here are some concise answers to your assumptions and questions:
Racket is a programmable programming language, meaning its macro system empowers programmers to tailor the language to special purposes. That’s what #lang is about. Another way of thinking about it is that this aspect of Racket is about coarse-grained DSL building and interactions among such DSLs.
Example 1: The construction of the teaching languages for HtDP is the first major example. These languages are widely used, and they inspired similar efforts.
Example 2: Typed Racket is a highly influential example. Its type-to-contracts compiler illustrates what we mean by making interactions between tailored Rackets safe.
Example 3: Rhombus, an ugly-syntax language for parenthophobes with the same powers as Racket when it comes to “tailoring”.
Racket also supports fine-grained DSL construction and interaction. The mechanisms (“macros that work”, “syntax spec”) support the construction of extensions of the expression and definition grammars.
Example 1: The Redex language is a domain-specific language for modeling the type systems and semantics of all kinds of programming languages.
Example 2: The syntax-spec system has just been used to teach undergraduate students how to construct such fine-grained examples,
ranging from logic-langauge extensions to contracts-for-students extensions and flight-planning ones. The system is still under development.
Racket is a plain programming language, more static than Lisp and Scheme (as of when we started) to ensure
— soundness (aka safety)
— compiler correctness reasoning (for an AOT compiler; the story might differe if it had a JIT compiler).
For this, we want a stable (not arbitrarily modifiable) ecosystem of libraries.
This last point implies that Racket is not a vehicle for exploring all possible points in the design space of languages. By moving away from, say, “everything is an S-expression” to somewhat more static and opaque structures (properties checkable before things go really bad), we have definitely restricted the explorable space to the HUGE space of notLisp/notSmalltalk.
In this light, the one question of yours that deserves a special answer is “are there any fundamental benefits of Racket's structure and runtime which cannot (unlike hygenic macros or the #lang reader system) be implemented via macros and reader macros in other Lisps?” Perhaps I fail to understand but the benefits of Racket’s macro systems and #lang approach to “language tailoring” are huge. If someone can implement “everything but those” in Lisp, I’d call this a half-baked quarter-interesting non-Racket. — Matthew, Robby, Sam, John, Jay and I (the core team) have written over 100 papers illustrating this point. I frankly don’t know how to condense them more into a reasonable message than this paragraph and the above basic points about what we think Racket is for.
While it is definitely true that “image-based development” in Racket is far different than many Lisp systems, I think it’s worth noting that there is a non-zero amount of support.
To quote one of @elibarzilay’s comments on “The DrScheme repl isn't the one in Emacs” (where “DrScheme” → “DrRacket” and “MzScheme” → “command-line racket”):
An incremental REPL is gone from PLT Scheme in a similarly superficial way: DrScheme does not support it — but it’s still part of the language, and you can still fire up MzScheme and talk to it like any other REPL.
Among the low-level mechanisms that exist are the compile-enforce-module-constants parameter and support for module redeclarations, e.g. via enter!, dynamic-rerequire, or explicit operations on namespaces.
Environments like XREPL integrate the low-level support into REPL meta-commands, and, as I wrote above:
Beyond the REPL, one tool for principled use of these features is the reloadable package, notably used in the implementation of https://pkgs.racket-lang.org.
Even DrRacket uses such features, e.g. to implement the “Reload #lang Extensions” menu item.
The other thing I want to point out is that this flexibility also extends to the language itself. Not only can programmers redefine code while a program is executing, but the code can be structured to take advantage of the compiler's metaprogramming abilities to rewrite itself; redefining forms at runtime, generating new packages, etc. In "normal" code this is not an advantage, as usually you have no need for self-rewriting code. However, the more you work on fundamental frameworks (e.g. writing DSLs with actually-complex behaviors, or full-fledged general-purpose languages), or alternatively the more esoteric your domain becomes, the more often you find need for code that manipulates other code at runtime, not just at compile time with macros (assuming you don't go for Greenspun's Tenth Rule to get around this constraint, ofc).
[…]
Aside from the above benefits of avoiding recompilation, the other benefit of image based development is that you get to change the behavior of a module you don't own, rather than being beholden to the designs (and mistakes!) of the original developer. There are two general ways this can be useful:
My (biased, less-than-nuanced) perspective is that Racket generally
takes the stance that, actually, compile-time macros are sufficient;
if you're building an inherently (extremely) dynamic system, you'll
probably want to wire things up to enable that kind of dynamism.
Racket's module boundaries are, well, important. Both for pedagogy
(organization of large systems) and safety (?)—students probably don't
need to mistakenly break DrRacket because they redefined something in
its core.
One way redefining systems you don't own is useful is if you're dealing with circumstances the original authors may not have predicted and are too un-responsive (or otherwise out-of-contact) to request a fix from. In that case, one option for image-based languages is to download the library module directly from the source, modify constructs which are in the API contract (and so have well-defined behaviors), and then use it as normal, including with any other libraries you use but don't own that invoke the initial library. This is something I have done multiple times in Common Lisp programs. The other main option in this circumstance is to instead maintain a full-fledged own fork of the library, which means you need to take up the friction of forking it, publishing it, and ensuring that anyone using your library also uses your fork of the library you needed to modify.
The burden is not always so high: I once put up my own version of a
dependency
so I could work around a
bug
that took the author some time to respond to (and later reverted
it.
Thus it is possible to replace a library in targeted ways somewhat
like you described doing for Common Lisp because Racket's package
sources are flexible.
No, I know it's not the same kind of "I write some code to reach into
your code and monkey patch everything"—I thought the Rails folks swore
that off, though?
Anyway, maybe this addresses your later question about "bus factor."
Working in the open and supporting package sources that are highly
flexible seems to mitigate a lot of possible issues.
Building off of the above, if we decide not to support modifying behavior inside existing modules, then that indirectly prohibits most non-mathematical libraries from being relied on independently of their maintainers. Users can't fix a library's flaws or gaps themselves unless they take up ownership and publishing of it, so as soon as a library's maintainer steps back from the job, the userbase needs to either immediately find another maintainer to take it up, or seek out an alternative even if it's technologically inferior. This dilemma remains even if the maintainer has a set time at which they promised they'll be coming back; it's not like users don't have to write code in the meanwhile!
Lacking the ability to reach into other people's modules also increases the friction to picking up maintainership of an open-source library. You can't play around with fixes and deploy them in your own code (incrementally working towards a place where you're confident enough to own the library and upstream your fixes there), so instead you have to either commit to maintainership from the start (and know for a fact that you have both the domain knowledge and bandwidth to handle it) or give up on using the library. This works against the ideal that a language ecosystem should be stable over time and only minimally require major-version-upgrades or library-switches in order to maintain availability.
Maybe it's me: I just don't see how being able to reach into all other
code and change it for your system promotes stability.
I suspect that I would prefer if you let me know if my package was
buggy in such a way (including being closed to the modifications you
wanted to make), so that it could be fixed for more than just you. Or,
if I disagreed, so that you knew my package and your goals were
incompatible.
Sometimes a package doesn't do what you need. That's ok. If you have
to fix it in your own, open source version, fine. I don't see how that
commits you to maintainership beyond the sort described
here.
[…]
However, this particular gap is a barrier to both my development style and parts of my domains of interest, and the maintainers don't seem to have any intention of filling it. I'm not willing to invest in getting a proper intuition for a language that doesn't fit either my niche or my normal use-cases unless it becomes required by an employer / community or gets such a larger ecosystem than other Lisps as to be worthwhile despite the flaws in the runtime itself.
Neat. Sorry to hear it's not for you. If you'll excuse that this might
sound snarky (I'm at least 50% genuinely curious): why jump into a
thread with this much text, then?
From the other side of things, to my understanding (as a novice Racketeer and somewhat-experienced Common Lisper) anything Racket does that image-based Lisps don't do can be implemented with macros and reader-macros,
That's a curious take. Maybe it's true in a Turing tarpit sense? My
days with Common Lisp (few, granted) didn't show me anything like what
Racket routinely does with language-oriented programming… and frankly
the extreme dynamism of the whole system was a lot to hold in my
brain.
I've found I tend to prefer Racket's "mostly less dynamic" point on
the spectrum—as I said earlier, when I need that much dynamism, I'll
pay the cost of building it. (Perhaps some libraries will
help.)
[…]
If some experienced Racketeers read this, please let me know your thoughts, and any misrepresentations w.r.t. things Racket can do that I may not have encountered yet! Lisps in general are a gift to the programming world in comparison to most other languages I've had to work with, and I'd love to see if there's a way to improve Racket to resolve the issues mentioned above.
I can buy that Racket could benefit from adding this capability in a
more structured way, but I have to imagine it would grow out of the
racket-reloadable library linked above as yet another facility for
writing and structuring some programs that need it (like structs,
classes, units, macros, etc.) rather than be so baked in that you can
change Racket itself as it runs.
Others have said things I might have thought, better, so I'll leave it here.
I think I'm mostly in alignment with @benknoble . I appreciate the thought and work you've put into your viewpoint, I think it's coherent and sensible, but I also think it's very different from mine. Specifically, I'm imagining putting together a PR when I discover a problem with a piece of software: if I'm going to tell you about a problem that I'm having with your code, it seems to me vital to be able to say that we're talking about a single piece of code with specific dependencies, not a (forgive me) monkey-patched intertangled updated version of it. You may point out that image-based programming gives me an easy way to capture my current state so that the developer of the code can examine it, but this will require you as developer to spend a probably-large amount of time figuring out what's going on with my environment, and possibly discover that the problem is that I modified your code in some way that I shouldn't have.
Am I misrepresenting your position?
I do wish that Racket had more native support for interactive workflows (somewhat similar to @swapneils I think), as it's particularly handy for long-running programs where there's a long setup time to arrive at the current state (e.g. data analysis, game engine, etc). You'd prefer to make a change anywhere in the system to tweak something without starting over and rebuilding all that state.
There are things like reloadable, but you have to mark a region of code to be reloaded in advance. The ideal state is the ability to change any part of the system without marking it ahead of time.
It's fairly clear to me Racket seems unlikely to gain this ability at this stage of its evolution though, so I'm left to search for other languages for these situations.
I'm totally spitballing here, but it seems like you might be willing to accept a line drawn between library code and target code here, no? That is, the ability to edit parts of your own code, but not other existing libraries? What about a setup that just marked all of your own code as reloadable? (Please note that I say this without any idea what the reloadable "thing" is...)
Or (still spitballing), what if the major change was the ability to freeze and reconstitute large pieces of data? It seems like this is the primary use case for something like this. I realize that in a world where functions are values, this proposal is arguably equivalent to just having image-based programming, but I suspect that you could get a substantial fraction of the desired functionality with only the abiilty to freeze and reconstitute things with easy mappings to (say) s-expressions.
Just wanted to mention Bogdan Popa's remote debugger.
The talk explores how Smalltalk's interactive environment and tools can inspire functional programming, emphasizing code exploration, visualization, and beginner-friendly abstractions.
Pharo ( a modern Smalltalk dialect) has put a high emphasis on discoverability and argues, the difference between a really skilled and a beginner functional programmer lies in how good they
can use the tooling to work around the pain points.
He discusses the implications of code visualization on library adoption and maintenance, highlighting the importance of having concrete metrics for code quality.
He stresses that teaching beginners should extend beyond pure, simple syntax by including strategies for managing and understanding medium-sized codebases. 
Then he speaks about the concept of domain-specific debuggers, and their role in improving the programming experience and they are proposing an alternative approach to managing memory by identifying and analyzing instances of classes.
Honestly, if you like to save time and get an overview as quickly as possible, I think you get the best impression by jumping directly into the IDE.
