I'm working on a library that implements Jack Wrenn's Dissertation work (https://jack.wrenn.fyi/publications/dissertation.pdf) -- essentially, taking student test suites and running them against known good and bad implementations.
My plan is to take a student implementation, which has code & tests, and replace the code with different implementations, running the test-engine suites after each to see the results.
With some namespace munging (that I only somewhat understand), this seems to work fine with ASL (which is not terrible helpful, since we don't use it), but doesn't work in either BSL or ISL(+) -- in BSL, the namespace-set-variable-value! seems to have no effect (the tests run afterwards don't use the new implementation), whereas in ISL, it causes an error (saying the variable being replaced is a constant).
I'm wondering if there is a way to make this work, or if there is another approach that would be more viable -- I thought about extracting out the tests, but given how they are written (imperatively updating something...), that seems somewhat challenging?
Hmm... I think I wouldn't do it like this. Slightly more specifically, I see a lot of top-level issues coming your way.
I can see two different ways of doing this.
The first one—if it works—would probably be a lot less code. Specifically, build a new #lang where the functions you're interested in are already defined in the language, and new definitions of these names are simply ignored. In my mind, this is using the #lang idea the way it's supposed to be used. I think doing this would be relatively straightforward if the students were writing in #lang racket, and I can imagine it being a bit more difficult for the teaching languages, because of the extra bits that are present in the teaching languages in & around syntax checking. But if you can get the right code in the right places, it could actually be just 20 or 30 lines of code total.
Another way to do it would be to manipulate the code yourself, and reassemble it into a program written in the appropriate teaching language. Doing a simple version of this—at the s-expression level, pre-expansion—could be really simple, albeit very fragile. If (say) you just stripped out everything that starts with (define ...) at the top level. Or, going the other direction, keep everything that contains a use of a check-expect. Or combining the two, keeping some, dropping others, and signalling an error for things that fall into both categories.
I think you could do a better job of this if you tackled things in the post-expansion world, though it could also get much more hairy. Fully expand everything, then replace the definitions you're interested in with fully-expanded definitions of your own. That sounds more painful, but maybe the right way to go. It depends on how robust you want this library to be?
The syntactic approach, as you say, feels like it could be fragile. For example: stripping defines could run into problems if they (as we suggest!) define example data and use that in their tests... (trying to grab just the check-expects could run into similar issues) Probably with enough work it could be made to work, but it doesn't quite feel like it is at the right point of interposition.
The other idea feels a lot better, though I'm a little wary of requiring students use different #langs for different assignments... In general, I'd rather not surface details that don't have any useful relevance for the students: how exactly we give them feedback on their tests feels like it is in that vein. But, someone (offline) suggested that I could effectively do that behind the scenes, using the module resolver (selectively redirecting certain modules), and that seems to be working perfectly. The students write in ISL, but the library that is doing this testing can load it as something else (for now, just turning it to ASL, but I may do something more targeted if that turns out to have consequences I don't want to deal with).