I wonder whether "remove" means raco pkg uninstall readline-gpl.
Searching for "readline" and "invalid memory reference" took me to this. Is libtermcap still in use? This looks like a stuck situation where building from Racket source is the only way to resolve the incompatibility: