Stable module reference through `raco exe`?

Questions & Answers
1

In racket-nonogram, I use serialize (with #:relative-directory) and serializable-struct to serialize messages sent over the wire. To prevent arbitrary code execution, I set deserialize-module-guard to a value that only allows relative library paths. This works okay when both machines are running from source, but it breaks if I attempt to distribute the application using raco distribute in two ways:

  1. After raco exe, module paths are no longer relative library paths and instead become either '#%mzc:main or things like '#%embedded:nonogram/lib/geometry:. I could recognize these paths specially and permit them, but the format doesn’t seem to be documented, so it seems sketchy to rely on that implementation detail.

  2. The module paths necessarily differ between a program running directly from source versus running from the result of raco exe. That is not the end of the world, but it would be nice if I could make it so that it is possible to run the server directly from source while a client runs from the result of raco exe.

I am curious if there is some way to arrange for this to work. In many cases, the answer to my challenges involving module paths has been to use module path indexes, instead, but even MPIs are not stable after raco exe. Is there anything resembling a stable module reference that will be portable across machines and remains identifiable after raco exe? Or should I resign myself to using something more explicit than serialize if I want to support this?

2

Actually, I now realize this isn’t true. It is documented with raco exe in the following paragraph:

When a module is embedded in an executable, it gets a symbolic name instead of its original filesystem-based name. The module-name resolver is configured in the embedding executable to map collection-based module paths to the embedded symbolic name, but no such mapping is created for filesystem paths. By default, a module’s symbolic name is generated in an unspecified but deterministic way where the name starts with #%embedded:, except that the main module is prefixed with #%mzc:. The relative lack of specification for module names can be a problem for language constructs that are sensitive to module names, such as serialization. To take more control over a module’s symbolic name, use the ++named-lib or ++named-file argument to specify a prefix that is appended before the module’s base name to generate a symbolic name.

It seems like maybe I could supply ++named-file '#%nonogram:' path/to/module.rkt for each module in my program that utilizes serializable-struct, then recognize such paths specially somehow. That is still a little annoying, as I would need to manually keep track of all the relevant modules, but maybe that is the intended solution here?

3

I'm confused about where the problem is. The serialization information should refer to the original module paths (otherwise an un-exeified program couldn't deserialize values sent from an exeified program). So if you allow module paths equivalent[*] to, eg, '(lib "nonogram/lib/geometry.rkt"), then I think that should work. Does it not?

The asterisk: you might have to normalize the module names for comparison, maybe with collapse-module-path.

4

Just to double-check, are the MPIs you’ve tried from define-runtime-module-path-index? I would have thought that was meant to address this kind of scenario. (There's a -runtime-module-path variant, too, if that stronger reference makes a difference.)

5

What you describe in your parenthetical seems to be the case? Here is a simple example program to illustrate:

#lang racket/base

(require racket/serialize)

(serializable-struct foo (value) #:transparent)

(module+ main
  (require racket/cmdline
           racket/file
           racket/match)

  (define what-to-do #f)
  (command-line
   #:once-any
   ["--write" "" (set! what-to-do 'write)]
   ["--read" "" (set! what-to-do 'read)])

  (match what-to-do
    ['write
     (call-with-output-file*
      "value.sexp"
      #:mode 'text
      #:exists 'replace
      (λ (out) (write (serialize (foo 'bar)) out)))]

    ['read
     (call-with-input-file*
      "value.sexp"
      #:mode 'text
      (λ (in) (println (deserialize (read in)))))]))

And here is what I see:

$ racket -u main.rkt --write
$ cat value.sexp
((3) 1 ((#"/tmp/serialize-test/main.rkt" . deserialize-info:foo-v0)) 0 () () (0 bar))⏎
$ raco exe main.rkt
$ ./main --read
standard-module-name-resolver: collection not found
  for module path: racket/base/lang/reader
  collection: "racket/base/lang"
  in collection directories:
  context...:
   .../private/serialize.rkt:719:2: deserialize
   .../serialize-test/main.rkt:30:6
   body of '|#%mzc:main(main)|
   body of top-level
$ ./main --write
$ cat value.sexp
((3) 1 (((quote #%mzc:main) . deserialize-info:foo-v0)) 0 () () (0 bar))⏎
$ racket -u main.rkt --read
dynamic-require: unknown module
  module name: '#%mzc:main
  context...:
   /Users/alexis/code/racket/racket/master/racket/collects/racket/private/serialize.rkt:719:2: deserialize
   /tmp/serialize-test/main.rkt:30:6
   body of (submod "/tmp/serialize-test/main.rkt" main)
6

No, I don’t think that is relevant here. Certainly, it’s true that define-runtime-module-path-index will produce an MPI that resolves to the right module whether the program is packaged with raco exe or not (within the version of the program that produced it), but that is not the issue here. The issue is that those MPIs are not stable between unpackaged and packaged versions of the program. See, for example, this program and output:

#lang racket/base

(require (for-syntax racket/base)
         racket/runtime-path)

(define-runtime-module-path-index this-mpi '(submod "."))
this-mpi

$ racket test.rkt
#<module-path-index:(submod ".") (file "test.rkt")>
$ raco exe test.rkt
$ ./test
#<module-path-index:'#%mzc:test>
7

Okay, it makes sense that it won't work for a module that doesn't live in the lib namespace. Is nonogram installed as a package when you run raco exe? If not, I think that might fix the issue.

If it is, then maybe raco exe needs to recognize that the module file path it's given corresponds to a lib module path. A workaround would be to create another module file as an entry point, require nonogram using an absolute module path, and call raco exe on the new entry point module. (That is, the new module should force raco exe to treat the rest of the source in "absolute" mode instead of "relative" mode. Or so I conjecture.)

Edited to add: In other words, lib module paths are the stable module references you asked for.

8

It seems like this does help, but only if the module is required using a lib path, regardless of whether it is installed into some collection. That is, if I split out my struct definition into a separate module:

;; serialize-test/struct.rkt
#lang racket/base
(require racket/serialize)
(provide (struct-out foo))
(serializable-struct foo (value) #:transparent)

Then make the main module the following:

;; serialize-test/main.rkt
#lang racket/base
(require racket/cmdline
         racket/file
         racket/match
         racket/serialize
         serialize-test/struct)

(define what-to-do #f)
(command-line
 #:once-any
 ["--write" "" (set! what-to-do 'write)]
 ["--read" "" (set! what-to-do 'read)])

(match what-to-do
  ['write
   (call-with-output-file*
    "value.sexp"
    #:mode 'text
    #:exists 'replace
    (λ (out) (write (serialize (foo 'bar)) out)))]

  ['read
   (call-with-input-file*
    "value.sexp"
    #:mode 'text
    (λ (in) (println (deserialize (read in)))))])

Now this works, assuming these modules are installed. However, if I instead change the require in main.rkt to use a relative path, as in

(require racket/cmdline
         racket/file
         racket/match
         racket/serialize
         "struct.rkt")

then things break again, as the imported module is assigned a new module name. This is sort of annoying, as I tend to follow the (I think fairly standard?) Racket convention of using relative paths for imports within the same package.

Yes, this seems like this would be sufficient to get things working in the way I expect. Thanks!

9

Actually, I’ve realized there’s an easy workaround for this. I can create a separate entrypoint as follows:

;; exe-main.rkt
(require (submod nonogram main))

Now, since this is a library import, raco exe will transitively treat all imported modules as library imports as well, even if they use relative paths. That resolves my issue without me having to change all the require paths in my program.