I'm having an issue where changes to parameters are not being noticed in a new thread. I think I know what's happening but would appreciate confirmation and suggestions on the best way to handle it.
In server.rkt I parse configuration data from the command line and other sources, combine them, and use them to set some parameters such as db-config-filepath which tells the system which database to use.
For testing I am running two instances of the server in parallel, one named 'alice' and the other named 'bob'. Each is run in a separate terminal with separate command line arguments.
The default DB path is db.conf but for testing I have db.alice.conf and db.bob.conf, each of which points the relevant peer to its own database.
When messages come into the server from another peer (e.g. when alice sends a message to bob), the message gets handled in a thread. These threads try to conclude as rapidly as possible in order to minimize the delay of responding to the other peer; long-running tasks get pushed into a job queue to be run separately as soon as a worker is available .
Tasks are added to the queue via a promise. Here is a simplified version of the relevant code from files.rkt:
I'm defining the task manager at the top of the file because I want there to be a single task manager instance shared across multiple message-handling threads. (Recall that each thread is created in response to a different message received by the server as alice and bob talk to each other.)
Both of the logging lines shown above have the correct db-config-filepath value (in this case db.alice.conf) but the code inside the promise has the default db.conf
(Note: As mentioned, this is simplified; although db-config-filepath is the parameter that led me to notice the issues, there are other parameters that are also not being updated. It's therefore not as simple as passing an already-initialized db connection.)
I think what's happening is this:
jarvis is created at startup time before the configuration data has been parsed and the parameters updated. jarvis therefore starts with the default values for all the parameters and for whatever reason it does not notice the updates.
Inside add-task, a new thread is created and the promise is forced inside it. As standard, this thread receives copies of all parameters as known to the surrounding context. Since 'the surrounding context' is the jarvis instance that is using the unconfigured default values, that means that the thread has the default db-config-filepath value and so it ends up talking to the default DB instead of alice's db.
Does this sound right? If so, what would y'all suggest for how to fix it -- would it be as simple as using a thunk instead of a promise, or is there a better solution?
Can you fill a but report so this problem is not forgoten?
It should be easier to fill because all your code is public.To see the error, it's enough to tun simplisp/trace-source-code.rkt at main · joskoot/simplisp · GitHub ? Is it possible to run it with a smaller expression instead of source-code so the output is not huge?
(It's nice to have a minimized example instead of a full repo, but for me it's good enough to get a reproducible foolproof instructions.)
#lang racket
#;(require simplisp/simplisp)
(require "simplisp.rkt")
; WARNING
; Running this module produces almost 100000 lines of output.
(simplisp
'(trace-align '5)
'(trace-width 90)
'(trace-option 'all)
source-code)
In DrRacket I got a loooooooooooooot of lines in 8.2[3m] . But in 8.2[cs] I got a few lines and a segfault. I don't have both versions of 8.4 to try now.
Just post an issue in Issues · racket/racket · GitHub explaining the problem and with a link to your repo and this program, so it's possible to use cut and paste see the problem.
It would be nice if you replace in the issue
'(trace-option 'all)
with a variant of
'(trace-option '(start finis selfi))
that prints less lines but still shows the difference between cs and bc.
Yes, that's right. Parameters are wrappers around thread-local state. Using a thunk instead of a promise wouldn't make a difference; when you run the code, it will see the parameter values of the thread it's running in.
And if your code wants to carry over other parameters, you have to save and restore them too. You can of course create a function or macro that does that for you, but you need to tell it what parameters to capture.
(In my RacketCon talk I said that I thought parameters were overused. This is one reason why.)
[Edit: Ryan's reply already covered this while I was typing it in.]
I'm not sure I follow, but it sounds like you're describing a situation like this:
(define param (make-parameter 'p-default))
(define var 'v-default) ; just to contrast with `param`
(define th
(thread (lambda ()
(sync (system-idle-evt)) ; waits until main thread is done
(printf "~s vs. ~s\n" (param) var))))
(param 'p-new)
(set! var 'v-new)
(sync th)
This example will print "p-default" and "v-new", because parameter mutation is thread-specific. Is that what you mean?
If this example is the kind of problem you're seeing, using a thunk instead of a promise won't change anything, because it's about how parameters and threads interact. If the goal is to share state across threads, you may have to use a mutable variable, mutable box, or something like that instead of a parameter.
I'm seeing two bugs in Racket CS when running simplisp:
An applicable structure as a prop:object-name value is not handled correctly. That's the source of the crash.
A structure type's prop:object-name procedure is called to get a printing name for the structure type itself (which is not a correct use of the property value). This bug only show up after the other one is fixed.
A continuation captures the values of parameters at the time it is
captured. So, if your parameterize is outside of the dispatcher that
calls the continuation, then the code inside the continuation will not
see the parameter change. That's a pretty important feature of
continuations.
Web parameters would work the same way as normal parameters and not help you.
The stateless Web language guarantees that the parameters will have
values they had when the servlet was initialized.
I think that you may want to use a thread-cell as the storage place.
That's a good point, and it works as long as you only parameterize parameters instead of setting them directly. But if you mix the two modes, you can still get surprises. Here's an example:
(define param (make-parameter 'original))
(define chan (make-channel))
(void (thread (lambda () (let loop () ((sync chan)) (loop)))))
(parameterize ((param 'changed))
(channel-put chan (lambda () (printf "#1 ~v\n" (param)))))
;; prints "#1 'original" -- this is the original problem
;; capture-params : (-> X) -> (-> X)
(define (capture-paramz thunk)
(define paramz (current-parameterization))
(lambda () (call-with-parameterization paramz thunk)))
(parameterize ((param 'changed))
(channel-put chan (capture-paramz (lambda () (printf "#2 ~v\n" (param))))))
;; prints "#2 'changed" -- fixed!
(parameterize ((param 'changed-once))
(param 'changed-twice)
(channel-put chan (capture-paramz (lambda () (printf "#3 ~v\n" (param))))))
;; prints "#3 'changed-once" -- doesn't see update after parameterize!
Those are good examples. The second one, especially, illustrates that sometimes when you mix two different kinds of magic (aka effects), you have to think carefully about how they interact.
With the info of Matthew Flatt I was able to reduce the code.
The following reveals the problem as explained by Matthew Flatt and does not produce much output.
Uncomment the subexpression marked by arrow to produce the problem.
I am aware of the fact that this problem is not related to the original post in this thread.
Apologies.
#lang racket
(require simplisp/simplisp)
(simplisp '(trace-option '(varef)) '(trace-width 50)
'(letrec-values ; Notice the quote.
((($trace-option) (make-parameter #t))
((inspector) (make-sibling-inspector))
((super-type super-type? super-type-name set-super-type-name!)
(let*-values
(((descr constr pred acc mut)
(make-struct-type
'super-type ; type-name
#f ; no super-type
1 ; fields: name (other fields to be provided by sub-types)
0 ; no auto fields
#f ; auto value n.a.
(list
; Uncommenting the following subexpr reveals the problem.
#; (cons prop:object-name
(λ (obj) (parameterize* (($trace-option #f)) (super-type-name obj)))))
inspector
#f ; no procedure properety
'() ; mutable name
#f))) ; no guard
(values descr pred
(make-struct-field-accessor acc 0 'name)
(make-struct-field-mutator mut 0 'name))))
((make-closure $closure?)
(let*-values
(((printer)
(λ (obj port mode)
(parameterize* (($trace-option #f))
(fprintf port "#<closure:~s>" (super-type-name obj)))))
((descr constr pred acc mut)
(make-struct-type
'closure ; type-name
super-type
1 ; nr of fields: procedure, name already in super-type
0 ; no auto fields
#f ; auto-value n.a.
(list (cons prop:custom-write printer))
inspector
0 ; procedure property
'(0) ; immutable proc
#f))) ; no guard
(values constr pred)))
((noot) (make-closure 'aap (lambda (x) x))))
(displayln (noot "this is displayed"))
noot))
It's nice that both problem were fixed. Remember that if you find a bug, you can post it in github (or here). They are very helpful to fix corner cases that have errors.
I'm embarrassed to say I'm overdue catching up on talks from that RacketCon, including yours. So apologies if this is an impoverished version of something you already said.
I think sometimes people just like the signature of parameters, without wanting or remembering the thread-local aspect. In other words they want something like:
Thank you all for helping me get my head around this. For the benefit of anyone else who stumbles over this thread, here's what I've taken away:
#lang racket
(define name (make-parameter 'bob))
(define db-path (make-parameter 'orig))
(define orig-parms (current-parameterization))
; Pointless use of call-with-parameterization since no params have changed
; outputs: 1 orig. bob
(call-with-parameterization
orig-parms
(thunk (displayln (~a 1 " " (db-path) ". " (name)))))
; parameterize db-path, leave name untouched
; outputs: 2 new. bob
(parameterize ([db-path 'new])
(displayln (~a 2 " " (db-path) ". " (name))))
; outside of the scope of the parameterize, so db-path has been automatically reset
; outputs: 3 orig. bob
(displayln (~a 3 " " (db-path) ". " (name)))
; set (as opposed to parameterize) db-path
(db-path 'new)
; parameterize name, then call with the original parameterization which did not include
; this and will therefore show the original value for name. Essentially, we update the
; value of name and then immediately override it back to its original value.
; outputs: 4 new. bob
(parameterize ([name 'alice])
(call-with-parameterization
orig-parms
(thunk (displayln (~a 3 " " (db-path) ". " (name))))))
Given that assigning directly to a parameter (e.g. (db-path "db.conf")) causes issues, what's the right way to handle command line arguments? The examples for command-line all work via assigning to parameters.
Ohhhhkay. Thought I had a handle on this and apparently I do not.
#lang racket
(define users (make-parameter '()))
(users (list "bob")) ; assign
(define outer (current-parameterization))
(parameterize ([users (list "alice")]) ; parameterize
(define inner (current-parameterization))
(println (users)) ; should print '("alice"). Works as expected.
(call-with-parameterization
inner
(thunk
; should print '("alice") because that was parameterized in
(println (format "inner users: ~v" (users))))) ; Works as expected.
(call-with-parameterization
outer
(thunk
; should print '("alice") because assignment is not captured by
; current-parameterization so the (users (list "bob") should not
; be seen
(println (format "outer users: ~v" (users)))))) ; prints '("bob") ??!?!?!?!
The second call-with-parameterization prints '("bob"), meaning that it did in fact capture the assignment. I thought it wasn't supposed to do that -- what am I missing?
Given that assigning directly to a parameter (e.g. (db-path "db.conf")) causes issues, what's the right way to handle command line arguments? The examples for command-line all work via assigning to parameters.
When I use parameters for command-line arguments, they aren't
application-level; rather, they are scoped very specifically to the
invocation of the command-line parser and the subsequent actions in
the main (sub)module that is the program.
For application-level parameters, I think parameterize is to be
preferred. But of course the greatest challenge is handling dynamic
scope: there's a reason PL has moved away from that by-and-large.
Parameters exist (I think) to re-enable it. I think this may be the
cause of the confusion in the original question; without reading too
closely, it sounds like there was confusion about which "dynamic
extent" (am I using this correctly?) some code in a thread had (or
"was under"?). And of course "dynamic extent" is far more than just
the immediate body forms of parameterize; it includes all code
that executes as a result of those body forms, excepting any
sub-parameterizations.
Perhaps this complexity is why some think they are over-used?
I haven't checked the reference, but I'm assuming current-parameterization does in fact capture the current values of all parameters, whether they were set via parameterize or the setter-procedure. If you move the (define outerbefore the line marked ; assign I would expect the empty list to be printed at the end.
So apologies if this is an impoverished version of something you already said.