How to best implement something like "defer"' in Go?

I was wondering if anyone had any ideas for how to elegantly implement something like defer in Racket which executes either at the end of the current scope and/or the end of the current function?

Obviously, this isn't that hard if you have either a runtime or macro level container which looks for some sort of marker in its children and moves those statements to the end. Ideally though I'd like to do this without introducing some sort of container/wrapper construct.

I don't think continuations will work without some sort of wrapper since the continuation would include everything up to the last mark without a way to separate out what is after the whole containing context executes vs what happens after the "defer" statement.

If nothing like this is possible, I'd also be interested if anyone has already made a package doing something similar. It seems like it would be a very useful primitive for doing stuff like automatic cleanup of files without introducing a bunch of nesting. So you could do something like this.

(define (foo)
  (define-file bar "./temp.txt")
  ...)

And in that example, you wouldn't need to close bar explicitly, it would close automatically before the function closes.

Another thought I had was to try and hook into the tracing system, but that would require the containing procedure to be traced so it runs into the container problem from before.

Maybe dynamic-wind is what you're looking for.

dynamic-wind doesn't quite work because all it does is this (plus magic to ensure post is always run, pre is always run before value-thunk, etc):

(dynamic-wind pre-thunk value-thunk post-thunk)
(let ([pre (pre-thunk)]
      [ret (value-thunk)]
      [post (post-thunk)])
  ret)

That allows for writing something like try/finally, but I don't think it allows you to write something as powerful as defer.

Looking through the docs, syntax-local-lift-module-end-declaration kind of does what I want, but only for modules, not for functions or one level upward (i.e. letting (let () pre (defer foo) post) expand to (let () pre post foo).

Whenever I want to manage resources I usually go with a function based approach.

(define (call-with-foo thunk args)
        ; here make a foo from args
       (let [(foo (make-foo args))]
           (thunk foo)
           ; remember to delete foo
           (delete-foo foo)))

This works well for simple stuff but you can extend it to have exception handlers and more. It introduces nesting but has two advantages.

  1. It is a known patter to schemers and they will pick it up easily.
  2. There is a syntactic scope for the resource you are manipulating.

And in that example, you wouldn't need to close bar explicitly, it would close automatically before the function closes.

I think the underlying problem here is that I like to be explicit about the resources I use and I will install exception handlers and dynamic wind as I see fit for my function based approach. If you are thinking instead about having a cleanup on thread end then I think custodians are the way to go.

1 Like

From a cursory read of the infamously incomplete Go spec, it seems like dynamic-wind is precisely the solution for a single defer. For multiple defers, you just have to stack dynamic-wind in the correct order.

Can you share an example where defer does something dynamic-wind can't?

2 Likes

I think file handling is a bad example, because there are already plenty of functions that handle the opening and closing for you (so I don't get why I would use defer to do this more manually, but still lexically scoped):

with-input-from-file
call-with-input-file

> (with-output-to-file some-file
    (lambda () (printf "hello")))
> (with-input-from-file some-file
    (lambda () (read-string 5)))

"hello"
1 Like

Here's a simple implementation of defer:

#lang racket

(define x null)

(define-syntax-rule (with-defer e ...)
  (dynamic-wind
    (lambda () (set! x null))
    (lambda () e ...)
    (lambda () (for-each (lambda (i) (i)) x))))

(define-syntax-rule (defer e)
  (set! x (cons (lambda () e) x)))

(with-defer
  (define f (open-input-file "/dev/random"))
  (defer (close-input-port f))
  (displayln (bytes->list (read-bytes-line f))))
7 Likes

I think this is interesting, but I think you may be unfamiliar with go's defer statement which works quite differently. See @benknoble's answer for a link.
For example the results of the deferred call are ignored in go, those deferred calls are executed for their side-effects and potentially triggering panics (go's sort-of "exceptions").

If the deferred function has any return values, they are discarded when the function completes.

Yes - I am unfamilar with Go. I'll make a new version based on that description.
Do you know of a test suite with examples of defer?

Here is a version that follows https://go.dev/ref/spec#Defer_statements

#lang racket
(require (for-syntax syntax/parse racket/base)
         syntax/parse racket/stxparam)

(define-syntax-parameter defer
  (λ (stx) (raise-syntax-error 'defer "used out of context" stx)))


(define-syntax (define/defer stx)
  (syntax-parse stx
    [(_define/defer formals . body)
     (with-syntax ([ooo #'(... ...)])
       (syntax/loc stx
         (define formals
           (let ([ds '()])
             (syntax-parameterize
                 ([defer (λ (so)
                           (syntax-parse so
                             ; A deferred expression must be a function call.
                             [(_defer (fun:expr arg:expr ooo))
                              (syntax/loc so
                                ; The function expression and arguments are evaluated now.
                                (let ([f fun])
                                  ; if f is "nil" panic now (not later)
                                  (unless (procedure? f) (error 'panic))
                                  ; evaluate the arguments
                                  (let ([as (list arg ooo)])
                                    ; the actual call to f is delayed
                                    (set! ds (cons (cons f as) ds)))))]))])
               (dynamic-wind
                 void
                 (λ () . body)
                 (λ () (for ([d ds]) ; specified to be in this order (so no reverse here)
                         (define f    (car d))
                         (define args (cdr d))
                         (apply f args)))))))))]))
        
(define/defer (foo)
  (define (bar x) (displayln (list 'later x)))
  (defer (bar (begin (displayln 'now) 11)))
  42)

(foo)

  
2 Likes

A good introduction to how it works could be this blog post: Defer, Panic, and Recover - The Go Programming Language

I think this may be go's test cases for defer, but I am not 100% sure, my go proficiency is a bit stale, and I am not to familiar with its internal code base. I think it is implemented in itself, but I don't know how everything is structured internally.

1 Like

One of Racket's design principles (inherited from Scheme) is to separate scoping from dynamic behavior as much as possible. There's nothing special about (that is, no dynamic behavior attached to) the body of a function or a let-like form. That's why, for example, Racket has no built-in equivalent to C's return statement. Instead, you can use let/ec to explicitly create a non-local return point.

One consequence of this design principle is that macros can use lambda in their implementation without accidentally "capturing" function-boundary behavior. The same benefit goes to a programmer refactoring code to use helper functions.

That's why several people have answered with solutions like dynamic-wind, call-with-foo, and with-defer. You might be interested in the disposable package (but beware, it's currently marked experimental).

5 Likes

Here is yet another defer in Racket. This was initially based on samth's example.

We already do have non-local effects via lifts to the module level so I don't entirely buy the argument that we couldn't have one at the scope level syntactically. However, I do agree that having it built in dynamically could get pretty messy.

That said, I think doing it dynamically is pretty interesting since it lets you add cleanup functions in a very composable way. However, using a bunch of continuations and dynamic parameters is going to have performance implications if you want defer to be fast. This implementation also doesn't handle dynamic-wind since that would require an explicit cleanup-dynamic parameter or something.

#lang racket
(require syntax/parse/define)
(require racket/stxparam)

(define dynamic-return (make-parameter #f))
(define-syntax-parameter return (λ _ #'undefined))

(define-syntax-parse-rule (define-proc head body ... last-expr)
  (define head
    (let/ec return-func
      ;; dynamic-wind would be better, but I like the elegance
      ;; of implementing this based on return only
      (parameterize ([dynamic-return return-func])
        (let ([new-return-func (λ args (apply (dynamic-return) args))])
          (syntax-parameterize ([return (make-rename-transformer #'new-return-func)])
            body ...
            (return last-expr))))))) ;; Ensure that return is always called explicitly at the end.

(define (defer-thunk thunk)
  (let ([old-return (dynamic-return)])
    (dynamic-return (λ args
                      (thunk)
                      (apply old-return args)))))

(define-syntax-parse-rule (defer body ...)
  (let ([body-thunk (λ () body ...)])
    (defer-thunk body-thunk)))

For those who say they prefer with-output-to-file style functions, does add-user2! look ok in the following example? Even if you made auto-lock2! accept multiple locks, that would both obfuscate order of lock acquisition and you really can't condense using some IO resource with some unrelated lock acquisition since that is very application specific.

(define (make-lock)
  (make-semaphore 1))

(define (lock! lock-val)
  (semaphore-wait lock-val))

(define (unlock! lock-val)
  (semaphore-post lock-val))

(define (auto-lock! lock-val)
  (lock! lock-val)
  (defer (unlock! lock-val)))

(define (printlnf template . args)
  (let ([to-print (apply format template args)])
    (println to-print)))

(define (auto-open-db! db-name)
  (printlnf "Opened ~a" db-name)
  (defer (printlnf "Closed ~a" db-name))
  db-name)

;; Maybe first lock is logical while the latter
;; protects some internal data structures for the DB.
(define user-table-lock (make-lock))
(define db-integrity-lock (make-lock))

(define-proc (add-user! username)
  (auto-lock! user-table-lock)
  (auto-lock! db-integrity-lock)
  (auto-open-db! "users")
  ;; logic manipulating db to save user excluding locks.
  (printlnf "Added ~a to users" username))

(define (auto-lock2! lock-val thunk)
  (lock! lock-val)
  (let ([ret (thunk)])
    (unlock! lock-val)
    ret))

(define (auto-open-db2! db-name thunk)
  (printlnf "Opened ~a" db-name)
  (let ([ret (thunk db-name)])
    (printlnf "Closed ~a" db-name)
    ret))

(define-proc (add-user2! username)
  (auto-lock2!
   user-table-lock
   (λ ()
     (auto-lock2!
      db-integrity-lock
      (λ ()
        (auto-open-db2!
         "users"
         (λ (db)
           ;; logic manipulating db to save user excluding locks.
           (printlnf "Added ~a to users" username))))))))

I think it does not have to look that way, 'call-with-` style functions can be combined with syntax that makes them look as nice or I would claim even nicer than defer.

defer

Personally I just don't find defer appealing at all. I don't think that having a minimum of 2 statements composes well at all. (because the return values are ignored and in case of the file, the file handle is bound to a name using a define statement)

When (if) I program go then I would use defer too, because lambdas/thunks are more awkward in go and it doesn't match what is expected there.

unnecessary dynamic behaviour (hidden from the user)

What I don't like about your implementation is that it splits the world of functions into 2 worlds: those that have to be used with define-proc or something similar and those that work without it.

Notice e.g. when I call auto-open-db! in the repl I get:

> (auto-open-db! "bar")
"Opened bar"
"bar"

Because I have to know that I have to call that function from within a function that uses define-proc or something similar.

That is what I mean with 2 worlds. Personally I don't like it when I can use a function in a context where its behavior only works partially, without it giving me an error message.
You could add some kind of checking to prevent that, but that would be even more dynamic.

While calling auto-open-db2! is less "pretty" I prefer that it is explicit and does less juggling underneath:

> (auto-open-db2! "bar" (lambda (x) (printlnf "x: ~a" x) x))
"Opened bar"
"x: bar"
"Closed bar"
"bar"

semantics vs syntax

I also think there are 2 different things being discussed here:

  1. semantics: what and how the code is executed under the hood
  2. syntax: having a convenient way to write it, without lots of nesting and the associated right-wards drift.

semantics

Regarding 1. I don't like that your solution, makes the code that ends up being executed more complicated. I don't think that parameters and escape continuations are necessary or desirable to implement this particular thing.

Why I don't like this?
Mostly I find that this makes the expanded code or when you want to understand the actual control-flow unnecessarily difficult and dynamic.
I only use escape continuations when they greatly simplify the code or the code can't be modeled without them, here it is possible without.

syntax

Your example defines add-user2! with lots of right-wards drift and I agree, I wouldn't want to write a lot of functions that way. Why not start creating a solution right from that code?

As a short aside:
I want to point out that add-user2! has a handle to db while add-user! does not. That needs to be added, without it the comparison isn't fair.

Having this "handle"/parameter is useful, so I am adding it in the 2nd version:

#lang racket
(require syntax/parse/define)

(define (make-lock) (make-semaphore 1))
(define printlnf (compose1 println format))

;; Maybe first lock is logical while the latter
;; protects some internal data structures for the DB.
(define user-table-lock (make-lock))
(define db-integrity-lock (make-lock))

(define-syntax-parser with
  [(_ ([func arg]) body ...+)
   #'(func arg (lambda () body ...))]
  [(_ ([func arg] next ...) body ...+)
   #'(func arg (lambda () (with (next ...) body ...)))])

;; my variant of auto-open-db!
(define (database-connection db-name thk)
  (printlnf "Opened ~a" db-name)
  (begin0
      db-name
    (thk)
    (printlnf "Closed ~a" db-name)))

;; why does add-user2! get an actual handle to the db
;; when add-user! does not?  That's not fair...
;; how would add-user! get the db handle?

(define locked call-with-semaphore) ;; renamed so it looks nicer
(define (add-user3! username)
  (with ([locked user-table-lock]
         [locked db-integrity-lock]
         [database-connection "users"])
    (printlnf "Added ~a to users" username)))

;; in general I think this code is too much pseudo code, 
;; would be better to work with real examples

;; getting some kind of handle could definitely be useful
;; so lets add them; and multiple arguments while we are at it...

(define-syntax-parser with2
  #:datum-literals (->)
  [(_ ([func:expr (~and args:expr (~not ->)) ...+ (~optional (~seq -> bindings:id ...))] next ...) body ...+)
   #'(func args ... (lambda ((~? (~@ bindings ...))) (with2 (next ...) body ...)))]
  [(_ () body ...+)
   #'(begin body ...)])

(define (database-connection2 db-name something-else thk)
  (printlnf "Opened ~a" db-name)
  (begin0
    db-name
    (thk 'db-connection)
    (printlnf "Closed ~a" db-name)))

(define (add-user4! username)
  (with2 ([locked user-table-lock]
          [locked db-integrity-lock]
          [database-connection2 "users" 'some-other-data -> db])
    (printlnf "Added ~a to users using connection ~a" username db)))

;; maybe you want to write code in between the different function calls
(define-syntax-parser with-block
  #:datum-literals (->)
  [(_ (~and before (~not #:with)) ... #:with [func:expr (~and args:expr (~not ->)) ...+ (~optional (~seq -> bindings:id ...))] after ...+)
   #'(begin before ... (func args ... (lambda ((~? (~@ bindings ...))) (with-block after ...))))]
  [(_ body ...+)
   #'(begin body ...)])

(define (add-user5! username)
  (with-block
    #:with (locked user-table-lock)
    ;; do something with table lock, before acquiring the next lock
    #:with (locked db-integrity-lock)
    ;; do something with db lock, before getting the db connection
    #:with (database-connection2 "users" 'some-other-data -> db)
    (printlnf "Added ~a to users using connection ~a" username db)))

;; or even integrate it into the define similar to define-proc,
;; but without the "use from wrong context"
(define-syntax-rule (define-with head body ...)
  (define head (with-block body ...)))

(define-with (add-user6! username)
  #:with (locked user-table-lock)
  ;; do something with table lock, before acquiring the next lock
  #:with (locked db-integrity-lock)
  ;; do something with db lock, before getting the db connection
  #:with (database-connection2 "users" 'some-other-data -> db)
  (printlnf "Added ~a to users using connection ~a" username db))

I especially like that doing it this way the macro expands to simple code just a bunch of lambdas, without me having to write it that way.
This following uses the macro steppers macro hiding, but without it, it is very similar just less readable.

(define (database-connection db-name thk)
 (printlnf "Opened ~a" db-name)
 (begin0 db-name (thk) (printlnf "Closed ~a" db-name)))
(define locked call-with-semaphore)
(define (add-user3! username)
 (locked
  user-table-lock
  (lambda ()
    (locked
     db-integrity-lock
     (lambda () (database-connection "users" (lambda () (printlnf "Added ~a to users" username))))))))

For comparison here is add-user!:

(define dynamic-return (make-parameter '#f))
(define (defer-thunk thunk)
  (let ((old-return (dynamic-return)))
    (dynamic-return (λ args (thunk) (apply old-return args)))))
(define (auto-lock! lock-val) (lock! lock-val) (let ([body-thunk (λ () (unlock! lock-val))]) (defer-thunk body-thunk)))
(define (auto-open-db! db-name)
 (printlnf "Opened ~a" db-name)
 (let ([body-thunk (λ () (printlnf "Closed ~a" db-name))]) (defer-thunk body-thunk))
 db-name)
(define (add-user! username)
 (let/ec
  return-func
  (parameterize
   ([dynamic-return return-func])
   (let ([new-return-func (λ args (apply (dynamic-return) args))])
     (syntax-parameterize
      ([return (make-rename-transformer #'new-return-func)])
      (auto-lock! user-table-lock)
      (auto-lock! db-integrity-lock)
      (auto-open-db! "users")
      (new-return-func (printlnf "Added ~a to users" username)))))))))

And here is add-user4! doing some more things, but still a simple expansion:

(define (database-connection2 db-name something-else thk)
   (printlnf "Opened ~a" db-name)
   (begin0 db-name (thk 'db-connection) (printlnf "Closed ~a" db-name)))
(define (add-user4! username)
  (locked
   user-table-lock
   (lambda ()
     (locked
      db-integrity-lock
      (lambda ()
        (database-connection2
         "users"
         'some-other-data
         (lambda (db) (printlnf "Added ~a to users using connection ~a" username db))))))))))

Personally most of the time I would stop with with2, but some may want to have the syntax integrated into the function declaration syntax and go all the way to define-with.
Currently with-block gives a not so great error message when the body is empty an extra case could detect that and state that it isn't allowed explicitly, or alternatively it could default to (void).

1 Like

This blog post by @jeapostrophe is related to the topic:
2013-09-23: Defer, Panic, and Recover in Racket