How can I improve my knitting?

Questions & Answers
1

Hi, Racket Discourse.

I have been messing around with the following little syntax, called knitting, which is basically a thin wrapper around @lexi.lambda's threading, which she recently updated.

The excitement got me thinking about theading values, which can become yucky to deal with. So why not build on the shoulders of giants.

My question is, how can I improve the macro-side of things? I have been reading through the older version of the source for threading and see for example that @lexi.lambda uses adjust-outer-context with the remark

; Adjusts the lexical context of the outermost piece of a syntax object;
; i.e. changes the context of a syntax pair but not its contents.

(define-for-syntax (adjust-outer-context ctx stx [srcloc #f])
  (datum->syntax ctx (syntax-e stx) srcloc))

Why is this necessary, or, a good thing? What other bits of smelly code are lurking in my definitions?

First, the definitions:

Edit: public gist of the definitions.

The basic idea is that we stitch together yarns. A yarn is any expression that isn't a "knitting phrase", such as %, #:knit, &, or #:purl. A "knit" is executed regardless, and a "purl" is executed conditionally.

The same idea applies as with threading, but we may enjoy greater pre-emption and control.

Re: the adjust-outer-context: I just ran into a bug caused by carelessly renaming things in the macro definitions, leading to a weird error (because it goes back to the macro source!).

But, after adjusting the context, it seems to yield a more intelligible result. Probably not the only reason to do that, but it is interesting.

2

@matteo-daddio, this made me think of you; might be convenient for using with those pesky n-ary streams.

#lang racket

(require knitting)

(define (nary-stream . ss)
  (if (ormap stream-empty? ss)
      empty-stream
      (stream-cons (apply values      (map stream-first ss))
                   (apply nary-stream (map stream-rest  ss)))))

(define 3-stream
  (nary-stream (stream  1   2   3)
               (stream 'a  'b  'c)
               (stream "x" "y" "z")))

;; looking sharp
(let loop ([3-stream 3-stream])
  (%> (stream-empty? 3-stream)
      #:purl {#false}
      (stream-first 3-stream)
      #:knit {x y z}
      (displayln `(,x ,y ,z))
      #:put
      (loop
       (stream-rest 3-stream))))

;=>
; (1 a x)
; (2 b y)
; (3 c z)
; #f
1 Like
3

For what it’s worth, the current version of threading no longer includes that definition.

Historically, that definition was used to transfer lexical context from input syntax to syntax in the expansion of ~>. Originally, the primary motivation was to cooperate with implicit introduction of #%app, which is given the lexical context of the syntax list it is inserted into. More broadly, this transferring of lexical context is simply the Right Thing To Do, as ~> disassembles and reassembles syntax in its expansion, so the lexical context should be preserved.

The old version of threading used syntax to reconstruct syntax lists in its expansion, which uses the lexical context of the syntax form itself. For example, if one writes

(syntax-parse stx
  [(x y z)
   #'(x y z)])

then the resulting syntax is not quite the same as the input syntax. The x, y, and z terms are all preserved exactly, but the lexical context, source location, and syntax properties of the enclosing syntax list are taken from the location of the syntax template itself.

Most macros do not disassemble and reassemble syntax in this way, so this behavior of syntax does not normally cause trouble. Moreover, the lexical context of syntax lists is rarely relevant at all: usually we only care about the lexical context of identifiers. However, there are a few reasons to care about preserving properties of the input syntax:

  • As mentioned above, implicit introduction of #%app uses the lexical context of the syntax list it is inserted into.
  • Macros can transfer lexical context arbitrarily, which can make the difference observable.
  • Source location information is used by error reporting tools like errortrace and as part of producing inferred value names, so preserving it is useful.
  • Syntax properties have many uses, from determination of whether syntax satisfies syntax-original? to tracking the bracket type via 'paren-shape to any number of other purposes. Usually they aren’t critical, but it’s still good hygiene[1] not to drop them.[2]

So if this is so important, why doesn’t the new version of threading use adjust-outer-context anymore? The answer is that the new version avoids the need for it altogether: it rebuilds syntax lists using datum->syntax rather than syntax, so the rebuilt syntax doesn’t need to be “fixed up” afterwards; it’s already put back together in the right way. With the benefit of hindsight, I can definitely recommend that approach, but it does require understanding how all the different pieces fit together, which I didn’t understand back then.

  1. In the colloquial sense, not the macro hygiene sense… though the macro hygiene sense comes from the colloquial sense, so perhaps this can be considered to fall under that same umbrella. ↩︎

  2. Note that the adjust-outer-context function actually failed to preserve syntax properties. The new version of threading does better. ↩︎

1 Like
4

Thank you for the reply. A lot to consider, indeed :blush:

So, if I am understanding you correctly, the "Ship of Theseus" only rears its head when I am actively breaking apart the syntax and making it anew in a different context. But what constitutes a "break" and what is merely rearrangement?

So, in my case, I don't really mess with the syntax that much (I think), because I am either wrapping it in procedures or I fall back onto threading for the heavier lifting. I would then assume this falls more into the latter category than the former.

The idea I have in my head, is that if I define something like:

(my-macro x y z) -> (lambda (x) (z (y x)))

This would constitute rearrangment, but, if I had something like:

(my-macro (lambda (x) ...) y z) -> (lambda (x y) ... z ...)

This would constitute "breakage" because I am inserting things where they were not before. Is this getting closer to the concept?

Edit: as a real example from the code, I have

(my-macro ex (yn ...)) -> (apply yn ... (knits ex list))

I doubt which case this falls under, although I would lean toward "breakage" because the apply and ex wasn't there before, so when the code executes, it will "happen" somewhere in the synthetic context as opposed to the original (I think), especially because (yn ...) is assumed to be a form that will turn into something else.

5

It is worth thinking about the fact that ~> is a very unusual macro, because it is quite explicitly about pushing (“threading”) forms inside other forms. When you write

(~> foo
    (bar baz))

then the expansion is (bar foo baz), and the idea is this is just that second ~> subform with foo slipped into the middle.

Most macros do not do this, and almost none do it so mindlessly. In fact, I would even argue that most macros should not do this. Macros should be syntactic abstractions, and their users should not need to reason about them in terms of what they expand into. If that is the case, it makes sense for syntax introduced by the macro to be attributed to the macro: the macro “takes responsibility” for whatever it expands into.

What makes ~> so useful is also what makes it so unusual: it’s really just a syntactic abbreviation, and it exists purely to rearrange syntax objects. ~> neither knows nor cares about what its subforms actually mean. For this reason, the user of ~> really does want to think of the above example as precisely equivalent to if they had written (bar foo baz) directly, in the original source file, and ~> should therefore take care to honor that expectation.

Here, you are introducing a use of apply in the expansion, and you are almost certainly expecting (apply ....) to mean “a procedure application”. This means that you are in fact depending on the binding of #%app being the one from racket/base, as your expansion would fail to work correctly if (#%app apply ....) meant something else. This is really just an illustration of why macro hygiene is so important: you want your expansion to mean what it means in the module where you defined my-macro, not what it would have meant in some entirely different module with some other binding for #%app.

I don’t think I can say terribly much more without understanding what my-macro in your example is actually intended to do, but at the very least, the lexical context of the outermost syntax list (i.e. the context “on the parentheses”) should come from the template, not the input to the macro.

1 Like
6

Interesting; I like the bit about syntactic abstraction as opposed to abbreviation (ad hoc abstraction?). This means something else (edit: but its meaning is codified? My words fail me; but I will think about this some more), vs., this means the same thing, but in a different form (from a context perspective).

I hear you; so, what that macro is doing, is applying (call-with-values (thunk ex) list) for an expression in the macro, and then applying the resulting values list in the tail position to (yn ...), which, as you note, is assumed to be a procedure.[0]

I couldn't think of a clean way to deal with the fact that I do not yet know what the values will be at the time (how many, precisely), so it can't be as "care-free" as ~> about inserting those values as syntax. I guess you could indicate the number explicitly and construct some temporary identifiers which you might splice into (yn ...) instead of applying them, but I have not tried.

Edit: so cursed!

(stitch
 (values apply list) % {2...}
 (1 2 3 '()))
=> '(1 2 3)

; I like synonyms
(stitch
 (values apply *)
 #:knit {2...}
 (1 2 3 '(4 5 6)))

(stitch
 (values apply *)
 #:knit-pick -2
 (1 2 3 '(4 5 6)))
=> 720

Edit: [0]
Which, now that I look at it again, is pretty redundant. {...} or #:knit-plug, already collects the values into a list and passes it on, so if we have threading in any case, why have a separate mnemonic for that? At least pruning is good for the tree.

; don't
(%> (values 3 4 5) % {...@} (list 1 2))
; do
(%> (values 3 4 5) % {...} (apply list 1 2 _))
7

An interesting but otherwise useless exercise.

Moral of the story, at least from what I have now seen, is that you should just use threading and something like an augmented match/values (maybe match/values*?) which allows for arbitrary arity among the values (so, basically, pump the values expression into a list and match on that).

Thanks, @lexi.lambda, for your insightful comments.

Something, such as, for example:

(define-syntax-rule (match/values* values-ex
                      [(pats ...) then ...]
                      ...)
  (match (call-with-values (thunk values-ex) list)
    [(list pats ...) then ...] ...))

(match/values* (values 1 2 3)
  [(1 2 3) #true]
  [(1 2)   #false])
8

For working with values, see also the Qi language, which augments a threading-like operator with many forms for computing with values.

9

Thank you, @benknoble. I keep putting it off for some reason, but perhaps on the second recommendation, it might be time to dig in :sweat_smile: