Apply values to a procedure

[damien_mattei]

@countvajhula

Is $bracket-apply$ a macro? If so, this section of the docs contains some recipes.

yes $bracket-apply$ is a macro and i had a strange bad syntax error i did not understood but now i have registered it with (define-qi-foreign-syntaxes $bracket-apply$) it works:

(define-qi-foreign-syntaxes $bracket-apply$)
(define Lplot (~> ((read-lines src)) list->vector {_[5 :]} vector->list (△ (~> string-split rest (△ string->number) vector)) ▽))

is parsed by SRFI 105 curly infix in :

(define-qi-foreign-syntaxes $bracket-apply$)
    (define Lplot
      (~>
       ((read-lines src))
       list->vector
       ($bracket-apply$ _ 5 :)
       vector->list
       (△ (~> string-split rest (△ string->number) vector))
       ▽))

the critical point being {_[5 :]} parsed and converted in ($bracket-apply$ _ 5 :)

i add too problem with infix parsing with macro, the problem being that you can manipulate a procedure like + or list but not a macro like and because you will get a bad syntax once evaluated outside the syntax for which it has been defined. The solution i found for manipulate macro is to use it like a syntax with syntax transformers. I do not know if Qi also use syntax transformers.

This day i tried to go further in associating Qi and infix parsing, i added for example ~> to the list of my known operator having an operator precedence priority given by it position in the list:

in file: operators-list.rkt

(define infix-operators-lst-for-parser-syntax

  (list
    exponential-operator-syntax
    (list #'* #'/ #'%)
    (list #'·) ; symbolic logic And
    (list #'⊕) ; symbolic logic Xor
    (list #'+ #'-)
	
    (list #'<< #'>>)

    (list #'& #'∣)

    (list #'< #'> #'= #'≠ #'<= #'>= #'<> #'equal?)

    (list #'and)

    (list #'or)

    (list #'~>)
    
    assignment-operator-syntax
    definition-operator-syntax 
    )

  )

it is necessary because this list use too being flatten just to know if a scheme symbol is an operator or another procedure.It is used in an infix? predicate to check if an expression is infix or prefix.

But when running the $nfx$ of SRFI-105 i have a strange error related to Qi modification of the syntax i think. Note that it previously worked well for $brackett-apply$ macro but not with $nfx$.

(define-qi-foreign-syntaxes $nfx$) 
(define Lplot {((read-lines src)) ~> list->vector ~> {_[5 :]} ~> vector->list ~> (△ {string-split ~> rest ~> (△ string->number) ~> vector}) ~> ▽})

it is a bit hard to understand for someone not in the code of Scheme+ and SRFI-105.

$brackett-apply$ is a macro called only when the parser meet some [ ]
$nfx$ is called when the parser meet an infix expression with need (or not depending of my code version) of applying an operator precedence rule when transforming the infix code in prefix sexpr.

On the previous code (sorry it is not simple expression) the code is pre-parsing this way:

(define-qi-foreign-syntaxes $nfx$)
(define Lplot
      ($nfx$
       ((read-lines src))
       ~>
       list->vector
       ~>
       ($bracket-apply$ _ 5 :)
       ~>
       vector->list
       ~>
       (△ ($nfx$ string-split ~> rest ~> (△ string->number) ~> vector))
       ~>
       ▽))

i suppose the more nested $nfx$ is first expanded, but this not important ; And i have this error:(a bit verbose i have activated debug info)

Welcome to DrRacket, version 8.14 [cs].
Language: Determine language from source; memory limit: 8192 MB.
infix? : expr=(.#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:16:39 -4>)
infix? : (a)  allowed or not  as infix ,check code
infix? : rv=#t
infix? : expr=(.#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:16:39 -4>)
infix? : (a)  allowed or not  as infix ,check code
infix? : rv=#t
infix? : expr=()
infix? : null expr
infix? : rv=#t
infix? : expr=()
infix? : null expr
infix? : rv=#t
infix? : expr=(.#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:23:46 -4>)
infix? : (a)  allowed or not  as infix ,check code
infix? : rv=#t
infix? : expr=(.#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:23:46 -4>)
infix? : (a)  allowed or not  as infix ,check code
infix? : rv=#t
infix? : expr=(.#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:24:75 4>)
infix? : (a)  allowed or not  as infix ,check code
infix? : rv=#t
infix? : expr=(.#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:24:75 4>)
infix? : (a)  allowed or not  as infix ,check code
infix? : rv=#t
infix? : expr=(.#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:29:55 4>)
infix? : (a)  allowed or not  as infix ,check code
infix? : rv=#t
infix? : expr=(.#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:29:55 4>)
infix? : (a)  allowed or not  as infix ,check code
infix? : rv=#t
infix? : expr=(.#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:32:38 5>)
infix? : (a)  allowed or not  as infix ,check code
infix? : rv=#t
infix? : expr=(.#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:32:38 5>)
infix? : (a)  allowed or not  as infix ,check code
infix? : rv=#t
infix? : expr=()
infix? : null expr
infix? : rv=#t
infix? : expr=()
infix? : null expr
infix? : rv=#t
infix? : expr=(.#<syntax:Applications/Racket v8.14/share/pkgs/qi-lib/macro.rkt:87:47 v> .#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:47:17 string-split> .#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:47:30 ~>> .#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:47:33 rest> .#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:47:38 ~>> (.#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:47:42 △> .#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:47:44 string->number>) .#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:47:60 ~>> .#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:47:63 vector>)
infix? : check not start with an operator:#t
infix-rec? : expr=(.#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:47:17 string-split> .#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:47:30 ~>> .#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:47:33 rest> .#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:47:38 ~>> (.#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:47:42 △> .#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:47:44 string->number>) .#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:47:60 ~>> .#<syntax:opt/homebrew/var/www/drive/parsed_files_directory/displayTrajectory3D+.rkt:47:63 vector>)
infix? forbids: op1 without e1 : (not (null? (cdr expr))) :#t
infix? check (op1 e1 ...) : (member-syntax (car expr) oper-lst) : #f
infix? check not (op1 op2 ...) : (not (member-syntax (cadr expr) oper-lst)) :#f
infix? : rv=#f
. . ../../../../../../Users/mattei/Scheme-PLUS-for-Racket/nfx.rkt:41:2: pre-check-!*-generic-infix-parser  : arguments do not form an infix expression :terms: '(#<syntax:/Applications/Racket v8.14/share/pkgs/qi-lib/macro.rkt:87:47 v> #<syntax:displayTrajectory3D+.rkt:47:17 string-split> #<syntax:displayTrajectory3D+.rkt:47:30 ~>> #<syntax:displayTrajectory3D+.rkt:47:33 rest> #<syntax:displayTrajectory3D...
> 

the main problem is that the Qi modified expression no more form an infix expression:

. . ../../../../../../Users/mattei/Scheme-PLUS-for-Racket/nfx.rkt:41:2: pre-check-!*-generic-infix-parser  : arguments do not form an infix expression :terms: '(#<syntax:/Applications/Racket v8.14/share/pkgs/qi-lib/macro.rkt:87:47 v> #<syntax:displayTrajectory3D+.rkt:47:17 string-split> #<syntax:displayTrajectory3D+.rkt:47:30 ~>> #<syntax:displayTrajectory3D+.rkt:47:33 rest> #<syntax:displayTrajectory3D...

. ../../../../../../Users/mattei/Scheme-PLUS-for-Racket/nfx.rkt:41:2: pre-check-!*-generic-infix-parser : arguments do not form an infix expression :terms: '(#<syntax:/Applications/Racket v8.14/share/pkgs/qi-lib/macro.rkt:87:47 v> #<syntax:displayTrajectory3D+.rkt:47:17 string-split> #<syntax:displayTrajectory3D+.rkt:47:30 ~>> #<syntax:displayTrajectory3D+.rkt:47:33 rest>

in fact the problem is that the Qi system has added this in front of my valid infix list expression:

#<syntax:/Applications/Racket v8.14/share/pkgs/qi-lib/macro.rkt:87:47 v>

is suppose v is a Qi macro or procedure used internally but as i'm modifying the syntax with syntax transformers after the pre-parsing of the reader and after the Qi modification of syntax my infix? predicate detect that the generated Qi expression is no more infix and it stop .

To be working my macros $nfx$ should be applied before Qi modification, i read Qi is doing it like compilation but i do not know more.

The problem is a bit like macro phase, the order of modifying syntax here is important.

I do not know if there is something to do easily to fix that? i did not know how the Qi system works internally.

[countvajhula]

I think you've correctly identified the problem. It has to do with the relationship between the reading and expansion stages, and the order of expansion.

The $nfx$ macros are inserted into the user syntax at the read stage, before we know anything yet about the meaning of the program.

At the expansion stage, the way I look at it, syntax is transformed in relation to unspecified meaning. That is, expressions are transformed according to rules of the form, "this syntax means what that syntax means." The ultimate meaning of expressions is finally determined when the expressions are evaluated after being expanded and compiled. Yet, the transformations during expansion are aware of meaning, at least in the sense that, unlike reading, expansion rules are context sensitive (cf. this great article by Shriram).

At this expansion stage, to take context into account, macros are expanded "outside in." The outermost macros get to define the meaning of the contained syntax, delegating to any additional macros as needed by expanding to them. So even though the $nfx$ is independently inserted at the reading stage in both places --- the outer one and the nested one --- when it gets to expansion, these are not expanded independently. Instead, the outer $nfx$ expands first to generate the Qi expression. This expression is then expanded by Qi (specifically by the flow macro which specifies the Qi language). And finally, the inner $nfx$ macro is expanded.

During the Qi expansion stage, if we don't tell Qi about $nfx$, it just assumes it is a function and uses it as a function (for example, it might try to compose it with other functions in the expansion). This, as you also mentioned encountering elsewhere, usually leads to a syntax error.

By using define-qi-foreign-syntaxes, we tell Qi to treat $nfx$ as a "foreign" macro, that is, a form that isn't actually a function but has the same syntax as a function and is typically used as if it were a function. Racket's and is a good example of this (so much so, that people are usually confused when they find they can't use and in higher-order functions like foldl!).

So at this point, having been informed about $nfx$, Qi first transforms it for use as a flow. To do this, it rewrites the $nfx$ expression by adding a wrapping lambda taking a single v argument that is then passed to the original macro --- basically just wrapping the macro so it can be used as a function.

And finally, expansion is now delegated to the inner $nfx$ macro. But of course, at this stage, the expression has already been rewritten in a way that can't then be parsed correctly by the $nfx$ macro. That is,

($nfx$ string-split ~> rest ...)

has become:

(lambda (v)
  ($nfx$ v string-split ~> rest ~> ...))

… as you observed!

The issue is that, unlike and, but more like let, $nfx$ is a Racket macro that doesn't have the same syntax as function application and doesn't behave like a function. So it can't be treated as a flow by using define-qi-foreign-syntaxes.

Regarding a solution, one option is to wrap the inner infix expression with (esc ...). This tells Qi that what follows is a Racket expression, and so it will not attempt to expand it.

Another way could be to write a $nfx$ Qi macro using define-qi-syntax-rule, which expands to an appropriate use of the $nfx$ Racket macro, so that, in a way, at the point where Qi is about to rewrite the $nfx$ macro, you get to take back control of expansion at that point. In fact, the previous solution is also similar, where instead of taking control back yourself, you give control to Racket, which is good because $nfx$ is a Racket macro, after all.

Possibly one simple way to write this Qi macro is just:

(define-qi-syntax-rule ($nfx$ e ...)
  (esc ($nfx$ e ...)))

I suspect this will work precisely because the $nfx$ was inserted at the reading stage in a context-free way, so that it doesn't refer to a specific binding (e.g., the Racket macro $nfx$), and the same $nfx$ syntax could refer to either the Qi macro or the Racket macro when the bindings are resolved at expansion time. I could be wrong, though!

If this approach works out, then it could be worth making an adapter library for curly infix syntax and Qi, which defines the necessary adapter macros like $nfx$ as Qi macros.

If you have any other ideas or thoughts (or if we need to debug further), you're welcome to stop by at the next (or any) Qi meetup, which happens on Fridays

[damien_mattei]

@countvajhula

this great article

Your message and the article is very interesting, i will explain how the curly infix parser works too. There are many stage:

1-file is parsed "lexically" character by character to construct the tokens and also the tree of sexpr i think (code was part of SRFI 105, i just modified it)
At this stage too the $nfx$ and $bracket-apply$ are inserted. But also some simple infix code expressions without operator precedence need is already converted in prefix code .Expressions are not evaluated but "quoted".
This is donne by #lang reader SRFI-105 or by externally by the same scheme code in a program called curly-infix2prefix4racket.rkt .

2-the previous code is passed to Scheme/Racket , then the macros $nfx$ and $bracket-apply$ (that are in Scheme+) are expanded , at this point this could conflict with Qi own macro expansion. Note that my macro use syntax transformers, with-syntax, manipulate syntax, not quoted expressions (there is no problem to manipulate macro so)

If there was no need of operator precedence it works because there is no $nfx$ inserted.

A few thoughts about Qi system:

@countvajhula

To do this, it rewrites the $nfx$ expression by adding a wrapping lambda taking a single v argument that is then passed to the original macro --- basically just wrapping the macro so it can be used as a function.

@countvajhula

And finally, expansion is now delegated to the inner $nfx$ macro. But of course, at this stage, the expression has already been rewritten in a way that can't then be parsed correctly by the $nfx$ macro. That is,

($nfx$ string-split ~> rest ...)

has become:

(lambda (v)
  ($nfx$ v string-split ~> rest ~> ...))

… as you observed!

at this point i do not understand how it could work with any macro because Qi has changed the number and place of arguments of the macro.

It would be better that Qi manipulate just the syntax with syntax transformers to keep the correct syntax of any existing macro and this will solve the procedure vs macro problem.

Also if i understand that we can encapsulate a macro in a procedure to be able to use it in a map or foldr ,the resulting procedure is not equivalent to the macro , for example the short-circuited and macro is no more short-circuited when encapsulated in a procedure/lambda because at some point we pass the argument to the procedure/lambda and they are immediatly evaluated.

A possibly workaround about that is for me to modify again the SRFI 105 code to make apply the $nfx$ in phase 1, then the Racket/Qi system will only saw normal already prefixed expression but this has some drawback:

-delegate much work to external parser or reader
-duplicate a lot of procedures (even if they already are generic sometimes) to work not with syntaxified expressions but quoted ones
-not easy possibility to add dynamically new operators with a special precedence order...

@countvajhula

Possibly one simple way to write this Qi macro is just:

(define-qi-syntax-rule ($nfx$ e ...)
  (esc ($nfx$ e ...)))

with this code:

(define-qi-syntax-rule ($nfx$ e ...)
	    (esc ($nfx$ e ...)))

(define Lplot {(tdl) ~> vector->list ~> (△ {string-split ~> rest ~> (△ string->number) ~> vector}) ~> ▽})

it is parsed in:

(define-qi-syntax-rule ($nfx$ e ...) (esc ($nfx$ e ...)))
(define Lplot
      ($nfx$
       (tdl)
       ~>
       vector->list
       ~>
       (△ ($nfx$ string-split ~> rest ~> (△ string->number) ~> vector))
       ~>
       ▽))

but i get an error :

Welcome to DrRacket, version 8.14 [cs].
Language: Determine language from source; memory limit: 8192 MB.
~>: bad syntax in: (~> string-split rest (△ string->number) vector)

with more info for debug:

note: post updated many times due to typos

[countvajhula]

at this point i do not understand how it could work with any macro because Qi has changed the number and place of arguments of the macro.

For a moment assuming that and is Racket's usual and macro, (~> (a) (and b c)) rewrites to a use of (and a b c). In fact, you could also do these variations:

(~>> (c) (and a b))
(~> (a c) (and _ b _))

These are all equivalent to (and a b c). If this is the expression you intended to write, the above variations are just different ways to write it.

(Note: in reality, and is a Qi form that is equivalent to Racket's conjoin so it can't be used this way --- but for the purpose of discussion, I can't think of any other examples of Racket macros that we might want to use in a flow!)

Also if i understand that we can encapsulate a macro in a procedure to be able to use it in a map or foldr ,the resulting procedure is not equivalent to the macro , for example the short-circuited and macro is no more short-circuited when encapsulated in a procedure/lambda because at some point we pass the argument to the procedure/lambda and they are immediatly evaluated.

You're right that there are short-circuiting implications, but can you think of any cases where they would be surprising in practice? Note that in the above examples, the argument is always evaluated, but the other expressions in the and form are shortcircuited as usual. When used in a threading flow, you would usually expect that the threaded arguments are evaluated, so this shouldn't be surprising. The main effect of the lambda isn't to prevent shortcircuiting but to delay evaluation.

With foldr and foldl, as they are functions and as they expect a function in the higher order position, I'm not sure there is any way to get shortcircuiting with them. But if you feel this kind of shortcircuiting fold would be a useful feature, we could consider adding it to Qi's foldl and foldr in some form (ideas welcome!).

with more info for debug:

That's helpful! Well, at present I'm not sure if defining a Qi macro to simplify the integration is straightforward, due to the somewhat complicated interaction between the read and expand stages, but what if you try just using esc on its own? Something like:

(define Lplot {(tdl) ~> vector->list ~> (△ (esc (lambda (v) {(v) ~> string-split ~> rest ~> (△ string->number) ~> vector}))) ~> ▽})

If this works, it should provide a starting point to understanding how the Qi macro might work. We need something that rewrites the resulting use of $nfx$ so that it correctly introduces the implicit argument (we can assume a single argument to start with). Something like this:

(define-qi-syntax-rule ($nfx$ e0 op e ...)
  (esc (lambda (v) ($nfx$ (v) op e0 op e ...)))

I'm not too familiar with the syntax of $nfx$ so I'm not sure if this is quite right. But some variation of this might work if we can at least get it syntactically to look right. I'm also still concerned about the fact that the curly infix transformations occur prior to expansion (and it sounds like, maybe even prior to the reading stage?). So in that case, there might be some issues with introducing the binding op here during expansion, as at the expansion stage it refers to a specific binding and is not just an identifier. But it's worth trying.

[benknoble]

A short note on short-circuiting

I frequently use a pattern like (flow (~> … do some work … (and _ more-work))) where the previous operations are fallible, and more-work should only get invoked if the previous work succeeded. For example:

(define ability (~> (mg) monster-group-set-name (hash-ref ads _ #f) (and _ ability-decks-current)))

If the hash-ref doesn't succeed, I don't want to do ability-decks-current. So you can still short-circuit anything in the and directly, but any values flowing to it must have already been evaluated. (You could delay them yourself if needed by making them thunks and then doing (and … apply …), but I can't come up with a realistic example.)

[countvajhula]

At first when I saw your code, I thought you were doing some elaborate computations on the monster group and was frightened.

But of course it must be Frosthaven

[damien_mattei]

@countvajhula

Note: in reality, and is a Qi form that is equivalent to Racket's conjoin so it can't be used this way -

oh ... i perheaps misunderstand what is really a flow, seems the and of flows is not really the and of scheme...

But i will continue learning Qi and the more concrete example i will get will be the best.

@countvajhula

For a moment assuming that and is Racket's usual and macro, (~> (a) (and b c)) rewrites to a use of (and a b c). In fact, you could also do these variations:

(~>> (c) (and a b))
(~> (a c) (and _ b _))

so it is short-circuited , it is normal to have at least the first argument evaluated

@countvajhula

Something like:

(define Lplot {(tdl) ~> vector->list ~> (△ (esc (lambda (v) {(v) ~> string-split ~> rest ~> (△ string->number) ~> vector}))) ~> ▽})

If this works, it should provide a starting point to understanding how the Qi macro might work.

yes ,it worked

but it force the coder to adapt its code to Qi for this reason i prefer the solutions that follow.... i prefer trying to modify our libraries to make them enought tolerant to be used simply.

@countvajhula

I'm not too familiar with the syntax of $nfx$ so I'm not sure if this is quite right.

The job of $nfx$ is simply to take an infix expression and convert it in prefix. So it is something like:

(define-syntax $nfx$
  (lambda (stx)
    (syntax-case stx ()

       .............

          (($nfx$ e1 op1 e2 op ...)
       
	          (with-syntax
           ...............

where e1 op1 e2 op ... are the argument.

For example {3 * 5 + 2} my favorite example test is transformed in prefix by a call to ($nfx$ 3 * 5 + 2) and will generate something like i expect: (+ (* 3 5) 2) .

This is a simplification because i added, more and more support for both infix and prefix and math expression in the next release and the code is becoming hard and perheaps i should refactor some thing.

for example there is not only a simple case but many depanding of the number of args:

(define-syntax $nfx$
  (lambda (stx)   
    (syntax-case stx ()

      (($nfx$ expr) 

       (with-syntax
			 
	   ((parsed-args
....

(($nfx$ op1 e1) 

       (with-syntax
			 
	   ((parsed-args


.........
(($nfx$ e1 op1 e2 op ...)
       
	 (with-syntax 
			 
	     ((parsed-args


......

source code of $nfx$
Perheaps i should use a macro taking simply a list as arguments but thinking of that will create problems i think.

@countvajhula

I'm also still concerned about the fact that the curly infix transformations occur prior to expansion (and it sounds like, maybe even prior to the reading stage?).

yes and no, it was the case in first versions where i just use SRFI 105 strict but it appeared SRFI 105 was a good idea but just the start of the problem of dealing with infix expression because it does not take in account operator precedence, for example in strict SRFI 105 3 * 5 + 2 should be written {{3 * 5} + 2} , so there is the same number of parenthesis than with sexpr, also different parenthesis, not a big deal !
SRFI 105 was great for expression with the same operator such as {1 + 2 + 3 + 4} because it transformed it in (+ 1 2 3 4) ,great !

And it was perfect for {(x) ~> expr1 ~> expr2 ~> expr3} which works when i reactivate the strict SRFI 105 mode of the parser.

But the latest version of my code SRFI-105/Scheme+ now allow using ( ) inside { } as mathematic parenthesis, so the code has again changed evolved and then some expression as {(x) ~> expr1 ~> expr2 ~> expr3} no more works because $nfx$ is now inserted in the expressions and convert it in prefix not the same way the original SRFI 105 did it.

@countvajhula

We need something that rewrites the resulting use of $nfx$ so that it correctly introduces the implicit argument (we can assume a single argument to start with). Something like this:

(define-qi-syntax-rule ($nfx$ e0 op e ...)
  (esc (lambda (v) ($nfx$ (v) op e0 op e ...)))

this worked again

but i'm a bit surprised because your macro use op e0 op but my infix macro $nfx$ as we see it use op1 e2 op , i mean the operators are not the same but it worked even if my op1 and op are differents

but for this reason i wrote another macro like this :

  (define-qi-syntax-rule ($nfx$ e0 op other ...)
	    (esc (lambda (v) ($nfx$ (v) op e0 other ...))))

that worked too.

i can not decide which macro to choose in the definitive version of my code, as the 2 seems to work?

i tested those macros with this code:

{Lplot := (tdl) ~> vector->list ~> (△ (string-split ~> rest ~> (△ string->number) ~> vector)) ~> ▽}

note := is like set! scheme , like <- too

this code is translated by the parser at step 1 in :

($nfx$
     Lplot
     :=
     (tdl)
     ~>
     vector->list
     ~>
     (△ (string-split ~> rest ~> (△ string->number) ~> vector))
     ~>
     ▽)

and at step 2 , what you probably call expansion stage in:

but it was not so easy to get this result, i have to insert more the ~> operator in my code for a silly reason: SRFI 105 was great to parse infix with same operator, but now i have := and ~> it required a precedence rule to apply and also the code with ~> was translated in some thing like (~> (~> (~> (e1)) e2) e3) (it is an example, not sure of the exact syntax as the code is correct now) which is a valid sexpr but was not compatible with the Qi flow syntax. It generated the famous bad syntax i did not understood ,it was in fact nothing related to Qi but with my parser not simplifying left (?) associative n-arity sexpr.

so i changed my code to get a result like (~> (e1) e2 e3) which is compatible with Qi.

It was not hard to fix but as the code is complex i have to change and modify those things to add the FLOW support:

operators.rkt:

(define (FLOW-op? oper)
  (datum=? oper '~>))

(define (is-associative-operator? op)
  (or (AND-op? op)
      (OR-op? op)
      (XOR-op? op)
      (EQUIV-op? op)
      (ADD-op? op)
      (MULTIPLY-op? op)
      (FLOW-op? op)
      (DEFINE-op? op)
      (ASSIGNMENT-op? op)))

(define (isFLOW? expr)
  (and (pair? expr) (FLOW-op?  (operator expr))))

n-arity.rkt:

;; return a closure of collect-leaves function associated with an operator (AND or OR or others)
(define (make-collect-leaves-operator oper)

  ;;(display "make-collect-leaves-operator") (newline) 
  (let ((ourOperation?
	 
	 (cond
	  
	  ((AND-op? oper) isAND?)
	  ((OR-op? oper) isOR?)
	  ((ADD-op? oper) isADD?)
	  ((DEFINE-op? oper) isDEFINE?)
	  ((ASSIGNMENT-op? oper) isASSIGNMENT?)
	  ((FLOW-op? oper) isFLOW?)

	  (else ; fonction generique pour tous les operateurs ( non syntaxiques )
	   (lambda (expr) (and (pair? expr) (equal? (car expr) oper)))))))

    
    (letrec ((collect-leaves-operator

	      (lambda (expr)
		(cond
		 ((not (list? expr)) (list expr)) ;; symbol,number,boolean,vector,hash table....
		 ((null? expr) (list expr))
		 ((function-without-parameters? expr) (list expr))
		 ((unary-operation? expr)
		  (list
		   (cons
		    (operator expr)
		    (list (n-arity (arg expr))))))
		 ((ourOperation? expr) ;; #;(eqv? oper (operator expr))
		  (apply append (map collect-leaves-operator (args expr))))
		 (else (list (n-arity expr)))))))

        collect-leaves-operator)))) ;  end module

and in $nfx$ :

(if ;;(not (isEXPONENTIAL? expr))
		    (or (isDEFINE? expr)
		       (isASSIGNMENT? expr)
			   (isFLOW? expr))
		    ;;  make n-arity for <- and <+ only (because could be false with ** , but not implemented in n-arity for now)

i have inserted those change for the next release of Scheme+/SRFI-105 to be compatible with Qi ~>.

[countvajhula]

i have inserted those change for the next release of Scheme+/SRFI-105 to be compatible with Qi ~> .

Cool stuff!

it appeared SRFI 105 was a good idea but just the start of the problem of dealing with infix expression because it does not take in account operator precedence

i added, more and more support for both infix and prefix and math expression in the next release and the code is becoming hard and perheaps i should refactor some thing.

Rhombus supports infix operators and allows you to indicate precedence. Maybe there's some inspiration to be found there?

that worked too.
i can not decide which macro to choose in the definitive version of my code, as the 2 seems to work?

In your example, you have one main wrapping {} for the whole expression, but the inner expression in (△ ...) isn't in curly braces:

{Lplot := (tdl) ~> vector->list ~> (△ (string-split ~> rest ~> (△ string->number) ~> vector)) ~> ▽}

So in this case, I suspect the inner expression never gets wrapped with $nfx$, and so, the Qi macro we've written never gets invoked! Assuming the outer $nfx$ traverses the full syntax and changes all infix expressions to prefix, it should work correctly as is, without needing the Qi macro.

Do you anticipate ever using curly brackets in an expression nested within a flow?

If you do, the curly expression would insert $nfx$ there, and in that case, as it is considered a Qi expression, the Qi macro would be invoked. The macro we've written translates the expression into an escaped Racket $nfx$ expression, delegating control back to Racket and to the $nfx$ macro, for processing as usual.

At this point, I would expect in your version of the macro that {f ~> g ...} would become (lambda (v) ($nfx$ (v) ~> f g ...)), so it would likely complain that ($nfx$ (v) ~> f g ...) isn't a valid infix expression since there is no ~> between f and g.

Also note the distinction between Qi's ~> form, and its embedding into Racket that's also named ~>. In our Qi macro here, we are rewriting to the Racket version, hence the initial form that passes the arguments (the Qi version does not mention any arguments). We do this so that $nfx$ is processed as an ordinary Racket expression, the way that was intended by you when you wrote the macro.

Technically, as you pointed out earlier, if we could only give control to $nfx$ before Qi parses the expression, then there would be no problems. For that matter, it should be the same with any macro, like this example:

(require syntax/parse/define)

(define-syntax-parser bool
  [(_ ((~datum &&) a b)) #'(and a b)]
  [(_ ((~datum ||) a b)) #'(or a b)])

(bool (&& 1 2))

If we try to use && as an infix operator as (bool {1 && 2}), then I suspect this macro would issue (the famous) bad syntax error, because the bool macro gets control of expanding the expression, but it doesn't know what to do with $nfx$. And unlike Qi where we are able to get "inside" the language using macros, there's no way to recover from this in this bool macro.

But I am not aware of a way in Racket to indicate that one macro has precedence over others. In fact, this would probably not be sound in general since there could be macros that do define an expansion for ($nfx$ ...), and it could cause an error if it is handled as an infix expression.

Racket's expansion rule is that syntax is expanded outside-in, and within this paradigm, we need to tell the outer macro how to handle the inner macro, even if just to say, as we are doing here with Qi, please do nothing, I promise I'll take care of it later

[LiberalArtist]

@countvajhula

Technically, as you pointed out earlier, if we could only give control to $nfx$ before Qi parses the expression, then there would be no problems. For that matter, it should be the same with any macro, like this example:

…

But I am not aware of a way in Racket to indicate that one macro has precedence over others. In fact, this would probably not be sound in general since there could be macros that do define an expansion for ($nfx$ ...), and it could cause an error if it is handled as an infix expression.

Racket's expansion rule is that syntax is expanded outside-in, and within this paradigm, we need to tell the outer macro how to handle the inner macro, even if just to say, as we are doing here with Qi, please do nothing, I promise I'll take care of it later

Is there a reason local-expand doesn't work in this context? (I haven't read this whole thread in detail, and I also haven't used Qi, so this may be an obvious question.)

[countvajhula]

I think that would require the bool macro, in the above example, to proactively local-expand the $nfx$ macro after requiring it from the Scheme+ library?

I believe Damien needs the $nfx$ macro to transform the source code without requiring cooperation from third party macros, as it is intended to be a general way to write expressions using infix syntax.

To summarize the case as I understand it: $nfx$ is inserted prior to expansion by rewriting all uses of {} in the source file to ($nfx$ ...), and $nfx$ is an ordinary macro invoked at expansion time. In cases where {} happened to be present inside uses of macros in the source file, can these ($nfx$ ...) forms be expanded by the $nfx$ macro first before the containing macros get the result?

[jgomo3]

A bit late to the party.

An alternative:

(call-with-values (curry values 1 2 3) list)

[damien_mattei]

@countvajhula

In your example, you have one main wrapping {} for the whole expression, but the inner expression in (△ ...) isn't in curly braces:

{Lplot := (tdl) ~> vector->list ~> (△ (string-split ~> rest ~> (△ string->number) ~> vector)) ~> ▽}

So in this case, I suspect the inner expression never gets wrapped with $nfx$, and so, the Qi macro we've written never gets invoked! Assuming the outer $nfx$ traverses the full syntax and changes all infix expressions to prefix, it should work correctly as is, without needing the Qi macro.

yes no recall to $nfx$ because there is a map recursively on the sub-sexpr and we all know we can not map with a macro so it is a different procedure (note there is 3 version: github, running for application, and developped for future release, the link is not exactly the one that run on my host but the map is in every versions ) used :

 (define deep-terms (map (lambda (x) (recall-infix-parser x operator-precedence creator))  terms))

yes in the latest version of Scheme+ the inner ( ) are treated as infix. Which could be source of problem ,consider (cons + L) ,is it a prefix one (for sure) but the parser will think it is the addition of cons and L (you can only avoid,it at user leve by (define add +) and do (cons add L) the next version will evalluate (with limited problems described in the great article you where referring previously) cons to find it in the scheme basename library , but again this is not fully reliable, think of (f ° g) with f and g user level procedure not known from basename lib and ° an operator (used infix here) of functions composition.... next release should try to fix all this the better way. But note that having both prefix and infix this way create what is called an 'ambiguous language' .

Yes in the linked code above there is just next this :

;;(display "!*prec-generic-infix-parser : rv : deep-terms:") (display deep-terms) (newline)
      ;; test for simple-infix (no operator precedence)
      (if (simple-infix-list-syntax? deep-terms)
	  (begin
	    ;;(display "!*prec-generic-infix-parser : deep-terms is a simple infix list") (newline)
	    ;;(display "!*prec-generic-infix-parser : deep-terms=") (display deep-terms) (newline)
	    (list ; cadr is op in arg1 op arg2 op ....
	     (cons (cadr deep-terms) (alternating-parameters deep-terms)))) ; we put it in a list because nfx take the car...

	  (begin
	    ;;(display "!*prec-generic-infix-parser : deep-terms is not a simple infix list") (newline)
            (pre-check-!*-generic-infix-parser  deep-terms ;terms
						operator-precedence
						creator)))))

so when there is some simple infix (with same operator) as in the nested flow of example it is put in the good syntax by the first branch of the if :

(list ; cadr is op in arg1 op arg2 op ....
	     (cons (cadr deep-terms) (alternating-parameters deep-terms))

else we apply the parser again:

(pre-check-!*-generic-infix-parser  deep-terms ;terms
						operator-precedence
						creator)

as the operator is taken from (cadr deep-terms) it works for every operator ,even for those that were not in the list like ~> , the new flow operator. (and even for non (left) associative operator wich could result in unknow result..... i shoud add a test in this code)

@countvajhula

Also note the distinction between Qi's ~> form, and its embedding into Racket that's also named ~>.

sure i do not well understand the differences, the arguments are not the same , so i suppose any misunderstanding and misuse by me is important indeed.....

@countvajhula

Do you anticipate ever using curly brackets in an expression nested within a flow?

yes, i understand the problem is that Qi dislike macro in flow , forbids perheaps, unless rewritten in Qi special macro syntax....

But there is no need of macro nested like $nfx$ as i can use ( ) in prefix expression too instead of { } now , the problem is when indexing object with [ ] that is converted in another macro $brackett-apply$ which is used with vector, arrays,strings, hash tables.....

Note that there is really no more concern with nested $nfx$ because as this macro recurse deeply on the sexpr tree you can parse a whole procedure with define-infix and in the future i should wrote an equivalent for a whole module parsing....

@countvajhula

Do you anticipate ever using curly brackets in an expression nested within a flow?

i made a lot of test, for example this works :

(define Lplot {(src) ~> read-lines ~> list->vector ~> (vector-copy _ 5) ~> vector->list ~> (△ {string-split ~> rest ~> (△ string->number) ~> vector}) ~> ▽})

without no need of define-qi-syntax-rule but this do not:

(define-qi-syntax-rule ($nfx$ e0 op other ...)
	    (esc (lambda (v)
	  	   (display "In $nfx$ e0 op other of define-qi-syntax-rule") (newline)
	  	   ($nfx$ e0 op other ...))))

(define Lplot {(src) ~> read-lines ~> list->vector ~> (vector-copy _ 5) ~> vector->list ~> (△ {string-split ~> rest ~> (△ string->number) ~> vector}) ~> ▽})

it expands in:

(define Lplot
      ($nfx$
       (src)
       ~>
       read-lines
       ~>
       list->vector
       ~>
       (vector-copy _ 5)
       ~>
       vector->list
       ~>
       (△ ($nfx$ string-split ~> rest ~> (△ string->number) ~> vector))
       ~>
       ▽))

and after:

but it works with your macro that i still do not understand...

(define-qi-syntax-rule ($nfx$ e0 op e ...)
	    (esc (lambda (v)
	  	   (display "In $nfx$ e0 op e of define-qi-syntax-rule") (newline)
	  	   ($nfx$ (v) op e0 op e ...))))

.........

In $nfx$ e0 op e of define-qi-syntax-rule
In $nfx$ e0 op e of define-qi-syntax-rule
In $nfx$ e0 op e of define-qi-syntax-rule
In $nfx$ e0 op e of define-qi-syntax-rule
In $nfx$ e0 op e of define-qi-syntax-rule
In $nfx$ e0 op e of define-qi-syntax-rule
In $nfx$ e0 op e of define-qi-syntax-rule
In $nfx$ e0 op e of define-qi-syntax-rule
In $nfx$ e0 op e of define-qi-syntax-rule
In $nfx$ e0 op e of define-qi-syntax-rule
after Lplot
Lplot length=1440
Lplot=
(#(-5.226162 -0.133805 1.462782) #(-5.239695 -0.133871 1.446325) #(-5.253049 -0.133932 1.429818) #(-5.266226 -0.133988 1.413263) #(-5.279225 -0.134038 1.396661) #(-5.292047 -0.134083 1.380013) #(-5.304692 -0.134123 1.363318) #(-5.317161 -0.134158 1.346578) #(-5.329453 -0.134187 1.329793) #(-5.34157 -0.134211 1.312965) #(-5.353512 -0.13423 1.296094) #(-5.365279 -0.134243 1.279181) #(-5.376871 -0.134252 1.262226) #(-5.388289 -0.134255 1.24523)



.......

$nfx$ is not a problem as i can parse a whole procedure ,module or even file with a single initial call to $nfx$ and as you said it the macro expand from outside-in or even i can use your mysterious macro.

The concern about [ ] ,alias $brackett-apply$ ,you perheaps noticed that i replace the macro with a scheme procedure (vector-copy _ 5) because this won't works: _[5 :] as it is transformed in a macro : ($brackett-apply$ _ 5 :) ,note this is a slicing notation such as the one in Python.

here is my definition of $bracket-apply$ :

;; > (define x 1)
;; > ($bracket-apply$ (vector 0 1 2 3 4) 2 + x)

;; bracket-apply : #'parsed-args={.#<syntax:Users/mattei/Scheme-PLUS-for-Racket/main/Scheme-PLUS-for-Racket/src/bracket-apply.scm:126:30 list> {.#<syntax:72-interactions from an unsaved editor:17:40 +> .#<syntax:72-interactions from an unsaved editor:17:38 2> .#<syntax:72-interactions from an unsaved editor:17:42 x>}}
;; 3
  
(define-syntax $bracket-apply$  ;;  this implements a possible bracket-apply as proposed in SRFI-105
  
  (lambda (stx)
    
    (syntax-case stx ()

      ;; a version that pre-compil the infix expression, should be faster
      (($bracket-apply$ container arg-bracket ...) ;  . args-brackets

      
       (with-syntax ((parsed-args

		      (cons #'list 
		      	    (parse-square-brackets-arguments-lister-syntax (syntax->list #'(arg-bracket ...)))))) 
	 
	 ;;(newline)
	 ;;(display "bracket-apply : #'parsed-args=") (display #'parsed-args) (newline)
	 
	 #'($bracket-apply$next4list-args container parsed-args))))))

For this to be working i can imagine a solution with one of your Qi macro :

(define-qi-syntax-rule ($bracket-apply$ container arg-bracket ...)
	    (esc (lambda (v)
	  	   (display "In $bracket-apply$ of define-qi-syntax-rule") (newline)
	  	   ($bracket-apply$ (v)  container arg-bracket ...))))

this one seems to works but i can not test it (but tested with a true vector not a _ wildcar, it worked ) because with this :

(define Lplot {(src) ~> read-lines ~> list->vector ~> _[5 :] ~> vector->list ~> (△ {string-split ~> rest ~> (△ string->number) ~> vector}) ~> ▽})

wildcard _ not allowed:

@countvajhula

If we try to use && as an infix operator as (bool {1 && 2}), then I suspect this macro would issue (the famous) bad syntax error, because the bool macro gets control of expanding the expression, but it doesn't know what to do with $nfx$. And unlike Qi where we are able to get "inside" the language using macros, there's no way to recover from this in this bool macro.

yes

 (require syntax/parse/define)


(require syntax/parse/define)


#<eof>
> (define-syntax-parser bool
  [(_ ((~datum &&) a b)) #'(and a b)]
  [(_ ((~datum ||) a b)) #'(or a b)])


(define-syntax-parser
 bool
 ((_ ((~datum &&) a b)) #'(and a b))
 ((_ ((~datum \|\|) a b)) #'(or a b)))


#<eof>
> (bool (&& 1 2))


(bool (&& 1 2))
2


#<eof>
> (bool {1 && 2})


(bool ($nfx$ 1 && 2))
bool: expected one of these literal symbols: `&&' or `\|\|' in: $nfx$

but i suppose define-syntax-parser is special to Racket.

any idea to replace the _ wildcard?

update:

with :

(define-qi-foreign-syntaxes $bracket-apply$)

all is working now ,even this one:

{Lplot := (src) ~> read-lines ~> list->vector ~> _[5 :] ~> vector->list ~> (△ {string-split ~> rest ~> (△ string->number) ~> vector}) ~> ▽}

and the red dot spacecraft trajectory are non continous only because the original data are not good

update 2:

this one (with parenthesis) works too:

{Lplot := (src) ~> read-lines ~> list->vector ~> _[5 :] ~> vector->list ~> (△ (string-split ~> rest ~> (△ string->number) ~> vector)) ~> ▽}

[countvajhula]

That's great!

your mysterious macro

You may have already figured it out, but we are trying to rewrite

($nfx$ f ~> g ~> ...)

occurring in a Qi context (i.e. the nested flow inside △ in your example) to

(esc (lambda (v) ($nfx$ (v) ~> f ~> g ~> ...)))

The initial op occurs one extra time because we are introducing an arguments clause (v) that wasn't there in the original expression written by the user.

(bool ($nfx$ 1 && 2))
bool: expected one of these literal symbols: `&&' or `\|\|' in: $nfx$

Yeah, note that unlike bool, the ability to get cooperation from Qi by writing a Qi macro is a special feature provided by only a few macros in Racket (I can only think of Qi's flow (implemented using Syntax Spec which provides this macro-extensibility) and Racket's match which provides "match expanders").

So doing this infix → prefix tranformation in two stages feels risky in general. One idea is to avoid the $nfx$ macro by doing the entire transformation from curly infix syntax to prefix syntax at the read stage, as that would allow you to do all the transformations before any macros (including bool) get control of the syntax.

I don't have much experience with this (others here may have suggestions), but you may find readtables and Beautiful Racket useful, to explore this option and see if it's right for your use case.

and the red dot spacecraft trajectory are non continous only because the original data are not good

Are you affiliated with BepiColombo? Seems very cool. I didn't realize Mercury had polar ice caps!