Copy-file and upi/cp fail to copy the entire file

Questions & Answers
1

hello,

a so weird error i do not expect answers but who knows....

when copying a file with copy-file or upi/cp:

https://docs.racket-lang.org/upi/upi-cp.html

it is truncated near the end (496kb file):

<TR>
      <TD>
        2021-01-01T23:53:00.000
      </TD>
      <TD>
        -0.479085

so i get error in browser :

Erreur d’analyse XML : aucun élément trouvé
Emplacement : http://localhost/Vl5OLGf.xml
Numéro de ligne 32998, Colonne 18 :
        -0.479085
-----------------^

just wanted to avoid a few lines of code by using copy-file to avoid launching an unix subprocess with cp, so i code it this way:

    ;; copy file to make it public to the remote web browser
	{basename-trajectory-xml <- (basename trajectory-xml)}
	;;(copy-file trajectory-xml (string-append "../" basename-trajectory-xml))
	(cp trajectory-xml (string-append "../" basename-trajectory-xml))

if i look to the source file on disk the end of file is correct:

<TD>
        -0.479085
      </TD>
      <TD>
        0.023290
      </TD>
      <TD>
        -2.295427
      </TD>
      <TD>
        13.800711743772244
      </TD>
      <TD>
        -0.9750889679715302
      </TD>
      <TD>
        13.615658362989326
      </TD>
    </TR>
    <TR>
      <TD>
        2021-01-01T23:54:00.000
      </TD>
      <TD>
        -0.428158
      </TD>
      <TD>
        0.021976
      </TD>
      <TD>
        -2.272187
      </TD>
      <TD>
        14.133451957295375
      </TD>
      <TD>
        -0.896797153024911
      </TD>
      <TD>
        14.160142348754448
      </TD>
    </TR>
    <TR>
      <TD>
        2021-01-01T23:55:00.000
      </TD>
      <TD>
        -0.377049
      </TD>
      <TD>
        0.020649
      </TD>
      <TD>
        -2.247945
      </TD>
      <TD>
        14.281138790035588
      </TD>
      <TD>
        -1.1245551601423487
      </TD>
      <TD>
        14.524911032028472
      </TD>
    </TR>
    <TR>
      <TD>
        2021-01-01T23:56:00.000
      </TD>
      <TD>
        -0.325759
      </TD>
      <TD>
        0.019309
      </TD>
      <TD>
        -2.222663
      </TD>
      <TD>
        14.583629893238435
      </TD>
      <TD>
        -1.208185053380783
      </TD>
      <TD>
        14.87544483985765
      </TD>
    </TR>
    <TR>
      <TD>
        2021-01-01T23:57:00.000
      </TD>
      <TD>
        -0.274306
      </TD>
      <TD>
        0.017956
      </TD>
      <TD>
        -2.196307
      </TD>
      <TD>
        14.818505338078293
      </TD>
      <TD>
        -1.1672597864768683
      </TD>
      <TD>
        15.21708185053381
      </TD>
    </TR>
    <TR>
      <TD>
        2021-01-01T23:58:00.000
      </TD>
      <TD>
        -0.222710
      </TD>
      <TD>
        0.016590
      </TD>
      <TD>
        -2.168841
      </TD>
      <TD>
        15.09608540925267
      </TD>
      <TD>
        -1.2188612099644127
      </TD>
      <TD>
        15.701067615658364
      </TD>
    </TR>
    <TR>
      <TD>
        2021-01-01T23:59:00.000
      </TD>
      <TD>
        -0.170992
      </TD>
      <TD>
        0.015211
      </TD>
      <TD>
        -2.140229
      </TD>
      <TD>
        15.45373665480427
      </TD>
      <TD>
        -1.3558718861209964
      </TD>
      <TD>
        16.144128113879002
      </TD>
    </TR>
  </TABLEDATA>
</DATA>
  </TABLE>
 </RESOURCE>
</VOTABLE>

when i look to the end of destination file on disk:

      <TD>
        2021-01-01T23:53:00.000
      </TD>
      <TD>
        -0.479085

it is truncated. So the copied file is really truncated this is not an effect of network web application or browser.

the source file size is :

499771 bytes, so i do not think it is big or slow to copy it....

i tested with copy-file, upi/cp many times , it does not works...

any idea about what cause that?

update:

here is the file info on disk:

ls -la Vh5RkaG.xml /opt/homebrew/var/www/Vh5RkaG.xml 
-rwxrw-rw-  1 _www  admin  497382  9 déc 15:27 /opt/homebrew/var/www/Vh5RkaG.xml
-rwxrw-rw-  1 _www  wheel  499771  9 déc 15:27 Vh5RkaG.xml

clearly it is truncated.

update 2:

but i can copy it manually in command line, just to test if there was some quota or something else activate n the filesystem:

_www@mbp-touch-bar-1 tmp % cp Vh5RkaG.xml /opt/homebrew/var/www/test.xml   
_www@mbp-touch-bar-1 tmp % ls -la Vh5RkaG.xml /opt/homebrew/var/www/test.xml
-rwxr--r--  1 _www  admin  499771  9 déc 15:47 /opt/homebrew/var/www/test.xml
-rwxrw-rw-  1 _www  wheel  499771  9 déc 15:27 Vh5RkaG.xml
2

That does seem strange.

I can imaging getting that result by having copy-file called in a thread, and either exiting the process or terminating the thread before it completes, since copy-file is not atomic. But probably you're not calling copy-file in a thread — right?

Is the truncation always at the same file size or always by the same amount? The copy-file function copies in increments of (up to) 4096 bytes, and the file-size delta is less than that in the example, but that may not mean anything.

3

i did not create a thread but this is a web app, the browser ,request a PHP method that call a web service, here on the same host, the web service is returned by a httpd/php server that launch PHP web service on the server , PHP backend calls Racket program that do some stuff on text and xml files ,etc, remove previous computed file (with rm in a subprocess) and call a Python interpolation code (4-5 minutes) and after Racket build an XML file, this is this XML file that in the last lines of Racket code should copy it on a public directory of the web server to make the remote browser possible access to the resulting XML file.(the file names are always differents in tmp dir)

I did not think there is something cutting the Racket call because it reach the end of the Racket code successfully (see log).

here is the end of Racket code:

(display "interpole_fields : end of post-processing XML file: ") (display (current-date)) (newline)

	
	(display "interpole_fields : trajectory-xml=") (display trajectory-xml) (newline)
	
	;; copy file to make it public to the remote web browser
	{basename-trajectory-xml <- (basename trajectory-xml)}
	(copy-file trajectory-xml (string-append "../" basename-trajectory-xml))
	;;(cp trajectory-xml (string-append "../" basename-trajectory-xml))
	
	(display " interpole_fields : last-text=") (display last-text) ; this last line should be in $message variable of PHP caller
       
	
	;; for my program the outport port of the subprocess are my input ports and vice versa
	(close-input-port stdout)
	(close-output-port stdin)
	(close-input-port stderr)

) ; end module

and here is the http log:

  [24] => interpole_fields : end of post-processing XML file: #(struct:date* 5 27 15 9 12 2024 1 343 #f 3600 216079235 CET)
    [25] => interpole_fields : trajectory-xml=/private/var/tmp/Vh5RkaG.xml
    [26] =>  interpole_fields : last-text=CPU finished interpolation computation
)

GeneralReturn  : this->url_output=/private/var/tmp/Vh5RkaG.xml

note: 24,25,26 are lines from Racket code displayed by PHP, GeneralReturn is from PHP code.

So all is running well, no hangup , or crash...

always at the same point, and strangely near the end as you noticed it, both when using copy-file or upi/cp.

Capture d’écran 2024-12-09 à 22.39.37
Capture d’écran 2024-12-09 à 22.39.37557×364 20.4 KB

Tonight i made a last test from Racket GUI REPL to test copy-fileand all is good here:

Welcome to DrRacket, version 8.14 [cs].
Language: racket, with debugging; memory limit: 8192 MB.
> copy-file
#<procedure:copy-file>
> (copy-file "/private/var/tmp/Vh5RkaG.xml" "/opt/homebrew/var/www/test-from-racket-gui.xml")

from the Racket GUI REPL the file are well copied:

_www@mbp-touch-bar-1 tmp % ls -la Vh5RkaG.xml /opt/homebrew/var/www/test-from-racket-gui.xml 
-rwxrw-rw-@ 1 mattei  admin  499771  9 déc 22:03 /opt/homebrew/var/www/test-from-racket-gui.xml
-rwxrw-rw-  1 _www    wheel  499771  9 déc 15:27 Vh5RkaG.xml

probably tomorrow i will launch a sub-process with a real cp command (as i did it for the rm of previous files before using Racket filesystem library`) to do the job. Hoping then it will work.

Another last test to see if there is some problem caused by my scheme+ library used in code or the SRFI 105 infix reader, here is a test again in a REPL for scheme+ and SRFI105:

Welcome to DrRacket, version 8.14 [cs].
Language: reader SRFI-105, with debugging; memory limit: 8192 MB.
Parsed curly infix code result = 

(module repl racket (provide (all-defined-out)) (require Scheme+))
>
> copy-file


copy-file
#<procedure:copy-file>


#<eof>
> (copy-file "/private/var/tmp/VK6D8Oa.xml" "/opt/homebrew/var/www/test-from-racket-gui-scheme+-reader-srfi105.xml")


(copy-file
 "/private/var/tmp/VK6D8Oa.xml"
 "/opt/homebrew/var/www/test-from-racket-gui-scheme+-reader-srfi105.xml")


#<eof>

and the result on disk is ok:

_www@mbp-touch-bar-1 tmp % ls -la VK6D8Oa.xml /opt/homebrew/var/www/test-from-racket-gui-scheme+-reader-srfi105.xml 
-rwxrw-rw-@ 1 mattei  admin  499771  9 déc 22:49 /opt/homebrew/var/www/test-from-racket-gui-scheme+-reader-srfi105.xml
-rwxrw-rw-  1 _www    wheel  499771  9 déc 22:35 VK6D8Oa.xml

but the new copied file is again truncated by copy-filein the web application at exactly the same byte (size):

_www@mbp-touch-bar-1 tmp % ls -la /opt/homebrew/var/www/VK6D8Oa.xml
-rwxrw-rw-  1 _www  admin  497382  9 déc 22:35 /opt/homebrew/var/www/VK6D8Oa.xml
4

more and more strange : even when launching cp the file is truncated :scream:

;; copy it by cp UNIX command
	(define web-root "/opt/homebrew/var/www")
	(define cp-prcss-lst (process*  "/bin/cp"
					"-p"
					trajectory-xml
					web-root))

	(display "interpole_fields : cp-prcss-lst =") (display cp-prcss-lst) (newline)
	(define cp-prcss-vct (list->vector cp-prcss-lst))
	{cp-stdout <- cp-prcss-vct[0]}
	{cp-stdin <- cp-prcss-vct[1]}
	{cp-id <- cp-prcss-vct[2]}
	{cp-stderr <- cp-prcss-vct[3]}
	{cp-proc-ctrl <- cp-prcss-vct[4]}
	
	(display "interpole_fields : passed cp") (newline)
	
	(define cp-err-lns (port->lines cp-stderr))
	(define cp-len-err-lns (length cp-err-lns))
	(display "number of lines in cp-stderr=") (display cp-len-err-lns) (newline)
	(when {cp-len-err-lns > 0}
	      (display "cp Error lines:") (display cp-err-lns) (newline))

	(define cp-out-lns (port->lines cp-stdout))
	(display "Output of cp:") (display cp-out-lns) (newline)
	
	(display " interpole_fields : last-text=") (display last-text) ; this last line should be in $message variable of PHP caller
       
	
	;; for my program the outport port of the subprocess are my input ports and vice versa
	(close-input-port stdout)
	(close-output-port stdin)
	(close-input-port stderr)

) ; end module

browser:

Erreur d’analyse XML : aucun élément trouvé
Emplacement : http://localhost/VK6D8Oa.xml
Numéro de ligne 32998, Colonne 18 :
        -0.479085
-----------------^

logs are good:

 [25] => interpole_fields : trajectory-xml=/private/var/tmp/VYFfEE0.xml
    [26] => interpole_fields : cp-prcss-lst =(#<input-port:subprocess-stdout> #<output-port:subprocess-stdin> 98850 #<input-port:subprocess-stderr> #<procedure:control>)
    [27] => interpole_fields : passed cp
    [28] => number of lines in cp-stderr=0
    [29] => Output of cp:()
    [30] =>  interpole_fields : last-text=CPU finished interpolation computation
)

GeneralReturn  : this->url_output=/private/var/tmp/VYFfEE0.xml

ok... writing that i understand now , if 3 methods failed the problem is in other place.

it is a port of code from Latmos laboratory (i'm in Lagrange lab.)

the Latmos code was in PHP and Fortran , our port keeps the main logic of PHP but use Python and Racket instead of Fortran.

the PHP code write the XML header, their fortran code compute but i have not the source code so i do not know where they write the XML body of data, i use the compute code in Python (code written by a student) ,then it generates the XML body data table in Racket and there was a GetURL.php post process at latmos they forget to give to us that should just made public to the world the xml computed locally. I do it in Racket but file was probably close in PHP missing code with perheaps the closing xml tag added unless it was the Fortran code that did it but i will not rely on that, it is what i do me in Racket code but as the XML file was open initially by PHP process i did not close it (this is the PHP missing code job) and the copy is made on a non closed,non flushed file and we miss the buffered end of output xml file in the copy.

solution : just add one of those lines:

(flush-output xml-file)

or

(close-output-port xml-file)

:man_facepalming: (should have think before)

and it works with any of the 3 methods (copy-file,upi/cp,UNIX cp) :

Capture d’écran 2024-12-10 à 12.09.38
Capture d’écran 2024-12-10 à 12.09.383778×2126 709 KB