Pyffi with numpy (<1.26) low-level lib and Python 3.1x

[damien_mattei]

Hi,

I try to use pyffi but i have a Mac with Python 3.9 by default from Apple and Python 3.12 for work for depandancies with some scientific libraries. pyffi is searching for version 3.10 so i did:

/Applications/Racket/bin/raco pyffi configure /opt/homebrew/bin/python3.12
Configuration tool for `pyffi`.
-------------------------------
This tool attempts to find the shared library `libpython3` 
with the help of the `python3` executable.

The executable
    /opt/homebrew/bin/python3.12
will be used to find the location of the shared library.

The preference for LIBDIR is now set to:
    /opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/lib

The preference for DATA is now set to:
    /opt/homebrew

But unfornunately pyffi still search for the 3.10 version:

(require pyffi)
. . ../../Applications/Racket v8.18/collects/ffi/unsafe.rkt:134:0: ffi-lib: could not load foreign library
  path: libpython3.10.dylib
  system error: dlopen(libpython3.10.dylib, 0x000A): tried: 'libpython3.10.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache), 'libpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache)
> 

I suppose there is nothing else to do but install also python 3.10 on my system?

[damien_mattei]

just installing python 3.10 solve the problem but would it be hard to make pyffi working for different versions of python because as-is this really restrict the use of this good (even if not yet tested) ffi library?

[soegaard]

would it be hard to make pyffi working for different versions of python

Probably not.

Try it!

Note: Look up if there are have been changes to the C-API:

Python/C API Reference Manual

I believe, they are rather conservative wrt making changes,
so I expect minor changes at the most.

[damien_mattei]

i was asking myself if it could just be enough to change the string

@damien_mattei

libpython3.10.dylib

in the source code?

[soegaard]

I think so - at least if the C-api hasn't changed.

[damien_mattei]

i have installed Numpy with pip3.10 after the pyffi install but then i got this error:

#lang racket

(require pyffi
         pyffi/numpy)

(initialize)
(post-initialize)
;(run "1+2")
(import sys)
(run "sys.version")

Welcome to DrRacket, version 8.18 [cs].
Language: racket, with debugging; memory limit: 128 MB.
. . get: not bound: yy numpy in numpy.array

this seems only related with pyffi/numpy

update:

i even started drracket an dreinstall pyffi from a python virtual environment of 3.10 version:

(myvenv3.10) mattei@mac ~ % /Applications/Racket/bin/raco pyffi show                                      
Current configuration for 'pyffi'.

    libdir = "/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib"
    data   = "/Users/mattei/myvenv3.10"

Meaning:

    libdir:  location of the shared library 'libpython'
    data:    location of bin/ lib/ share/ etc.

and about numpy:

(myvenv3.10) mattei@mac ~ % ls -la myvenv3.10/lib/python3.10/site-packages/
total 8
drwxr-xr-x   5 mattei  staff   160  2 déc.  14:24 _distutils_hack
drwxr-xr-x  11 mattei  staff   352  2 déc.  14:48 .
drwxr-xr-x   3 mattei  staff    96  2 déc.  14:23 ..
-rw-r--r--   1 mattei  staff   151  2 déc.  14:24 distutils-precedence.pth
drwxr-xr-x  55 mattei  staff  1760  2 déc.  14:48 numpy
drwxr-xr-x   9 mattei  staff   288  2 déc.  14:48 numpy-2.2.6.dist-info
drwxr-xr-x   9 mattei  staff   288  2 déc.  14:24 pip
drwxr-xr-x   9 mattei  staff   288  2 déc.  14:24 pip-25.3.dist-info
drwxr-xr-x   7 mattei  staff   224  2 déc.  14:24 pkg_resources
drwxr-xr-x  52 mattei  staff  1664  2 déc.  14:24 setuptools
drwxr-xr-x  10 mattei  staff   320  2 déc.  14:24 setuptools-80.9.0.dist-info

if i start python i can import numpy but not with Racket

[damien_mattei]

(post-initialize)
Fatal Python error: pyinit_main_reconfigure: fail to reconfigure Python
Python runtime state: initialized
Traceback (most recent call last):
  File "/Users/mattei/myvenv3.10/lib/python3.10/site-packages/numpy/__init__.py", line 400, in __getattr__
    raise AttributeError(
AttributeError: `np.add_newdoc` was removed in the NumPy 2.0 release. It's still available as `np.lib.add_newdoc`.

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator (total: 11)

problem seems to come from the numpy version used at year 2022, because each time i change version i got a different error of unbound numerical type

with numpy 1.23:

get: not bound: yy float128 in numpy.float128

ok, finally testing 3 years of numpy release i get the 1.26 working with this code:

#lang racket

(require pyffi
         pyffi/numpy
         pyffi/numpy-core)

(initialize)
(display "initialize passed") (newline)
;(initialize-numpy)           ; load the `numpy` module
;(display "initialize passed") (newline)
(import-numpy)
(display "import-numpy passed") (newline)
;(post-initialize)


(finish-initialization)
(display "finish-initialization passed") (newline)
(declare-special-prefix numpy)

;(run "1+2")
(import sys)
;(run "sys.version")
sys.version_info

;numpy.__version__
numpy.dot

Welcome to DrRacket, version 8.18 [cs].
Language: racket, with debugging; memory limit: 128 MB.
initialize passed
import-numpy passed
finish-initialization passed
(obj "version_info" : sys.version_info(major=3, minor=10, micro=19, releaselevel='final', serial=0))
#<procedure:dot>

but it seems there is only a subset of numpy, how can i add new API in the code,
i need for example :

numpy.lib.format.read_magic

read_array_header_1_0 or read_array_header_2_0

seems to be in numpy-core.rkt, i'm interested in modifying this code, but what is the way to do it?

[damien_mattei]

Done

after a clone/fork and a bit of reverse engineering i conclude there was no need of modifying the source code. Just the way to use numpy in racket is important because there is some double methods for importing (import and import-numpy) and for initializing ( post-initialize and finish-initialization) that sometimes do the same sometimes i have no idea what they are for. Reading the doc and testing the example was indeed helpful.

here is a working solution of this problem , that was reading just the numpy header of .npy file very big (21Gb), a 3D cube of magnetic field, read the header to extract the math type , is it 1D,3D, scalar,vector? and the numeric type, float64? or other.....

ls -la /opt/homebrew/var/www/Data/BplusBdip64array_it0_rank.npy
-rw-r-xr-x  1 mattei  admin  24192000128 10 nov.  11:40 /opt/homebrew/var/www/Data/BplusBdip64array_it0_rank.npy

a scheme solution:

#lang racket

(require pyffi
         pyffi/numpy
         pyffi/numpy-core)

;(set-environment-variables) ; do not know what it does?
(initialize)
(display "initialize passed") (newline)

;(display "initialize passed") (newline)
(import-numpy)
(display "import-numpy passed") (newline)

;; 2 methods below seems to work
(post-initialize)
(display "post-initialize passed") (newline)
;(finish-initialization)
;(display "finish-initialization passed") (newline)

(declare-special-prefix numpy)

;(run "1+2")
(import sys)
;(run "sys.version")
;sys.version_info
(import numpy)

;; does not works at any place
;;(initialize-numpy)           ; load the `numpy` module


;numpy.__version__
;numpy.dot
(define name "/opt/homebrew/var/www/Data/BplusBdip64array_it0_rank.npy")
(define fobj (builtins.open name "rb"))
(define version-file (numpy.lib.format.read_magic fobj)) ;  determine the format version of the file
(define version-file-vect (pytuple->vector version-file))
(define func_name (apply string-append (cons "read_array_header" (vector->list (vector-map (λ (n) (string-append "_" (number->string n)))
                                                                                       version-file-vect)))))
(define func (getattr numpy.lib.format func_name)) ; returns the shape of the array, whether or not it is F-contiguous, and the dtype
(define Varray-header (pytuple->vector (func fobj)))
(define dim-obj (pytuple->vector (vector-ref Varray-header 0)))
dim-obj
(define data-type (vector-ref Varray-header 2))
(define type-obj data-type.name)
type-obj

and a scheme+ solution:

#lang reader SRFI-110 ; 105
(require (rename-in Scheme+
                    (:= :=s)
                    (-> s->)))
(require pyffi
         pyffi/numpy
         pyffi/numpy-core)

;(set-environment-variables) ; do not know what it does?

(initialize)
(display "initialize passed") (newline)

;(import-numpy) ; does not seem mandatory
;(display "import-numpy passed") (newline)


;; 2 methods below seems to work
(post-initialize)
(display "post-initialize passed") (newline)

;(finish-initialization)
;(display "finish-initialization passed") (newline)


(declare-special-prefix numpy) ; does not seem mandatory, at least for my code

;(run "1+2")
(import sys)
;(run "sys.version")
;sys.version_info

(import numpy)
(display "import numpy passed") (newline)

;; does not works at any place
;;(initialize-numpy)           ; load the `numpy` module

;numpy.__version__
;numpy.dot

{name <- "/opt/homebrew/var/www/Data/BplusBdip64array_it0_rank.npy"}
{fobj <- (builtins.open name "rb")}
{version-file <- (numpy.lib.format.read_magic fobj)} ;  determine the format version of the file
{version-file-vect <- (pytuple->vector version-file)}
{func_name <- (apply string-append (cons "read_array_header" (vector->list (vector-map (λ (n) (string-append "_" (number->string n)))
                                                                                       version-file-vect))))}
{func <- (getattr numpy.lib.format func_name)} ; returns the shape of the array, whether or not it is F-contiguous, and the dtype
{Varray-header <- (pytuple->vector (func fobj))}
{dim-obj <- pytuple->vector(Varray-header[0])}
dim-obj
{data-type <- Varray-header[2]}
{type-obj <- data-type.name}
type-obj

they both output the dimension and type:

Welcome to DrRacket, version 8.18 [cs].
Language: Determine language from source; memory limit: 128 MB.
SRFI 110 Curly Infix,Neoteric and Sweet expressions v1.2 for Scheme+
SRFI-110 Curly Infix parser for Racket Scheme and R6RS by Damien MATTEI
(based on code from David A. Wheeler and Alan Manuel K. Gloria.)

Possibly skipping some header's lines containing space,tabs,new line,etc  or comments.

SRFI-110 Curly Infix reader : number of skipped lines (comments, spaces, directives,...) at header's beginning : 2

Parsed curly infix code result = 
(require (rename-in Scheme+ (:= :=s) (-> s->)))
(require pyffi pyffi/numpy pyffi/numpy-core)
(initialize)
(display "initialize passed")
(newline)
(post-initialize)
(display "post-initialize passed")
(newline)
(declare-special-prefix numpy)
(import sys)
(import numpy)
(display "import numpy passed")
(newline)
(<- name "/opt/homebrew/var/www/Data/BplusBdip64array_it0_rank.npy")
(<- fobj (builtins.open name "rb"))
(<- version-file (numpy.lib.format.read_magic fobj))
(<- version-file-vect (pytuple->vector version-file))
(<-
 func_name
 (apply
  string-append
  (cons
   "read_array_header"
   (vector->list
    (vector-map
     (λ (n) (string-append "_" (number->string n)))
     version-file-vect)))))
(<- func (getattr numpy.lib.format func_name))
(<- Varray-header (pytuple->vector (func fobj)))
(<- dim-obj (pytuple->vector ($bracket-apply$ Varray-header 0)))
dim-obj
(<- data-type ($bracket-apply$ Varray-header 2))
(<- type-obj data-type.name)
type-obj

Scheme+ v17.2 by Damien Mattei

initialize passed
post-initialize passed
import numpy passed
'#(3 700 1200 1200)
"float64"
> 

this could not be done simply in scheme as it requires to use low-level numpy API to just read the .npy numpy file.

Note: this should be not critical to use only python 3.10 as other part of the data treatment sequence which is sequential can use other version of Python and numpy,if the multiple Python,Numpy versions are correctly managed on the system.

[soegaard]

Great to hear you succeeded in getting Pyffi up and running.

The release notes for Numpy says the oldest supported Python version is 3.11.
So it seems it is about time to update Pyffi to handle something newer than 3.10.

Are you interested in updating Pyffi to use Python 3.14?

The funny looking (initialize) and (post-initialize) is explained here:

6 Initialization of the Python Interpreter

Basically the issue is that initialize is run after any required Racket modules are instantiated. This means that a Racket module for numpy doesn't have an Python interpreter instance available. Instead it uses add-initialization-thunk to delay anything that needs an instance. When initialize has created an instance, then post-initialize can invoke any initialization thunks.

[damien_mattei]

@soegaard

The release notes for Numpy says the oldest supported Python version is 3.11.
So it seems it is about time to update Pyffi to handle something newer than 3.10.

yes, i used python 3.11 on the previous system, now 3.12, going back to 3.11 seems feasible, i say that about error (see below) , even if it exists 3.13 and 3.14 , the scientific library (can not remember exactly because there is a lot, matplotlib,scipy,numpy of course) i use only deal with at max python 3.12
But 3.11 is still ok.

@soegaard

Are you interested in updating Pyffi to use Python 3.14?

going so far is not necessary, but i was wrong when thinking i could melt multiple python version on the whole application:

The application run on a web server,http->PHP->Racket/scheme->Python/Numpy/Matpplotlib/Scipy

i set up the web server account to use the 3.11 or 3.12 python virtual environment, for this reason, pyffi for racket now fails to start on the web server code:

(module interpolate-field-caller racket
	
	(require (rename-in Scheme+
                            (:= :=s) ; for compatibility with pyffi
                            (-> s->)))


        (require pyffi
                 pyffi/numpy
                 pyffi/numpy-core)

.....


;;;;;;;;;;;;;;;;;;; numpy with pyffi ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
        ;(set-environment-variables) ; do not know what it does?

        (initialize)
        (display "Scheme+ : interpole_fields : initialize passed") (newline)

        ;(import-numpy) ; does not seem mandatory
        ;(display "Scheme+ : interpole_fields : import-numpy passed") (newline)

        ;; 2 methods below seems to work
        (post-initialize)
        (display "Scheme+ : interpole_fields : post-initialize passed") (newline)

        ;(finish-initialization)
        ;(display "Scheme+ : interpole_fields : finish-initialization passed") (newline)

        (declare-special-prefix numpy) ; does not seem mandatory, at least for my code

        (import numpy)
        (display "Scheme+ : interpole_fields : import numpy passed") (newline)
        ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

when i insert the above pyffi for racket support i got this error in the httpd log:

generateCmdField  : CubeURL = Data/Dipole3D_rhoe0_6000.vtk
generateCmdField : this->cmd=./drive/interpole_fields /private/var/tmp/VPLASMAetnijevqqel59Gf8dgt Data/Dipole3D_rhoe0_6000.vtk
CoordinateSpecified  : this->warning= Following parameters ignored=OutputFileType SCHEME+PYTHON command= ./drive/interpole_fields /private/var/tmp/VPLASMAetnijevqqel59Gf8dgt Data/Dipole3D_rhoe0_6000.vtk
GeneralReturn : this->url_input=/private/var/tmp/VPLASMAetnijevqqel59Gf8dgt
GeneralReturn : executing command.....
ffi-lib: could not load foreign library
  path: libpython3.10.dylib
  system error: dlopen(libpython3.10.dylib, 0x000A): tried: 'libpython3.10.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache), 'libpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache)
  context...:
   /Applications/Racket/collects/ffi/unsafe.rkt:134:0: get-ffi-lib
   body of "/Users/mattei/pyffi/pyffi-lib/pyffi/libpython.rkt"
GeneralReturn  : message=
GeneralReturn : command terminated.

related pyffi for racket error is:

ffi-lib: could not load foreign library
  path: libpython3.10.dylib
  system error: dlopen(libpython3.10.dylib, 0x000A): tried: 'libpython3.10.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache), 'libpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache)
  context...:
   /Applications/Racket/collects/ffi/unsafe.rkt:134:0: get-ffi-lib
   body of "/Users/mattei/pyffi/pyffi-lib/pyffi/libpython.rkt"

i have ideas to solve that:

• solve the above error by modifying some seach path if it is enough?? i'm not sure at all
• use Python 3.11 and upgrade source code of pyffi for racket if necessary as on the previous system it ran with : : python version= 3.11.2 (v3.11.2:878ead1ac1, Feb 7 2023, 10:02:41) [Clang 13.0.0 (clang-1300.0.29.30)] and : numpy version= 1.26.4 .
• use python 3.12 as i see on my current system it uses 1.26.4 also. So it is same solution as previous one.
• do not use python for this

I do not see what is necessary to change in code to go to 3.11 or 3.12, and going more far to 3.14 then require to modify also the numpy support to use numpy 2.0 ,and it is not necessary and should force to modify more the pyffi for racket source code

[damien_mattei]

about the previous error, it is a bit strange because
i have installed pyffi and started racket from command line from the python 3.10 environment, but if i start in command line without the 3.10 environment it works too with pyffi/numpy , so i do not understand why it fails from web server
dlopen is a C call to dynamic library loader, perheaps just a path fix could solve the problem

[damien_mattei]

it is more and more suprising because , the initial way to install and run my test code that worked is from a python 3.10 virtual environment:

(myvenv3.10) mattei@mac ~ % /Applications/Racket/bin/drracket

but if i force a python 3.12 environment to test if it confuse pyffi for racket:

((my-env) ) mattei@mac ~ % /Applications/Racket/bin/drracket

here my-env is a python 3.12 environment it also works perfect.

I think there is only a path confusing dlopen with the server environment, i tried to see my LD_LIBRARY_PATH but it is not defined in command line when it worked ,what variable should be used?

it seems it has searched in:

@damien_mattei

dlopen(libpython3.10.dylib, 0x000A): tried: 'libpython3.10.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache), 'libpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache)

i should search this way perheaps.

[damien_mattei]

i understand a bit more of the problem:

even if pyffi is installed in 'installation' mode but with a link (i the GUI package manager) to the cloned/fork of pyffi in my user dir the web server which another user _www under /Library/Webserver on mac os has no configuration for pyffi for python 3.10 , so i log in _www and do the configuration for him with an python 3.10 virtual environment:

_www@mac ~ % source myvenv3.10/bin/activate
(myvenv3.10) _www@mac ~ % /Applications/Racket/bin/raco pyffi configure
Configuration tool for `pyffi`.
-------------------------------
This tool attempts to find the shared library `libpython3` 
with the help of the `python3` executable.

The executable
    /Users/mattei/myvenv3.10/bin/python3
will be used to find the location of the shared library.

The previous value of LIBDIR was:
    /opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/lib
The preference for LIBDIR is now set to:
    /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib

The previous value of DATA was:
    /Library/WebServer/my-env
The preference for DATA is now set to:
    /Users/mattei/myvenv3.10
(myvenv3.10) _www@mac ~ % /Applications/Racket/bin/drracket            
2025-12-04 23:21:41.019 DrRacket[11656:102821] +[NSXPCSharedListener endpointForReply:withListenerName:replyErrorCode:]: an error occurred while attempting to obtain endpoint for listener 'ClientCallsAuxiliary': Connection interrupted
Swift/SwiftNativeNSArray.swift:77: Fatal error: Array index out of range
zsh: trace trap  /Applications/Racket/bin/drracket
(myvenv3.10) _www@mac ~ % /Applications/Racket/bin/racket 
Welcome to Racket v8.18 [cs].
> (require pyffi)
(libpython-path /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib/libpython3.10.dylib)

and then pyffi start well with _www user.

But it again fails when from the apache web server, i suppose it fails because for the scientific code the web server use python3.12 and then at some point pyffi....

> (find-system-path 'pref-file)
(libpython-path /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib/libpython3.10.dylib)
#<path:/Library/WebServer/Library/Preferences/org.racket-lang.prefs.rktd>

ok i understand the problem, under Mac OS ,Racket , if i'm with httpd ran by user _www i got in my logs:

[3] => interpole-fields : Racket/Scheme+ (find-system-path 'pref-file): /Users/mattei/Library/Preferences/org.racket-lang.prefs.rktd

that is Racket do not get pref-file from _www home dir but from mattei home dir.

I think it is a bug in Racket because the doc says:

15.2 Filesystem

and pyffi get it by calling:

(get-preference 'pyffi:libdir (λ () #f))

15.2 Filesystem

which said : Extracts a preference value from the file designated by (find-system-path 'pref-file),

Racket should use preference file of the user running the process, which is _www, not me (mattei)

update: and it fails because _www has. no right to read my preference file:

open-input-file: cannot open input file
  path: /Users/mattei/Library/Preferences/org.racket-lang.prefs.rktd
  system error: Permission denied; errno=13
  context...:
   body of "/opt/homebrew/var/www/drive/interpole_fields"

when i test it from the web server application

so it default to #f of (λ () #f)

which then set it like this :

(define libpython-path
  (or (for/first ([name '("libpython3.10" "libpython310" "libpython3")]
                  #:when (file-exists? (build-full-path name)))
        (build-full-path name))
      ;; Github Action (Ubuntu)
      "libpython3.10")) 

from source code of pyffi , it fall back to libpython3.10 which is not a full path and cannot be find by dlopen ....

[damien_mattei]

at some point when running pyffi from a web server running racket on Mac OS i got this:

althought the problem it is still using mattei environment unless _www which is not a big problem i have now this fatal error:

Python path configuration:
  PYTHONHOME = '/Users/mattei/myvenv3.10'
  PYTHONPATH = (not set)
  program name = 'python3.10'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/Users/mattei/myvenv3.10/lib/python3.10'
  sys.base_prefix = '/Users/mattei/myvenv3.10'
  sys.base_exec_prefix = '/Users/mattei/myvenv3.10'
  sys.platlibdir = 'lib'
  sys.executable = '/Users/mattei/myvenv3.10/lib/python3.10'
  sys.prefix = '/Users/mattei/myvenv3.10'
  sys.exec_prefix = '/Users/mattei/myvenv3.10'
  sys.path = [
    '/Users/mattei/myvenv3.10/lib/python310.zip',
    '/Users/mattei/myvenv3.10/lib/python3.10',
    '/Users/mattei/myvenv3.10/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000001fcfee080 (most recent call first):
  <no Python frame>

does this error already rings the bell to anyone? i searched the internet and find it could happen on django server too....

seems to be the same error as this:

3.11 Regression, Py_Initialize; Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding

there is a lot of litterature about it but not a clear solution

[damien_mattei]

trying to upgrade the code of pyffi from 3.10 to 3.12 changing all the occurence of 310 to 312 and 3.10 to 3.12 (which is pretty naive ) i just succeeded to make crash Racket and strangely get the same error as on web server but in terminal:

#lang racket

(require pyffi
         pyffi/numpy
         pyffi/numpy-core)

;(set-environment-variables) ; do not know what it does?
(initialize)
(display "initialize passed") (newline)

never pass initialize:

Python path configuration:
  PYTHONHOME = '/Users/mattei/my-env'
  PYTHONPATH = (not set)
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/Users/mattei/my-env/lib/python3.12/python3.12'
  sys._base_executable = '/Applications/Racket/bin/../DrRacket.app/Contents/MacOS/DrRacket'
  sys.base_prefix = '/Users/mattei/my-env'
  sys.base_exec_prefix = '/Users/mattei/my-env'
  sys.platlibdir = '/Users/mattei/my-env/lib/python3.12'
  sys.executable = '/Applications/Racket/bin/../DrRacket.app/Contents/MacOS/DrRacket'
  sys.prefix = '/Users/mattei/my-env'
  sys.exec_prefix = '/Users/mattei/my-env'
  sys.path = [
    '/Users/mattei/my-env/lib/python3.12/python312.zip',
    '/Users/mattei/my-env/lib/python3.12/python3.12',
    '/Users/mattei/my-env/lib/python3.12/python3.12/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000001fcfee080 (most recent call first):
  <no Python frame>

so i suspect a confict between pyffi (with any version of code ,original or upgraded) for python 3.10 and python 3.12

[damien_mattei]

news about this strange error, i think it fails on the web server because i ran Racket with racket not drracket that is in command line without GUI, so perheaps racket in command line do the things differently, here is the test in command line which fail:

mattei@map-mattei ~ % source myvenv3.10/bin/activate 
(myvenv3.10) mattei@map-mattei ~ % /Applications/Racket/bin/racket
Welcome to Racket v8.18 [cs].
> (require pyffi
               pyffi/numpy
               pyffi/numpy-core)
(libpython-folder /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib)
(libpython-folder /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib)
> (set-environment-variables)
> (initialize)
Python path configuration:
  PYTHONHOME = '/Library/WebServer/myvenv3.10'
  PYTHONPATH = (not set)
  program name = 'python3.10'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/Library/WebServer/myvenv3.10/lib/python3.10'
  sys.base_prefix = '/Library/WebServer/myvenv3.10'
  sys.base_exec_prefix = '/Library/WebServer/myvenv3.10'
  sys.platlibdir = 'lib'
  sys.executable = '/Library/WebServer/myvenv3.10/lib/python3.10'
  sys.prefix = '/Library/WebServer/myvenv3.10'
  sys.exec_prefix = '/Library/WebServer/myvenv3.10'
  sys.path = [
    '/Library/WebServer/myvenv3.10/lib/python310.zip',
    '/Library/WebServer/myvenv3.10/lib/python3.10',
    '/Library/WebServer/myvenv3.10/lib/python3.10/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00000001fcfee080 (most recent call first):
  <no Python frame>

if i run the same code in drracket it runs perfectly , all tests made with or without Python virtual environment activated or no. That is the error is not related to the virtual environment in my opinion.

[damien_mattei]

finally it works ,pyffi for Racket works in the web application,here is the httpd logs:

    [0] => (libpython-folder /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib)
    [1] => 
    [2] => Scheme+ v17.2 by Damien Mattei
    [3] => 
    [4] => (libpython-folder /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib)
    [5] => Scheme+ : interpole_fields : pyffi : initialize passed
    [6] => Scheme+ : interpole_fields : pyffi : import-numpy passed
    [7] => Scheme+ : interpole_fields : pyffi : post-initialize passed
    [8] => Scheme+ : interpole_fields : pyffi : import numpy passed
    [9] => interpole-fields : Racket/Scheme+ : numpy.__version__ : 1.26.0

i even asked to chatGPT,possibly this command was usefull as the

" Gatekeeper (macOS) only blocks the CLI binary

Sometimes macOS treats binaries differently depending on the launch method.
A .dylib that loads fine inside DrRacket may be quarantined for command-line loading." :

xattr -dr com.apple.quarantine /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib/libpython3.10.dylib

but i'm not sure it is mandatory.

For the code i used this strict order :

;;;;;;;;;;;;;;;;;;; numpy with pyffi ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
  ;; (set-environment-variables) ; do not know what it does? make it crash (at least in web server app)!

  (initialize)
  (display "Scheme+ : interpole_fields : pyffi : initialize passed") (newline)

  (import-numpy)
  (display "Scheme+ : interpole_fields : pyffi : import-numpy passed") (newline)

  ;; 2 methods below seems to work
  (post-initialize)
  (display "Scheme+ : interpole_fields : pyffi : post-initialize passed") (newline)

  ;(finish-initialization)
  ;(display "Scheme+ : interpole_fields : pyffi : finish-initialization passed") (newline)

  (declare-special-prefix numpy) ; does not seem mandatory, at least for my code

  (import numpy)
  (display "Scheme+ : interpole_fields : pyffi : import numpy passed") (newline)

  (display "interpole-fields : Racket/Scheme+ : numpy.__version__ : ") (display numpy.__version__) (newline)
  ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

[soegaard]

Hi Damien,

Great to hear you got to work with your web-server.

In the meantime I have worked on updating Pyffi to allow other versions than version 3.10.

The updated docs are here:

pyffi - Use Python from Racket

The idea is as before:

1. Make sure python3 in the terminal invokes the right Python version.
2. Run raco pyffi configure . This stored the Python settings in Racket preferences.
3. Use pyffi from Racket.

I haven't tested extensively, but so far Python version 3.14 seems to work as expected on my machine (macOS).

[damien_mattei]

Hello @soegaard ,

Great, i see your code:

pyffi-lib/pyffi/libpython.rkt

that allow other versions.
I had planned to do something like that today, by defining the version and subversion in 2 variable defined by use, but with your auto configure feature this should works automagically.

I will try to test it ASAP but it's 3 days i'm almost full time on this special problem.

With the web server account and my user account, as Racket persist to get my preferences from my user account i have set the pyffi ... oh it is too long to explain....

It will be simpler now using one single Python version on the host and with all the web app i think , i will test this ASAP, this will simplify the dev. Thank you.

Because i finally last evening not really understood what was the blocking point.But it was now working on my account and web server, so it was just fine.

Yes about this i finally understood

@damien_mattei

(set-environment-variables)

was crashing the app on web server at least. What is doing (set-environment-variables) ? i should read your code about it to understand.

Also it is unclear if the problem of MacOS Quarantine for security have some consequence.

I never understood why my pyffi 3.12 version does not work , oh i checked your code and it seems you just changed the string lib name so finally my idea was good . My 3.12 problem was not related to source code but just an environment or quarantine issue. And i changed just the string as you, it should had worked too...
Of course Mac OS is not the best host for web dev, i should test all under Linux as finally web server app are hosted on linux box... It will take just more time.

Great again you updated your code, i hope to test it today or sunday...

[soegaard]

What is doing (set-environment-variables) ?

I think, it is leftover from an experiment.
It's not used anywhere.

The function is defined like this:

(define (set-environment-variables)
  (define (decode s) (Py_DecodeLocale s #f))
  (Py_SetProgramName (decode "python3.10"))
  (Py_SetPythonHome  (decode home)))

The function Py_SetProgramName sets the name of the executable used to invoke Python. The function has been deprecated since Python version 3.11.

The function Py_SetPythonHome sets PYTHONHOME.
This function has also been deprecated.