Pyffi with numpy (1.26) low-level lib and Python 3.10

[damien_mattei]

Hi,

I try to use pyffi but i have a Mac with Python 3.9 by default from Apple and Python 3.12 for work for depandancies with some scientific libraries. pyffi is searching for version 3.10 so i did:

/Applications/Racket/bin/raco pyffi configure /opt/homebrew/bin/python3.12
Configuration tool for `pyffi`.
-------------------------------
This tool attempts to find the shared library `libpython3` 
with the help of the `python3` executable.

The executable
    /opt/homebrew/bin/python3.12
will be used to find the location of the shared library.

The preference for LIBDIR is now set to:
    /opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/lib

The preference for DATA is now set to:
    /opt/homebrew

But unfornunately pyffi still search for the 3.10 version:

(require pyffi)
. . ../../Applications/Racket v8.18/collects/ffi/unsafe.rkt:134:0: ffi-lib: could not load foreign library
  path: libpython3.10.dylib
  system error: dlopen(libpython3.10.dylib, 0x000A): tried: 'libpython3.10.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache), 'libpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache)
> 

I suppose there is nothing else to do but install also python 3.10 on my system?

[damien_mattei]

just installing python 3.10 solve the problem but would it be hard to make pyffi working for different versions of python because as-is this really restrict the use of this good (even if not yet tested) ffi library?

[soegaard]

would it be hard to make pyffi working for different versions of python

Probably not.

Try it!

Note: Look up if there are have been changes to the C-API:

Python/C API Reference Manual

I believe, they are rather conservative wrt making changes,
so I expect minor changes at the most.

[damien_mattei]

i was asking myself if it could just be enough to change the string

@damien_mattei

libpython3.10.dylib

in the source code?

[soegaard]

I think so - at least if the C-api hasn't changed.

[damien_mattei]

i have installed Numpy with pip3.10 after the pyffi install but then i got this error:

#lang racket

(require pyffi
         pyffi/numpy)

(initialize)
(post-initialize)
;(run "1+2")
(import sys)
(run "sys.version")

Welcome to DrRacket, version 8.18 [cs].
Language: racket, with debugging; memory limit: 128 MB.
. . get: not bound: yy numpy in numpy.array

this seems only related with pyffi/numpy

update:

i even started drracket an dreinstall pyffi from a python virtual environment of 3.10 version:

(myvenv3.10) mattei@mac ~ % /Applications/Racket/bin/raco pyffi show                                      
Current configuration for 'pyffi'.

    libdir = "/opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib"
    data   = "/Users/mattei/myvenv3.10"

Meaning:

    libdir:  location of the shared library 'libpython'
    data:    location of bin/ lib/ share/ etc.

and about numpy:

(myvenv3.10) mattei@mac ~ % ls -la myvenv3.10/lib/python3.10/site-packages/
total 8
drwxr-xr-x   5 mattei  staff   160  2 déc.  14:24 _distutils_hack
drwxr-xr-x  11 mattei  staff   352  2 déc.  14:48 .
drwxr-xr-x   3 mattei  staff    96  2 déc.  14:23 ..
-rw-r--r--   1 mattei  staff   151  2 déc.  14:24 distutils-precedence.pth
drwxr-xr-x  55 mattei  staff  1760  2 déc.  14:48 numpy
drwxr-xr-x   9 mattei  staff   288  2 déc.  14:48 numpy-2.2.6.dist-info
drwxr-xr-x   9 mattei  staff   288  2 déc.  14:24 pip
drwxr-xr-x   9 mattei  staff   288  2 déc.  14:24 pip-25.3.dist-info
drwxr-xr-x   7 mattei  staff   224  2 déc.  14:24 pkg_resources
drwxr-xr-x  52 mattei  staff  1664  2 déc.  14:24 setuptools
drwxr-xr-x  10 mattei  staff   320  2 déc.  14:24 setuptools-80.9.0.dist-info

if i start python i can import numpy but not with Racket

[damien_mattei]

(post-initialize)
Fatal Python error: pyinit_main_reconfigure: fail to reconfigure Python
Python runtime state: initialized
Traceback (most recent call last):
  File "/Users/mattei/myvenv3.10/lib/python3.10/site-packages/numpy/__init__.py", line 400, in __getattr__
    raise AttributeError(
AttributeError: `np.add_newdoc` was removed in the NumPy 2.0 release. It's still available as `np.lib.add_newdoc`.

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator (total: 11)

problem seems to come from the numpy version used at year 2022, because each time i change version i got a different error of unbound numerical type

with numpy 1.23:

get: not bound: yy float128 in numpy.float128

ok, finally testing 3 years of numpy release i get the 1.26 working with this code:

#lang racket

(require pyffi
         pyffi/numpy
         pyffi/numpy-core)

(initialize)
(display "initialize passed") (newline)
;(initialize-numpy)           ; load the `numpy` module
;(display "initialize passed") (newline)
(import-numpy)
(display "import-numpy passed") (newline)
;(post-initialize)


(finish-initialization)
(display "finish-initialization passed") (newline)
(declare-special-prefix numpy)

;(run "1+2")
(import sys)
;(run "sys.version")
sys.version_info

;numpy.__version__
numpy.dot

Welcome to DrRacket, version 8.18 [cs].
Language: racket, with debugging; memory limit: 128 MB.
initialize passed
import-numpy passed
finish-initialization passed
(obj "version_info" : sys.version_info(major=3, minor=10, micro=19, releaselevel='final', serial=0))
#<procedure:dot>

but it seems there is only a subset of numpy, how can i add new API in the code,
i need for example :

numpy.lib.format.read_magic

read_array_header_1_0 or read_array_header_2_0

seems to be in numpy-core.rkt, i'm interested in modifying this code, but what is the way to do it?

[damien_mattei]

Done

after a clone/fork and a bit of reverse engineering i conclude there was no need of modifying the source code. Just the way to use numpy in racket is important because there is some double methods for importing (import and import-numpy) and for initializing ( post-initialize and finish-initialization) that sometimes do the same sometimes i have no idea what they are for. Reading the doc and testing the example was indeed helpful.

here is a working solution of this problem , that was reading just the numpy header of .npy file very big (21Gb), a 3D cube of magnetic field, read the header to extract the math type , is it 1D,3D, scalar,vector? and the numeric type, float64? or other.....

ls -la /opt/homebrew/var/www/Data/BplusBdip64array_it0_rank.npy
-rw-r-xr-x  1 mattei  admin  24192000128 10 nov.  11:40 /opt/homebrew/var/www/Data/BplusBdip64array_it0_rank.npy

a scheme solution:

#lang racket

(require pyffi
         pyffi/numpy
         pyffi/numpy-core)

;(set-environment-variables) ; do not know what it does?
(initialize)
(display "initialize passed") (newline)

;(display "initialize passed") (newline)
(import-numpy)
(display "import-numpy passed") (newline)

;; 2 methods below seems to work
(post-initialize)
(display "post-initialize passed") (newline)
;(finish-initialization)
;(display "finish-initialization passed") (newline)

(declare-special-prefix numpy)

;(run "1+2")
(import sys)
;(run "sys.version")
;sys.version_info
(import numpy)

;; does not works at any place
;;(initialize-numpy)           ; load the `numpy` module


;numpy.__version__
;numpy.dot
(define name "/opt/homebrew/var/www/Data/BplusBdip64array_it0_rank.npy")
(define fobj (builtins.open name "rb"))
(define version-file (numpy.lib.format.read_magic fobj)) ;  determine the format version of the file
(define version-file-vect (pytuple->vector version-file))
(define func_name (apply string-append (cons "read_array_header" (vector->list (vector-map (λ (n) (string-append "_" (number->string n)))
                                                                                       version-file-vect)))))
(define func (getattr numpy.lib.format func_name)) ; returns the shape of the array, whether or not it is F-contiguous, and the dtype
(define Varray-header (pytuple->vector (func fobj)))
(define dim-obj (pytuple->vector (vector-ref Varray-header 0)))
dim-obj
(define data-type (vector-ref Varray-header 2))
(define type-obj data-type.name)
type-obj

and a scheme+ solution:

#lang reader SRFI-110 ; 105
(require (rename-in Scheme+
                    (:= :=s)
                    (-> s->)))
(require pyffi
         pyffi/numpy
         pyffi/numpy-core)

;(set-environment-variables) ; do not know what it does?

(initialize)
(display "initialize passed") (newline)

;(import-numpy) ; does not seem mandatory
;(display "import-numpy passed") (newline)


;; 2 methods below seems to work
(post-initialize)
(display "post-initialize passed") (newline)

;(finish-initialization)
;(display "finish-initialization passed") (newline)


(declare-special-prefix numpy) ; does not seem mandatory, at least for my code

;(run "1+2")
(import sys)
;(run "sys.version")
;sys.version_info

(import numpy)
(display "import numpy passed") (newline)

;; does not works at any place
;;(initialize-numpy)           ; load the `numpy` module

;numpy.__version__
;numpy.dot

{name <- "/opt/homebrew/var/www/Data/BplusBdip64array_it0_rank.npy"}
{fobj <- (builtins.open name "rb")}
{version-file <- (numpy.lib.format.read_magic fobj)} ;  determine the format version of the file
{version-file-vect <- (pytuple->vector version-file)}
{func_name <- (apply string-append (cons "read_array_header" (vector->list (vector-map (λ (n) (string-append "_" (number->string n)))
                                                                                       version-file-vect))))}
{func <- (getattr numpy.lib.format func_name)} ; returns the shape of the array, whether or not it is F-contiguous, and the dtype
{Varray-header <- (pytuple->vector (func fobj))}
{dim-obj <- pytuple->vector(Varray-header[0])}
dim-obj
{data-type <- Varray-header[2]}
{type-obj <- data-type.name}
type-obj

they both output the dimension and type:

Welcome to DrRacket, version 8.18 [cs].
Language: Determine language from source; memory limit: 128 MB.
SRFI 110 Curly Infix,Neoteric and Sweet expressions v1.2 for Scheme+
SRFI-110 Curly Infix parser for Racket Scheme and R6RS by Damien MATTEI
(based on code from David A. Wheeler and Alan Manuel K. Gloria.)

Possibly skipping some header's lines containing space,tabs,new line,etc  or comments.

SRFI-110 Curly Infix reader : number of skipped lines (comments, spaces, directives,...) at header's beginning : 2

Parsed curly infix code result = 
(require (rename-in Scheme+ (:= :=s) (-> s->)))
(require pyffi pyffi/numpy pyffi/numpy-core)
(initialize)
(display "initialize passed")
(newline)
(post-initialize)
(display "post-initialize passed")
(newline)
(declare-special-prefix numpy)
(import sys)
(import numpy)
(display "import numpy passed")
(newline)
(<- name "/opt/homebrew/var/www/Data/BplusBdip64array_it0_rank.npy")
(<- fobj (builtins.open name "rb"))
(<- version-file (numpy.lib.format.read_magic fobj))
(<- version-file-vect (pytuple->vector version-file))
(<-
 func_name
 (apply
  string-append
  (cons
   "read_array_header"
   (vector->list
    (vector-map
     (λ (n) (string-append "_" (number->string n)))
     version-file-vect)))))
(<- func (getattr numpy.lib.format func_name))
(<- Varray-header (pytuple->vector (func fobj)))
(<- dim-obj (pytuple->vector ($bracket-apply$ Varray-header 0)))
dim-obj
(<- data-type ($bracket-apply$ Varray-header 2))
(<- type-obj data-type.name)
type-obj

Scheme+ v17.2 by Damien Mattei

initialize passed
post-initialize passed
import numpy passed
'#(3 700 1200 1200)
"float64"
> 

this could not be done simply in scheme as it requires to use low-level numpy API to just read the .npy numpy file.

Note: this should be not critical to use only python 3.10 as other part of the data treatment sequence which is sequential can use other version of Python and numpy,if the multiple Python,Numpy versions are correctly managed on the system.

[soegaard]

Great to hear you succeeded in getting Pyffi up and running.

The release notes for Numpy says the oldest supported Python version is 3.11.
So it seems it is about time to update Pyffi to handle something newer than 3.10.

Are you interested in updating Pyffi to use Python 3.14?

The funny looking (initialize) and (post-initialize) is explained here:

6 Initialization of the Python Interpreter

Basically the issue is that initialize is run after any required Racket modules are instantiated. This means that a Racket module for numpy doesn't have an Python interpreter instance available. Instead it uses add-initialization-thunk to delay anything that needs an instance. When initialize has created an instance, then post-initialize can invoke any initialization thunks.

[damien_mattei]

@soegaard

The release notes for Numpy says the oldest supported Python version is 3.11.
So it seems it is about time to update Pyffi to handle something newer than 3.10.

yes, i used python 3.11 on the previous system, now 3.12, going back to 3.11 seems feasible, i say that about error (see below) , even if it exists 3.13 and 3.14 , the scientific library (can not remember exactly because there is a lot, matplotlib,scipy,numpy of course) i use only deal with at max python 3.12
But 3.11 is still ok.

@soegaard

Are you interested in updating Pyffi to use Python 3.14?

going so far is not necessary, but i was wrong when thinking i could melt multiple python version on the whole application:

The application run on a web server,http->PHP->Racket/scheme->Python/Numpy/Matpplotlib/Scipy

i set up the web server account to use the 3.11 or 3.12 python virtual environment, for this reason, pyffi for racket now fails to start on the web server code:

(module interpolate-field-caller racket
	
	(require (rename-in Scheme+
                            (:= :=s) ; for compatibility with pyffi
                            (-> s->)))


        (require pyffi
                 pyffi/numpy
                 pyffi/numpy-core)

.....


;;;;;;;;;;;;;;;;;;; numpy with pyffi ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
        ;(set-environment-variables) ; do not know what it does?

        (initialize)
        (display "Scheme+ : interpole_fields : initialize passed") (newline)

        ;(import-numpy) ; does not seem mandatory
        ;(display "Scheme+ : interpole_fields : import-numpy passed") (newline)

        ;; 2 methods below seems to work
        (post-initialize)
        (display "Scheme+ : interpole_fields : post-initialize passed") (newline)

        ;(finish-initialization)
        ;(display "Scheme+ : interpole_fields : finish-initialization passed") (newline)

        (declare-special-prefix numpy) ; does not seem mandatory, at least for my code

        (import numpy)
        (display "Scheme+ : interpole_fields : import numpy passed") (newline)
        ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

when i insert the above pyffi for racket support i got this error in the httpd log:

generateCmdField  : CubeURL = Data/Dipole3D_rhoe0_6000.vtk
generateCmdField : this->cmd=./drive/interpole_fields /private/var/tmp/VPLASMAetnijevqqel59Gf8dgt Data/Dipole3D_rhoe0_6000.vtk
CoordinateSpecified  : this->warning= Following parameters ignored=OutputFileType SCHEME+PYTHON command= ./drive/interpole_fields /private/var/tmp/VPLASMAetnijevqqel59Gf8dgt Data/Dipole3D_rhoe0_6000.vtk
GeneralReturn : this->url_input=/private/var/tmp/VPLASMAetnijevqqel59Gf8dgt
GeneralReturn : executing command.....
ffi-lib: could not load foreign library
  path: libpython3.10.dylib
  system error: dlopen(libpython3.10.dylib, 0x000A): tried: 'libpython3.10.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache), 'libpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache)
  context...:
   /Applications/Racket/collects/ffi/unsafe.rkt:134:0: get-ffi-lib
   body of "/Users/mattei/pyffi/pyffi-lib/pyffi/libpython.rkt"
GeneralReturn  : message=
GeneralReturn : command terminated.

related pyffi for racket error is:

ffi-lib: could not load foreign library
  path: libpython3.10.dylib
  system error: dlopen(libpython3.10.dylib, 0x000A): tried: 'libpython3.10.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache), 'libpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache)
  context...:
   /Applications/Racket/collects/ffi/unsafe.rkt:134:0: get-ffi-lib
   body of "/Users/mattei/pyffi/pyffi-lib/pyffi/libpython.rkt"

i have ideas to solve that:

• solve the above error by modifying some seach path if it is enough?? i'm not sure at all
• use Python 3.11 and upgrade source code of pyffi for racket if necessary as on the previous system it ran with : : python version= 3.11.2 (v3.11.2:878ead1ac1, Feb 7 2023, 10:02:41) [Clang 13.0.0 (clang-1300.0.29.30)] and : numpy version= 1.26.4 .
• use python 3.12 as i see on my current system it uses 1.26.4 also. So it is same solution as previous one.
• do not use python for this

I do not see what is necessary to change in code to go to 3.11 or 3.12, and going more far to 3.14 then require to modify also the numpy support to use numpy 2.0 ,and it is not necessary and should force to modify more the pyffi for racket source code

[damien_mattei]

about the previous error, it is a bit strange because
i have installed pyffi and started racket from command line from the python 3.10 environment, but if i start in command line without the 3.10 environment it works too with pyffi/numpy , so i do not understand why it fails from web server
dlopen is a C call to dynamic library loader, perheaps just a path fix could solve the problem

[damien_mattei]

it is more and more suprising because , the initial way to install and run my test code that worked is from a python 3.10 virtual environment:

(myvenv3.10) mattei@mac ~ % /Applications/Racket/bin/drracket

but if i force a python 3.12 environment to test if it confuse pyffi for racket:

((my-env) ) mattei@mac ~ % /Applications/Racket/bin/drracket

here my-env is a python 3.12 environment it also works perfect.

I think there is only a path confusing dlopen with the server environment, i tried to see my LD_LIBRARY_PATH but it is not defined in command line when it worked ,what variable should be used?

it seems it has searched in:

@damien_mattei

dlopen(libpython3.10.dylib, 0x000A): tried: 'libpython3.10.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OSlibpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache), 'libpython3.10.dylib' (no such file), '/usr/lib/libpython3.10.dylib' (no such file, not in dyld cache)

i should search this way perheaps.

[damien_mattei]

i understand a bit more of the problem:

even if pyffi is installed in 'installation' mode but with a link (i the GUI package manager) to the cloned/fork of pyffi in my user dir the web server which another user _www under /Library/Webserver on mac os has no configuration for pyffi for python 3.10 , so i log in _www and do the configuration for him with an python 3.10 virtual environment:

_www@mac ~ % source myvenv3.10/bin/activate
(myvenv3.10) _www@mac ~ % /Applications/Racket/bin/raco pyffi configure
Configuration tool for `pyffi`.
-------------------------------
This tool attempts to find the shared library `libpython3` 
with the help of the `python3` executable.

The executable
    /Users/mattei/myvenv3.10/bin/python3
will be used to find the location of the shared library.

The previous value of LIBDIR was:
    /opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12/lib
The preference for LIBDIR is now set to:
    /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib

The previous value of DATA was:
    /Library/WebServer/my-env
The preference for DATA is now set to:
    /Users/mattei/myvenv3.10
(myvenv3.10) _www@mac ~ % /Applications/Racket/bin/drracket            
2025-12-04 23:21:41.019 DrRacket[11656:102821] +[NSXPCSharedListener endpointForReply:withListenerName:replyErrorCode:]: an error occurred while attempting to obtain endpoint for listener 'ClientCallsAuxiliary': Connection interrupted
Swift/SwiftNativeNSArray.swift:77: Fatal error: Array index out of range
zsh: trace trap  /Applications/Racket/bin/drracket
(myvenv3.10) _www@mac ~ % /Applications/Racket/bin/racket 
Welcome to Racket v8.18 [cs].
> (require pyffi)
(libpython-path /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib/libpython3.10.dylib)

and then pyffi start well with _www user.

But it again fails when from the apache web server, i suppose it fails because for the scientific code the web server use python3.12 and then at some point pyffi....

> (find-system-path 'pref-file)
(libpython-path /opt/homebrew/opt/python@3.10/Frameworks/Python.framework/Versions/3.10/lib/libpython3.10.dylib)
#<path:/Library/WebServer/Library/Preferences/org.racket-lang.prefs.rktd>

ok i understand the problem, under Mac OS ,Racket , if i'm with httpd ran by user _www i got in my logs:

[3] => interpole-fields : Racket/Scheme+ (find-system-path 'pref-file): /Users/mattei/Library/Preferences/org.racket-lang.prefs.rktd

that is Racket do not get pref-file from _www home dir but from mattei home dir.

I think it is a bug in Racket because the doc says:

15.2 Filesystem

and pyffi get it by calling:

(get-preference 'pyffi:libdir (λ () #f))

15.2 Filesystem

which said : Extracts a preference value from the file designated by (find-system-path 'pref-file),

Racket should use preference file of the user running the process, which is _www, not me (mattei)

update: and it fails because _www has. no right to read my preference file:

open-input-file: cannot open input file
  path: /Users/mattei/Library/Preferences/org.racket-lang.prefs.rktd
  system error: Permission denied; errno=13
  context...:
   body of "/opt/homebrew/var/www/drive/interpole_fields"

when i test it from the web server application

so it default to #f of (λ () #f)

which then set it like this :

(define libpython-path
  (or (for/first ([name '("libpython3.10" "libpython310" "libpython3")]
                  #:when (file-exists? (build-full-path name)))
        (build-full-path name))
      ;; Github Action (Ubuntu)
      "libpython3.10")) 

from source code of pyffi , it fall back to libpython3.10 which is not a full path and cannot be find by dlopen ....