Pyffi - Use Python from Racket

Show & Tell
1

Hi All,

For a while I have been hacking on using Python/Python libraries from Racket.
The library is now so advanced, that it makes sense to present it here.

The purpose is simple: make it easy to use Python libraries from Racket.

Libraries written in Python ought to work out of the box. Due to the way Python represent all values as objects, it is possible to use reflection to bridge Python values nicely back and forth between Python and Racket. Most Python extensions (libraries are implemented in C) are well-behaved
and work as if they were native libraries. Some Python extensions miss the reflection
information. If you happen to need such a library there is a low-level interface available.
The low-level interface has an interface similar to the the C FFI.

Currently you can't pass Racket functions to the Python side - but if there is popular demand...

The introduction to Pyffi is a good place to begin

5 An introduction to pyffi

Examples from using Random Python libraries:

pyffi/pyffi-tutorials at main · soegaard/pyffi · GitHub

Enjoy,
Jens Axel

16 Likes
2

Can't wait to have an excuse to try this out!

3

That is a very cool project indeed!

I found recently the Python interface for Gambit Scheme, but it was hard to get it working ... still, one thing that I found interesting there was the interactive call to pip, which installs new packages in a dedicated virtualenv. And you can even call it from REPL, in theory.

Any chance you are considering to add something like this to pyffi? Leveraging the Python standard library and Numpy is good, but being able to install any PyPI package would open many doors.

1 Like
4

Hi logc,

With the pyffi configuration tool you can choose which Python installation to use - virtual or not.
So install Python as normal, then use python3 -m pip install <your-python-package> to install your packages. Finally use raco pyffi configure to configure pyffi.

If you look at

pyffi/pyffi-tutorials at main · soegaard/pyffi · GitHub

you will see an examples using Pygment and Skia.

I don't have any plans of making adding a pip-install, but you can add one yourself.
Gambit's pip-install just calls pip:

python/python.scm at main · gambit/python · GitHub

3 Likes
5

Perhaps I’m missing something obvious but my:

raco pyffi configure

Gives me:

Unrecognized command: pyffi

6

@saludes

The command raco pyffi is available after installation of pyffi.

Now, ideally, you could write raco pkg install pyffi and be done, but I haven't put pyffi on the package server yet [1]. For now, clone the pyffi Github repo

https://github.com/soegaard/pyffi

and use raco pkg update pyffi to install it.

[1] The documentation of pyffi contains examples that invokes the Python, so to build the documentation a Python installation is needed. That is, I need help from the maintainers
of the build server to get the documentation built.

That's why I put the documentation here:

https://soegaard.github.io/pyffi/

7

If I understand it correctly, I should clone the repository and then issue:

raco pkg update pyffi

But I tried it inside the main pyffi directory, the parent directory and ** pyffi/pyffi** always getting:

raco pkg update: package not currently installed
package: pyffi
current scope: user

8

Sorry, not "update" but "install":

raco pkg install pyffi
9

It would be lovely if the garbage collectors of the two systems could interact properly.

10

I think you meant raco pkg install inside the cloned pyffi directory.

Re examples that invokes the Python: can you use make-log-based-eval instead?

2 Likes
11

No luck. it gives:

raco pkg install pyffi
Resolving "pyffi" via Package Catalog
Resolving "pyffi" via https://pkgs.racket-lang.org
Resolving "pyffi" via https://planet-compats.racket-lang.org
raco pkg install: cannot find package on catalogs
package: pyffi

from inside the cloned pyffi directory or from the parent directory.

12

@saludes

Thanks for the patience - I usually don't install from directly from GIthub.

I think, you need a / after pyffi, to indicate the installation is from a folder and not from the package server:

raco pkg install pyffi/

(You might need to enter /, space before hitting enter. I think my shell eats the slash otherwise).

13

It would be lovely if the garbage collectors of the two systems could interact properly.

Python uses reference counting (there is not much to do about that).

When a Python value is allocated from Racket, the Racket side gets a pointer to a C-struct.
This C-struct contains a reference count, which we increment (so the Python garbage collector
doesn't remove values used by Racket). On the Racket side the pointer is associated with a "will executor". When Racket determines that the value is dead, we decrement the Python
reference count. This allows the Python side to reclamin the memory.

2 Likes
14

You might need --link, e.g.:

# inside the cloned pyffi folder
$ raco pkg install --link

(note: no need to specify "pyffi" again as it's inferred from the current directory name)

... although, I just tried that and, while the package did install, I saw some errors:

pyffi-doc/pyffi/scribblings/manual-pyffi.scrbl:9:21: cannot open module file
  module path: pyffi
  path: /Users/siddhartha/work/racket/pyffi/main.rkt
  system error: no such file or directory; rkt_err=3

and

ffi-lib: could not load foreign library
  path: libpython3.10.dylib
  system error: dlopen(libpython3.10.dylib, 10): image not found
  context...:
   /Applications/Racket-Latest/collects/ffi/unsafe.rkt:131:0: get-ffi-lib
   body of "/Users/siddhartha/work/racket/pyffi/pyffi-lib/pyffi/libpython.rkt"

...

raco pkg install: packages installed, although setup reported errors

I haven't followed this thread in detail so apologies if I've missed something above.

15

Nevermind, I see this in the docs:

In order to use pyffi you need Python 3.10.

And mine is:

$ python --version
Python 3.9.9

So presumably, the command in my previous comment would work if you have python 3.10 installed.

16

I’m using Python 3.10 and I’m getting the same error

17

First, what's the output of:

python3 --version

Second, what's the output of:

raco pyffi show

And what do you get when you run:

raco pyffi configure

Btw - It might me faster to use the Racket Discord while we figure out, what's wrong.

For comparison, here is my output on macOS:

soegaard@mbp2 stxmat % raco pyffi show
Current configuration for 'pyffi'.

    libdir = "/Library/Frameworks/Python.framework/Versions/3.10/lib"
    data   = "/Library/Frameworks/Python.framework/Versions/3.10"

Meaning:

    libdir:  location of the shared library 'libpython'
    data:    location of bin/ lib/ share/ etc.
18

python --version

Python 3.10.6

raco pyffi show

open-input-file: cannot open module file
module path: pyffi/configure-pyffi
path: /Users/saludes/pyffi/configure-pyffi.rkt
system error: no such file or directory; rkt_err=3
context...:
/Applications/Racket/collects/raco/raco.rkt:41:0
body of "/Applications/Racket/collects/raco/raco.rkt"
body of "/Applications/Racket/collects/raco/main.rkt"

raco pyffi configure

open-input-file: cannot open module file
module path: pyffi/configure-pyffi
path: /Users/saludes/pyffi/configure-pyffi.rkt
system error: no such file or directory; rkt_err=3
context...:
/Applications/Racket/collects/raco/raco.rkt:41:0
body of "/Applications/Racket/collects/raco/raco.rkt"
body of "/Applications/Racket/collects/raco/main.rkt

19

We experimented a little on Discord.

Some notes:

  1. Since the documentation doesn't built correctly until after pyffi has been configured,
    it's a good idea to use --no-docs when pyffi is installed.

  2. The command python3 -m sysconfig can be used to check whether the correct Python installation is used.

  3. The command raco pyffi diagonistics shows all paths that pyffi gets from Python.

  4. Getting pyffi on the package server would make it much easier to install.

1 Like
20

this!

  • Signed, John Brinckerhoff Name of 20 characters Clements
1 Like