If you're looking for a way to tokenize python using its canonical tokenizer implementation, but wish to have the results in racket, this library may be useful.
Source code here GitHub - abhillman/py-tokenizer-rkt: Python tokenizer that delegates to libpython written in Racket.
I had a lot of fun writing it as pyffi is quite nice. it could allow for micro-interpreters to be written for Python or Python-like languages like bazel in Racket.
Future work may include:
- supporting versions of python other than 3.10
- implementing parsing support
- writing a mini-interpreter, perhaps alternatively for
starlark(the subset of Python implemented forbazel)
Related:
- the announcement of
pyffiPyffi - Use Python from Racket - the
python-tokenizerlibrary which is written in pure racket, albeit last updated in 2017