# Python Tokenizer with \`libpython\` via \`pyffi\`

**URL:** <https://racket.discourse.group/t/python-tokenizer-with-libpython-via-pyffi/2493>\
**Category:** Show & Tell\
**Created:** [November 11, 2023, 6:45am UTC](https://racket.discourse.group/t/python-tokenizer-with-libpython-via-pyffi/2493 "2023-11-11T06:45:30Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![abhillman](https://yyz2.discourse-cdn.com/free1/user_avatar/racket.discourse.group/abhillman/32/1516_2.png) [@abhillman](https://racket.discourse.group/u/abhillman)\
**Post date:** [November 11, 2023, 6:45am UTC](https://racket.discourse.group/t/python-tokenizer-with-libpython-via-pyffi/2493/1 "2023-11-11T06:45:30Z")

</div>

If you're looking for a way to tokenize python using its canonical tokenizer implementation, but wish to have the results in racket, this library may be useful.

Source code here [GitHub - abhillman/py-tokenizer-rkt: Python tokenizer that delegates to libpython written in Racket](https://github.com/abhillman/py-tokenizer-rkt/tree/main).

I had a lot of fun writing it as [`pyffi`](https://soegaard.github.io/pyffi/) is quite nice. it could allow for micro-interpreters to be written for Python or Python-like languages like `bazel` in Racket.

Future work may include:

- supporting versions of python other than 3.10
- implementing parsing support
- writing a mini-interpreter, perhaps alternatively for `starlark` (the subset of Python implemented for `bazel`)

Related:

- the announcement of `pyffi` [Pyffi - Use Python from Racket](https://racket.discourse.group/t/pyffi-use-python-from-racket/1362)
- the [`python-tokenizer`](https://pkgs.racket-lang.org/package/python-tokenizer) library which is written in pure racket, albeit last updated in 2017