pythonrs 0.1.12

Python as a fusevm frontend: lexer, parser, and compiler to fusevm::Chunk on a PyHost heap — the first compiled standalone Python runtime
Documentation
```
██████╗ ██╗   ██╗████████╗██╗  ██╗ ██████╗ ███╗   ██╗██████╗ ███████╗
██╔══██╗╚██╗ ██╔╝╚══██╔══╝██║  ██║██╔═══██╗████╗  ██║██╔══██╗██╔════╝
██████╔╝ ╚████╔╝    ██║   ███████║██║   ██║██╔██╗ ██║██████╔╝███████╗
██╔═══╝   ╚██╔╝     ██║   ██╔══██║██║   ██║██║╚██╗██║██╔══██╗╚════██║
██║        ██║      ██║   ██║  ██║╚██████╔╝██║ ╚████║██║  ██║███████║
╚═╝        ╚═╝      ╚═╝   ╚═╝  ╚═╝ ╚═════╝ ╚═╝  ╚═══╝╚═╝  ╚═╝╚══════╝
```

[![CI](https://github.com/MenkeTechnologies/pythonrs/actions/workflows/ci.yml/badge.svg)](https://github.com/MenkeTechnologies/pythonrs/actions/workflows/ci.yml)
![Rust](https://img.shields.io/badge/Rust-2021-05d9e8?style=flat-square)
[![Docs](https://img.shields.io/badge/docs-online-blue.svg)](https://menketechnologies.github.io/pythonrs/)
![license](https://img.shields.io/badge/license-MIT-ff2a6d?style=flat-square)
![status](https://img.shields.io/badge/status-active%20%C2%B7%20in%20development-9b5de5?style=flat-square)

### `[PYTHON, COMPILED TO BYTECODE — rkyv-CACHED ON EVERY RUN, AOT-NATIVE]`

> *"CPython compiles to its own bytecode and walks it. pythonrs lowers Python to a shared machine, caches the result on every run, and can bake a script into a native binary."*

**pythonrs** is Python as a [`fusevm`](https://github.com/MenkeTechnologies/fusevm)
frontend — a lexer/parser and compiler that lowers Python 3 to `fusevm::Chunk`
bytecode running on the fusevm three-tier Cranelift JIT, over a `PyHost` object
heap. There is no bespoke VM and no bespoke JIT: pythonrs is a pure front end;
execution and codegen live in `fusevm` — the same engine behind
[`zshrs`](https://github.com/MenkeTechnologies/zshrs),
[`strykelang`](https://github.com/MenkeTechnologies/strykelang),
[`awkrs`](https://github.com/MenkeTechnologies/awkrs),
[`vimlrs`](https://github.com/MenkeTechnologies/vimlrs),
[`elisprs`](https://github.com/MenkeTechnologies/elisprs), and
[`rubylang`](https://github.com/MenkeTechnologies/rubylang).

It is the first compiled standalone Python runtime that both
**transparently caches bytecode via rkyv on every run** and **AOT-compiles a
script to a native executable**.

### [`Read the Docs`]https://menketechnologies.github.io/pythonrs/ · [`Engineering Report`]https://menketechnologies.github.io/pythonrs/report.html · [`Builtin Reference`]https://menketechnologies.github.io/pythonrs/reference.html

---

## Table of Contents

- [\[0x00\] Overview]#0x00-overview
- [\[0x01\] Install]#0x01-install
- [\[0x02\] Usage]#0x02-usage
- [\[0x03\] Language Features]#0x03-language-features
- [\[0x04\] Command-Line Flags]#0x04-command-line-flags
- [\[0x05\] Architecture]#0x05-architecture
- [\[0x06\] Parity Harness]#0x06-parity-harness
- [\[0x07\] Status & Roadmap]#0x07-status--roadmap
- [\[0x08\] Documentation]#0x08-documentation
- [\[0xFF\] License]#0xff-license

---

## [0x00] OVERVIEW

pythonrs keeps Python the language and throws away CPython's execution model. It
lexes and parses Python to an AST, lowers the AST to `fusevm` bytecode, and runs
the bytecode on a stack VM with a Cranelift JIT. Arithmetic and comparisons lower
to native ops; Python-specific behavior — truthiness, `str`/`list` concat, bignum
promotion, exact `int`-against-`float` comparison, attribute and method dispatch —
runs through a strict numeric hook and a numbered builtin-call protocol into the
`PyHost` object heap.

Two things set it apart from every other standalone Python:

- **Transparent rkyv bytecode cache — on every run.** `python foo.py` hashes the
  source, consults `~/.pythonrs/scripts.rkyv`, and on a hit runs the compiled
  chunks directly with lex/parse/lower skipped entirely. No flags, no separate
  build step, no `__pycache__` ritual. The shard is an archived INDEX followed by
  a raw blob region, and a lookup validates only the index: what the cache costs
  a run is set by how many programs it holds, never by how large they are.
- **AOT to a native executable.** `python --build foo.py` emits a standalone
  native binary (via `fusevm::aot`, linked against the pythonrs runtime
  staticlib) that runs the script with no interpreter present. This path needs
  the libpython-free build (`cargo build --no-default-features`).

## [0x01] INSTALL

```sh
# Via the Homebrew tap (bumped by each release; formula is `pythonrs`)
brew install menketechnologies/menketech/pythonrs

# From crates.io — the default build links libpython, so it needs a CPython
# >= 3.9 on the machine; `--no-default-features` builds the CPython-free one
cargo install pythonrs
cargo install pythonrs --no-default-features

# Or from source
git clone https://github.com/MenkeTechnologies/pythonrs
cd pythonrs && cargo build --release
# binary: target/release/python  (+ libpythonrs.a for AOT linking)
```

The default build links an embedded libpython (the `stdlib-ffi` bridge), so it
needs CPython **3.9 or newer** present at build time — the `abi3-py39` feature
sets that floor, and the stable ABI keeps the resulting binary usable against any
newer CPython at run time. pyo3 finds the interpreter by looking up `python3` on
`PATH`; point it elsewhere with

```sh
PYO3_PYTHON=$(command -v python3.14) cargo build
```

which is also the escape hatch when `python3` on `PATH` is pythonrs itself —
pyo3 rejects it with "no Python 3.x interpreter found".

The built binary links that interpreter's libpython, and finding its standard
library at RUN time needs `PYTHONHOME` set to the matching prefix (see
[FFI_STDLIB.md](FFI_STDLIB.md)); without it `import os` raises `ModuleNotFoundError`
and `sys.path` comes back nearly empty:

```sh
PYTHONHOME=$(python3.14 -c 'import sys; print(sys.prefix)') ./target/debug/python script.py
```

`cargo build --no-default-features` drops pyo3/libpython entirely and serves
`import` from the vendored `pylib/` tree, with no version floor and no
`PYTHONHOME` to set.

#### Self-contained install (macOS)

```sh
scripts/install.sh --release
```

Installs into `~/.pythonrs` — the binary, the CPython runtime, and every
transitive dylib the C extensions touch — with all load commands rewritten to
`@rpath` and re-signed, so nothing under `/opt/homebrew` is referenced and
`brew uninstall python` leaves pythonrs working. Put `~/.pythonrs/bin` on `PATH`
(or **symlink** `bin/python`; a bare `cp` breaks the `@executable_path` rpath).

#### Zsh tab completion

```sh
cp completions/_python "${fpath[1]}/"
# or: fpath=(/path/to/pythonrs/completions $fpath) in .zshrc
```

## [0x02] USAGE

```sh
python foo.py               # run a script (transparently rkyv-cached)
python -c 'print(1 + 1)'    # run a one-liner
python --build foo.py       # AOT-compile to a native ./foo executable
python --dump-bytecode f.py # print the lowered fusevm bytecode
python --tiers f.py         # run it, then report which fusevm tiers took it
python --repl               # interactive REPL
python --lsp                # Language Server Protocol over stdio
python --doctor             # runtime / CPython / cache / env diagnostic report
python --cacheview          # list the compiled programs in the bytecode cache
python --cache-clear        # delete the bytecode cache shard
```

The REPL is a `reedline` line editor: **Tab** pops a columnar completion menu
(Shift+Tab / BackTab cycles backward). On a bare word it offers the language
keywords, builtins, `math.*`, per-type method names, and the live module
globals / class names of the persistent session. After a `name.` it switches to
**type-aware attribute completion** — it reads the receiver's live runtime type
and offers exactly that surface: `str`/`list`/`dict`/`set`/`tuple`/`int`/`float`
methods for a builtin value, an imported module's own namespace, or an instance's
attributes plus every method reachable along its class MRO. History persists to
`~/.pythonrs/history`.

Set `PYTHONRS_TRACE=1` to log cache hit/miss to stderr (silent otherwise). Set
`PYTHONRS_CACHE=0` (or `false`/`no`) to disable the transparent bytecode cache
entirely — every run recompiles and nothing is stored. `PYTHONRS_STDLIB` overrides
the embedded-CPython stdlib prefix (checked before the bundled and system
locations); `PYTHONRS_LIB` overrides the vendored `pylib/` search path used by a
`--no-default-features` build.

`PYTHONHASHSEED` is honoured exactly as CPython honours it. A pinned seed
installs the same `_Py_HashSecret` CPython derives (`lcg_urandom` over the seed;
`0` zeroes it), so `hash('abc')` is byte-identical to `PYTHONHASHSEED=N python3`
for every `N` in `[0, 4294967295]` — not just for `0`. Unset or `random` draws
per-process entropy, as CPython does, and a value CPython rejects is rejected
here with the same message and exit code.

`PYTHONINTMAXSTRDIGITS` seeds `sys.get_int_max_str_digits()` the same way:
`0` or at least 640, anything else refused at startup with CPython's fatal
message. The limit bounds every base-ten int conversion — `int(str)`, `str`/
`repr`, `%d`, decimal format specs, and decimal literals (a `SyntaxError`) —
and `sys.set_int_max_str_digits()` changes it at run time; hex, octal and
binary are never limited.

## [0x03] LANGUAGE FEATURES

Arbitrary-precision integers, real closures, classes with inheritance, operator
dunders, generators (`yield` / `yield from` / lazy generator expressions, backed
by stackful `corosensei` coroutines), `match`/`case` structural pattern matching,
own-scope comprehensions (list/dict/set) and proper `nonlocal`, f-strings,
exceptions, and full call-site and literal `*` / `**` unpacking. The `PyHost`
heap implements the `str` / `list` / `dict` / `tuple` / `set` / instance object
model with the operator, attribute, item, and iteration protocols. A program
that does not compile is reported the way CPython reports it — the
`File "…", line N` header, the source line and a caret run, then CPython's
message — and a `SyntaxError` carries its `lineno`/`offset`/`end_offset`/`text`
/`filename`; `compile()`, `exec` and `eval` share that path. See
[\[0x07\]](#0x07-status--roadmap) and [BUGS.md](BUGS.md) for the honest list of
what is not yet implemented.

## [0x04] COMMAND-LINE FLAGS

| Flag | Effect |
|---|---|
| *(none)* | Run the script/one-liner, transparently rkyv-cached. |
| `-c SRC` | Execute a one-liner (`python -c 'print(1+1)'`). |
| `-m MODULE …` | Run a library module as a script. Delegates to the embedded CPython (`runpy`), so `-m pip` / `-m venv` / `-m http.server` / `-m json.tool` behave exactly like `python3 -m`; every token after the module is the module's own `sys.argv`. Needs the `stdlib-ffi` bridge (default build). |
| `-u` | Sets `PYTHONUNBUFFERED` for the embedded interpreter. pythonrs's own `print` is already unbuffered on every stream, so the flag changes nothing on that side — see [BUGS.md]BUGS.md for the buffering divergence this implies. |
| `-E -I -O -S -B -W` | CPython interpreter flags, accepted for drop-in compatibility (`-u`/`-W` take real effect via the embedded interpreter; the rest are tolerated no-ops). |
| `--build` | AOT-compile the script to a standalone native executable. Needs a libpython-free runtime — build with `--no-default-features`; a `stdlib-ffi` build refuses up front (its CPython symbols can't be statically linked). |
| `--dump-bytecode` | Print the lowered `fusevm` bytecode and exit. |
| `--dump-tokens` | Print the lexer token stream and exit. |
| `--dump-ast` | Print the parsed AST and exit. |
| `--disasm` | Print a `fusevm` bytecode disassembly listing and exit. |
| `--tiers` | Run the script, then report which fusevm execution tier took each of its chunks. |
| `--repl` | Start the interactive REPL. |
| `--lsp` | Run the Language Server Protocol server over stdio. |
| `--dap` | Run the Debug Adapter Protocol server over stdio — breakpoints, stepping, stack trace, locals, expression `evaluate`. |
| `--doctor` | Print a diagnostic report — runtime, embedded CPython, fusevm engine, bytecode cache, `PYTHON*` env, and every `python*` interpreter on `PATH` — and exit. |
| `--cacheview` | List the compiled programs held in the rkyv bytecode cache (`~/.pythonrs/scripts.rkyv`): per-entry hashes, blob size, and op/function/try/warning counts. |
| `--cache-clear` | Delete the rkyv bytecode cache shard and exit. |

## [0x05] ARCHITECTURE

```
lexer  →  parser  →  AST  →  compiler  →  fusevm::Chunk  →  fusevm VM + JIT
                                              │                    │
                                              └── CallBuiltin ─────→ host (PyHost heap)
```

- `lexer.rs` — indentation-significant tokenizer (INDENT/DEDENT/NEWLINE, f-strings).
- `parser.rs` — recursive-descent Python grammar → `ast.rs`.
- `compiler.rs` — lowers the AST to fusevm ops + `CallBuiltin` dispatches.
- `host.rs` — the `PyHost` object heap (str/list/dict/tuple/set/instances/…), the
  operator/attribute/item/iteration semantics, and the fusevm run plumbing.
- `builtins.rs` — the `CallBuiltin` handler table, the numeric hook, the Kernel
  builtin functions (`print` / `len` / `range` / …), and per-type methods.
- `cache.rs` — the rkyv-shard bytecode cache.
- `aot_native.rs` — native-executable emission via `fusevm::aot`.
- `pylib/` — the vendored CPython pure-Python standard library (`.py` sources)
  shipped **with** pythonrs. In the native build these are imported by compiling
  and executing them on pythonrs's own interpreter — no libpython.

### Standard library: two build modes

The `import` path resolves a module from native inline arms first, then:

| Build | Command | `import <stdlib>` source |
|---|---|---|
| **Native (CPython-free)** | `cargo build --no-default-features` | The vendored `pylib/*.py`, compiled on pythonrs and run on fusevm. No pyo3, no libpython — CPython is not in the dependency graph. This is the shipping target (`brew install pythonrs` lays `pylib/` beside the binary). |
| **Bridged (drop-in)** | `cargo build` | The real CPython stdlib over an embedded libpython (pyo3, the `stdlib-ffi` feature). Kept primary while the native build's C-accelerator floor (`posix`/`_io`/`_sre`/…) is completed. |

Imports are memoized through the host's `sys.modules` cache, so a module's
vendored `.py` executes at most once (CPython run-once identity semantics).
`$PYTHONRS_LIB` overrides the `pylib/` search path.

## [0x06] PARITY HARNESS

Correctness is measured, not asserted: an example corpus runs through both
pythonrs and the reference `python3`, and the output is diffed byte-for-byte.
pythonrs runs a large, real subset of Python 3, verified against CPython on that
corpus.

Beyond the fixed corpus, the `parity-fuzz` binary is a differential fuzzer. It
generates thousands of grammar-driven, deterministic-output snippets — biased
toward the historically fragile areas (float `repr`, integer `//`/`%` sign
rules, bignum, slices, the `format` mini-language, string methods, containers
whose elements key through user `__hash__`/`__eq__`, `re` match positions over
subjects mixing 1-, 2-, 3- and 4-byte characters, and the exception boundary
between user code and the standard library — a stdlib `KeyError` coming back
with its key, a `@contextlib.contextmanager` driving a user generator through
`next`/`throw`/`close`) — runs each
through `python -c` and the reference `python3 -c`, and reports every case where
stdout or accept/reject diverges. Each case is seeded, so any divergence is
delta-debugged to a minimal reproducer and replays exactly:

A curated corpus has one structural blind spot worth naming: it can only report
constructs somebody thought to write down. A keyword-only argument, a function
nobody happened to call, or a method missing from the note-taker's mental list
stays invisible no matter how many cases run. Diffing the names `src/` dispatches
against the identifiers the corpus actually contains is what turns that blind
spot into a work list — `--mode containertail` exists because that diff surfaced
a dozen `itertools`/`collections`/`math` gaps at once, several of them silently
wrong answers rather than errors.

The same audit run over the generator corpus itself — which identifiers do the
modes never emit? — is what `--mode binop` and `--mode numproto` came from.
Not one case in the corpus had ever written `__radd__`, or any other reflected
dunder; nine of the thirteen forward operators and twelve of the thirteen
in-place ones were equally absent, as were `__round__`, `__trunc__`, `__floor__`,
`__ceil__`, `__complex__`, `__pos__` and `__invert__`. Behind that hole sat the
whole of CPython's operator negotiation — the rule that a subclass's reflected
dunder runs BEFORE its base's forward one, its mirror that two operands of the
same type never consult the reflected half for arithmetic, and the augmented
form's own `op=` wording — plus a `math` module that read every argument as
`as_f(v).unwrap_or(0.0)`, so `math.sqrt("s")`, `math.cos(None)` and
`math.sqrt(10**30)` all answered a plausible NUMBER rather than raising. A
saturated fuzzer is evidence about the grammar, not about the frontend.

`numproto`'s single divergence was one ulp of `math.erf(3)`, and the ulp was the
finding. Sweeping `[-6, 6]` in hundredths put `erf` 312 points out of 1201 away
from CPython, `erfc` 390, `gamma` 907 and `lgamma` 976 — pythonrs answered all
four out of the pure-Rust `libm` crate, and CPython answers none of them from
there: `erf`/`erfc` are the platform's, while `gamma`/`lgamma` are CPython's own
Lanczos code, carried in `mathmodule.c` precisely because the platform's are not
good enough. Porting that code closed three of the four; the fourth needed the
FMA contraction clang applies to `num*x + coeff` by default, which is one
rounding where a literal Rust translation has two.

Counting the corpus again against the 68 modes that preceded them produced
`--mode itertail2` and `--mode numintro`. `pairwise`, `starmap`, `groupby`,
`zip_longest`, `filterfalse`, `dropwhile`, `takewhile`, `compress`,
`permutations`, `combinations_with_replacement` and `chain.from_iterable` are all
implemented and every one occurred ZERO times — `gen_itertools` covered the
builtin `zip`/`map`/`filter`/`enumerate`/`reversed`, never the module. So did
`bit_length`, `bit_count`, `as_integer_ratio`, `is_integer`, `int.from_bytes` and
`numerator`/`denominator`. The counted hole held `groupby(xs, key=f)` grouping by
the raw element while reporting it as the key (the positional `groupby(xs, f)`
was correct, so only the keyword form was wrong), a negative `r` yielding an
empty result instead of raising, and `True.real` answering `True` where CPython
answers `1`.

Two things about those modes are the method rather than the subject. A key
function has to be NON-identity: a `key=` that returns its argument unchanged
cannot distinguish "the key was applied" from "the key was ignored", which is the
shape that hid `groupby` for as long as it was hidden. And a case whose oracle
raises prints NOTHING and is counted barren — measured nothing at all — so
`itertail2`'s first run reported 138 barren cases, every one of them an error
path the mode existed to test. Printing the exception instead of letting it
propagate turned all 138 into comparisons.

A generated corpus has the mirror blind spot: it can only report combinations
the grammar can produce. `--mode buffer` exists because of one — the grammar
wrote `ba[i] = <int literal>` and read `a[Idx()]`, so the literal store and the
`__index__` load were both covered while nothing ever reached a store or delete
whose subscript or element was an OBJECT, or any write through a `memoryview` at
all. Thousands of clean cases ran either side of that hole while a view was
advertising `readonly is False` and refusing every write, and `L[Idx(1)] = 9`
was raising where `L[Idx(1)]` returned. Where a mode's cases are refusals as
often as successes, the refusal is printed rather than raised, so the exact
wording is compared on stdout and generic text for a specific diagnosis fails.

```sh
cargo build --bin parity-fuzz
./target/debug/parity-fuzz --count 5000          # fuzz every mode
./target/debug/parity-fuzz --formatspec          # one surface only
./target/debug/parity-fuzz --seed 51 --once      # replay + minimize one case
```

The generator is written not to emit nondeterministic output, so a reported
divergence is meant to be a real gap — but that is a rule about the generator,
not something the harness can verify: `gen_dataclass` printed a sentinel whose
`repr` carries an object address and produced a permanent false divergence
until it was caught by reading the report. `PYTHONHASHSEED` is pinned to the
same value on both children and **swept** across cases rather than frozen at
`0`, so `str`/`bytes` hashing and string-keyed container order are measured
across the seed axis instead of at one point of it.
`PYTHONRS_ORACLE` names the reference interpreter (`PYTHONRS_FUZZ_PYTHON` still
works); a `--baseline` allowlist keeps known gaps from failing while new ones
exit non-zero. Every harness — this one, the `parity` corpus runner, and the
`tests/parity.rs` integration test — resolves the reference to an ABSOLUTE path
and prints it before comparing anything, so a result can be attributed to the
interpreter that produced it. A bare `python3` is a PATH lookup rather than a
toolchain: a shim, a venv or a pyenv/Homebrew shadow all answer to it, and a run
that silently compared against the wrong one is indistinguishable from a correct
one. An explicitly-named oracle that will not run is a hard error, never a
fallback onto a different CPython.
A clean run has to be a run that measured something: cases the reference did not
answer (timed out, exited non-zero, or printed nothing) are reported as `barren`
and excluded from `productive`, and a run that executed no case — or none the
reference answered — exits non-zero instead of printing `divergences : 0`.

## [0x07] STATUS & ROADMAP

Active, in development. The runtime executes a substantial real subset of Python
3. The full CPython standard library is importable by default — the `stdlib-ffi`
bridge (on by default) delegates `import os`/`json`/`random`/… to an embedded
libpython, so only a `--no-default-features` build is limited to the native
module subset. `re` and `itertools` are the exception: they are native in BOTH
builds and never reach CPython. [BUGS.md]BUGS.md is the honest ledger of remaining gaps. A DAP
debug adapter (`--dap`) — source-line and function breakpoints, stepping, call
stack, locals, and expression `evaluate` — ships today, alongside man pages and
the generated `reference.html`.

## [0x08] DOCUMENTATION

- **Docs site** — <https://menketechnologies.github.io/pythonrs/>
- **Engineering report** — <https://menketechnologies.github.io/pythonrs/report.html>
- **Builtin reference** — <https://menketechnologies.github.io/pythonrs/reference.html>
- **The shared VM** — [`fusevm`]https://github.com/MenkeTechnologies/fusevm, also behind `zshrs`, `strykelang`, `awkrs`, `vimlrs`, `elisprs`, `rubylang`.

## [0xFF] LICENSE

MIT.