bytecode-lang 0.3.0

The model

• A Module is one compilation unit: a string table, a type table (function signatures, structs with single inheritance and methods, arrays, ordered maps, cells, iterators), a constant pool, imports (host functions), globals, functions, exports, dynamic-operation hooks, and per-function debug line tables and local names.
• A function is a frame of typed 64-bit registers plus a flat array of **Inst**ructions: 200 of them, covering moves and constants, OPS integer and float arithmetic with their policies (PHP's int-to-float promote and shift saturate included, pow in every form), conversions, dynamic arithmetic with a numeric fast path, dynamic truthiness and logical not, calls (direct, indirect, host, tail, dynamic, and shaped dynamic calls with named and spread arguments), closures and cells, structs, arrays, insertion-ordered hash maps (PHP arrays) with copy-on-write separation and references into their slots, iteration, byte strings, type tests and casts, exceptions (raise of any catchable kind with a payload), GC safepoints, and stackful coroutines for generators and async tasks (coro_new, yield, yield_kv, await, resume, resume_throw, coro_close, spawn).
• A function or import may carry a ParamList (names, rest parameters, defaults with a presence mask, by-reference parameters); ParamList::bind is the one binding rule every tier applies to dynamic calls.
• encode and decode convert between a module and its canonical bytes; disassemble prints it.
• ModuleBuilder builds modules: branches take labels, strings and constants are deduplicated, and simultaneous register moves are sequentialized correctly.

The normative semantics of every instruction, the encoding, and the verifier's rules are in the LexerSketch spec specs/LSB.md.

What it guarantees, and how each guarantee is checked:

Installation

[dependencies]
bytecode-lang = "0.3"

Without the standard library:

[dependencies]
bytecode-lang = { version = "0.3", default-features = false }

Quick start

Build a function, encode it, decode it, read it:

use bytecode_lang::{decode, disassemble, encode, Inst, IntOp, IntTy, ModuleBuilder, ValType};

let mut m = ModuleBuilder::new();
let mut f = m.function("add", &[ValType::I64, ValType::I64], &[ValType::I64]);
let (a, b) = (f.param(0), f.param(1));
let sum = f.reg(ValType::I64);
f.emit(Inst::IAdd { dst: sum, lhs: a, rhs: b, op: IntOp::new(IntTy::I64) });
f.ret(sum);
m.add_function(f).unwrap();
let module = m.finish().unwrap();

let bytes = encode(&module);
let back = decode(&bytes).unwrap();
assert_eq!(back, module);
assert!(disassemble(&back).contains("  0000 iadd.i64 r2, r0, r1\n  0001 ret r2"));

Loops, labels, and policies

The overflow policy is part of the instruction: iadd.i64 raises ArithOverflow, iadd.i64.wrap wraps, on every tier.

use bytecode_lang::{disassemble, Inst, IntOp, IntTy, ModuleBuilder, Overflow, Policy, ValType};

let wrap = IntOp::new(IntTy::I64).with_policy(Policy::new().with_overflow(Overflow::Wrap));
let mut m = ModuleBuilder::new();
let mut f = m.function("hash", &[ValType::I64], &[ValType::I64]);
let n = f.param(0);
let (h, i, one, more) = (f.reg(ValType::I64), f.reg(ValType::I64), f.reg(ValType::I64), f.reg(ValType::Bool));
f.emit(Inst::LoadInt { dst: h, val: 17, ty: IntTy::I64 });
f.emit(Inst::LoadInt { dst: i, val: 0, ty: IntTy::I64 });
f.emit(Inst::LoadInt { dst: one, val: 1, ty: IntTy::I64 });
let (top, done) = (f.label(), f.label());
f.bind(top);
f.emit(Inst::ILt { dst: more, lhs: i, rhs: n, ty: IntTy::I64 });
f.jmp_if_not(more, done);
f.emit(Inst::IMul { dst: h, lhs: h, rhs: h, op: wrap });
f.emit(Inst::IAdd { dst: i, lhs: i, rhs: one, op: IntOp::new(IntTy::I64) });
f.emit(Inst::Safepoint {}); // every loop passes a safepoint: GC and fuel
f.jmp(top);
f.bind(done);
f.ret(h);
m.add_function(f).unwrap();

let text = disassemble(&m.finish().unwrap());
assert!(text.contains("jmp_if_not r4, L1"));
assert!(text.contains("imul.i64.wrap r1, r1, r1"));
assert!(text.contains("L1:\n  0009 ret r1"));

Dynamic code

dyn registers, property access by name, and hooks that let a language define its own slow path:

use bytecode_lang::{Callee, Hook, Inst, ModuleBuilder, Policy, ValType};

let mut m = ModuleBuilder::new();
// PHP-style loose `+` on strings and arrays lives in the language runtime.
let sig = m.func_type(&[ValType::Dyn, ValType::Dyn], &[ValType::Dyn]);
let loose_add = m.import("mox.rt", "add", sig);
m.hook(Hook::Add, Callee::Import(loose_add));

let total = m.string("total");
let mut f = m.function("bump", &[ValType::Dyn, ValType::Dyn], &[ValType::Dyn]);
let (obj, amount) = (f.param(0), f.param(1));
let (old, new) = (f.reg(ValType::Dyn), f.reg(ValType::Dyn));
let name = f.name_ref(total);
f.emit(Inst::GetProp { dst: old, obj, name });
f.emit(Inst::DAdd { dst: new, lhs: old, rhs: amount, pol: Policy::new() });
f.emit(Inst::SetProp { obj, name, src: new });
f.ret(new);
m.add_function(f).unwrap();
let module = m.finish().unwrap();
assert_eq!(module.hook(Hook::Add), Some(Callee::Import(loose_add)));

PHP's calls, arrays, and references

A dynamic call to a function with a real signature, a nested write that separates only a shared array, and a reference into a map slot:

use bytecode_lang::{ArgItem, Bound, Inst, ModuleBuilder, Param, ParamKind, ParamList, ValType};

let d = ValType::Dyn;
let mut m = ModuleBuilder::new();
let (a, k) = (m.string("a"), m.string("k"));
// function f(&$a, $k = 0, ...$rest): the signature ends with the presence mask.
let list = ParamList::new(vec![
    Param::normal(a).by_ref(),
    Param::normal(k).with_default(),
    Param::new(ParamKind::RestMap, None),
]);
let mut f = m.function("f", &[d, d, d, ValType::I64], &[]);
f.set_params(list.clone());
let (arr, inner, r) = (f.reg(d), f.reg(d), f.reg(d));
f.emit(Inst::CellGet { dst: arr, cell: f.param(0) });          // $a is a reference
f.emit(Inst::DSepIndex { dst: inner, obj: arr, key: f.param(1) }); // $a[$k][] = ...
f.emit(Inst::DRefIndex { dst: r, obj: arr, key: f.param(1) });    // $r = &$a[$k]
f.ret_void();
m.add_function(f).unwrap();
let module = m.finish().unwrap();

// f($x, extra: 1): `$k` takes its default, `extra` lands in `$rest`.
let bound = list.bind(&module, &[ArgItem::Positional, ArgItem::Named(b"extra")]).unwrap();
assert_eq!(bound.slots(), &[Bound::Arg(0), Bound::Default, Bound::Collected(vec![1])]);
assert_eq!(bound.presence(), 0b101);

Hostile input

use bytecode_lang::{decode_with, DecodeErrorKind, Limit, Limits};

// A header followed by a string section claiming four billion strings.
let mut bytes = b"LSB\0\x02\0\0\0\0\0\0\0".to_vec();
bytes.extend([1, 0, 0, 0, 4, 0, 0, 0, 0xff, 0xff, 0xff, 0xff]);
let mut limits = Limits::default();
limits.max_strings = 1000;
let err = decode_with(&bytes, &limits).unwrap_err();
assert_eq!(err.kind(), &DecodeErrorKind::LimitExceeded(Limit::Strings));
assert_eq!(err.to_string(), "at byte 20: string count limit exceeded");

Examples

Runnable programs in examples/:

Performance

Encoding writes eight bytes per instruction from a value already laid out as the word; decoding validates each word with one mask test and one jump-table dispatch, and reserves every list once, after proving the input can fill it. Section lengths are computed by running the same writer against a byte counter, so the output buffer is allocated exactly once.

Measured with the benchmarks in benches/ on a module of 1,000 functions × 1,000 instructions (a realistic mix: constants, typed and dynamic arithmetic, a property read, a compare and branch, a call, a safepoint), x86_64, Rust stable, release profile:

Decoding costs about twice from_bytes alone: the difference is allocating and first touching the 8 MB of decoded code. Ranges are across runs on a machine shared with other builds. The builder's overflow = promote check, fused into its branch-target pass, costs about 0.1–0.5 ms per million instructions (≈3–12%) in interleaved measurements. These are library-only numbers; nothing here measures a VM.

0.3.0 against 0.2.0, both built and run back to back on the same (loaded) machine: encoding and building are unchanged within the run-to-run spread; decoding measured faster in 0.3.0 (≈2.3–2.8 ms against ≈4.2–4.4 ms for the million-instruction module, a difference not isolated, so not claimed as a gain); disassembly measured ≈7% slower (≈10.6–10.9 ms against ≈9.9–10.2 ms per 100,000 instructions), not isolated either. The binder allocates its result (one slot per parameter, one list per rest parameter); a VM that binds on a hot path uses its own fast path for all-positional exact calls and checks the rest against ParamList::bind.

cargo bench --bench bench

Criterion writes per-benchmark reports to target/criterion/. Numbers vary by CPU; use the trend across runs, not a single absolute.

Design notes

• A register machine whose decoded form is its executed form. The Inst enum is eight bytes, the same as its encoding, so there is no second representation to keep in sync and no parse step between loading and running.
• Typed registers, dyn included. Declared register types make GC roots exact and let the verifier check every operand in one linear pass. Dynamic languages declare dyn registers and use the d* instructions; static languages use unboxed typed registers. Mixed code converts explicitly.
• Constants are format values. The pool holds exact i64, u64, IEEE bit patterns, chars, UTF-8 and byte strings, and arrays and maps of earlier constants; the runtime value layout is the VM's business at load time. That is why the crate does not depend on value-lang, whose i32 integers and process-local symbols cannot represent or serialize these.
• Fixed-width, canonical encoding. No variable-length integers: every value has one encoding, decoding is branch-light, and a decoded file re-encodes byte for byte.
• Structure here, meaning in the verifier. The decoder checks the format; whether an index is in range or a register is used at its type is the verifier's (0.5). Every accessor that follows an index returns Option, and the disassembler prints bad references as <invalid>.
• No recursion anywhere input reaches. Aggregate constants refer only to earlier constants, so their depth is computed in one forward pass; nothing else in the format nests.

Testing

cargo test                       # unit + integration + property + doctests
cargo test --no-default-features # no_std + alloc
cargo clippy --all-targets --all-features -- -D warnings
cargo bench --bench bench

The property tests in tests/properties.rs generate instructions from the crate's own opcode metadata (so every opcode and modifier is covered without a hand-kept list), build arbitrary modules (parameter lists and call shapes included), and check the round trip, canonicity, determinism, decoder robustness against arbitrary, framed, and mutated bytes, budgets, label resolution against a reference resolver, and parallel moves against parallel assignment. tests/codec.rs hand-assembles inputs for every decode error, independently of the encoder. tests/disasm.rs pins the listing format with a golden file. tests/binding.rs checks the dynamic-call binding rule against an independent reference, and tests/format2.rs every instruction, rename, and modifier change of format version 2. Every rust example in this README and in docs/API.md is compiled and run as a doctest.

Cross-platform support

• Linux (x86_64, aarch64)
• macOS (x86_64, Apple Silicon)
• Windows (x86_64)

The format is little-endian by definition and the crate uses no operating-system facilities, so the same module encodes to the same bytes on every platform.

Contributing

See REPS.md for the engineering standards every change is held to, dev/DIRECTIVES.md for the definition of done, and dev/ROADMAP.md for what comes next. Before a PR: cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, and cargo test --all-features must be clean.