Skip to main content

Crate pagoda

Crate pagoda 

Source
Expand description

An SGLang-inspired LLM serving runtime, implemented from scratch in Rust.

This crate distills the architecture of SGLang (sgl-project/sglang) into a compact, dependency-free reference implementation:

  • engine — offline batch engine + continuous-batching scheduler
  • radix_cache— RadixAttention-style prefix cache (compute-skip)
  • kv_cache — paged KV cache with reference counting and copy-on-write
  • sampler — temperature / top-k / top-p / penalty sampling
  • grammar — regex / JSON constrained-decoding logit mask
  • dsl — an SGLang-style frontend language (gen/select/fork)
  • model — model backend trait + deterministic n-gram toy LM
  • tokenizer — tokenizer trait + deterministic byte-level tokenizer
  • server — minimal dependency-free HTTP serving frontend

The implementation deliberately keeps external dependencies to zero so the whole runtime compiles and runs entirely offline. See docs/DESIGN.md for the architecture, the mapping to SGLang, and the parity roadmap.

Re-exports§

pub use apc::ApcCache;
pub use dsl::Op;
pub use dsl::Program;
pub use dsl::StreamResult;
pub use engine::CacheBackend;
pub use engine::CheckpointId;
pub use engine::Engine;
pub use engine::EngineConfig;
pub use engine::EngineStats;
pub use engine::SchedulePolicy;
pub use engine::ToyEngine;
pub use grammar::mask_logits;
pub use grammar::ByteRegex;
pub use grammar::Grammar;
pub use kv_cache::BlockId;
pub use kv_cache::PagedKvCache;
pub use model::ModelEngine;
pub use model::ModelSession;
pub use model::NGramModel;
pub use radix_cache::RadixCache;
pub use sampler::Sampler;
pub use spec::FinishReason;
pub use spec::GenerationOutput;
pub use spec::RejectReason;
pub use spec::SamplingParams;
pub use spec::TokenId;
pub use spec::WriteRequest;
pub use tokenizer::ByteTokenizer;
pub use tokenizer::Tokenizer;

Modules§

apc
Block-level chained-hash prefix cache, in the style of vLLM’s Automatic Prefix Caching (APC).
dsl
An SGLang-style frontend language, expressed as an embeddable builder API.
engine
Offline batch engine plus the continuous-batching scheduler.
grammar
Dependency-free constrained decoding.
http_client
A minimal, dependency-free blocking HTTP/1.1 client.
json
A minimal JSON value, parser, and serializer. Self-contained (no serde) because the runtime must compile offline.
kv_cache
Paged KV cache with reference counting and copy-on-write.
model
Model backend abstraction plus a deterministic n-gram toy language model.
radix_cache
Radix-tree prefix cache — the analogue of SGLang’s RadixAttention.
rng
A tiny deterministic PRNG (SplitMix64) so sampling is reproducible with no external dependencies.
sampler
Logit post-processing and categorical sampling.
server
A minimal, dependency-free HTTP serving frontend.
spec
Public data model shared across the runtime and the SGLang-style frontend.
tokenizer
Tokenizer abstraction plus a deterministic byte-level reference tokenizer.
triage
Laya triage: a System-1 gate in front of generation (System 2).