Skip to main content

Crate logit_loom_llamacpp

Crate logit_loom_llamacpp 

Source
Expand description

§logit-loom-llamacpp

The safe llama.cpp adapter for Logit Loom.

It maps llama.cpp candidate logits and admitted tokens into the transform and observer boundaries supplied by logit-loom. It also exposes causal prefill, compatibility-bound checkpoints, topology-bound activation capture and transactions, target-authoritative MTP/EAGLE-3 generation, and scoped LoRA and control vectors. Session::generate_v2 additionally executes versioned eager or bounded trigger-driven lazy grammar plans without changing the original generation-plan identity.

§Select a backend

The crate enables no llama.cpp backend feature by default. Select the feature for the deployment explicitly:

[dependencies]
logit-loom = "=0.2.0"
logit-loom-llamacpp = { version = "=0.2.0", features = ["vulkan"] }

ModelOptions::default requires accelerator participation and does not retry a rejected load as CPU-only inference. Use DevicePolicy::Any only when that fallback is intentional.

§Example

The adapter never downloads a model. Supply a local GGUF explicitly:

use logit_loom::{GenerationPlan, SamplingPlan};
use logit_loom_llamacpp::{
    Model, ModelOptions, Runtime, SessionOptions, Tokenization,
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let model_path = std::path::Path::new("model.gguf");
    let mut runtime = Runtime::initialize()?;
    runtime.silence_native_logs();
    let model = Model::load(&runtime, model_path, ModelOptions::default())?;
    let prompt = model.tokenize("Hello", Tokenization { add_bos: true })?;
    let mut session = model.session(&runtime, SessionOptions::default())?;
    session.prefill(&prompt, true)?;

    let output = session.generate(
        &GenerationPlan {
            sampling: SamplingPlan::default(),
            max_tokens: 16,
            biases: Vec::new(),
            grammar: None,
            stops: Vec::new(),
        },
        None,
        None,
    )?;
    let _exact_bytes = output.bytes;
    Ok(())
}

Local resource authorities may avoid whole-file digest passes for immutable sealed artifacts. Model::load_preverified accepts a caller-established BLAKE3 identity, while Model::load_authorized accepts a provider-owned stable generation identity and exact length without hashing model payload bytes. Model::load_lora_authorized provides the same provider-owned boundary for an already authenticated and sealed LoRA. Mutable and ordinary filesystem paths continue to use Model::load and Model::load_lora, which verify content before and after native loading.

Generated pieces remain arbitrary bytes. Call GenerationOutput::text only when the complete output is expected to be valid UTF-8. Model::tokenize rejects NUL bytes and inputs larger than MAX_TOKENIZATION_BYTES before calling the native binding.

§Structured projection

StructuredProjectionController captures one exact prefilled causal boundary, builds a content-identified table of every tokenizer ID, exact piece bytes, and EOG classification, and runs eager or lazy native grammar together with caller-identified exact-byte feedback. An installed authoritative validator then either selects the complete output or restores the boundary.

attempt performs exactly one attempt. It never retries internally. Cancellation, validation rejection, feedback errors, validator errors, and contained callback panics restore the common checkpoint when native state remains recoverable. The caller may then explicitly call attempt again within the plan bound. Compiler and validator semantics remain outside Logit Loom; their exact implementation/configuration identities and content-free receipts are inside the public mechanical contract.

§Aggregate execution

execute_text_mechanics is the complete TextMechanicsPlanV2 lowering. A TextMechanicsRequest supplies exact prompt tokens, target/draft context geometry, installed activation runtimes, borrowed transform and observer sets, loaded target LoRAs, an optional target control vector, and optional process-local parent state. Every identity and bound is checked before context allocation.

The ordinary and MTP/EAGLE-3 paths share one lifecycle: apply ordered target steering, admit bounded prompt chunks, generate under the selected activation and callback mechanics, explicitly clear steering in reverse order, then optionally capture a quiescent checkpoint. A PrefillMonitor may stop either fresh path at a complete synchronized chunk boundary. That terminal result contains prefill and cleanup evidence, publishes neither generation nor a partial checkpoint, and performs no retry or fallback.

Ordinary aggregate checkpoints use TextMechanicsCheckpointReceiptV2 and retain the opaque native sampler plus cross-operation stop prefix; speculative snapshots retain their complete aggregate plan and native sampler. Both forms are process-local and thread-affine. A continuation must preserve every mechanic apart from its explicit parent-checkpoint field. Because native state omits the next-token logits, exact steering is reapplied while reconstructing the restored boundary and cleared again before successor capture.

§Activation and speculation

ActivationConfiguration validates exact tensor sites, bounded capture plans, ordered operators, sparse vector banks, and the loaded model topology before allocating a context. Graph-node selectors are bound to the exact adapter, binding, llama.cpp revision, and architecture profile. A selected tensor is copied into owned storage; one complete finite result is written back only after every declared operator succeeds.

generate_speculative drives either same-model MTP or a separately loaded EAGLE-3 draft. The supplied SpeculationPlanV1 must name speculation_implementation_identity(), both exact model/topology identities, one supported mechanism, and an explicit activation policy. Before allocating contexts, the adapter also applies llama.cpp HEAD’s vocabulary-compatibility rules, checks the MTP hidden-row width, and requires an eagle3 draft with exactly three in-range target extraction layers:

use logit_loom::{
    GenerationPlan, SamplingPlan, SpeculationActivationPolicyV1,
    SpeculationPlanV1, TextSpeculativeMechanismV1,
};
use logit_loom_llamacpp::{
    Model, ModelOptions, Runtime, SpeculativeRequest, Tokenization,
    generate_speculative, speculation_implementation_identity,
};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let runtime = Runtime::initialize()?;
    let model = Model::load(&runtime, "mtp-model.gguf", ModelOptions::default())?;
    let prompt = model.tokenize("A small mechanical experiment:", Tokenization {
        add_bos: true,
    })?;
    let generation = GenerationPlan {
        sampling: SamplingPlan::default(),
        max_tokens: 16,
        biases: Vec::new(),
        grammar: None,
        stops: Vec::new(),
    };
    let topology = model.topology().digest()?;
    let speculation = SpeculationPlanV1 {
        target_model: model.artifact_digest().clone(),
        target_topology: topology.clone(),
        draft_model: model.artifact_digest().clone(),
        draft_topology: topology,
        implementation: speculation_implementation_identity(),
        mechanism: TextSpeculativeMechanismV1::Mtp,
        sequences: 1,
        maximum_draft_tokens: 4,
        minimum_draft_tokens: 0,
        probability_floor_bits: 0.0_f32.to_bits(),
        activation: SpeculationActivationPolicyV1::None,
    };
    let output = generate_speculative(
        &runtime,
        &model,
        &model,
        SpeculativeRequest::new(&prompt, &generation, &speculation),
    )?;
    let _exact_bytes = output.generation.bytes;
    Ok(())
}

The target sampler verifies the longest proposal prefix. Rejected proposals and end-of-generation selections never reach token observers. Verification captures remain provisional until rollback and native acceptance establish their admitted/rejected positions. The current high-level operation supports one sequence and requires explicit context headroom; larger sequence plans are rejected before allocation rather than silently lowered.

The runnable speculative_mtp example takes one compatible caller-supplied GGUF and writes exact generated bytes:

cargo run -p logit-loom-llamacpp --example speculative_mtp \
  --features vulkan -- /path/to/mtp-model.gguf "Draft from here:"

generate_speculative_checkpointed captures exact target, draft, and MTP/EAGLE-3 implementation state at a quiescent boundary together with an opaque target-sampler clone, activation configuration, cross-operation stop state, and causal lineage. resume_speculative_checkpointed validates the complete parent before allocation, clones the sampler for an independent branch, restores contexts and implementation state, and captures the next boundary. Target steering selected through SpeculativeRequest or SpeculativeContinuationRequest spans the native operation and is cleared before that boundary is captured.

The checkpoint object is process-local and thread-affine because llama.cpp provides no portable target-sampler encoding. Its SpeculativeCheckpointReceiptV1 is serializable mechanical evidence, but the receipt alone cannot reconstruct the native sampler. No partial portable checkpoint is advertised.

§Coordinator integration

Runtime::native_backend and Model::native_model provide read-only borrows for an in-process coordinator that must compose other llama.cpp mechanics, such as multimodal projection or embeddings, with the same backend and loaded model. Logit Loom retains backend and model lifecycle ownership. Calls made through those handles are outside Logit Loom’s plan and receipt surface; a coordinator must give those calls their own bounded contract and evidence.

Checkpoints bind the model bytes, exact safe-binding source, literal llama.cpp revision, adapter build identity, and exact session allocation options; native state is opaque and is not a portable interchange format. A failed automatic steering cleanup or partial checkpoint restore poisons the session, and subsequent mutation returns Error::Poisoned.

Applications may persist StateSnapshot::into_parts in their own container format and reconstruct it with StateSnapshot::from_parts. Reconstruction validates internal byte and token-lineage identities; restore additionally requires the original model and compatible backend build. Because the pinned llama.cpp state format omits next-token logits, restore re-decodes the final recorded token at its exact position after restoring causal memory. A backend that cannot remove that one position is rejected, and a failed refresh poisons the session.

See the compatibility policy and capability status before selecting features or interpreting validation results.

Structs§

ActivationCaptureOutput
Captured records and per-plan aggregate receipts drained from a session.
ActivationConfiguration
Validated activation program, vector banks, captures, and graph profile.
ActivationProgramOutput
Transaction records and aggregate program receipt drained from a session.
AuthorizedLoraArtifact
Provider-owned identity and exact length for an immutable LoRA artifact.
AuthorizedModelArtifact
Provider-owned identity and exact length for an immutable model artifact.
ByteFeedbackCandidate
One exact model-tokenizer candidate presented to byte feedback.
ByteFeedbackError
Bounded implementation error from exact-byte feedback.
ControlVector
In-memory native control-vector data and layer contract.
ControlVectorScope
Active reversible control-vector application.
GenerationOutput
Successful bounded generation output.
InstalledByteFeedback
Installed byte-feedback implementation bound to its public identity.
InstalledStructuredValidator
Installed validator implementation bound to its public identity.
LlamaCppTensorProfile
Explicit llama.cpp graph-site compatibility profile.
LoraAdapter
Loaded model-compatible LoRA adapter.
LoraApplication
One borrowed LoRA and its exact application scale.
LoraScope
Active reversible LoRA application.
Model
Immutable loaded model with content and placement identity.
ModelOptions
Model-loading options exposed by the first adapter release.
OrdinaryTextMechanicsSnapshot
Opaque ordinary causal state plus its complete aggregate mechanics.
PrefillOutput
Result of one plain or controlled prefill call.
PreverifiedModelArtifact
Exact content evidence for an immutable model artifact verified by its caller.
Runtime
Process-wide initialized llama.cpp backend.
Session
Single-owner llama.cpp context with explicit causal lineage.
SessionOptions
Context-allocation options.
SpeculativeActivationOutput
Activation evidence produced by one context in a speculative operation.
SpeculativeActivations
Explicit activation runtimes supplied to one speculative operation.
SpeculativeCheckpointOutput
Successful generation plus a reusable quiescent continuation checkpoint.
SpeculativeCheckpointRequest
Aggregate-plan lineage selected for one quiescent speculative checkpoint.
SpeculativeContinuationRequest
Borrowed mechanics for continuation from a speculative checkpoint.
SpeculativeGenerationOutput
Complete successful output of one target-authoritative operation.
SpeculativeRequest
Borrowed mechanics and owned runtime configuration for one operation.
SpeculativeSessionOptions
Target and draft context-allocation options for one speculative operation.
SpeculativeStateSnapshot
Opaque process-local continuation state for one quiescent speculative boundary.
StateSnapshot
Opaque llama.cpp causal state with compatibility-bound identity metadata.
SteeringScope
Active aggregate steering application.
StructuredAttemptOutput
Runtime output from one explicit projection attempt.
StructuredProjectionController
Single-owner controller for exact rejection recovery and explicit retry.
StructuredValidationContext
Read-only authoritative validation input.
StructuredValidationOutcome
Authoritative mechanical validation output before evidence bytes are reduced to digests.
StructuredValidatorError
Bounded implementation error from an authoritative validator.
TextMechanicsOutput
Successful whole-operation output and content-free terminal evidence.
TextMechanicsRequest
Borrowed callbacks/resources and owned runtime configuration for one aggregate text operation.
Tokenization
Tokenization flags for exact text admission.

Enums§

DevicePolicy
Runtime placement requirement checked after model loading.
Error
Failures from native execution, callback boundaries, or identity checks.
TextMechanicsCheckpoint
Opaque causal state captured at a successful aggregate boundary.
TextMechanicsExecutionOutput
Mechanic-specific output retained beside the aggregate receipt.
TextMechanicsResume
Exact causal state selected as the parent of one aggregate operation.

Constants§

LLAMA_CPP_BINDING_SOURCE_REVISION
Exact source revision of the reviewed native Rust binding successor.
LLAMA_CPP_BINDING_VERSION
Exact native Rust binding version used by this adapter release.
LLAMA_CPP_REVISION
Exact llama.cpp revision carried by the ADR 0002 binding successor.
MAX_STRUCTURED_TOKENIZER_TABLE_BYTES
Maximum aggregate token-piece bytes retained by a prepared tokenizer table.
MAX_STRUCTURED_VALIDATION_EVIDENCE_BYTES
Maximum canonical artifact or validation-report bytes hashed per attempt.
MAX_TOKENIZATION_BYTES
Maximum UTF-8 bytes accepted by one native tokenization call.

Traits§

ByteFeedback
Stateful exact-byte candidate feedback for one structured attempt.
StructuredValidator
Caller-supplied authoritative parser and constraint validator.

Functions§

execute_text_mechanics
Executes one complete topology-bound text-mechanics plan.
generate_speculative
Runs one bounded MTP or EAGLE-3 operation.
generate_speculative_checkpointed
Runs one bounded speculative operation and captures complete quiescent continuation state.
resume_speculative
Continues one compatible quiescent checkpoint without capturing a successor boundary.
resume_speculative_checkpointed
Continues one compatible quiescent checkpoint and captures the next quiescent boundary.
speculation_implementation_identity
Returns the exact native and safe-wrapper implementation identity.