Struct Session 

pub struct Session { /* private fields */ }

Stateful inference session. Owns refcounted handles to the model and tokenizer — no borrow lifetime — so Session values can flow across an FFI boundary or be returned from a constructor without tying them to an owning CeraEngine.

The Arc-based design replaced the earlier Session<'a> { model: &'a dyn Model, tokenizer: &'a BpeTokenizer } form because UniFFI and bindgen tools don’t marshal Rust lifetimes — the exposed type has to own its dependencies.

Implementations

impl Session

pub fn new( model: Arc<dyn Model>, tokenizer: Arc<BpeTokenizer>, capabilities: ModalityCapabilities, config: SessionConfig, ) -> Result<Self, CeraError>

Construct a new session backed by an already-loaded model + tokenizer. Both are taken by Arc — in-process callers typically clone from crate::CeraEngine (see crate::CeraEngine::new_session); FFI callers wrap owned handles.

capabilities declares what the loaded model accepts / emits. Direct callers (tests, standalone Model loaders) that don’t have a Manifest handy can pass ModalityCapabilities::text_only.

pub fn default_generate_opts(&self) -> &GenerateOpts

Borrow the default generation options configured for this session.

pub fn set_default_generate_opts(&mut self, opts: GenerateOpts)

Override the default generation options for this session.

pub fn attach_drafter(&mut self, drafter: &dyn Drafter)

Attach a speculative decoding drafter (e.g. DSpark sidecar model).

pub fn attach_vocoder( &mut self, decoder: Arc<AudioDecoderWeights>, detok: Arc<DetokenizerWeights>, )

Attach vocoder weights so Self::generate can synthesize PCM audio frames from model hidden states when the model enters audio mode.

pub fn attach_gpu_audio_decoder(&mut self, decoder: Arc<dyn AudioGpu>)

Attach a GPU audio decoder backend. When present, Self::generate passes it to AudioOutputDecoder so detokenization and iSTFT run on the GPU. The CPU vocoder weights must still be attached via Self::attach_vocoder. Preserved across reset().

pub fn has_gpu_audio_decoder(&self) -> bool

Whether a GPU audio decoder backend is attached to this session.

pub fn attach_audio_encoder(&mut self, encoder: Arc<AudioEncoderWeights>)

Attach an audio encoder so Self::append_audio can encode PCM samples into LLM-ready embeddings. Callers load the encoder from the bundle’s multimodal_projector GGUF via crate::model::audio_encoder::AudioEncoderWeights::from_gguf.

Replaces any previously-attached encoder. Preserved across Self::reset — the encoder is independent of KV state.

Does not validate that the encoder’s llm_hidden_size matches the LLM’s hidden_size; that check lives on Self::append_audio so a stub encoder used in test setup doesn’t have to wire up matching dimensions just to be attached. Real callers should always pair the encoder with the LLM it was trained against.

pub fn attach_gpu_audio_encoder(&mut self, encoder: Arc<dyn AudioGpuEncode>)

Attach a cached GPU audio encoder. When present, Self::append_audio runs the Conformer stack on the GPU, falling back to the CPU audio_encoder for chunks the GPU kernels cannot take (see crate::model::audio_encoder_gpu::MAX_AUDIO_TOKENS) and for any other encode error. The CPU encoder must still be attached via Self::attach_audio_encoder: it backs that fallback and the capability/dimension checks. Preserved across reset().

pub fn attach_vision_encoder(&mut self, encoder: Arc<VisionEncoderWeights>)

Attach a vision encoder so Self::append_image can encode PNG / JPEG bytes into LLM-ready image embeddings. Callers load the encoder from the bundle’s multimodal_projector GGUF via crate::model::vision_encoder::VisionEncoderWeights::from_gguf. Mirrors Self::attach_audio_encoder’s semantics: replaces any prior attachment, preserved across reset(), no dimension check at attach time (it lives on append_image where we have a real input to size against).

pub fn attach_gpu_vision_encoder(&mut self, encoder: Arc<dyn VisionGpuEncode>)

Attach a cached GPU vision encoder. When present, Self::append_image_with_opts runs the ViT on the GPU for patch grids within the GPU kernel’s capacity (crate::model::vision_encoder_gpu::MAX_VIT_TOKENS), falling back to the CPU vision_encoder otherwise. The CPU encoder must still be attached via Self::attach_vision_encoder (it backs the fallback and the capability/dimension checks). Preserved across reset().

pub fn attach_lora_adapters( &mut self, adapters: Arc<LoraAdapterWeights>, ) -> Result<(), CeraError>

Attach a LoRA adapter (from crate::lora::LoraAdapterWeights). It’s applied to every subsequent forward pass — generation and hidden-states extraction — until replaced or removed. Replaces any prior adapter (hot-swap) and is preserved across Self::reset. Load the adapter once and share the Arc across sessions.

The adapter’s dimensions are validated against the model up front, so an adapter built for a different model is rejected with CeraError::LoraDimMismatch rather than silently corrupting output. An adapter that fits the model but targets something this backend cannot apply is rejected separately, with CeraError::LoraUnsupportedByBackend, since that one is fixed by changing backend rather than by changing the adapter.

Note: this only affects tokens processed after the call — it does not retroactively re-adapt KV already in the cache. Attach before prefilling the context you want adapted (or Self::reset first).

pub fn remove_lora_adapters(&mut self)

Remove any attached LoRA adapter, returning to base-model inference.

pub fn has_lora_adapters(&self) -> bool

Whether a LoRA adapter is currently attached.

pub fn set_image_max_long_size(&mut self, max_long_size: Option<u32>)

Set the session-default cap on the longest side of an appended image, in pixels (None = no cap). Every image-append path honors it — including Self::append_chat_with_images, the recommended multimodal path — so callers can bound image-encode cost once instead of per call. Self::append_image_with_opts takes an explicit per-call override. See that method for the cap semantics (shrinks the encoded target, never upscales, takes precedence over the model’s image_min_pixels floor).

pub fn capabilities(&self) -> ModalityCapabilities

What this session accepts as input / emits as output. Derived from the model’s inference_type at construction — see ModalityCapabilities::from_inference_type for the mapping.

pub fn tokenizer(&self) -> &BpeTokenizer

Borrow the tokenizer the session was constructed with. Useful for callers (tests, FFI wrappers) that want to encode / decode without threading the tokenizer through separately.

pub fn tokenizer_arc(&self) -> Arc<BpeTokenizer>

Clone the Arc-wrapped tokenizer the session was constructed with.

pub fn model(&self) -> &dyn Model

Borrow the model the session was constructed with. Primarily for introspection (vocab size, max_seq_len, etc.); hot-path forward calls still go through Session’s own methods.

pub fn position(&self) -> u32

Current KV position — tokens live. Atomic; safe from any thread.

pub fn position_handle(&self) -> Arc<AtomicU32>

Shared handle to the position counter. Clone into another thread to watch an in-flight generate’s progress without holding &self (which would block on generate’s &mut self borrow).

pub fn cancel_handle(&self) -> Arc<AtomicBool>

Shared handle to the cancel flag. Clone it into another thread and call .store(true, Relaxed) to interrupt an in-flight generate. The convenience cancel() method does the same for the owning thread.

pub fn cancel(&self)

Flip the cancel flag. Safe from any thread.

pub fn reset(&mut self) -> Result<(), CeraError>

Clear KV state and reset position to 0. Rebuilds the sampler from SessionConfig::seed so a seeded session is fully reproducible after reset. Does NOT touch the engine-level disk prefix cache (which lives on CeraEngine, not Session).

pub fn hidden_size(&self) -> usize

The model’s hidden dimension D — the per-token width of the vectors returned by Self::hidden_states_for_tokens. Callers reshape the flattened [T*D] result into [T][D] with this.

pub fn last_logits(&self) -> Option<&[f32]>

The [vocab_size] logits for the last processed token, i.e. the next-token distribution the sampler draws from. Some immediately after a successful Self::append_tokens / Self::append_text; None before any input has been appended, and cleared to None on a cancelled or partial prefill and on a context shift. Read it right after an append_* call — e.g. backend parity checks comparing the same prompt’s logits across --device cpu vs metal.

pub fn hidden_states_for_tokens( &mut self, tokens: &[u32], ) -> Result<Vec<f32>, CeraError>

Extract the model’s per-token last-layer hidden states (post-final RMSNorm, matching llama.cpp --pooling none) for tokens.

Returns a flattened row-major [n_tokens * hidden_size] buffer (token t, channel c at t * hidden_size + c). This is a side-effect-free one-shot prefill: it uses a reused, prompt-sized scratch state and does NOT advance or disturb the session’s generation KV, so it composes with an already-primed conversation.

Errors: CeraError::EmptyInput on empty input; CeraError::UnsupportedModality if the backend doesn’t implement hidden-state extraction (probe via Model::supports_hidden_states); CeraError::InvalidToken if any id is >= vocab_size.

pub fn hidden_states_mean_pooled( &mut self, tokens: &[u32], ) -> Result<Vec<f32>, CeraError>

Like Self::hidden_states_for_tokens but mean-pools over tokens, returning a single [hidden_size] vector. This is the common classifier path (their head consumes the mean-pooled hidden state) and avoids shipping the full [T*D] matrix across an FFI/WASM boundary.

pub fn hidden_states_for_text( &mut self, text: &str, ) -> Result<Vec<f32>, CeraError>

Tokenize text and return its per-token hidden states. Convenience over Self::hidden_states_for_tokens (Swift hiddenStates(for:)).

pub fn append_text(&mut self, text: &str) -> Result<(), CeraError>

Tokenize text and append. Convenience over append_tokens.

pub fn append_audio( &mut self, samples: &[f32], sample_rate: u32, ) -> Result<(), CeraError>

Append PCM audio input to the session’s context. Runs the samples through the attached audio encoder (mel + Conformer + MLP adapter) and prefills the resulting per-frame hidden states into KV via Self::append_embeddings.

samples is f32 PCM in [-1, 1], mono. Non-16kHz inputs are automatically linearly resampled to 16 kHz.

Errors are checked in the order listed. The first matching condition wins — e.g. an empty samples buffer paired with an audio-incapable session returns UnsupportedModality, not EmptyInput.

1. CeraError::UnsupportedModality when the loaded model doesn’t support audio input (Self::capabilities.audio_in == false).
2. CeraError::Backend with a “no encoder attached” message when the model supports audio but Self::attach_audio_encoder hasn’t been called yet.
3. CeraError::Backend when the attached encoder’s llm_hidden_size doesn’t match the LLM’s hidden_size (wrong-bundle encoder).
4. CeraError::EmptyInput when samples is empty or the audio is too short to produce any encoder frames (less than one window after center-padded STFT). Non-16kHz inputs are automatically resampled to 16kHz.
5. CeraError::ContextOverflow / CeraError::Cancelled propagated from the underlying Self::append_embeddings call.

pub fn append_tokens(&mut self, tokens: &[u32]) -> Result<(), CeraError>

Append raw token IDs, running a prefill pass from the current position over just the new tail.

Prefill runs through Model::forward_prefill_chunked with SessionConfig::ubatch_size so long prompts can be cancelled mid-flight. Returns CeraError::Cancelled when cancel fires before the full slice is consumed, or CeraError::InvalidToken if any token ID is >= vocab_size.

On cancellation:

• Tokens already fed through the kernel stay in KV; position() advances to reflect how many were actually consumed.

• last_logits is cleared (not set to the partial-prefill logits). This forces a subsequent generate() to return EmptyInput rather than silently producing tokens from mid-prompt state. The caller’s contract is to clear the flag via Self::clear_cancel and resume by appending the unconsumed tail before generating. Sketch:

  let before = session.position() as usize;
  match session.append_tokens(&tokens) {
      Err(CeraError::Cancelled) => {
          let consumed = session.position() as usize - before;
          session.clear_cancel();
          session.append_tokens(&tokens[consumed..])?;
      }
      other => other?,
  }

pub fn append_embeddings( &mut self, embeddings: &[f32], n_tokens: usize, ) -> Result<(), CeraError>

Append a sequence of pre-computed hidden-dim embeddings — the soft-token analog of Self::append_tokens.

Each row of embeddings is fed straight into the model at the LLM’s input-embedding stage, bypassing the embed_tokens lookup. Used for non-text input modalities where an external encoder produces hidden states directly (e.g. the LFM2A audio encoder’s per-frame output via crate::model::audio_encoder::encode_audio_pcm).

embeddings is a flat row-major buffer of length n_tokens * hidden_size. Position advances by n_tokens. Returns Err(EmptyInput) for n_tokens == 0, Err(Backend(...)) for shape mismatch.

Mirrors append_tokens’s context-shift logic (n_keep-aware) and last_logits semantics. Cancellation is checked between ubatch_size-sized chunks (granularity: one chunk); on cancel, last_logits is cleared and Err(Cancelled) is returned with the frames processed so far still in KV (caller can clear_cancel and resume from position() like append_tokens).

Dispatches to Model::forward_prefill_from_embeddings in ubatch-sized chunks, mirroring Model::forward_prefill_chunked for tokens. Backends with a true batched embedding-prefill path (CPU Lfm2Model) process a whole chunk per call and amortize per-layer GEMM dispatch across frames; the trait default falls back to a per-frame forward_from_embedding loop, preserving correctness for backends that haven’t overridden.

pub fn clear_cancel(&self)

Clear the cancel flag. Call this after handling a CeraError::Cancelled from append_tokens / generate when you want to resume work on the same session (append more tokens, generate again) without rebuilding it via Self::reset.

pub fn append_image(&mut self, bytes: &[u8]) -> Result<(), CeraError>

Append an image input. Decodes PNG / JPEG bytes, resizes to the encoder’s native input size, normalises with the encoder’s per-channel mean / std, runs the ViT + projector forward to produce 64 image tokens × projection_dim, and splices them into the LLM prefill stream at the current position via Self::append_embeddings.

Placement matters. The model was trained on a specific surrounding-token envelope (LFM2-VL: <|image_start|> / <|image_end|> inside the user-turn opening <|im_start|>user\n…<|im_end|> block). Calling append_image at the wrong stream position — before the <bos> token, outside the user turn, or without the model-specific markers — leaves the LLM unable to interpret the embeddings as visual content; the visible failure mode is a generic non-image-conditioned description (e.g. “I see a complex and abstract scene with various shapes…”).

Recommended path: Self::append_chat_with_images handles render + marker-walk + envelope splice in one call. It’s the right path for any LFM2-VL inference driven by the standard chat template.

Use this method directly only when you need the manual splice — e.g. a non-LFM2-VL model with a different envelope convention, or custom token routing the helper doesn’t support. Manual recipe:

let img_start = tokenizer.special_token_id("<|image_start|>")?;
let img_end   = tokenizer.special_token_id("<|image_end|>")?;
session.append_tokens(&prefix_tokens)?;          // BOS + <|im_start|>user\n
session.append_tokens(&[img_start])?;
session.append_image(&jpeg_bytes)?;
session.append_tokens(&[img_end])?;
session.append_tokens(&suffix_tokens)?;          // user text + <|im_end|>\n + asst tag
session.generate(&opts, &mut sink)?;

cera/tests/vl_bundle_load.rs::vl_bundle_appends_synthetic_image is the reference integration recipe (now via the helper).

Errors:

1. CeraError::EmptyInput when bytes is empty.
2. CeraError::UnsupportedModality if the session capabilities don’t include image_in (non-VL bundle).
3. CeraError::Backend when no vision encoder is attached (text-only construction of a VL bundle, or test setup that skipped attach_vision_encoder).
4. CeraError::Backend when image decode / resize fails (corrupt PNG, unsupported format, etc.).
5. CeraError::Backend when the encoder’s projection_dim doesn’t match the LLM’s hidden_size (mismatched mmproj loaded against a different LLM).
6. CeraError::ContextOverflow / CeraError::Cancelled propagated from Self::append_embeddings.

pub fn append_image_with_opts( &mut self, bytes: &[u8], max_long_size: Option<u32>, ) -> Result<(), CeraError>

Like Self::append_image, but with an explicit per-call cap (max_long_size) on the longest side of the encoded image, overriding the session default (Self::set_image_max_long_size).

When Some(n), the resize target is shrunk (aspect-preserving, re-aligned) so its longer side is at most n pixels — a caller-controlled quality/cost knob (smaller = fewer image tokens, faster, less detail). Each dimension is floored at one aligned patch block (patch_size · scale_factor), so a very small n rounds the encoded long side up to that minimum rather than below it. The cap only ever shrinks the target (it never upscales) and takes precedence over the model’s image_min_pixels floor — passing a small n is an explicit request to trade detail for cost, down to one aligned patch block. None (or 0) applies no cap. See crate::model::vision_preprocessor::preprocess_image_with_opts.

Errors are identical to Self::append_image.

pub fn append_chat_with_images( &mut self, messages: &[ChatMessageMultimodal], images: &[&[u8]], add_generation_prompt: bool, ) -> Result<(), CeraError>

Append a multimodal chat conversation in one call: render the chat template (which emits <image> markers for each crate::tokenizer::ContentItem::Image in the messages), then walk the resulting token stream and splice in <|image_start|> + image embeddings + <|image_end|> for each marker. The images slice maps positionally onto <image> markers in render order — the order they appear in the messages’ content lists.

This is the recommended path for VL inference: it replaces the manual splicing example documented on Self::append_image. The manual path stays available for callers with non-LFM2-VL chat templates or custom token routing.

All validation runs before any Self::append_tokens / Self::append_image call, so a failed render or a marker-count mismatch leaves session state untouched.

Errors:

1. CeraError::UnsupportedModality when the session capabilities don’t include image_in (non-VL bundle).
2. CeraError::Backend when the model has no chat template, when <image> doesn’t tokenize to a single token (model isn’t VL-shaped), when the tokenizer is missing the <|image_start|> / <|image_end|> special tokens, or when the marker count doesn’t match the supplied images.len().
3. Any error from Self::append_tokens / Self::append_image propagates once splicing begins.

pub fn append_user_message( &mut self, message: &UserMessage, ) -> Result<(), CeraError>

Append a user multimodal message to the session’s context, automatically placing media in the model’s canonical order.

pub fn generate<S: ModalitySink + ?Sized>( &mut self, opts: &GenerateOpts, sink: &mut S, ) -> Result<GenerateSummary, CeraError>

Run autoregressive decode, emitting token chunks through the sink. Returns a summary with timing + finish reason. The sink also receives on_done(finish_reason) at the end; callers can treat the Result as authoritative and use on_done for UI cleanup.