Struct SmolVlm 

pub struct SmolVlm { /* private fields */ }

SmolVLM-256M-Instruct on candle: SigLIP tower, pixel-shuffle connector, Llama decoder.

Chosen in Gate 1.2 over SmolVLM2 for one decidable reason: only v1 has a published OCRBench row to be scored against, which is what makes Arm 1 an external number rather than our own claim.

Implementations

impl SmolVlm

pub fn new() -> Self

An engine reading manifests from the workspace’s models/ directory.

pub const fn with_manifest_dir(dir: PathBuf) -> Self

pub fn from_bytes(w: ArgusBytes) -> Result<Self>

Build from weights the caller already holds — no std::fs, no mmap and no manifest on this path at all.

new() and with_manifest_dir defer loading to first use, where they reach a manifest, the model cache and an mmap. A browser has none of the three, so this is the constructor a wasm build uses.

Eager, not lazy: the caller has already paid to get the bytes here, so there is nothing left to defer, and a malformed checkpoint surfaces at construction rather than on the first image.

pub fn is_loaded(&self) -> bool

Are the weights already resident?

Loading is ~1 GB of safetensors and happens once, on first use. A caller timing a caption needs to know whether that cost landed inside the measurement, because a first call that includes it is not the same event as a warm one and averaging the two describes neither.

pub fn warm(&self) -> Result<()>

Force the load now, so a later timed call does not pay for it.

Errors

Whatever loading the checkpoint would return.

impl SmolVlm

pub fn describe_image_unsplit( &self, image: &ImageBuffer, opts: &VlmOptions, ) -> Result<String>

VlmEngine::describe_image, with a stage-by-stage timing trace.

Same code path as the untraced call — it is the untraced call, with a trace threaded through — so the numbers describe what actually runs rather than a parallel implementation that exists to be measured.

Errors

Whatever describe_image would return. Caption a still with tile splitting off — one tile, not seventeen.

VlmEngine::describe_image splits, because seventeen tiles is what lets the model read fine print and a single still has all 8192 text positions to itself. That is the right default when the compute is free. It is not free everywhere.

A split still is 17 tiles / 1088 image tokens; unsplit it is 1 tile / 64 tokens, so the vision tower does roughly a seventeenth of the work. describe_video has always taken this path — not to trade quality for speed, but because a split frame caps a window at seven frames against the tower’s 8192 positions. Wasm wants it for the other reason: the browser has no threads, and 17 tiles of scalar-plus-gemm arithmetic is minutes per image.

The detail loss is bounded and known. The unsplit tile is exactly the global thumbnail the split path already computes and prepends, so this is not a different preprocessing path with its own oracle — it is the same path, stopping before the tiles. What goes is fine print and small objects; what stays is the whole-image gist the thumbnail carries.

Prefer VlmEngine::describe_image wherever the seventeen tiles are affordable.

pub fn describe_image_traced( &self, image: &ImageBuffer, opts: &VlmOptions, ) -> Result<(String, CaptionTrace)>

Trait Implementations

impl Default for SmolVlm

fn default() -> Self

Returns the “default value” for a type.

impl VlmEngine for SmolVlm

fn info(&self) -> EngineInfo

fn describe(&self, prompt: &VlmPrompt<'_>, opts: &VlmOptions) -> Result<String>

The general path: an ordered, interleaved multimodal prompt.

fn describe_video( &self, frames: &[VideoFrame], opts: &VlmOptions, ) -> Result<Vec<TimedSegment<String>>>

Video understanding over sampled frames → a timed caption track.

fn describe_image( &self, image: &ImageBuffer, opts: &VlmOptions, ) -> Result<String, Error>

One image, with VlmOptions::prompt as the instruction.