Skip to main content

Module engine

ffai_argus

Module engine 

Source
Expand description

The VlmEngine implementation: an ImageBuffer in, a caption out.

Step 6 of docs/plans/argus-launch-plan.md. Steps 3, 4 and 5 each gated one brick against the reference in isolation; this is the one that composes them and is therefore the one where a plumbing mistake — a stale cache, a dropped alpha channel, a grid the prompt and the pixels disagree about — can appear without any brick being wrong.

§The four things this file does, in order

  1. RGB8. Every image arrives as an ImageBuffer in one of three pixel formats. The tower takes one. Converting here, once, means no other stage has to know that grayscale exists.
  2. Preprocess + tower, per tile, into a (tiles, tokens_per_tile, d) block of image embeddings.
  3. Assemble the chat turn, tokenize it, embed the ids, and splice the image embeddings over the <image> placeholders.
  4. Decode greedily (or sampled, per [Decoding]) and detokenize.

§Why the geometry is carried and not recomputed

[preprocess_rgb8] returns the tile grid it chose, and that grid is what builds the <row_r_col_c> markers. The alternative — deriving the grid twice, once for pixels and once for text — is how the two silently disagree, and the failure mode is not an error but a caption of the wrong thing.

Structs§

ArgusBytes
The three artefacts a SmolVLM load needs, supplied by the caller.
CaptionTrace
Where the time went, stage by stage, for one caption.
SmolVlm
SmolVLM-256M-Instruct on candle: SigLIP tower, pixel-shuffle connector, Llama decoder.

Constants§

MODEL
The checkpoint this engine is gated against.