pub struct VisionModel {
pub config: VisionConfig,
pub patch_embed: VisionLinear,
pub blocks: Vec<VisionBlock>,
pub norm: Vec<f32>,
pub aligner_w1: VisionLinear,
pub aligner_w2: VisionLinear,
pub image_start: Vec<f32>,
pub image_end: Vec<f32>,
pub image_newline: Vec<f32>,
}Expand description
Loaded V4.1 vision tower plus projector/marker embeddings.
Fields§
§config: VisionConfig§patch_embed: VisionLinear§blocks: Vec<VisionBlock>§norm: Vec<f32>§aligner_w1: VisionLinear§aligner_w2: VisionLinear§image_start: Vec<f32>§image_end: Vec<f32>§image_newline: Vec<f32>Implementations§
Source§impl VisionModel
impl VisionModel
Sourcepub fn from_model(
model: &Arc<CmfModel>,
config: VisionConfig,
) -> Result<Self, String>
pub fn from_model( model: &Arc<CmfModel>, config: VisionConfig, ) -> Result<Self, String>
Load canonical V4.1 names from a CMF model. The marker vectors are required whenever the tower is enabled because they replace the image-start/end/newline token embeddings in the text hidden state.
Sourcepub fn encode_image(
&self,
image: &ImageInput,
pool: Option<&Pool>,
) -> Result<Vec<f32>, String>
pub fn encode_image( &self, image: &ImageInput, pool: Option<&Pool>, ) -> Result<Vec<f32>, String>
Compute a vision image embedding with the official ViT and aligner.
The returned rows correspond only to IMAGE positions, in reading
order; marker vectors remain separate for the text runtime to insert.
Sourcepub fn fill_image_span(
&self,
image: &ImageInput,
span: &mut [f32],
pool: Option<&Pool>,
) -> Result<(), String>
pub fn fill_image_span( &self, image: &ImageInput, span: &mut [f32], pool: Option<&Pool>, ) -> Result<(), String>
Insert image marker/projector rows into a text hidden-state span.
span must have the exact ImageInput::types length and be laid out
with the text hidden size in its last dimension.
Sourcepub fn image_span(
&self,
image: &ImageInput,
pool: Option<&Pool>,
) -> Result<Vec<f32>, String>
pub fn image_span( &self, image: &ImageInput, pool: Option<&Pool>, ) -> Result<Vec<f32>, String>
Return one complete text-hidden image span (markers plus projected image rows) for callers that do not already own a preallocated hidden buffer. The runtime’s batched prefill can copy these rows directly over the corresponding placeholder positions.