Struct CmfStreamWriter 

pub struct CmfStreamWriter { /* private fields */ }

Implementations

impl CmfStreamWriter

pub fn head_reserve_for(n_tensors: usize, avg_name: usize) -> u64

A gap that comfortably holds the head for n_tensors whose names run to avg_name bytes: the directory’s fixed records, the name pool, the envelope, and a header JSON with room for arch metadata — then doubled, because being wrong here costs a whole re-run.

pub fn new(path: impl AsRef<Path>, gap: u64) -> Result<Self, CmfError>

gap bytes are reserved for envelope + header + directory.

pub fn push( &mut self, name: &str, dtype: TensorDtype, shape: &[usize], data: &[u8], ) -> Result<(), CmfError>

Append one tensor. The payload is consumed here, so the caller can drop it immediately — that is the entire point of this writer.

pub fn push_bounded( &mut self, name: &str, dtype: TensorDtype, shape: &[usize], data: &[u8], chunk_bytes: usize, ) -> Result<(), CmfError>

Append one tensor while limiting each write syscall to chunk_bytes.

The source is normally an mmap, so this does not change the caller’s memory residency. It does make the bounded-copy contract explicit for very large auxiliary tensors (DeepSeek-V4.1’s native Engram tables are hundreds of gigabytes): the writer never asks an I/O layer to stage the whole payload as one buffer. Hashing remains one streaming pass over the borrowed bytes, and the directory entry is identical to Self::push.

pub fn with_manifest(self, path: impl AsRef<Path>) -> Result<Self, CmfError>

Start recording a sidecar manifest at path. One JSON line per tensor, flushed as it goes, plus a first line pinning the gap size.

pub fn appending_manifest( self, path: impl AsRef<Path>, ) -> Result<Self, CmfError>

Keep recording into an existing manifest — for a writer from CmfStreamWriter::resume, whose earlier lines must survive.

pub fn resume( path: impl AsRef<Path>, manifest: impl AsRef<Path>, ) -> Result<(Self, ResumeState), CmfError>

Rebuild a writer over an output file whose payloads are already on disk, from the manifest that recorded them. The file is reopened for writing without truncation and the cursor is placed after the last recorded tensor, so finish can complete a conversion that died.

pub fn mark(&mut self, note: &str) -> Result<(), CmfError>

Note a milestone in the manifest — a source shard fully consumed, say. Resume reads these back, which is what lets a restart skip work whose payloads are already in the file rather than only skipping tensors it happens to recognise by name.

pub fn tensor_count(&self) -> usize

pub fn data_len(&self) -> u64

Bytes of weight blob written so far.

pub fn finish( self, header: &CmfHeader, masks: Option<&MaskCatalog>, vocab: Option<&[u8]>, ) -> Result<(), CmfError>

Write the trailing sections, then patch the head into the reserved gap.