rusttorch-text 0.4.0

Typed text pipelines for RustTorch
Documentation

RustTorch Text

Prepare local text for embedding, classification and Transformer models using Hugging Face tokenizers, explicit length policies and typed loader batches. Nothing downloads a vocabulary or model implicitly.

Installation

Enable rusttorch = { version = "0.4", features = ["text"] } to use rusttorch::text. Direct consumers can use rusttorch-text with download-libtorch. Advanced tokenizer types are available through rusttorch_text::tokenizers.

Features

The text facade feature enables local Hugging Face tokenization, padding and token-budget batch sampling. The direct package has no default features. download-libtorch obtains the tensor runtime; doc-only builds documentation without native linking and cannot execute tensor operations.

Native runtime

Tensor operations require LibTorch 2.13.0. Enable download-libtorch, or set LIBTORCH to an extracted distribution and add its library directory to PATH on Windows or the platform's shared-library search path. All RustTorch packages in one application must share the same runtime. Documentation can be checked with cargo doc -p rusttorch-text --no-default-features --features doc-only.

Example: Tokenize, budget and batch

use rusttorch_text::{HfTokenizer, Padding, TextCollator, TextDataset,
    TokenBudgetConfig, TokenBudgetSampler, Truncation};
use rusttorch_data::{DataLoader, ResourceLimits};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let limits = ResourceLimits::default();
    let tokenizer = HfTokenizer::from_file(
        "tokenizer.json", Truncation::Reject(512), true, limits,
    )?;
    println!("vocabulary entries: {}", tokenizer.vocabulary_size());
    let data = TextDataset::new(vec![
        "RustTorch prepares model inputs.".into(),
        "A shorter sentence.".into(),
    ], tokenizer)?;
    let batches = TokenBudgetSampler::new(data.lengths()?, TokenBudgetConfig {
        max_tokens: 2048, max_samples: 16, shuffle: true, seed: 42,
        ..Default::default()
    }, limits)?;
    let collator = TextCollator::new(Padding::Longest { pad_id: 0 }, limits)?;
    let mut loader = DataLoader::builder(data)
        .batch_sampler(batches)
        .collate(collator)
        .workers(2)
        .build()?;
    for batch in loader.iter() {
        let batch = batch?;
        // All four model-input tensors are Int64 [batch, padded_length].
        println!("{:?}", batch.input_ids.size());
        assert_eq!(batch.attention_mask.size(), batch.input_ids.size());
    }
    Ok(())
}

Use the padding ID from your model's vocabulary; zero is an example, not a universal token ID. attention_mask is one for real tokens and zero for padding. token_type_ids preserves the tokenizer's segment IDs, and special_tokens_mask marks original special tokens. Original lengths remain available separately. Padding values have zero type/special-token masks.

The budget accounts for padding: a batch costs number of samples × longest sequence length. Every sequence must fit by itself, and max_samples also bounds batches of empty sequences. Measured lengths must use the same tokenizer and policy as loading; stochastic tokenizer configurations are inappropriate for fixed-length plans and exact replay.

Choose truncation and padding deliberately

use rusttorch_text::{EncodedSequence, Padding, TextCollator};
use rusttorch_data::Collate;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut collate = TextCollator::new(
        Padding::Fixed { pad_id: 9, length: 4 }, Default::default(),
    )?;
    let batch = collate.collate(vec![
        EncodedSequence::from_ids(vec![2, 3]),
        EncodedSequence::from_ids(vec![4]),
    ])?;
    assert_eq!(Vec::<Vec<i64>>::try_from(&batch.input_ids)?,
        [[2, 3, 9, 9], [4, 9, 9, 9]]);
    Ok(())
}

Truncation::Reject(n) returns an error above n; Right(n) keeps the first n tokens after special-token insertion. Right truncation can remove a trailing separator. Models that require preserving closing tokens should use a suitable tokenizer/postprocessor policy and reject unexpected overlength input. The adapter disables native padding/truncation to avoid applying two competing length policies.

Padding::Longest selects each batch's maximum width. Padding::Fixed rejects longer input; it never silently truncates. An all-empty longest-padded batch has width zero, so check whether your model accepts it.

Distributed sampling and resume

TokenBudgetConfig includes replicas, rank, epoch-based seeded shuffle and rank-tail policy. It delegates index partitioning to the shared distributed sampler. Equal sample counts do not guarantee equal batch counts when sequence lengths differ. Synchronized training must handle uneven iteration explicitly.

TokenBudgetSampler implements the shared batch-source checkpoint contract. Its versioned state records length/configuration identity and the committed sample/batch boundary; incompatible restoration fails. This does not make an arbitrary external text source or stochastic tokenizer replay-safe. Exact loader resume additionally requires the existing source/transform contracts.

Limits and evidence

Text/vocabulary files, input strings, sequence lengths and padded tensor bytes have finite limits. Large outputs are rejected; the upstream tokenizer still owns temporary encoding allocations. Tests cover actual WordLevel tokenization, worker batches, padding errors, budget overflow, disjoint ranks and restored remaining batches in tests/pipelines.rs. The package adds no HTTP downloader, plugin registry or separate loader.

Operation Executable evidence
Local WordLevel tokenization, workers and masks tokenizer_to_worker_loader_produces_ids_masks_and_budget_batches
Explicit padding and invalid sequence rejection padding_policies_and_limits_reject_invalid_sequences
Resume a nonzero epoch into a fresh sampler token_budget_resume_replays_only_remaining_complete_batches
Disjoint distributed rank indices distributed_token_batches_preserve_disjoint_rank_indices