Skip to main content

Crate rusttorch_text

Crate rusttorch_text 

Source
Expand description

Tokenize local text and produce padded model inputs with explicit policies.

use rusttorch_text::{EncodedSequence, Padding, TextCollator};
use rusttorch_data::Collate;
let sequences = vec![EncodedSequence::from_ids(vec![2, 3]), EncodedSequence::from_ids(vec![4])];
let batch = TextCollator::new(Padding::Longest { pad_id: 0 }, Default::default())?.collate(sequences)?;
assert_eq!(batch.input_ids.size(), [2, 2]);
assert_eq!(Vec::<Vec<i64>>::try_from(&batch.attention_mask)?, [[1,1],[1,0]]);

§RustTorch Text

Prepare local text for embedding, classification and Transformer models using Hugging Face tokenizers, explicit length policies and typed loader batches. Nothing downloads a vocabulary or model implicitly.

§Installation

Enable rusttorch = { version = "0.4", features = ["text"] } to use rusttorch::text. Direct consumers can use rusttorch-text with download-libtorch. Advanced tokenizer types are available through rusttorch_text::tokenizers.

§Features

The text facade feature enables local Hugging Face tokenization, padding and token-budget batch sampling. The direct package has no default features. download-libtorch obtains the tensor runtime; doc-only builds documentation without native linking and cannot execute tensor operations.

§Native runtime

Tensor operations require LibTorch 2.13.0. Enable download-libtorch, or set LIBTORCH to an extracted distribution and add its library directory to PATH on Windows or the platform’s shared-library search path. All RustTorch packages in one application must share the same runtime. Documentation can be checked with cargo doc -p rusttorch-text --no-default-features --features doc-only.

§Example: Tokenize, budget and batch

use rusttorch_text::{HfTokenizer, Padding, TextCollator, TextDataset,
    TokenBudgetConfig, TokenBudgetSampler, Truncation};
use rusttorch_data::{DataLoader, ResourceLimits};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let limits = ResourceLimits::default();
    let tokenizer = HfTokenizer::from_file(
        "tokenizer.json", Truncation::Reject(512), true, limits,
    )?;
    println!("vocabulary entries: {}", tokenizer.vocabulary_size());
    let data = TextDataset::new(vec![
        "RustTorch prepares model inputs.".into(),
        "A shorter sentence.".into(),
    ], tokenizer)?;
    let batches = TokenBudgetSampler::new(data.lengths()?, TokenBudgetConfig {
        max_tokens: 2048, max_samples: 16, shuffle: true, seed: 42,
        ..Default::default()
    }, limits)?;
    let collator = TextCollator::new(Padding::Longest { pad_id: 0 }, limits)?;
    let mut loader = DataLoader::builder(data)
        .batch_sampler(batches)
        .collate(collator)
        .workers(2)
        .build()?;
    for batch in loader.iter() {
        let batch = batch?;
        // All four model-input tensors are Int64 [batch, padded_length].
        println!("{:?}", batch.input_ids.size());
        assert_eq!(batch.attention_mask.size(), batch.input_ids.size());
    }
    Ok(())
}

Use the padding ID from your model’s vocabulary; zero is an example, not a universal token ID. attention_mask is one for real tokens and zero for padding. token_type_ids preserves the tokenizer’s segment IDs, and special_tokens_mask marks original special tokens. Original lengths remain available separately. Padding values have zero type/special-token masks.

The budget accounts for padding: a batch costs number of samples × longest sequence length. Every sequence must fit by itself, and max_samples also bounds batches of empty sequences. Measured lengths must use the same tokenizer and policy as loading; stochastic tokenizer configurations are inappropriate for fixed-length plans and exact replay.

§Choose truncation and padding deliberately

use rusttorch_text::{EncodedSequence, Padding, TextCollator};
use rusttorch_data::Collate;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut collate = TextCollator::new(
        Padding::Fixed { pad_id: 9, length: 4 }, Default::default(),
    )?;
    let batch = collate.collate(vec![
        EncodedSequence::from_ids(vec![2, 3]),
        EncodedSequence::from_ids(vec![4]),
    ])?;
    assert_eq!(Vec::<Vec<i64>>::try_from(&batch.input_ids)?,
        [[2, 3, 9, 9], [4, 9, 9, 9]]);
    Ok(())
}

Truncation::Reject(n) returns an error above n; Right(n) keeps the first n tokens after special-token insertion. Right truncation can remove a trailing separator. Models that require preserving closing tokens should use a suitable tokenizer/postprocessor policy and reject unexpected overlength input. The adapter disables native padding/truncation to avoid applying two competing length policies.

Padding::Longest selects each batch’s maximum width. Padding::Fixed rejects longer input; it never silently truncates. An all-empty longest-padded batch has width zero, so check whether your model accepts it.

§Distributed sampling and resume

TokenBudgetConfig includes replicas, rank, epoch-based seeded shuffle and rank-tail policy. It delegates index partitioning to the shared distributed sampler. Equal sample counts do not guarantee equal batch counts when sequence lengths differ. Synchronized training must handle uneven iteration explicitly.

TokenBudgetSampler implements the shared batch-source checkpoint contract. Its versioned state records length/configuration identity and the committed sample/batch boundary; incompatible restoration fails. This does not make an arbitrary external text source or stochastic tokenizer replay-safe. Exact loader resume additionally requires the existing source/transform contracts.

§Limits and evidence

Text/vocabulary files, input strings, sequence lengths and padded tensor bytes have finite limits. Large outputs are rejected; the upstream tokenizer still owns temporary encoding allocations. Tests cover actual WordLevel tokenization, worker batches, padding errors, budget overflow, disjoint ranks and restored remaining batches in tests/pipelines.rs. The package adds no HTTP downloader, plugin registry or separate loader.

OperationExecutable evidence
Local WordLevel tokenization, workers and maskstokenizer_to_worker_loader_produces_ids_masks_and_budget_batches
Explicit padding and invalid sequence rejectionpadding_policies_and_limits_reject_invalid_sequences
Resume a nonzero epoch into a fresh samplertoken_budget_resume_replays_only_remaining_complete_batches
Disjoint distributed rank indicesdistributed_token_batches_preserve_disjoint_rank_indices

Re-exports§

pub use tokenizers;

Structs§

EncodedSequence
One unpadded sequence, including post-processor special tokens.
HfTokenizer
Adapter over a locally configured Hugging Face tokenizer.
TextBatch
Model-ready integer tensors and original unpadded lengths.
TextCollator
Stateless deterministic collator for encoded sequences.
TextDataset
Local strings encoded lazily during map-style loading.
TokenBudgetConfig
Token-budget and distributed-rank policy. Padded token cost is batch_size * longest_sequence, and every sequence must fit by itself.
TokenBudgetSampler
Checkpointable variable-size batches over measured sequence lengths. Distributed ranks receive equal sample counts, but can receive different batch counts; synchronized training must account for uneven batch counts.
TokenBudgetState
Versioned token-budget checkpoint containing immutable sequence lengths.

Enums§

Padding
Right-padding policy; long sequences are rejected, never silently truncated.
TextError
Text processing failure with preserved native source chains.
Truncation
Explicit right-truncation policy. Truncation occurs after special-token insertion and can remove trailing special tokens; use tokenizer-native postprocessing when a model requires preserving a closing separator.

Type Aliases§

Result
Result returned by text operations.