Expand description
Tokenize local text and produce padded model inputs with explicit policies.
use rusttorch_text::{EncodedSequence, Padding, TextCollator};
use rusttorch_data::Collate;
let sequences = vec![EncodedSequence::from_ids(vec![2, 3]), EncodedSequence::from_ids(vec![4])];
let batch = TextCollator::new(Padding::Longest { pad_id: 0 }, Default::default())?.collate(sequences)?;
assert_eq!(batch.input_ids.size(), [2, 2]);
assert_eq!(Vec::<Vec<i64>>::try_from(&batch.attention_mask)?, [[1,1],[1,0]]);§RustTorch Text
Prepare local text for embedding, classification and Transformer models using Hugging Face tokenizers, explicit length policies and typed loader batches. Nothing downloads a vocabulary or model implicitly.
§Installation
Enable rusttorch = { version = "0.4", features = ["text"] } to use
rusttorch::text. Direct consumers can use rusttorch-text with
download-libtorch. Advanced tokenizer types are available through
rusttorch_text::tokenizers.
§Features
The text facade feature enables local Hugging Face tokenization, padding and
token-budget batch sampling. The direct package has no default features.
download-libtorch obtains the tensor runtime; doc-only builds documentation
without native linking and cannot execute tensor operations.
§Native runtime
Tensor operations require LibTorch 2.13.0. Enable download-libtorch, or set
LIBTORCH to an extracted distribution and add its library directory to PATH
on Windows or the platform’s shared-library search path. All RustTorch packages
in one application must share the same runtime. Documentation can be checked
with cargo doc -p rusttorch-text --no-default-features --features doc-only.
§Example: Tokenize, budget and batch
use rusttorch_text::{HfTokenizer, Padding, TextCollator, TextDataset,
TokenBudgetConfig, TokenBudgetSampler, Truncation};
use rusttorch_data::{DataLoader, ResourceLimits};
fn main() -> Result<(), Box<dyn std::error::Error>> {
let limits = ResourceLimits::default();
let tokenizer = HfTokenizer::from_file(
"tokenizer.json", Truncation::Reject(512), true, limits,
)?;
println!("vocabulary entries: {}", tokenizer.vocabulary_size());
let data = TextDataset::new(vec![
"RustTorch prepares model inputs.".into(),
"A shorter sentence.".into(),
], tokenizer)?;
let batches = TokenBudgetSampler::new(data.lengths()?, TokenBudgetConfig {
max_tokens: 2048, max_samples: 16, shuffle: true, seed: 42,
..Default::default()
}, limits)?;
let collator = TextCollator::new(Padding::Longest { pad_id: 0 }, limits)?;
let mut loader = DataLoader::builder(data)
.batch_sampler(batches)
.collate(collator)
.workers(2)
.build()?;
for batch in loader.iter() {
let batch = batch?;
// All four model-input tensors are Int64 [batch, padded_length].
println!("{:?}", batch.input_ids.size());
assert_eq!(batch.attention_mask.size(), batch.input_ids.size());
}
Ok(())
}Use the padding ID from your model’s vocabulary; zero is an example, not a
universal token ID. attention_mask is one for real tokens and zero for padding.
token_type_ids preserves the tokenizer’s segment IDs, and
special_tokens_mask marks original special tokens. Original lengths remain
available separately. Padding values have zero type/special-token masks.
The budget accounts for padding: a batch costs number of samples × longest
sequence length. Every sequence must fit by itself, and max_samples also
bounds batches of empty sequences. Measured lengths must use the same tokenizer
and policy as loading; stochastic tokenizer configurations are inappropriate
for fixed-length plans and exact replay.
§Choose truncation and padding deliberately
use rusttorch_text::{EncodedSequence, Padding, TextCollator};
use rusttorch_data::Collate;
fn main() -> Result<(), Box<dyn std::error::Error>> {
let mut collate = TextCollator::new(
Padding::Fixed { pad_id: 9, length: 4 }, Default::default(),
)?;
let batch = collate.collate(vec![
EncodedSequence::from_ids(vec![2, 3]),
EncodedSequence::from_ids(vec![4]),
])?;
assert_eq!(Vec::<Vec<i64>>::try_from(&batch.input_ids)?,
[[2, 3, 9, 9], [4, 9, 9, 9]]);
Ok(())
}Truncation::Reject(n) returns an error above n; Right(n) keeps the first
n tokens after special-token insertion. Right truncation can remove a
trailing separator. Models that require preserving closing tokens should use a
suitable tokenizer/postprocessor policy and reject unexpected overlength input.
The adapter disables native padding/truncation to avoid applying two competing
length policies.
Padding::Longest selects each batch’s maximum width. Padding::Fixed rejects
longer input; it never silently truncates. An all-empty longest-padded batch has
width zero, so check whether your model accepts it.
§Distributed sampling and resume
TokenBudgetConfig includes replicas, rank, epoch-based seeded shuffle and
rank-tail policy. It delegates index partitioning to the shared distributed
sampler. Equal sample counts do not guarantee equal batch counts when
sequence lengths differ. Synchronized training must handle uneven iteration
explicitly.
TokenBudgetSampler implements the shared batch-source checkpoint contract.
Its versioned state records length/configuration identity and the committed
sample/batch boundary; incompatible restoration fails. This does not make an
arbitrary external text source or stochastic tokenizer replay-safe. Exact
loader resume additionally requires the existing source/transform contracts.
§Limits and evidence
Text/vocabulary files, input strings, sequence lengths and padded tensor bytes
have finite limits. Large outputs are rejected; the upstream tokenizer still
owns temporary encoding allocations. Tests cover actual WordLevel tokenization,
worker batches, padding errors, budget overflow, disjoint ranks and restored
remaining batches in tests/pipelines.rs. The package adds no HTTP downloader,
plugin registry or separate loader.
| Operation | Executable evidence |
|---|---|
| Local WordLevel tokenization, workers and masks | tokenizer_to_worker_loader_produces_ids_masks_and_budget_batches |
| Explicit padding and invalid sequence rejection | padding_policies_and_limits_reject_invalid_sequences |
| Resume a nonzero epoch into a fresh sampler | token_budget_resume_replays_only_remaining_complete_batches |
| Disjoint distributed rank indices | distributed_token_batches_preserve_disjoint_rank_indices |
Re-exports§
pub use tokenizers;
Structs§
- Encoded
Sequence - One unpadded sequence, including post-processor special tokens.
- HfTokenizer
- Adapter over a locally configured Hugging Face tokenizer.
- Text
Batch - Model-ready integer tensors and original unpadded lengths.
- Text
Collator - Stateless deterministic collator for encoded sequences.
- Text
Dataset - Local strings encoded lazily during map-style loading.
- Token
Budget Config - Token-budget and distributed-rank policy. Padded token cost is
batch_size * longest_sequence, and every sequence must fit by itself. - Token
Budget Sampler - Checkpointable variable-size batches over measured sequence lengths. Distributed ranks receive equal sample counts, but can receive different batch counts; synchronized training must account for uneven batch counts.
- Token
Budget State - Versioned token-budget checkpoint containing immutable sequence lengths.
Enums§
- Padding
- Right-padding policy; long sequences are rejected, never silently truncated.
- Text
Error - Text processing failure with preserved native source chains.
- Truncation
- Explicit right-truncation policy. Truncation occurs after special-token insertion and can remove trailing special tokens; use tokenizer-native postprocessing when a model requires preserving a closing separator.
Type Aliases§
- Result
- Result returned by text operations.