rusttorch 0.4.0

Build, train, and run neural networks in Rust with LibTorch
Documentation
# Sequence models

RustTorch provides recurrent networks for ordered observations and attention
models for token or feature sequences. Layers register their parameters in a
`VarStore`, so a model can combine embeddings, sequence processing, prediction
heads, optimizers, and checkpoints without copying weights between components.

## Classify a time series with an LSTM

Each observation below contains three measurements. A unidirectional LSTM reads
five observations and a linear head predicts one of two classes from the final
output. The example performs a complete training update.

```rust
use rusttorch::{
    Device, Kind, Tensor,
    nn::{LinearConfig, RnnConfig, VarStore, functional},
    optim::Adam,
};

fn main() -> rusttorch::Result<()> {
    let store = VarStore::new(Device::Cpu);
    let encoder = RnnConfig::new(3, 8)
        .num_layers(2)
        .dropout(0.1)
        .batch_first(true)
        .build_lstm(&(store.root() / "encoder"))?;
    let classifier = LinearConfig::new(8, 2).build(&(store.root() / "classifier"))?;
    let mut optimizer = Adam::builder().learning_rate(0.001).build(&store)?;

    let observations = Tensor::arange(30, (Kind::Float, Device::Cpu)).reshape([2, 5, 3]) / 30.;
    let labels = Tensor::from_slice(&[0_i64, 1]);
    let (features, (_hidden, _cell)) = encoder.forward_t(&observations, None, true)?;
    let logits = classifier.forward(&features.select(1, 4))?;
    let loss = functional::cross_entropy(&logits, &labels)?;
    optimizer.backward_step(&loss)?;
    assert_eq!(logits.size(), [2, 2]);
    assert!(loss.double_value(&[]).is_finite());
    Ok(())
}
```

`RnnConfig::build_gru` creates a smaller gated encoder with one hidden-state
tensor. `build_rnn` creates a vanilla network with `RnnActivation::Tanh` or
`RnnActivation::Relu`. All three families support stacked layers, optional
biases, reverse directions, and dropout between layers. A single-layer model
has no intervening layer to which recurrent dropout can apply.

Input defaults to `[time, batch, features]`; `batch_first(true)` switches to
`[batch, time, features]`. Unbatched input is always `[time, features]`.
Hidden states remain `[layers * directions, batch, hidden_width]` in either
batched layout. For unbatched input the batch dimension is omitted.
Bidirectional output concatenates the two directions along the feature axis.
LSTM `projection(width)` reduces the output and hidden-state widths; its cell
state retains the original `hidden_size`.

## Continue a stream with explicit recurrent state

Carry the final state into the next chunk when observations belong to the same
stream. Reset to `None` at a new independent sequence. This example uses a GRU
in evaluation mode and detaches the state, which also prevents retaining the
previous chunk's autograd graph during truncated sequence training.

```rust
use rusttorch::{
    Device, Kind, Tensor,
    nn::{RnnConfig, VarStore},
};

fn main() -> rusttorch::Result<()> {
    let store = VarStore::new(Device::Cpu);
    let encoder = RnnConfig::new(3, 6)
        .batch_first(true)
        .build_gru(&store.root())?;
    let first_chunk = Tensor::ones([2, 4, 3], (Kind::Float, Device::Cpu));
    let next_chunk = Tensor::zeros([2, 2, 3], (Kind::Float, Device::Cpu));
    let (_first_features, state) = encoder.forward_t(&first_chunk, None, false)?;
    let state = state.f_detach()?;
    let (next_features, final_state) = encoder.forward_t(&next_chunk, Some(&state), false)?;
    assert_eq!(next_features.size(), [2, 2, 6]);
    assert_eq!(final_state.size(), [1, 2, 6]);
    Ok(())
}
```

The same protocol uses `(hidden, cell)` for an LSTM. Continuous forward-only
streams require a unidirectional model: a reverse direction needs future
observations and changes its result when chunk boundaries change. Packed
variable-length recurrent sequences and standalone recurrent cells are not
exposed yet; padding a recurrent sequence does not automatically stop its
state updates.

## Encode token features with attention

An encoder lets each position use information from the other visible positions.
Use a padding mask to exclude padded keys and a causal mask when predicting
later tokens must not reveal their contents. `true` means a position is blocked;
floating-point masks instead add directly to attention scores.

```rust
use rusttorch::{
    Device, Kind, Tensor,
    nn::{AttentionMask, TransformerActivation, TransformerConfig, VarStore},
};

fn main() -> rusttorch::Result<()> {
    let store = VarStore::new(Device::Cpu);
    let encoder = TransformerConfig::new(8, 2)
        .dim_feedforward(16)
        .activation(TransformerActivation::Gelu)
        .norm_first(true)
        .batch_first(true)
        .build_encoder(&(store.root() / "encoder"), 2, true)?;

    // Supply embeddings and positional information from your model.
    let token_features = Tensor::randn([2, 4, 8], (Kind::Float, Device::Cpu));
    let padding =
        Tensor::from_slice(&[false, false, false, true, false, false, true, true]).reshape([2, 4]);
    let encoded = encoder.forward_t(
        &token_features,
        AttentionMask {
            key_padding: Some(&padding),
            ..Default::default()
        },
        false,
    )?;
    assert_eq!(encoded.size(), [2, 4, 8]);
    Ok(())
}
```

Padding masks exclude keys, not query positions. Exclude padded targets from
the training loss separately. `MultiheadAttentionConfig` provides attention
without feed-forward or normalization layers; `forward_t` returns the output
and averaged attention maps, `forward_per_head_t` returns maps for each head,
and `forward_without_weights_t` computes only the output using native scaled
dot-product attention. Output-only attention contributes zero features for a
fully blocked row before its output projection. Returned attention maps use
ordinary softmax, which yields NaNs for a fully blocked row.

## Translate source features into a target sequence

The complete `Transformer` first encodes the source and then lets every decoder
layer attend to both the target prefix and the encoder's memory. Its result is
a feature vector per target position; add a linear vocabulary head and a token
loss for a language task, or a regression head for predicted measurements.

```rust
use rusttorch::{
    Device, Kind, Tensor,
    nn::{AttentionMask, TransformerConfig, TransformerMasks, VarStore},
};

fn main() -> rusttorch::Result<()> {
    let store = VarStore::new(Device::Cpu);
    let transformer = TransformerConfig::new(8, 2)
        .dim_feedforward(16)
        .num_encoder_layers(2)
        .num_decoder_layers(2)
        .batch_first(true)
        .build(&store.root())?;

    let source = Tensor::randn([2, 5, 8], (Kind::Float, Device::Cpu));
    let target_prefix = Tensor::randn([2, 3, 8], (Kind::Float, Device::Cpu));
    let source_padding = Tensor::from_slice(&[
        false, false, false, false, true, false, false, false, true, true,
    ])
    .reshape([2, 5]);
    let source_mask = AttentionMask {
        key_padding: Some(&source_padding),
        ..Default::default()
    };
    let decoded = transformer.forward_t(
        &source,
        &target_prefix,
        TransformerMasks {
            source: source_mask,
            target: AttentionMask {
                causal: true,
                ..Default::default()
            },
            memory: source_mask,
        },
        false,
    )?;
    assert_eq!(decoded.size(), [2, 3, 8]);
    Ok(())
}
```

Pass source padding to both the encoder and the decoder's memory attention.
The target's causal restriction only controls decoder self attention. During
teacher-forced next-token training, supply the target sequence shifted right
as decoder input and compare predictions with the unshifted targets.

Use `forward_t(..., true)` to enable dropout during training and `false` for
deterministic evaluation. Recurrent modules, multihead attention, encoder
layers, and encoder stacks also implement `Module`, so they can be constructed
with `Sequential::builder().layer(...)`. That interface uses zero recurrent
state, unmasked self attention, and output features only; use the explicit
methods for state, masks, or decoder memory. `Module::forward` defaults to
evaluation, while `Sequential` passes its current training mode.

## Save and inspect a sequence model

Use `rusttorch::interop::save_state_dict` and `load_state_dict` with the shared
`VarStore`, and rebuild the same configuration before loading. Recurrent names
include `encoder.weight_ih_l0`, `encoder.weight_hh_l0`, and `_reverse` suffixes.
Transformer stacks use `encoder.layers.0.self_attn.in_proj_weight`,
`decoder.layers.0.multihead_attn.in_proj_weight`, and named `linear`/`norm`
parameters. All layers, directions, and normalization parameters participate
in optimizer updates and state dictionaries.

The implemented attention interface uses equal query, key, and value feature
widths. Separate key/value widths, appended bias or zero tokens, nested tensors,
and key/value caches remain outside this API. Transformer stacks operate on
dense tensors and supply neither token embeddings nor positional encodings.
Tests currently provide CPU numerical evidence; accelerator support must be
verified on the intended hardware before making backend-specific claims.