Skip to main content

Crate fountain_arena_operator

Crate fountain_arena_operator 

Source
Expand description

§fountain-arena-operator

An owned, contiguous-memory implementation of fountain_engine::DataOperator, with reusable vector slots and CPU arithmetic with portable fallback. It supports the Fountain 2.x operation interface and can be passed directly to its encoders and decoders.

This repository contains the operator extracted from CART for independent maintenance and review by Fountain maintainers. It can be used as a standalone crate with the Fountain 2.x engine.

§Use the operator

Add the crates.io dependencies:

[dependencies]
fountain-arena-operator = "0.1.1"
fountain_engine = "2.0.1"

The Rust import name is fountain_arena_operator. Encoding-plan caches belong to the codec or caller; this package supplies vector storage and arithmetic.

The only normal dependency is fountain_engine, which depends on smallvec. Raptor code schemes, the reference operator, and JSON output support are development dependencies. The crate builds without CART, RDMA, or native libraries.

use fountain_arena_operator::MemoryDataOperator;
use fountain_engine::{DataOperator, Operation};

let mut operator = MemoryDataOperator::with_capacity(4, 3);
operator.insert_vector(&[1, 2, 3, 4], 0);
operator.insert_vector(&[4, 3, 2, 1], 1);
operator.execute(&Operation::AddTwoToVector {
    s0: 0,
    s1: 1,
    target_id: 2,
});
assert_eq!(operator.get_vector(2), &[5, 1, 1, 5]);

To use an encoder, populate source IDs 0..K before handing over the operator:

use fountain_arena_operator::MemoryDataOperator;
use fountain_engine::{DataOperator, Encoder};
use fountain_raptor_10::Raptor10SysCode;

let code = Raptor10SysCode::new_with_default_setting(32);
let mut operator = MemoryDataOperator::new(64);
for id in 0..32 {
    operator.insert_vector(&vec![id as u8; 64], id);
}
let encoder = Encoder::new_with_operator_execute_only(&code, Box::new(operator));

Full recovery examples:

cargo run --locked --example roundtrip -- raptor10
cargo run --locked --example roundtrip -- raptorq

The RaptorQ example uses the engine’s explicit source count to cover K < K′ padding. Packet framing, repair IDs, encoding plans, and scheduling are owned by the codec and its caller.

The round-trip example constructs and runs the codec on a caller-owned thread with an 8 MiB stack. This accommodates the RaptorQ debug example on Windows, where the default main-thread stack overflowed. Applications should validate their codec stack budget for their own build and workload.

§Public API and behavior

  • MemoryDataOperator::new(width) creates an empty operator.
  • MemoryDataOperator::with_capacity(width, count) reserves vector storage.
  • operator.clear() removes all IDs and retains storage for another block of the same width. The backend and field configuration are preserved. This is an explicit caller decision, not an automatic codec pooling policy.
  • MemoryDataOperator::with_backend(width, backend) selects a concrete VectorOperationBackend for statically dispatched arithmetic.
  • CpuVectorBackend provides fused two/three-source XOR, scalar operations, and finite-field configuration.

Every live ID owns one slot. Move transfers ownership; removal and overwritten targets make slots reusable. The arena retains its high-water capacity until drop. Dense IDs use direct indexing, and larger IDs use sparse metadata.

The default CpuVectorBackend uses vectorizable Rust loops for fused XOR and multiplication by 2. General GF(256) multiplication and multiply-add select NEON on aarch64 or AVX2 on x86/x86_64 at runtime, with scalar fallback and cached field tables.

RuntimeVectorBackend is an optional bulk-vector backend. It shares the standard field arithmetic and selects AVX2 XOR at runtime for x86 slices of at least 128 bytes. Four independent 32-byte accumulators and bounded 16/8/1-byte tail handling support unaligned buffers. Other CPUs and short slices fall back to the standard loops. No target-cpu=native flag is required. Select a backend when constructing an operator with MemoryDataOperator::with_backend_and_capacity. The standard backend remains the default: whole-codec measurements found gains for packet-sized symbols but regressions for short symbols in a CART integration candidate. See the qualification report.

Storage and operation routing forbid unsafe code. Private arithmetic modules contain the SIMD intrinsics behind checked slice boundaries and CPU feature detection. Performance depends on the compiler, CPU, width, and workload.

The public contract specifies zero-length vectors, missing IDs, panic behavior, aliasing, field configuration, and memory lifetime. It also identifies three differences from fountain_utility 2.0.0::VecDataOperater: zero-coefficient MulAdd, source-aliasing BroadcastAdd, and missing arithmetic targets. See the operation contract.

§Validate and measure

Rust 1.89 is the supported compiler baseline. CI checks that version and stable.

cargo test --locked
cargo clippy --locked --all-targets -- -D warnings
cargo doc --locked --no-deps
cargo package --locked
mkdir -p results
cargo bench --locked --bench operator_comparison -- 12 > results/samples.ndjson
python3 scripts/analyze.py results/samples.ndjson --output results/summary

The benchmark verifies bytes for every measured encoder/decoder pair. It reports both identical operation replay and the full upstream codec with only the operator replaced. Historical aarch64 measurements, raw samples, timing definitions, and reproduction instructions are in the benchmark guide. The three-scope measurements separately compare operator replay, full upstream encoding/decoding and encoding-plan cache hits for Arena, Slab and Simd, with raw rounds, setup costs and host-load evidence.

The detailed operator comparison covers current Slab, Simd and Arena performance, storage and operation semantics. The GF(256) optimization results compare Arena before and after its SIMD changes.

The reports are included in the source package. Their raw benchmark archives remain in the GitHub repository and require collaborator access while that repository is private.

§Maintenance and license

See CONTRIBUTING.md for review and validation, and the release procedure for registry releases.

This crate is licensed under AGPL-3.0-only, retaining the license of the source from CART. fountain_engine has its own AGPL/commercial licensing terms. See LICENSE and provenance.

Memory initialization and explicit reuse measurements are documented in Memory reuse qualification. These APIs are available in version 0.1.1.

Structs§

CpuVectorBackend
CPU kernels with vectorizable XOR and runtime-selected GF(256) SIMD. Binary sessions allocate no field tables. Default-field tables are shared and initialized lazily; other fields are owned by the backend.
MemoryDataOperator
Owned equal-width vectors for incremental fountain codecs.
RuntimeVectorBackend
CPU backend for bulk vectors, with runtime-selected AVX2 XOR on x86.

Traits§

VectorOperationBackend
Arithmetic on equal-length, disjoint source and target slices. The operator resolves IDs and aliasing before calling the backend. A concrete backend is selected at construction so its contiguous byte kernels can be inlined.