fv-compute-container 0.2.0

The container backend for fv-compute: a transform is a process or image that reads Arrow IPC (or row JSON) on stdin and writes it on stdout — for heavy, GPU or arbitrary-dependency transforms.
Documentation
  • Coverage
  • 75%
    3 out of 4 items documented0 out of 3 items with examples
  • Size
  • Source code size: 59.6 kB This is the summed size of all the files inside the crates.io package for this release.
  • Documentation size: 159.7 kB This is the summed size of all files generated by rustdoc for all configured targets
  • Ø build duration
  • this release: 15s Average build duration of successful builds.
  • all releases: 15s Average build duration of successful builds in releases after 2024-10-23.
  • Links
  • Repository
  • crates.io
  • Dependencies
  • Versions
  • Owners
  • dan-garden

fv-compute-container

The process backend for FusionVault Kinetics. A transform is any program that reads its input on stdin and writes its output on stdout. Use it where the WASM sandbox is the wrong fit: GPU work, heavy native dependencies, or an interpreter and package set you already have.

Run a process transform

use fv_compute::Runtime;
use fv_compute_container::ContainerBackend;
# fn batch() -> arrow::array::RecordBatch { unimplemented!() }

let runtime = Runtime::builder()
    .root("./transforms")            // <dir>/transform.toml with impl = "container"
    .backend(ContainerBackend::new())
    .build()?;

let out = runtime.run("riskBand", &[batch()])?;   // spawns `entry`, streams the batch through
# Ok::<(), Box<dyn std::error::Error>>(())

cargo run -p fv-compute-container --example quickstart runs this against a twenty-line Python transform in the repository.

Write a process transform

The manifest's entry is the command line, run with the transform's directory as the working directory. Two wire formats, chosen by protocol:

  • json: one JSON document in, one out. No dependencies beyond a JSON parser.

    stdin:  {"ref": "riskBand", "config": {}, "mode": "batch",
             "inputs": [{"dataset": "in", "columns": ["id", "score"], "rows": [{"id": 1, "score": 0.2}]}]}
    stdout: {"rows": [{"id": 1, "score": 0.2, "band": "low"}]}     or     {"error": "why"}
    
  • arrow-ipc (the default): an Arrow IPC stream in and out, with the batch's real schema. Faster, and the natural choice when the process links Arrow. The reference guest is src/bin/refguest.rs.

Output rows must carry exactly the declared output columns; the backend checks the result against the manifest. A non-zero exit fails the batch, with the last line of stderr as the reason. One input per transform today.

Apache-2.0.