fv-compute 0.2.0

The FusionVault transform/compute contract: a transform is a typed function over Arrow data declared by a manifest, discovered by a registry, and run by a pluggable backend.
Documentation 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65

# fv-compute

The transform contract for [FusionVault Kinetics](https://github.com/FusionVault/kinetics). A
transform is a typed function over Arrow data, declared once in a `transform.toml`, discovered by
a registry, and run by whichever backend its `impl` names. This crate defines the manifest and
its validation, the typed signature, the capability envelope, the registry, the `Runtime`, and
the two traits a backend implements. It links no execution engine: `fv-compute-wasm` and
`fv-compute-container` are the backends that ship, and you can add your own.

## Validate a manifest

```rust
use fv_compute::{parse_and_validate, ImplKind};

let manifest = parse_and_validate(r#"
    id = "riskBand"
    version = "1.0.0"
    impl = "container"
    entry = "python3 transform.py"
    protocol = "json"
    [[inputs]]
    columns = [{ name = "id", type = "int64" }, { name = "score", type = "float64" }]
    [output]
    columns = [{ name = "id", type = "int64" }, { name = "band", type = "utf8" }]
"#).unwrap();

assert_eq!(manifest.impl_kind, ImplKind::Container);
assert!(manifest.capabilities.deterministic);   // the default envelope: pure, CPU, batch
```

## Run transforms

```rust,no_run
use fv_compute::Runtime;
# use fv_compute::{Compute, ComputeError, ImplKind, TransformBackend, TransformManifest};
# struct MyBackend;
# impl TransformBackend for MyBackend {
#     fn kind(&self) -> ImplKind { ImplKind::Wasm }
#     fn load(&self, _: &TransformManifest, _: &std::path::Path) -> Result<Box<dyn Compute>, ComputeError> { unimplemented!() }
# }
# fn batch() -> arrow::array::RecordBatch { unimplemented!() }

let runtime = Runtime::builder()
    .root("./transforms")          // every <dir>/transform.toml under it
    .backend(MyBackend)            // e.g. fv_compute_wasm::WasmBackend::new()?
    .build()?;                     // every manifest validated here

let out = runtime.run("riskBand@1.0.0", &[batch()])?;   // loaded once, cached, run per batch
# Ok::<(), Box<dyn std::error::Error>>(())
```

## What is here

- `manifest`: `TransformManifest`, `parse_and_validate`, the `impl` kinds.
- `types`: `FvType`, `ColumnSpec`, `SchemaSpec` and their Arrow bindings.
- `capability`: `CapabilityEnvelope`, `Binding`, and the single `Binding::accepts` rule.
- `registry`: `Registry`, roots (`DirRoot`, `MemoryRoot`, the `Root` trait), precedence, a
  serializable index.
- `runtime`: `Runtime`, the registry plus backends plus a cache of loaded transforms.
- `contract`: the `TransformBackend` and `Compute` traits.
- `catalog`: UI-friendly descriptors of what a registry holds, with no backend linked.

The [repository README](https://github.com/FusionVault/kinetics#readme) explains the concepts,
walks through writing a transform in Rust and Python, and shows how transforms run inside
pipelines and streams. Apache-2.0.