tokitai-dl

Languages: English | 中文 (Simplified Chinese)

A vendored/additive bridge crate connecting two deep-learning libraries.

Position statement

tokitai-dl is a vendored, additive bridge. It provides translation code that lets tokitai-operator operator DAGs be inspected with the god-graph algorithm library. The verification/proof boundary remains in the upstream crates; this crate does not add new formal verification for graph algorithms, LLM weight loading, or tensor-ring compression.

The two upstream crates are wired as follows:

tokitai-operator — sibling path-dep (path = "../tokitai-operator") in the development manifest. The publish template switches this to a crates.io version dependency. It is not modified and not vendored.
god-graph — vendored at vendor/god-graph/. The source manifests use path = "vendor/god-graph". The vendored copy tracks god-graph v0.6.0-alpha (edition 2021, MSRV 1.85). This is a deliberate departure from the original "100% non-invasive" position stated in ADR-0001; see ADR-0004 for the rationale.

The feature surface is unchanged: if a user enables only the graph feature, they get a build of tokitai-dl that does not pull in tokitai-operator. If they enable only the operator feature, they get a build that does not pull in god-graph. Neither side is required.

Feature matrix

Feature	Path dep enabled	Use case
(none)	none	Empty crate; useful for docs and CI smoke.
`graph`	`god-graph`	Pull in `Graph<T,E>` and the algorithm library alone.
`operator`	`tokitai-operator`	Pull in verified kernels + IR types alone.
`llm`	`god-graph` via `graph`	Enable god-graph Safetensors loader support; `llm_bridge` also needs `operator`.
`mcp`	(reserved)	Reserved for future MCP-server glue.

All features are additive: turning on graph does not require operator, and vice versa. Path deps are marked optional = true so the dependency graph stays minimal when only one side is needed.

Quickstart

A minimal crates.io consumer that enables both core bridges:

[dependencies]
tokitai-dl = { version = "0.1.2", features = ["graph", "operator"] }

For Safetensors / LLM loading into a tokitai TensorStore<i64>, enable the LLM loader feature together with operator:

[dependencies]
tokitai-dl = { version = "0.1.2", features = ["llm", "operator"] }

For local development from sibling checkouts, use a path dependency explicitly:

[dependencies]
tokitai-dl = { path = "../tokitai-dl", features = ["graph", "operator"] }

Building this crate with default features (empty) is intentionally fast and is the docs.rs-safe baseline:

cargo build              # < 1s, no upstream deps
cargo build --features graph
cargo build --features operator
cargo build --features "graph operator"
cargo build --features "llm operator"

docs.rs

The default feature set is empty, so a normal docs.rs build does not need the sibling ../tokitai-operator checkout and should not enable the operator or llm feature implicitly. The release manifests keep that docs.rs configuration explicit:

[package.metadata.docs.rs]
no-default-features = true
features = []
targets = ["x86_64-unknown-linux-gnu"]

Do not use all-features = true for docs.rs unless the published package has crate-registry dependencies for every optional dependency and that combination has been checked with cargo doc --all-features --no-deps.

Layout

tokitai-dl/
  Cargo.toml
  Cargo.publish.toml
  README.md
  LICENSE
  .gitignore
  vendor/
    god-graph/        # vendored copy (236 files); see ADR-0004
  docs/
    INTEGRATION.md
    adr/
      0001-non-invasive-design.md   # superseded
      0004-vendored-god-graph.md    # active position
  src/
    lib.rs
    op_dag_graph.rs   # gated on `feature = "operator"`

License

MIT OR Apache-2.0. See LICENSE.