# ic-host-tools
Shared Internet Computer host artifact tooling.
The workspace contains one host-only Rust library with bounded file reads,
streaming SHA-256 identities, core Wasm inspection, admitted host-tool execution,
Candid extraction, verified gzip-tar member selection, Git provenance observations
and bounded ICP CLI response decoding.
Canic and IcyDB have not yet adopted this library.
The intended boundary covers neutral Wasm/tool verification, artifact inspection,
Candid extraction, build provenance, and shared ICP command/configuration
mechanics. Consumer optimization policy, runtime code, deployment orchestration,
and release decisions stay with their products. [The extraction contract](docs/extraction.md)
records the source evidence and adoption requirements.
## Development
Install Rust through rustup, Git, GNU Make, Bash 3.2 or newer, and a SHA-256
utility. The pinned toolchain is Rust 1.99.0; the initial MSRV is 1.88.0.
On macOS, `shasum -a 256` qualifies as the checksum utility and native
`/bin/bash` is supported.
Developer setup uses `cargo-sort` 2.1.4, recorded in
[the local tool pins](ci/tool-versions.env). Provision it explicitly with
`cargo install cargo-sort --version 2.1.4 --locked`, then run
`make install-hooks` once per clone. Setup enables repository-local staged-file
formatting, handles logical/physical repository aliases and refuses
to replace existing hook obligations. `make fmt` sorts workspace manifests before
rustfmt, and `make fmt-check` independently checks both. Hooks reject partially
staged files and preserve unrelated edits. `make hooks-tools-check` exercises
the actual targets in isolated fixtures, without commits, builds or network.
`make help` lists commands. Use `make fmt`, `make check`, `make clippy`,
and `make docs-check` for focused validation. Prepare the selected dependency
cache with `cargo fetch --locked --offline` before offline validation.
Run module-filtered tests for the boundary being changed:
```bash
cargo test -p ic-host-tools --lib --locked --offline artifact::
cargo test -p ic-host-tools --lib --locked --offline archive::tests::
cargo test -p ic-host-tools --locked --offline wasm::tests::
cargo test -p ic-host-tools --lib --locked --offline tool::tests::
cargo test -p ic-host-tools --lib --locked --offline tool::resolution::tests::
cargo test -p ic-host-tools --lib --locked --offline candid::tests::
cargo test -p ic-host-tools --lib --locked --offline provenance::tests::
cargo test -p ic-host-tools --lib --locked --offline response::tests::
cargo test -p ic-host-tools --test consumer_contracts --locked --offline
```
`make ci` is the full gate,
requiring an explicit request outside configured CI.
Configured CI prepares the workspace's locked dependency cache before any
offline release fixtures or library checks. Installing `cargo-edit` prepares
that tool's dependencies, not this workspace's complete lockfile. Native and
MSRV validation run offline after the explicit cache preparation step.
See [agent rules](AGENTS.md), [host support](docs/hosts.md), and
[the current handoff](docs/status/current.md). Shared tooling is vendored at an
exact reviewed revision; normal checks need no sibling checkout.
The library does not install tools or select versions, credentials, or targets.
## Artifact inspection
`artifact::read_file` bounds file reads independently of metadata and uses
fallible allocation. `artifact::hash_reader` and `verify_reader` use constant
working memory and read at most one byte beyond the caller's allowance to
detect overflow. `hash_file` hashes regular files without loading them.
`Sha256Digest` accepts exactly 64 lowercase hexadecimal bytes and records raw
SHA-256 identities without selecting a product's pins or wire format.
`artifact::read_opened_file` consumes an already-open regular file, validates
its complete metadata length, then bounds the read independently. It starts at
the current cursor; open afresh or rewind to read the whole file. It preserves
the selected descriptor through pathname replacement, not immutable contents.
Callers retain opening, confinement, permission and concurrent-writer policy.
On Unix, `artifact::read_file_no_follow` opens with final-component symlink
rejection and nonblocking flags, then applies the same regular-file boundary.
A FIFO without a writer is rejected without waiting for one. Ancestor symlinks
are followed, so this does not replace a confined filesystem owner such as
`ic-query`'s capability-based opening. Existing `read_file` and `hash_file`
continue to follow symlinks. Filesystem deadlines remain caller-owned.
The read-only Unix example prints the bounded file's byte count and SHA-256:
```bash
cargo run -p ic-host-tools --example inspect_regular_file --locked --offline -- \
path/to/artifact.wasm 16777216
```
`wasm::inspect` uses `wasmparser` for core module framing and reports exact code
and data payload sizes, defined-function/data-segment counts, export kinds and
indices, and borrowed custom sections. Callers must supply module-byte,
section-count, export-count, and custom-section-count limits. Duplicate custom
sections are retained in encounter order. Product-specific Candid selection,
method classification, transform comparisons, and acceptance budgets stay with
the consumer. No gzip size or optimization policy is selected here.
Inspection is not full Wasm validation: it decodes the reported vectors and
module framing, while unreported section contents, instructions, feature
admission, and type/index validity require the consumer's validator. Digest
verification identifies the bytes read; it does not freeze a path for execution
or verify a tool version. File paths must be selected from a caller-controlled
filesystem, and blocking-reader timeouts remain caller-owned.
The example reads an existing artifact without modifying it. All limits and the
optional admitted digest are explicit command inputs:
```bash
cargo run -p ic-host-tools --example inspect_artifact --locked --offline -- \
path/to/artifact.wasm 16777216 1000 1000 1000
```
The numbers above are example policy choices, not library defaults. Append a
64-character lowercase SHA-256 digest to verify the artifact before inspection.
## Verified archive members
`archive::extract_tar_gz` requires the compressed archive digest, exact relative
POSIX member name, selected payload digest, and explicit limits for compressed
bytes, complete inflated tar bytes, raw records and selected payload bytes.
It verifies compressed bytes before decoding, checks gzip integrity, scans the
complete tar for duplicates, and verifies the selected bytes before returning
them with both identities. `artifact::read_reader` supplies bounded owned
stream storage with typed read/limit/allocation failures.
The API accepts one gzip stream and ordinary regular files/directories with
raw GNU or USTAR headers. It rejects links, special/sparse files, GNU long-name
records, PAX extensions, concatenated gzip streams and trailing data. Names are
matched exactly; archive paths, modes, owners and timestamps are never applied
to the filesystem. Compressed input, inflated tar and selected payload may all
be resident together under their caller-selected bounds.
This is the shared archive/member verification step, not a downloader or
installer. Consumers retain distribution pins, HTTPS transport, staging,
executable version admission and publication. Qualify the real distribution's
format before adoption; synthetic fixtures do not establish Binaryen archive
compatibility on any host.
```bash
cargo run -p ic-host-tools --example inspect_archive --locked --offline -- \
/path/to/tool.tar.gz "$ARCHIVE_SHA256" package/bin/tool "$MEMBER_SHA256" \
67108864 268435456 10000 67108864
```
The example only prints identities; the numbers are explicit example choices.
It never writes extracted files or executes their bytes.
## Verified tools and Candid
On supported Unix hosts, `tool::resolve_executable` selects a canonical absolute
candidate from a requested path/name, an explicit absolute working directory
and an ordered directory list. Requests containing `/`, including `./tool`,
are literal paths. Bare names search only the supplied list; relative or empty
entries use the supplied working directory. No ambient PATH, HOME, default
installation directory or shell expansion is added. Search skips missing,
nonregular and nonexecutable candidates, then stops on other filesystem errors.
Symlinks are followed using filesystem semantics rather than lexical cleanup.
Selection checks Unix execute bits, not digest/version authority or effective
user access. An admission failure must not trigger selection of a later tool.
The read-only example resolves a local name/path without executing it:
```bash
cargo run -p ic-host-tools --example resolve_tool --locked --offline -- \
wasm-opt /absolute/workdir /explicit/tool-directory /other/tool-directory
```
Consumers select the name/path, ordered directories and any ambient configuration
they explicitly admit. `tool::AdmittedTool::admit` still requires an absolute
executable path, its admitted digest and byte bound, explicit version arguments,
and the exact expected UTF-8 version identity. It hashes before executing and
requires a successful version exit. Every subsequent `run` rechecks executable
bytes/permission. Callers supply an absolute working directory, the complete
environment, stdout/stderr limits, and a positive capture deadline. Ambient
environment is cleared, stdin is null, and arguments are passed directly.
Capture drains both streams through nonblocking pipes. On overflow or deadline,
it closes the pipes, kills/reaps the direct child when necessary, and returns
typed failure evidence containing bounded output prefixes and cleanup results.
Formatting evidence/errors omits captured bytes, arguments, and environment.
Consumers can inspect and handle the retained raw bytes explicitly. Calls never
retry. An unsuccessful or interrupted command may have had external effects;
retry/reconciliation policy remains with the consumer.
This is not a sandbox or process-tree supervisor. Descendants remain caller-owned.
File verification and execution are separate operations, so callers must control
the tool/source directories and exclude concurrent writers. Digest checks do not
admit dynamic libraries or interpreters. The capture deadline covers execution
and pipe EOF; filesystem verification, spawn syscalls and kill/reap may take
longer. Portable pipe flags are handled through `rustix`; native macOS evidence
is still required.
`candid::extract` calls an admitted extractor with one absolute Wasm path,
bounds source hashing and output, and rejects observed source changes after
successful extraction. It retains source/tool identities and original process
bytes alongside normalized text. `candid::normalize` matches Canic's line contract:
trim trailing Unicode whitespace on each line, emit LF, preserve blank lines.
Its bound covers both input and normalized output, including added newlines.
Candid grammar, expected methods, and sidecar publication stay with consumers.
Select a governed read-only extractor: source-change detection does not restore
files changed by a misbehaving tool or replace filesystem isolation.
The runnable example requires every authority and bound explicitly and uses an
empty environment. Its version command is `--version`; the library itself has
no implicit version arguments. It prints normalized Candid without creating a
sidecar:
```bash
cargo run -p ic-host-tools --example extract_candid --locked --offline -- \
/absolute/candid-extractor "$ADMITTED_SHA256" 'candid-extractor VERSION' \
/absolute/input.wasm /absolute/workdir \
1048576 16777216 1048576 65536 5000
```
The numbers are example consumer choices in this order: executable bytes,
source bytes, stdout bytes, stderr bytes, and capture deadline in milliseconds.
Set the digest and exact version from your reviewed tool authority before
running against a real extractor. Deterministic test fixtures do not qualify an
installed extractor or a production tool pin.
## ICP CLI response decoding
`response::decode` decodes a caller-selected format into opaque bytes with
explicit `ResponseLimits` for complete input and decoded output. It supports
the top-level JSON `response_bytes` string used by Canic, and the plain/labeled
hex used by IcyDB. `serde_json` owns JSON grammar and escapes; unknown metadata
is skipped without building a value tree. Parsing storage is bounded by the
input allowance, and decoded output is reserved fallibly after validation.
Typed errors report categories and positions without response contents.
JSON hex is compact; text hex permits Rust's ASCII whitespace set. The labeled
format requires `response (hex):` at the start after leading whitespace and
rejects preambles or repeated labels. Duplicate JSON response fields, malformed
or truncated input and allowance overflow are rejected. JSON empty hex returns
empty bytes; text hex requires digits. Candid decoding, endpoint/result schemas,
command/version admission, capture limits and reconciliation remain caller-owned.
This API performs no process or network effects.
The local example reads a saved CLI response and prints only its decoded byte
count and raw SHA-256 identity:
```bash
cargo run -p ic-host-tools --example inspect_response --locked --offline -- \
/path/to/saved-response.json json 1048576 524288
```
Select `json`, `hex` or `labeled` explicitly. The limits are illustrative caller
choices. A decoded payload is not proof of successful canister execution or
permission to replay a request.
## Git provenance observations
On Unix, `provenance::capture_git` uses an admitted Git executable to observe
the HEAD object ID, HEAD tree ID and raw porcelain-v1 NUL-terminated status.
Untracked-file and submodule scope are explicit `StatusOptions`; no default
product policy is supplied. The result retains executable identity, bounded
stdout/stderr from each query and the raw status byte count/SHA-256. Pathname
bytes need not be UTF-8, and formatting omits captured bytes. Typed failures
retain earlier captures and the failed query's available evidence.
Commands disable optional locks and the filesystem monitor. Each query has
its own output bounds/deadline and runs once. They are separate observations,
not an atomic source snapshot or canonical dirty-tree identity. Consumers own
Git configuration/environment admission, repository selection, exclusion of
concurrent source/ref changes, incomplete-provenance policy and report schemas.
An empty status means no changes under the selected scope, including Git's
ignored-file rules; it does not prove all source files are unchanged.
The example selects an empty environment and prints facts without raw paths:
```bash
cargo run -p ic-host-tools --example inspect_git --locked --offline -- \
/absolute/git "$ADMITTED_SHA256" 'git version EXACT' /absolute/worktree \
67108864 1048576 65536 5000 all none
```
The bounds are example choices: executable bytes, per-query stdout/stderr
bytes and deadline milliseconds. The final two arguments choose untracked
scope (`no`, `normal`, `all`) and submodule scope (`none`, `untracked`, `dirty`,
`all`). Supply your reviewed digest/version authority before using a real Git.
## Adoption and publication
Maintainers can use `make release-patch`, `make release-minor`, or
`make release-major` to validate, update Cargo metadata and release notes,
commit, tag and atomically push the selected branch/tag. Defaults are `main`
and `origin`. Rerun the same target to reconcile an interrupted release at its
saved version; `make release-resume VERSION=X.Y.Z` selects that version explicitly.
These commands retain build outputs and evidence; crate publication remains a
separate action. See [release instructions](docs/releasing.md) for prerequisites,
the offline gate, exact effects and recovery. `make release-tools-check` checks
the workflow in isolated fixtures without publishing anything.
`make package` verifies the packaged crate offline, allowing local changes for
review. `make publish-dry-run` performs Cargo's registry checks without uploading;
it may access crates.io and also allows local changes. `make publish-check`
checks the publication source without uploading: the checkout must be clean,
Cargo and lockfile versions must agree, and the matching annotated `vX.Y.Z`
tag must identify HEAD. `make publish` applies that gate, then invokes Cargo
once with locked dependencies and the explicit crates.io registry. Provision
Cargo credentials yourself; the tooling never installs or prints them.
Creating this repository does not change any consumer dependency. Adoption
requires a concrete API, a reviewed dependency update in each consumer, removal
of superseded implementations, and focused behavioral evidence.
The manifest permits crates.io publication. Publish from a clean checkout at
the annotated tag matching the Cargo version; `make publish-check` verifies
that source contract without uploading. The maintainer's `v0.1.3` release
includes the publication tooling; the earlier `v0.1.2` tag retains disabled
publication. Source is hosted publicly at
[dragginzgame/ic-host-tools](https://github.com/dragginzgame/ic-host-tools).
The [extraction contract](docs/extraction.md) records the audited consumer
sources and the distinction between implementation and downstream adoption.