ic-host-tools

Shared Internet Computer host artifact tooling.

The workspace contains one host-only Rust library with bounded file reads, streaming SHA-256 identities, core Wasm inspection, admitted host-tool execution, Candid extraction, verified gzip-tar member selection, Git provenance observations and bounded ICP CLI response decoding. Consumer integration and qualification remain owned by each product.

The intended boundary covers neutral Wasm/tool verification, artifact inspection, Candid extraction, build provenance, and shared ICP command/configuration mechanics. Consumer optimization policy, runtime code, deployment orchestration, and release decisions stay with their products. The extraction contract records the source evidence and adoption requirements.

Development

Install Rust through rustup, Git, GNU Make, Bash 3.2 or newer, and a SHA-256 utility. The pinned toolchain is Rust 1.99.0; the initial MSRV is 1.88.0. On macOS, shasum -a 256 qualifies as the checksum utility and native /bin/bash is supported.

Developer setup uses cargo-sort 2.1.4, recorded in the local tool pins. Provision it explicitly with cargo install cargo-sort --version 2.1.4 --locked, then run make install-hooks once per clone. Setup enables repository-local staged-file formatting, handles logical/physical repository aliases and refuses to replace existing hook obligations. make fmt sorts workspace manifests before rustfmt, and make fmt-check independently checks both. Hooks reject partially staged files and preserve unrelated edits. make hooks-tools-check exercises the actual targets in isolated fixtures, without commits, builds or network.

make help lists commands. Use make fmt, make check, make clippy, and make docs-check for focused validation. Prepare the selected dependency cache with cargo fetch --locked --offline before offline validation. Registry dependency requirements allow compatible updates from the declared minimum versions. This workspace's committed lockfile and --locked checks retain its exact tested selection; consumers resolve their own dependency graph. make dependency-pins-check checks manifests, action references and tracked lockfiles offline under the shared pinning rules. Use make install-tools for explicit checksum-verified local jq/yq and IC-tool setup, then make tools-check for offline verification. Make and native CI select .tools/host/bin and .tools/ic/bin; interactive shells need the PATH export in local setup. Maintainer release preflight automatically prepares these pinned sets before offline validation. Ordinary checks and the validation hook require existing tools and never install them. The reviewed parser pins live in ci/tool-versions.env, and the six IC executable pins have one owner in ci/ic-tools.tsv. The library still requires explicit consumer admission; repository setup does not choose production tool identities. Run module-filtered tests for the boundary being changed:

cargo test -p ic-host-tools --lib --locked --offline artifact::
cargo test -p ic-host-tools --lib --locked --offline archive::tests::
cargo test -p ic-host-tools --locked --offline wasm::tests::
cargo test -p ic-host-tools --lib --locked --offline tool::tests::
cargo test -p ic-host-tools --lib --locked --offline tool::resolution::tests::
cargo test -p ic-host-tools --lib --locked --offline candid::tests::
cargo test -p ic-host-tools --lib --locked --offline provenance::tests::
cargo test -p ic-host-tools --lib --locked --offline response::tests::
cargo test -p ic-host-tools --test consumer_contracts --locked --offline

make ci is the full gate, requiring an explicit request outside configured CI. Configured CI prepares the workspace's locked dependency cache before any offline release fixtures or library checks. Installing cargo-edit prepares that tool's dependencies, not this workspace's complete lockfile. Native and MSRV validation run offline after the explicit cache preparation step.

See agent rules, host support, and the current handoff. Shared tooling is vendored at an exact reviewed revision; normal checks need no sibling checkout. The library does not install tools or select versions, credentials, or targets.

artifact::BoundedWriter<W> applies an inclusive output allowance to any caller-owned Write sink. It rejects oversized buffers before writing, counts short writes accurately and retains a limit-exceeded observation when an encoder hides its IO error. io::sink() gives bounded counting; SHA-256's existing Write implementation gives bounded hashing without a separate encoded copy. The caller chooses the codec and exact bytes; this library does not select a canonical JSON protocol or publish output.

use ic_host_tools::artifact::BoundedWriter;
fn encoded_length(value: &impl serde::Serialize, limit: u64) -> Result<u64, serde_json::Error> {
    let mut counter = BoundedWriter::new(std::io::sink(), limit);
    serde_json::to_writer(&mut counter, value)?;
    Ok(counter.bytes_written())
}

The hash_serialized_json example reads a bounded JSON value from stdin and prints the byte count and SHA-256 of its compact serde_json encoding. It does not print the input or retain a complete encoded output. This encoding may differ from the source bytes; product codecs remain with their owners.

printf '%s' '{"rows":["a","b"]}' | cargo run -p ic-host-tools --locked --offline \
  --example hash_serialized_json -- 4096 4096

artifact::copy_reader(source, &mut staging_sink, max_bytes) copies and hashes one stream using the same bounded traversal as reads/hashing. It observes at most one byte beyond the bound and returns distinct CopyError::Input and CopyError::Output failures. Partial output remains caller-owned on failure. Impossible reader/writer byte counts return IO InvalidData through their corresponding error variant; sink failures retain the shared WriterError when available. Reads, hashing and copying validate reader counts at one boundary. The returned identity describes bytes copied, not durability or digest admission: use a private staging sink, compare its identity with your expected digest, validate it, and let the existing publisher own flush/sync/rename and recovery.

Artifact inspection

artifact::read_file bounds file reads independently of metadata and uses fallible allocation. artifact::hash_reader and verify_reader use constant working memory and read at most one byte beyond the caller's allowance to detect overflow. hash_file hashes regular files without loading them. Sha256Digest accepts exactly 64 lowercase hexadecimal bytes and records raw SHA-256 identities without selecting a product's pins or wire format.

artifact::read_opened_file consumes an already-open regular file, validates its complete metadata length, then bounds the read independently. It starts at the current cursor; open afresh or rewind to read the whole file. It preserves the selected descriptor through pathname replacement, not immutable contents. Callers retain opening, confinement, permission and concurrent-writer policy.

On Unix, artifact::read_file_no_follow opens with final-component symlink rejection and nonblocking flags, then applies the same regular-file boundary. A FIFO without a writer is rejected without waiting for one. Ancestor symlinks are followed, so this does not replace a confined filesystem owner such as ic-query's capability-based opening. Existing read_file and hash_file continue to follow symlinks. Filesystem deadlines remain caller-owned.

The read-only Unix example prints the bounded file's byte count and SHA-256:

cargo run -p ic-host-tools --example inspect_regular_file --locked --offline -- \
    path/to/artifact.wasm 16777216

artifact::decode_gzip decodes one complete gzip member with independent compressed-input and decoded-output limits. It checks payload CRC/length and rejects concatenated members or trailing bytes, returning typed GzipError failures without partial decoded output. Valid empty members are accepted. Input and output buffers may be resident together under the caller's limits. Consumers retain digest admission, raw/compressed equality checks, Wasm validation and artifact budgets. Archive extraction uses the same decoder after verifying its compressed digest, with its existing errors preserved.

The read-only example prints compressed and decoded byte counts and digests:

cargo run -p ic-host-tools --example inspect_gzip --locked --offline -- \
    path/to/artifact.wasm.gz 16777216 67108864

The numbers are example policy choices. Decoded bytes are not written to disk.

wasm::inspect uses wasmparser for core module framing and reports exact code and data payload sizes, defined-function/data-segment counts, export kinds and indices, and borrowed custom sections. Callers must supply module-byte, section-count, export-count, and custom-section-count limits. Duplicate custom sections are retained in encounter order. Product-specific Candid selection, method classification, transform comparisons, and acceptance budgets stay with the consumer. No gzip size or optimization policy is selected here.

Inspection is not full Wasm validation: it decodes the reported vectors and module framing, while unreported section contents, instructions, feature admission, and type/index validity require the consumer's validator. Digest verification identifies the bytes read; it does not freeze a path for execution or verify a tool version. File paths must be selected from a caller-controlled filesystem, and blocking-reader timeouts remain caller-owned.

The example reads an existing artifact without modifying it. All limits and the optional admitted digest are explicit command inputs:

cargo run -p ic-host-tools --example inspect_artifact --locked --offline -- \
    path/to/artifact.wasm 16777216 1000 1000 1000

The numbers above are example policy choices, not library defaults. Append a 64-character lowercase SHA-256 digest to verify the artifact before inspection.

Verified archive members

archive::extract_tar_gz requires the compressed archive digest, exact relative POSIX member name, selected payload digest, and explicit limits for compressed bytes, complete inflated tar bytes, raw records and selected payload bytes. It verifies compressed bytes before decoding, checks gzip integrity, scans the complete tar for duplicates, and verifies the selected bytes before returning them with both identities. artifact::read_reader supplies bounded owned stream storage with typed read/limit/allocation failures.

The API accepts one gzip stream and ordinary regular files/directories with raw GNU or USTAR headers. It rejects links, special/sparse files, GNU long-name records, PAX extensions, concatenated gzip streams and trailing data. Names are matched exactly; archive paths, modes, owners and timestamps are never applied to the filesystem. Compressed input, inflated tar and selected payload may all be resident together under their caller-selected bounds.

This is the shared archive/member verification step, not a downloader or installer. Consumers retain distribution pins, HTTPS transport, staging, executable version admission and publication. For a tool bundle, select each required executable/library from the same immutable archive using its own digest. Admit every member before publishing the complete bundle, retaining failed staging and the previous installation. Each call independently verifies and inflates the archive; consumers own the aggregate staged-byte budget and durable bin/lib layout. The consumer contract exercises that boundary without choosing an installer or adding another extraction API.

The existing example qualifies a retained, pinned Linux Binaryen 132 archive; exact inputs and limits record that observation. No selected executable was run or installed. Qualify each real platform distribution before adoption: Linux decoding and synthetic bundle fixtures do not establish native macOS distribution compatibility.

cargo run -p ic-host-tools --example inspect_archive --locked --offline -- \
    /path/to/tool.tar.gz "$ARCHIVE_SHA256" package/bin/tool "$MEMBER_SHA256" \
    67108864 268435456 10000 67108864

The example only prints identities; the numbers are explicit example choices. It never writes extracted files or executes their bytes.

Verified tools and Candid

On supported Unix hosts, tool::resolve_executable selects a canonical absolute candidate from a requested path/name, an explicit absolute working directory and an ordered directory list. Requests containing /, including ./tool, are literal paths. Bare names search only the supplied list; relative or empty entries use the supplied working directory. No ambient PATH, HOME, default installation directory or shell expansion is added. Search skips missing, nonregular and nonexecutable candidates, then stops on other filesystem errors. Symlinks are followed using filesystem semantics rather than lexical cleanup. Selection checks Unix execute bits, not digest/version authority or effective user access. An admission failure must not trigger selection of a later tool.

The read-only example resolves a local name/path without executing it:

cargo run -p ic-host-tools --example resolve_tool --locked --offline -- \
    wasm-opt /absolute/workdir /explicit/tool-directory /other/tool-directory

Consumers select the name/path, ordered directories and any ambient configuration they explicitly admit. tool::AdmittedTool::admit still requires an absolute executable path, its admitted digest and byte bound, explicit version arguments, and the exact expected UTF-8 version identity. It hashes before executing and requires a successful version exit. Every subsequent run rechecks executable bytes/permission. Callers supply an absolute working directory, the complete environment, stdout/stderr limits, and a positive capture deadline. Ambient environment is cleared, stdin is null, and arguments are passed directly.

Capture drains both streams through nonblocking pipes. On overflow or deadline, it closes the pipes, kills/reaps the direct child when necessary, and returns typed failure evidence containing bounded output prefixes and cleanup results. Formatting evidence/errors omits captured bytes, arguments, and environment. Consumers can inspect and handle the retained raw bytes explicitly. Calls never retry. An unsuccessful or interrupted command may have had external effects; retry/reconciliation policy remains with the consumer.

This is not a sandbox or process-tree supervisor. Descendants remain caller-owned. File verification and execution are separate operations, so callers must control the tool/source directories and exclude concurrent writers. Digest checks do not admit dynamic libraries or interpreters. The capture deadline covers execution and pipe EOF; filesystem verification, spawn syscalls and kill/reap may take longer. Portable pipe flags are handled through rustix; native macOS evidence is still required.

candid::extract calls an admitted extractor with one absolute Wasm path, bounds source hashing and output, and rejects observed source changes after successful extraction. It retains source/tool identities and original process bytes alongside normalized text. candid::normalize matches Canic's line contract: trim trailing Unicode whitespace on each line, emit LF, preserve blank lines. Its bound covers both input and normalized output, including added newlines. Candid grammar, expected methods, and sidecar publication stay with consumers. Select a governed read-only extractor: source-change detection does not restore files changed by a misbehaving tool or replace filesystem isolation.

The runnable example requires every authority and bound explicitly and uses an empty environment. Its version command is --version; the library itself has no implicit version arguments. It prints normalized Candid without creating a sidecar:

cargo run -p ic-host-tools --example extract_candid --locked --offline -- \
    /absolute/candid-extractor "$ADMITTED_SHA256" 'candid-extractor VERSION' \
    /absolute/input.wasm /absolute/workdir \
    1048576 16777216 1048576 65536 5000

The numbers are example consumer choices in this order: executable bytes, source bytes, stdout bytes, stderr bytes, and capture deadline in milliseconds. Set the digest and exact version from your reviewed tool authority before running against a real extractor. Deterministic test fixtures do not qualify an installed extractor or a production tool pin.

ICP CLI response decoding

response::decode decodes a caller-selected format into opaque bytes with explicit ResponseLimits for complete input and decoded output. It supports the top-level JSON response_bytes string used by Canic, and the plain/labeled hex used by IcyDB. serde_json owns JSON grammar and escapes; unknown metadata is skipped without building a value tree. Parsing storage is bounded by the input allowance, and decoded output is reserved fallibly after validation. Typed errors report categories and positions without response contents.

JSON hex is compact; text hex permits Rust's ASCII whitespace set. The labeled format requires response (hex): at the start after leading whitespace and rejects preambles or repeated labels. Duplicate JSON response fields, malformed or truncated input and allowance overflow are rejected. JSON empty hex returns empty bytes; text hex requires digits. Candid decoding, endpoint/result schemas, command/version admission, capture limits and reconciliation remain caller-owned. This API performs no process or network effects.

The local example reads a saved CLI response and prints only its decoded byte count and raw SHA-256 identity:

cargo run -p ic-host-tools --example inspect_response --locked --offline -- \
    /path/to/saved-response.json json 1048576 524288

Select json, hex or labeled explicitly. The limits are illustrative caller choices. A decoded payload is not proof of successful canister execution or permission to replay a request.

Git provenance observations

On Unix, provenance::capture_git uses an admitted Git executable to observe the HEAD object ID, HEAD tree ID and raw porcelain-v1 NUL-terminated status. Untracked-file and submodule scope are explicit StatusOptions; no default product policy is supplied. The result retains executable identity, bounded stdout/stderr from each query and the raw status byte count/SHA-256. Pathname bytes need not be UTF-8, and formatting omits captured bytes. Typed failures retain earlier captures and the failed query's available evidence.

Commands disable optional locks and the filesystem monitor. Each query has its own output bounds/deadline and runs once. They are separate observations, not an atomic source snapshot or canonical dirty-tree identity. Consumers own Git configuration/environment admission, repository selection, exclusion of concurrent source/ref changes, incomplete-provenance policy and report schemas. An empty status means no changes under the selected scope, including Git's ignored-file rules; it does not prove all source files are unchanged.

The example selects an empty environment and prints facts without raw paths:

cargo run -p ic-host-tools --example inspect_git --locked --offline -- \
    /absolute/git "$ADMITTED_SHA256" 'git version EXACT' /absolute/worktree \
    67108864 1048576 65536 5000 all none

The bounds are example choices: executable bytes, per-query stdout/stderr bytes and deadline milliseconds. The final two arguments choose untracked scope (no, normal, all) and submodule scope (none, untracked, dirty, all). Supply your reviewed digest/version authority before using a real Git.

Adoption and publication

make check-doc-links checks local Markdown references offline using the shared verification helper. Its focused fixtures cover supported link syntax and failure handling; document selection stays local. Rust API documentation uses make docs-check.

Maintainers can use make release-patch, make release-minor, or make release-major to validate, update Cargo metadata and release notes, commit, tag and atomically push the selected branch/tag. Defaults are main and origin. Rerun the same target to reconcile an interrupted release at its saved version; make release-resume VERSION=X.Y.Z selects that version explicitly. These commands retain build outputs and evidence; crate publication remains a separate action. See release instructions for prerequisites, the offline gate, exact effects and recovery. make release-tools-check checks the workflow in isolated fixtures without publishing anything.

make package verifies the packaged crate offline, allowing local changes for review. make publish-dry-run performs Cargo's registry checks without uploading; it may access crates.io and also allows local changes. make publish-check checks the publication source without uploading: the checkout must be clean, Cargo and lockfile versions must agree, and the matching annotated vX.Y.Z tag must identify HEAD. make publish applies that gate, then invokes Cargo once with locked dependencies and the explicit crates.io registry. Provision Cargo credentials yourself; the tooling never installs or prints them.

Creating this repository does not change any consumer dependency. Adoption requires a concrete API, a reviewed dependency update in each consumer, removal of superseded implementations, and focused behavioral evidence.

The manifest permits crates.io publication. Publish from a clean checkout at the annotated tag matching the Cargo version; make publish-check verifies that source contract without uploading. The maintainer's v0.1.3 release includes the publication tooling; the earlier v0.1.2 tag retains disabled publication. Source is hosted publicly at dragginzgame/ic-host-tools. The extraction contract records the audited consumer sources and the distinction between implementation and downstream adoption.