Toko Feed

Toko Feed is a standalone Internet Computer canister that builds and serves a durable Pokémon set and card-price catalog. It owns its JustTCG HTTPS integration, canonical identities, pagination checkpoint, and embedded IcyDB database. Consumers see only its Candid endpoints; whether and how it stores the data is an internal implementation detail.

The workspace has two deliberately separated packages. toko-feed builds as an rlib for shared protocol types and native tests plus a cdylib canister. toko-feed-cli is the host-only operator package and installs the toko-feed executable. There is no separate canisters/ tree, schema crate, Canic dependency, or Toko dependency. A future Canic fleet can install many copies of the compiled Wasm as opaque feed roles.

Current behavior

• A controller configures the canister's JustTCG API key.
• toko_feed_ingest_collections fetches JustTCG's complete bounded game catalog, interprets those provider-specific game categories as Toko-owned collectible collections, and stores each provider ID as a source mapping.
• toko_feed_ingest_sets fetches all Pokémon sets in one bounded JustTCG v1 sets request and upserts their complete current metadata before card ingestion. The current provider fields include release date, total set value, card/variant/sealed counts, and 7/30/90-day value changes. Complete raw set objects are retained so newly introduced provider fields are not discarded.
• Each toko_feed_ingest call fetches and checkpoints at most 50 Pokémon cards. The requests are split into 20, 20, and 10 cards so they remain compatible with JustTCG's documented free-tier page limit. Repeated calls resume from the stored offset; reaching the provider's end resets the next manually triggered pass to offset zero. This release does not run an internal refresh timer.
• Raw successful pages and complete provider card records are retained in IcyDB. Public methods return bounded projections rather than raw provider JSON.
• Re-importing a provider card updates its existing row. Each canonical card receives a Toko-owned generated ULID; provider IDs are stored in a separate source mapping rather than used as the public identity.
• Card ingestion requires an initial provider set import. JustTCG can occasionally return a valid card whose set is absent from its sets endpoint; that card is retained with its provider set ID and name instead of failing the entire bounded ingestion job.
• Summary queries expose the lowest current USD price. Detail queries expose condition, printing, language, current price, provider timestamp, and recent percentage changes for each market variant.

Canonical identity

A ULID is an identifier, not a deduplication rule. Canonical collections therefore use two records:

1. Collection owns the Toko ULID, versioned canonical key, and display name.
2. CollectionSource maps (provider, provider_key) to that ULID and retains the provider's complete category record.

The curated registry assigns a namespace-stable Toko ULID to every supported collection. The current 18 JustTCG category records map to 17 canonical collections because pokemon and pokemon-japan both identify Pokémon. one-piece-card-game maps to the broader One Piece collection while retaining the provider's name in its source record. Every canonical ID is identical in every feed canister and is not derived from a provider ID.

There is deliberately no uncurated or locally generated collection state. A provider category absent from the registry makes collection ingestion fail before any collection rows are written. Supporting it requires an explicit canonical name, key, and Toko ULID plus tests. game and game_id remain raw JustTCG vocabulary; they are not canonical Toko concepts.

Cards likewise keep two related records:

1. PokemonCard owns the Toko ULID and canonical query projection.
2. PokemonCardSource maps (provider, provider_key) to that ULID and retains the complete provider record.

Canonical card ULIDs are generated and remain the public identity. Source and page rows also have locally generated ULID primary keys, while provider-owned keys and cursors are stored separately and uniquely indexed. Re-imports locate the row by its provider key and then update it through its stable local ID.

When a source has not been seen before, the initial conservative match key is the normalized Pokémon set name, collector number, and card name. This lets a future provider attach to an existing printing without pretending that unrelated provider IDs are comparable. The key is explicitly versioned so a future reconciliation migration can improve matching without changing public ULIDs.

Canister interface

The authoritative interface is toko-feed.did.

The HTTPS transform query is part of the IC management-canister outcall protocol and is not an application endpoint.

Query pagination

Start a listing with toko_feed_collections(null, limit), toko_feed_sets(null, limit), or toko_feed_cards(null, limit), then pass the returned next_after value into the corresponding next call. Results are ordered by ascending canonical ULID and limit must be between 1 and 100. Missing records are returned as Ok(null) by their singular query; malformed IDs and invalid limits return a bounded FeedError.

Operating the canister

1. Build and install the generated Wasm with the checked-in Candid interface. The service constructor takes the empty Candid argument ().
2. Fund the canister with enough cycles for HTTPS outcalls. One ingestion can make up to three JustTCG requests.
3. Have a controller invoke toko_feed_configure with the API key through a secure deployment or administration path.
4. Have a controller invoke toko_feed_ingest_collections to refresh canonical collection identities and provider mappings.
5. Have a controller invoke toko_feed_ingest_sets to establish or refresh the Pokémon set catalog. Set results include the mapped canonical_collection_id.
6. Have a controller invoke toko_feed_ingest whenever a card refresh is required. Its receipt reports next_offset and provider_complete; when completion is reported, the following invocation begins a new pass at offset zero.
7. Let consumers use the public status, collection, set, card-list, and card-detail queries.

There is no deployable hub canister or automatic scheduler yet. A future Toko or Canic fleet can coordinate multiple copies through this public interface while the feed canister continues to own its database and provider integration.

Build and installation

Install the pinned Rust toolchain, its Wasm target, ICP CLI, and ic-wasm, then build the complete canister package:

rustup target add wasm32-unknown-unknown
make canister

The installable artifacts are:

• target/wasm32-unknown-unknown/release/toko_feed.wasm
• crates/toko-feed/toko-feed.did

The full interface gate also expects didc and candid-extractor on PATH so the checked-in file can be compared structurally with the compiled Wasm.

make canister builds and validates the canister but does not install or deploy it. The checked-in icp.yaml uses the official pinned Rust recipe for ICP lifecycle builds, including Candid metadata and Wasm shrinking. Use make wasm when only the compiler Wasm artifact is needed and interface validation is not required.

Install or replace the checkout's operator CLI with:

make install
toko-feed --help

This follows the same package split and local-development pattern as ic-query/ic-query-cli: Make performs the Cargo installation, then toko-feed is invoked directly as a normal executable.

On install and post-upgrade, the canister applies its embedded IcyDB schema. Keep the API key out of shell history, logs, committed argument files, and fixtures. The credential is never returned by status, data, debug, or error surfaces.

Local deployment

The checked-in icp.yaml defines a project-local managed network on port 8003 and the standalone canister build. Create an ignored local credential file, restrict its permissions, then start, deploy, configure, and seed the canister:

cp .env.example .env.local
chmod 600 .env.local
# Edit .env.local and set JUST_TCG_API_KEY without committing it.

make local-start
make local-ready

make local-ready builds and upgrades or installs the canister, configures it from .env.local, imports the provider category catalog as collections, and then imports every Pokémon set. If an intentionally clean local database is required, make local-reset performs a reinstall first and therefore erases all existing local Toko Feed data.

The collection and set entities are newer than the 0.1.4 database layout. A local canister last installed from that layout needs one make local-reset. Do not upgrade a persistent production database from the pre-catalog layout until an explicit IcyDB migration artifact has been prepared and exercised.

Local setup and CLI calls default to ICP's anonymous identity, which the managed development network funds and which becomes the local canister controller. Override setup with make local-ready LOCAL_IDENTITY=<name> when testing another identity. Protected identities can be used by the installed CLI with --identity and --identity-password-file.

Use the installed CLI for every operational and query endpoint. It targets the local ICP environment and toko-feed canister by default, passes raw typed arguments to ICP CLI internally, and prints clean typed JSON:

toko-feed status
toko-feed collections ingest
toko-feed collections list --all
toko-feed collections get 00000000000000000000000001
toko-feed sets ingest
toko-feed sets list --limit 10
toko-feed sets list --limit 100 --all
toko-feed sets list --limit 10 --after 01KZ9GFKW3SY1G000000000001
toko-feed sets get 01KZ9GFKW3SY1G000000000001
toko-feed cards ingest
toko-feed cards list --limit 100 --all
toko-feed cards get 01KZ9GFKW3SY1G000000000001

Without --all, a list call returns one page and its next_after continuation, which can be passed back through --after. With --all, the CLI follows that cursor, combines the records, and reports its page and record counts. Automatic pagination stops on a repeated cursor and defaults to at most 1,000 pages; use --max-pages to choose a lower operational bound.

Run toko-feed --help for the complete syntax. Global options must precede the command. --environment, --canister, and --identity select another ICP deployment; --project-root overrides icp.yaml discovery; --identity-password-file supports protected non-interactive identities; and --compact emits one-line JSON. For example:

toko-feed --environment ic --canister <principal> --identity operator status

The CLI deliberately has no configuration or API-key argument. Configuration continues to read the ignored .env.local file so credentials are not copied into command history.

ICP CLI keeps ephemeral local network state under ignored .icp/cache/ paths. Connected-network mappings under .icp/data/ are durable deployment records and may be committed when an IC canister is deployed.

make ic-deploy builds and deploys the same canister to the IC using the active ICP identity. It intentionally does not copy a local credential or trigger a paid provider import; configure those production operations through an appropriately protected controller workflow.

API access does not itself grant permission to cache or redistribute provider data. Review the provider's current subscription, attribution, caching, and redistribution terms before exposing a production feed publicly.

Layout

Future APIs belong under api/<provider>. Their wire types should remain provider-specific; only deliberately selected Pokémon identity and price fields belong in the canonical projection.

Development

The repository is pinned to Rust 1.97.1. Run every required native, Candid, and Wasm check with:

make check

Useful individual targets include make fmt, make clippy, make test, make docs, make wasm, and make candid-check. Tests use sanitized fixtures and never call the live provider.

Releases and publication

Release notes live in CHANGELOG.md. Prepare its dated version entry and commit all non-version work before invoking a release target. cargo-edit is required for version management.

make release-patch
make release-minor
make release-major

These guarded targets run the full checks, bump the workspace version, create an annotated v<version> tag, and atomically push the release commit and tag. Minor and major releases require typed confirmation.

After the release is pushed, validate and publish the canister package followed by the CLI package with:

make publish-dry-run
make publish

License

Toko Feed is licensed under the MIT License.