Toko Feed

Toko Feed is a standalone Internet Computer canister for building a durable, provider-corroborated source of collectible identities, metadata, prices, and eventually real-time market feeds. It currently starts with Pokémon and JustTCG, but its authority comes from retained evidence and explicit Toko identities—not from treating any API as truth. It owns its HTTPS integration, polling engine, canonical identities, historical observations, pagination checkpoints, and embedded IcyDB database. Consumers see only its Candid endpoints; whether and how it stores the data is an internal implementation detail.

The workspace has two deliberately separated packages. toko-feed builds as an rlib for shared protocol types and native tests plus a cdylib canister. toko-feed-cli is the host-only operator package and installs the toko-feed executable. There is no separate canisters/ tree, schema crate, Canic dependency, or Toko dependency. A future Canic fleet can install many copies of the compiled Wasm as opaque feed roles.

Current behavior

• A controller configures the canister's JustTCG API key, whether polling is enabled, and an optional card cadence. Polling defaults to every 15 minutes; the accepted range is one minute through seven days.
• toko_feed_ingest_collections fetches JustTCG's complete bounded game catalog, interprets those provider-specific game categories as Toko-owned collectible collections, and stores each provider ID as a source mapping.
• toko_feed_ingest_sets fetches all Pokémon sets in one bounded JustTCG v1 sets request and upserts their complete current metadata before card ingestion. A newly discovered set also seeds a separate Toko-owned canonical row in candidate state. The current provider fields include release date, total set value, card/variant/sealed counts, and 7/30/90-day value changes. Complete raw set objects are retained so newly introduced provider fields are not discarded.
• Each bounded card run fetches and checkpoints at most 50 Pokémon cards. The requests are split into 20, 20, and 10 cards so they remain compatible with JustTCG's documented free-tier page limit. Repeated calls resume from the stored offset; reaching the provider's end resets the next pass to offset zero.
• Three durable jobs poll collections, Pokémon sets, and Pokémon cards. IC one-shot timers execute only the next due job, so intervals cannot overlap. Successful collection, set, and card runs recur after 24 hours, six hours, and the configured card cadence respectively. Failed runs retry from their durable checkpoint with exponential backoff beginning at 30 seconds, deterministic jitter, and a six-hour cap.
• Timer handles are recreated after upgrade. A run is written before its HTTPS await; an upgrade that interrupts it records an interrupted failure and schedules a retry instead of silently losing the attempt.
• Complete current provider records remain available to the internal projection path. Separately, content-addressed SourceObservation rows keep each distinct provider payload, while PriceObservation rows retain each distinct provider/variant/timestamp/price sample. Repeated identical payloads are deduplicated without erasing the IngestionRun that proved the poll took place. These rows use strong IcyDB relations to their run and typed canonical target rather than relying on unvalidated ULID-shaped values.
• Public methods return bounded, attributed projections and price observations rather than raw provider JSON.
• Re-importing a provider card updates its existing row. Each local card receives a Toko-owned generated ULID; provider IDs are stored in a separate source mapping rather than used as the public identity.
• Card ingestion requires an initial provider set import. JustTCG can occasionally return a valid card whose set is absent from its sets endpoint; that card is retained with its provider set ID and name instead of failing the entire bounded ingestion job. When the provider set is known, the card also carries a validated relation to its Toko canonical set; otherwise that optional relation remains empty until a later card refresh can reconcile it.
• Summary queries expose the lowest current USD price. Detail queries expose condition, printing, language, current price, provider timestamp, and recent percentage changes for each market variant.
• Every manual or timed attempt creates an append-only IngestionRun and sanitized operational events. Logs contain fixed messages and bounded error codes, never credentials, upstream bodies, or raw transport diagnostics.

Evidence, truth, and API versions

Provider data is evidence, not authority. The set layer makes that boundary physical: PokemonSet is a replaceable normalized provider projection, PokemonSetSource retains the provider record and identifier mapping, SourceObservation retains distinct historical payloads, and CanonicalPokemonSet contains only Toko-owned truth.

A new canonical set begins as candidate, where later evidence may still refresh its proposed name and release date. A controller can explicitly curate the complete editable row to verified; provider ingestion then continues to record and refresh evidence but cannot alter the canonical row. A verified row can be moved to locked, after which no canonical field—including its active or retired lifecycle state—can ever be changed through the canister. There is deliberately no unlock operation.

Every canonical operator write includes the revision returned by the latest read. A stale revision returns CanonicalConflict, preventing two curators from silently overwriting one another. sets get returns the canonical row and the latest normalized evidence together so a curator can compare them before accepting anything.

Every job, run, source observation, and price observation records its provider API version. This lets v1 and v2—or several unrelated providers—coexist without forcing their wire formats into one premature universal schema.

The first price-history layer samples current JustTCG variant prices whenever the provider timestamp or price changes. The full provider payload is retained internally, so future reconciliation can derive richer statistics without rewriting the original evidence. Future providers should add their own source records and mapping rules; agreement, disagreement, freshness, market, currency, and confidence must remain visible rather than being flattened into an unexplained “correct” value.

JustTCG currently exposes card identity and external IDs, variants, current prices, statistics, and optional price history through its card and variant schemas. Its v2 card API is documented as beta, so this release records its active integration explicitly as v1 and does not silently mix v2 responses into v1 observations.

Canonical identity

A ULID is an identifier, not a deduplication rule. Canonical collections therefore use two records:

1. Collection owns the Toko ULID, versioned canonical key, and display name.
2. CollectionSource maps (provider, provider_key) to that ULID and retains the provider's complete category record.

The curated registry assigns a namespace-stable Toko ULID to every supported collection. The current 18 JustTCG category records map to 17 canonical collections because pokemon and pokemon-japan both identify Pokémon. one-piece-card-game maps to the broader One Piece collection while retaining the provider's name in its source record. Every canonical ID is identical in every feed canister and is not derived from a provider ID.

There is deliberately no uncurated or locally generated collection state. A provider category absent from the registry makes collection ingestion fail before any collection rows are written. Supporting it requires an explicit canonical name, key, and Toko ULID plus tests. game and game_id remain raw JustTCG vocabulary; they are not canonical Toko concepts.

Set ingestion applies the same boundary: a set whose provider game_id has no curated source mapping is rejected before its provider projection, source record, or canonical candidate is written.

The current card layer keeps two related records:

1. PokemonCard owns the Toko ULID and current provider-derived query projection.
2. PokemonCardSource maps (provider, provider_key) to that ULID and retains the complete provider record.

The embedded schema enforces its single-target links. Collection, set, card, job, run, source-observation, price-observation, and operational-log references are strong IcyDB relations with restricted target deletion. The cross-kind SourceObservation.canonical_id remains as an indexed lookup key, accompanied by exactly one application-selected typed relation to Collection, CanonicalPokemonSet, or PokemonCard. Provider keys remain ordinary evidence fields and are never treated as foreign keys.

Canonical curation is intentionally implemented for sets first. Card rows are still provisional provider-derived projections and must not yet be treated as reviewed Toko truth. Their local ULIDs are generated and remain the public identity. Source and page rows also have locally generated ULID primary keys, while provider-owned keys and cursors are stored separately and uniquely indexed. Re-imports locate the row by its provider key and then update it through its stable local ID.

When a source has not been seen before, the initial conservative match key is the normalized Pokémon set name, collector number, and card name. This lets a future provider attach to an existing printing without pretending that unrelated provider IDs are comparable. The key is explicitly versioned so a future reconciliation migration can improve matching without changing public ULIDs.

Canister interface

The authoritative interface is toko-feed.did.

Method	Access	Purpose
toko_feed_configure	Controller update	Store or rotate the JustTCG credential and configure automatic card polling.
toko_feed_ingest_collections	Controller update	Refresh canonical collections from JustTCG source mappings.
toko_feed_ingest_sets	Controller update	Fetch and upsert every current JustTCG Pokémon set.
toko_feed_curate_set	Controller update	Verify a candidate or correct a verified canonical set using its expected revision.
toko_feed_lock_set	Controller update	Irreversibly lock a verified canonical set using its expected revision.
toko_feed_ingest	Controller update	Fetch, upsert, and checkpoint up to 50 cards.
toko_feed_status	Public query	Read credential-free cursor and health state.
toko_feed_scheduler	Public query	Read every durable job's cadence, next run, failure count, and active run.
toko_feed_runs	Public query	Read append-only ingestion attempts using a descending compound cursor.
toko_feed_logs	Public query	Read sanitized operational events using a descending compound cursor.
toko_feed_collections	Public query	Read up to 100 canonical collections using an optional ULID continuation.
toko_feed_collection	Public query	Read one canonical collection and all known provider mappings.
toko_feed_sets	Public query	Read up to 100 canonical set truth rows using an optional ULID continuation.
toko_feed_set	Public query	Read one canonical set together with its latest replaceable provider evidence.
toko_feed_cards	Public query	Read up to 100 summaries using an optional ULID continuation.
toko_feed_card	Public query	Read one provider-derived local card projection and its latest price variants.
toko_feed_card_prices	Public query	Read attributed, immutable price observations for one local card identity.

The HTTPS transform query is part of the IC management-canister outcall protocol and is not an application endpoint.

Query pagination

Start a listing with toko_feed_collections(null, limit), toko_feed_sets(null, limit), or toko_feed_cards(null, limit), then pass the returned next_after value into the corresponding next call. Results are ordered by ascending canonical ULID and limit must be between 1 and 100. Missing records are returned as Ok(null) by their singular query; malformed IDs and invalid limits return a bounded FeedError.

Run, log, and price-history pages are newest first. Begin with a null TimeCursor, then pass the returned next_before record—both its timestamp and ULID—to the next call. The compound cursor prevents equal timestamps from dropping or repeating rows. These limits are also between 1 and 100.

Operating the canister

1. Build and install the generated Wasm with the checked-in Candid interface. The service constructor takes the empty Candid argument ().
2. Fund the canister with enough cycles for HTTPS outcalls. One ingestion can make up to three JustTCG requests.
3. Have a controller invoke toko_feed_configure with the API key and desired polling settings through a secure deployment or administration path. The canister creates and starts all three durable jobs; Toko does not need to run a separate scheduler or database.
4. Use the controller-only ingestion methods for an immediate seed, repair, or operator-requested refresh. They use exactly the same run, evidence, checkpoint, logging, and retry bookkeeping as timer execution.
5. Monitor toko_feed_scheduler, toko_feed_runs, and toko_feed_logs. Fund enough cycles for the configured request rate and choose a cadence that fits the provider plan's rate and quota limits.
6. Curate set candidates deliberately, then lock only identities and metadata that should never change again.
7. Let consumers query canonical collections and sets. Treat current card and price surfaces as provider-derived evidence until their own curation model is introduced.

There is no deployable hub canister yet. A future Toko or Canic fleet can coordinate multiple copies through this public interface while each feed canister continues to own its database, provider integration, and retry loop. The proposed topology, current Canic compatibility gap, opaque-role contract, and secret/configuration boundaries are documented in Canic fleet integration.

Build and installation

Install the pinned Rust toolchain, its Wasm target, ICP CLI, and ic-wasm, then build the complete canister package:

rustup target add wasm32-unknown-unknown
make canister

The installable artifacts are:

• target/wasm32-unknown-unknown/release/toko_feed.wasm
• crates/toko-feed/toko-feed.did

The full interface gate also expects didc and candid-extractor on PATH so the checked-in file can be compared structurally with the compiled Wasm.

make canister builds and validates the canister but does not install or deploy it. The checked-in icp.yaml uses the official pinned Rust recipe for ICP lifecycle builds, including Candid metadata and Wasm shrinking. Use make wasm when only the compiler Wasm artifact is needed and interface validation is not required.

Install or replace the checkout's operator CLI with:

make install
toko-feed --help

This follows the same package split and local-development pattern as ic-query/ic-query-cli: Make performs the Cargo installation, then toko-feed is invoked directly as a normal executable.

Operators who also need direct database inspection can install both CLIs with:

make install-tools

This installs the checkout's toko-feed binary and exactly icydb-cli 0.220.1, matching the embedded IcyDB dependency. It deliberately does not install an unconstrained latest pre-1.0 CLI.

On install and post-upgrade, the canister applies its embedded IcyDB schema. Keep the API key out of shell history, logs, committed argument files, and fixtures. The credential is never returned by status, data, debug, or error surfaces.

Local deployment

The checked-in icp.yaml defines a project-local managed network on port 8003 and the standalone canister build. Create an ignored local credential file, restrict its permissions, then start, deploy, configure, and seed the canister:

cp .env.example .env.local
chmod 600 .env.local
# Edit .env.local and set JUST_TCG_API_KEY without committing it.

make local-start
make local-ready

make local-ready builds and upgrades or installs the canister, configures it from .env.local, starts its default polling jobs, imports the provider category catalog as collections, and then imports every Pokémon set. The ignored file contains the provider credential only; scheduler policy is non-secret control-plane state and does not belong there. If an intentionally clean local database is required, make local-reset performs a reinstall first and therefore erases all existing local Toko Feed data.

Direct local database inspection

Toko Feed exports IcyDB's controller-only readonly SQL and accepted-schema inspection endpoints. It does not export SQL mutation or DDL endpoints. After make local-ready, open the interactive database shell with:

icydb sql --environment local --canister toko-feed

Terminate interactive statements with ;. For one-shot inspection:

icydb sql -e local -c toko-feed --sql "SHOW ENTITIES"
icydb sql -e local -c toko-feed --sql "SELECT * FROM canonical_pokemon_set LIMIT 20"
icydb schema show toko-feed --environment local

IcyDB 0.220.1 currently reports an incorrect zero in the SHOW ENTITIES relations summary even when accepted relation constraints are active. Inspect the authoritative constraints instead:

icydb sql -e local -c toko-feed --sql "SHOW CONSTRAINTS FROM PokemonCard"
icydb sql -e local -c toko-feed --sql "SHOW CONSTRAINTS FROM SourceObservation"

Accepted links appear as validated relation constraints with restrict deletion semantics.

The ICP identity used by icydb must control the canister. The default local workflow installs under the anonymous identity; use the same explicit identity for deployment and inspection when overriding LOCAL_IDENTITY.

The observation, scheduler, run, log, and canonical-set truth entities are newer than the 0.1.5 database layout. CanonicalPokemonSet is an additive version-1 entity; PokemonCard and the observability entities now declare version 2 for their new relation contracts. Existing provider projection tables are intentionally preserved instead of being renamed or rewritten. A local canister last installed from 0.1.5 or from an earlier unreleased build needs one make local-reset; no migration artifact has yet been exercised for these pre-release schema changes. Do not upgrade a persistent production database from that layout until the complete IcyDB migration path has been prepared and tested.

Local setup and CLI calls default to ICP's anonymous identity, which the managed development network funds and which becomes the local canister controller. Override setup with make local-ready LOCAL_IDENTITY=<name> when testing another identity. Protected identities can be used by the installed CLI with --identity and --identity-password-file.

Use the installed CLI for every operational and query endpoint. It targets the local ICP environment and toko-feed canister by default, passes raw typed arguments to ICP CLI internally, and prints clean typed JSON:

toko-feed status
toko-feed scheduler
toko-feed runs list --limit 100
toko-feed logs list --all
toko-feed collections ingest
toko-feed collections list --all
toko-feed collections get 00000000000000000000000001
toko-feed sets ingest
toko-feed sets list --limit 10
toko-feed sets list --limit 100 --all
toko-feed sets list --limit 10 --after 01KZ9GFKW3SY1G000000000001
toko-feed sets get 01KZ9GFKW3SY1G000000000001
toko-feed sets curate 01KZ9GFKW3SY1G000000000001 --revision 0 --name "Base Set" --release-date 1999-01-09
toko-feed sets curate 01KZ9GFKW3SY1G000000000001 --revision 1 --name "Base Set" --release-date 1999-01-09 --retired
toko-feed sets lock 01KZ9GFKW3SY1G000000000001 --revision 2
toko-feed cards ingest
toko-feed cards list --limit 100 --all
toko-feed cards get 01KZ9GFKW3SY1G000000000001
toko-feed cards prices 01KZ9GFKW3SY1G000000000001 --limit 100

Without --all, a list call returns one page and its next_after continuation, which can be passed back through --after. With --all, the CLI follows that cursor, combines the records, and reports its page and record counts. Automatic pagination stops on a repeated cursor and defaults to at most 1,000 pages; use --max-pages to choose a lower operational bound.

History commands use next_before rather than next_after. For a manual next page, copy both returned fields into --before-time and --before-id. Use --all to have the CLI follow compound cursors and enforce the same repeated- cursor and maximum-page safeguards automatically.

Set curation is controller-only. Read the row with sets get, compare its set and latest_evidence objects, then pass the returned set.revision to sets curate. The command requires either --release-date YYYY-MM-DD or --no-release-date, so an omitted option cannot accidentally erase a known date. It defaults to --active; use --retired deliberately. Read the row again before sets lock and pass its new revision. Locking is final.

Run toko-feed --help for the complete syntax. Global options must precede the command. --environment, --canister, and --identity select another ICP deployment; --project-root overrides icp.yaml discovery; --identity-password-file supports protected non-interactive identities; and --compact emits one-line JSON. For example:

toko-feed --environment ic --canister <principal> --identity operator status

The CLI deliberately has no configuration or API-key argument. Configuration continues to read the ignored .env.local file so credentials are not copied into command history.

ICP CLI keeps ephemeral local network state under ignored .icp/cache/ paths. Connected-network mappings under .icp/data/ are durable deployment records and may be committed when an IC canister is deployed.

make ic-deploy builds and deploys the same canister to the IC using the active ICP identity. It intentionally does not copy a local credential or trigger a paid provider import; configure those production operations through an appropriately protected controller workflow.

API access does not itself grant permission to cache or redistribute provider data. JustTCG's current commercial-use guidance and terms distinguish displaying or combining provider-derived information from redistributing raw data or offering a substitute bulk pricing API. Toko Feed therefore keeps raw payload evidence internal and exposes bounded, attributed product views. Review the active provider plan and terms before exposing any production feed publicly.

Layout

Path	Responsibility
crates/toko-feed/src/api/just_tcg	JustTCG request construction, wire models, pagination, compatibility, and sanitized tests.
crates/toko-feed/src/canister	Candid endpoints, IC HTTPS adapter, durable timer/retry orchestration, bounded ingestion, and IcyDB access.
crates/toko-feed/src/collection.rs	Curated collectible collections, stable Toko IDs, provider aliases, and fallback matching policy.
crates/toko-feed/src/pokemon	Canonical set/card match keys, set projections, and card price-summary rules.
crates/toko-feed/src/schema	Embedded IcyDB canister plus job, run, log, observation, collection, set/card, provider-source, and future hub declarations. No hub actor is built yet.
crates/toko-feed-cli	Host-only installed CLI, typed ICP process boundary, lookup commands, and safe manual or automatic pagination.
icp.yaml	Official Rust recipe and local/IC project lifecycle definition.

Future APIs belong under api/<provider>. Their wire types should remain provider-specific; only deliberately selected Pokémon identity and price fields belong in the canonical projection.

Development

The repository is pinned to Rust 1.97.1. Run every required native, Candid, and Wasm check with:

make check

Useful individual targets include make fmt, make clippy, make test, make docs, make wasm, and make candid-check. Tests use sanitized fixtures and never call the live provider.

Releases and publication

Release notes live in CHANGELOG.md. Prepare its dated version entry and commit all non-version work before invoking a release target. cargo-edit is required for version management.

make release-patch
make release-minor
make release-major

These guarded targets run the full checks, bump the workspace version, create an annotated v<version> tag, and atomically push the release commit and tag. Minor and major releases require typed confirmation.

After the release is pushed, validate and publish the canister package followed by the CLI package with:

make publish-dry-run
make publish

License

Toko Feed is licensed under the MIT License.