Toko Feed

Toko Feed is a standalone Internet Computer canister for building a durable, provider-corroborated source of collectible identities, metadata, prices, and eventually real-time market feeds. It currently starts with Pokémon evidence from JustTCG, TCGdex, and Scrydex, plus a schema-only Magic: The Gathering template, but its authority comes from retained evidence and explicit Toko identities—not from treating any API as truth. It owns its HTTPS integration, polling engine, canonical identities, historical observations, pagination checkpoints, and embedded IcyDB database. Consumers see only its Candid endpoints; whether and how it stores the data is an internal implementation detail.

The workspace has two deliberately separated packages. toko-feed builds as an rlib for shared protocol types and native tests plus a cdylib canister. toko-feed-cli is the host-only operator package and installs the toko-feed executable. There is no separate canisters/ tree, schema crate, Canic dependency, or Toko dependency. A future Canic fleet can install many copies of the compiled Wasm as opaque feed roles.

Current behavior

A controller configures the canister's JustTCG API key, optional paired Scrydex API key and team ID, whether polling is enabled, and an optional card cadence. Polling defaults to every 15 minutes; the accepted range is one minute through seven days. Credentials are stored together in a versioned, private configuration payload and never returned by public status methods.
toko_feed_ingest_collections fetches JustTCG's complete bounded game catalog, interprets those provider-specific game categories as Toko-owned collectible collections, and stores each provider ID as a source mapping.
toko_feed_ingest_sets fetches all Pokémon sets in one bounded JustTCG v1 sets request and upserts their complete current metadata into PokemonSetSource. Ingestion never creates or updates canonical PokemonSet rows. Each new source starts as pending until a controller maps, rejects, or curates it. Provider fields include release date, total set value, card/variant/sealed counts, and 7/30/90-day value changes. Complete raw set objects are retained so newly introduced provider fields are not discarded.
toko_feed_ingest_tcgdex_set fetches one credential-free TCGdex v2 English set resource. toko_feed_ingest_tcgdex_card then consumes exactly one card ID from that stored set response per call. Both write only provider source rows and immutable observations; neither promotes data into canonical truth. The current local Aquapolis evidence contains its complete 185-card index and 177 usable detail responses. Eight indexed Cube 01 cards currently return provider-data failures and remain explicit acquisition holes rather than disappearing from the comparison. TCGdex reports 147 official cards, while the current JustTCG evidence reports 186 total printings, immediately demonstrating why provider totals are evidence rather than authority.
toko_feed_ingest_scrydex_set fetches one exact English Scrydex expansion; toko_feed_ingest_scrydex_cards fetches one expansion-scoped page of at most 10 cards. Both calls run through the canister, omit price expansions, retain provider wire records, and stage only source evidence. The default Aquapolis comparison uses Scrydex's ecard2 expansion ID. A full acquisition follows 19 rate-paced pages and retains all 182 card responses. Scrydex reports 147 printed cards, disagreeing with both other providers' total-printing models while matching TCGdex's official count.
Each bounded product run fetches and checkpoints at most 10 JustTCG records, staying below both IcyDB's per-request execution budget and JustTCG's documented free-tier page limit. Repeated calls resume from the stored offset; reaching the provider's end resets the next pass to offset zero.
JustTCG returns unopened products through its card feed. Records with a nonempty variant list whose conditions are all explicitly Sealed are routed to PokemonSealed; they are never inserted into PokemonCard. Packaging remains one entity with the broad BoosterBox, BoosterPack, Blister, Box, Bundle, Collection, Deck, Tin, Kit, Case, and Other enum rather than one table per product format.
A controller can instead target one canonical set with toko_feed_ingest_set_cards. Toko Feed resolves the set's JustTCG identifier through the provider translation table, applies the provider's exact set filter, and returns an explicit continuation without advancing the global Pokémon cursor. Page evidence is partitioned by provider set and offset.
The CLI's bootstrap operation builds the development corpus from provider collections, all current Pokémon set metadata, and the complete Expedition, Aquapolis, and Skyridge card feeds. With --source, retained records are replayed before a requested refresh. Refreshing then updates only catalogs and set metadata and resumes each card feed at its retained count; complete local card sets make no card API requests. The operation composes ordinary bounded canister calls with hard page bounds and no-progress detection.
The CLI never connects to JustTCG and contains no provider HTTP transport. Live bootstrap invokes typed controller-only canister methods through icp; only the canister's IC HTTPS adapter constructs and sends provider requests. The canister also owns provider decoding, card/sealed classification, set partitioning, and replay validation. The CLI only writes the returned raw JSON evidence to local files or sends those files back for canister replay.
The same canister-only rule applies to TCGdex. toko-feed provider set tcgdex ecard2 replays source/providers/tcgdex/v2/en when present; --refresh first replays that cache, performs one set-metadata request, and requests only card IDs not already retained. The default acquisition target is a 10-card comparison sample; --all follows the complete ordered set index. Full acquisition checkpoints every ten new details and reports missing or unprojectable indexed cards without preventing later records from being acquired. A gap behind a later retained index position is treated as a previously attempted provider hole and is not retried on every refresh. Removing the TCGdex source directory is the explicit full-refetch path. This command takes TCGdex's exact set ID; after ingestion, operator-visible names can be inspected with toko-feed sets sources list --all but do not replace the provider identity.
toko-feed provider set scrydex ecard2 follows the same cache-first flow for Scrydex. With no local cache it makes one expansion request and one bounded 10-card page request through the canister, then exports the retained evidence. --refresh --all follows the provider's continuation through rate-paced pages with page-continuity and no-progress checks. A refresh replays and validates the cached contiguous provider prefix first, then starts at the first uncached page; a complete cache makes no card-page requests.
The CLI persists the resulting provider evidence as neutral, ignored source data rather than as a canonical backup. The file source/providers/just-tcg/v1/collections.json contains provider categories, sets.json contains every set, and cards/<provider-set-id>.json contains each fetched set's card and sealed-product records. TCGdex and Scrydex use equivalent provider-versioned trees under source/providers/. toko-feed bootstrap --source replays every file present without HTTPS outcalls, preserving original observation times. If the directory is absent, it performs the live bootstrap and writes the split dataset; --refresh refreshes provider catalogs and set metadata but retains immutable card payloads and fetches only missing card positions. This is provider evidence replay, not a canonical snapshot or an IcyDB physical backup. Tracked canonical JSON belongs under data/.
Three durable jobs poll collections, Pokémon sets, and Pokémon cards. IC one-shot timers execute only the next due job, so intervals cannot overlap. Successful collection, set, and card runs recur after 24 hours, six hours, and the configured card cadence respectively. Failed runs retry from their durable checkpoint with exponential backoff beginning at 30 seconds, deterministic jitter, and a six-hour cap.
Timer handles are recreated after upgrade. A run is written before its HTTPS await; an upgrade that interrupts it records an interrupted failure and schedules a retry instead of silently losing the attempt.
Complete current provider records remain in their source tables. Separately, content-addressed SourceObservation rows keep each distinct provider payload, while PriceObservation rows retain each distinct provider/variant/timestamp/price sample. Repeated identical payloads are deduplicated without erasing the IngestionRun that proved the poll took place. These rows use strong IcyDB relations to their run and typed provider source rather than relying on unvalidated ULID-shaped values.
Public methods return bounded, attributed projections and price observations rather than raw provider JSON.
Card and sealed-product identity rows never store a price, a lowest-price summary, or a provider price timestamp. Current detail prices are derived from the latest provider source payload and explicitly identify that provider and its item key; historical queries read append-only observations instead.
Re-importing a provider card updates only its existing PokemonCardSource row and observations. A canonical PokemonCard exists only after explicit curation. Sealed products retain their current source-plus-identity flow.
Card ingestion requires an initial provider set import. JustTCG can occasionally return a valid card whose set is absent from its sets endpoint; that card is retained with its provider set ID and name instead of failing the entire bounded ingestion job. When the provider set source is mapped, the card source also carries a validated relation to its Toko canonical set; otherwise that optional relation remains empty until a later refresh can reconcile it.
Detail queries expose source-attributed condition, printing, language, current price, provider timestamp, and recent percentage changes for each provider price variant. Price-history queries preserve different prices from different providers rather than collapsing them into one canonical value.
Every manual or timed attempt creates an append-only IngestionRun and sanitized operational events. Logs contain fixed messages and bounded error codes, never credentials, upstream bodies, or raw transport diagnostics.
Every live response and logical cache replay is screened at ingress. The non-destructive screen detects invalid UTF-8; invisible, directional, private-use, tag, variation-selector, or excessive combining Unicode; confusable mixed-script tokens; and suspicious encoded or repeated-text patterns. It accepts visible multilingual text such as accented Latin and Japanese rather than treating all non-ASCII data as hostile.
A finding appends a sanitized warning operational event with a bounded ProviderIntegrity code, while the original valid provider bytes continue through decoding and persistence unchanged. The warning identifies only the category; suspect provider text is never copied into the alert. Disagreement with canonical data is expected source evidence and is not an integrity finding, even when the related canonical row is locked.
The initial Magic template is canonical-schema only: MagicSet, MagicCard, and ordered MagicCardFace rows are related to the existing Toko collection identity for Magic: The Gathering. Face-specific mana cost, type line, rules text, colors, power, toughness, loyalty, and defense remain separate from Pokémon HP and attacks. No Magic provider, ingestion endpoint, or automatic canonical write path is enabled yet.

Evidence, truth, and API versions

Provider data is evidence, not authority. That boundary is physical for both sets and cards:

PokemonSetSource and PokemonCardSource retain replaceable current provider records, provider keys, free-form provider vocabulary, raw payloads, and a pending, mapped, or rejected resolution state.
SourceObservation and PriceObservation retain immutable historical evidence and point to those provider source rows.
PokemonSet and PokemonCard contain only explicitly curated Toko truth. Provider polling has no write path into either table.

Curating a pending source creates a canonical row directly in verified state, or maps it to an existing canonical row using that row's expected revision. A verified row remains correctable by a controller. It can then be moved to locked, after which no canonical field can ever be changed through the canister. There is deliberately no unlock operation. Additional provider sources can still be mapped to an already locked row when the supplied canonical values match it exactly.

Every canonical operator write includes the revision returned by the latest read. A stale revision returns CanonicalConflict, preventing two curators from silently overwriting one another. sets sources get and cards sources get expose unresolved evidence for review. Canonical detail queries return only mapped evidence and derived pricing data, so pending or rejected provider rows never masquerade as truth.

Every job, run, source observation, and price observation records its provider API version. This lets v1 and v2—or several unrelated providers—coexist without forcing their wire formats into one premature universal schema.

Toko-owned closed domains are variants throughout storage and Candid: ingestion operations, manual versus timer triggers, observation target kinds, log severity, and persisted failure categories cannot acquire accidental new spellings. Normalized prices currently support the deliberately small currency set Usd, Cad, Eur, Gbp, Jpy, and Aud. Toko does not assign an inferred market to a price; any provider-specific market context stays in the attributed raw evidence. Raw condition, printing, rarity, and other wire labels also remain provider-owned text.

A price has no canonical value. PriceObservation is an append-only fact that links a card source (or current sealed-product identity) to a provider, provider item, provider variant, API version, currency, provider timestamp, observed timestamp, and integer-micro price. Two providers quoting the same card create two rows; a later quote creates another row. Nothing updates a price on the identity record, and prices in different currencies are not compared implicitly. Any future “best price” or aggregate is a derived query or cache, not canonical card data.

The price-history layer samples current provider variants whenever the source, provider item, variant, timestamp, or price changes. Each immutable row retains the provider, API version, currency, source relation, and provider keys. The full provider payload is also retained internally, so future reconciliation can derive richer statistics without rewriting the original evidence. Future providers should add their own source records and mapping rules; agreement, disagreement, freshness, currency, and confidence remain visible rather than being flattened into an unexplained “correct” value.

JustTCG currently exposes card identity and external IDs, variants, current prices, statistics, and optional price history through its card and variant schemas. Its v2 card API is documented as beta, so this release records its active integration explicitly as v1 and does not silently mix v2 responses into v1 observations.

TCGdex is integrated as an independent Pokémon evidence source using its v2 English REST resources. It requires no API key, so no new secret is stored. The adapter still uses bounded calls and the local ignored cache rather than treating the absence of a published hard rate limit as permission to send bursts. Its richer card payload—including HP and attacks—is retained raw in PokemonCardSource. category describes a card kind such as Pokemon, Trainer, or Energy; all are valid Pokémon TCG evidence. Those game-specific details will be normalized only after the cross-provider canonical model is chosen.

Scrydex is a third independent Pokémon evidence source. Its adapter sends the required API-key and team-ID headers only from the canister, marks both as sensitive, and requests the English Pokémon v1 expansion and card resources without price includes. Full acquisition remains deliberately paced at one bounded 10-card page per call. Raw responses remain ignored local evidence; their normalized match keys let operators compare them with JustTCG and TCGdex without silently merging provider identities.

Aquapolis evidence baseline

The ignored local capture completed on 2026-08-07. These are provider claims and acquisition results, not canonical rows:

Provider	Set key	Set/index claim	Retained card details	Important difference
JustTCG v1	`aquapolis-pokemon`	186	186	Also exposes one sealed product.
TCGdex v2 English	`ecard2`	185	177	Omits #128 from its index; eight `Cube 01` details fail provider decoding.
Scrydex Pokémon v1 English	`ecard2`	182	182	Collapses four `a`/`b` artwork pairs into unsuffixed numbers.

After removing harmless zero-padding from collector numbers, JustTCG contains all 185 identities indexed by TCGdex plus Memory Berry #128. JustTCG and TCGdex both distinguish 50a/50b, 74a/74b, 95a/95b, and 103a/103b; Scrydex represents each pair as one unsuffixed card. Scrydex independently contains #128. The eight missing TCGdex details are #119, #121, #122, #124, #127, #129, #132, and #140; both other providers contain them.

The strongest current canonical candidate is therefore 186 distinct English printings: retain the four a/b pairs and #128, but do not copy JustTCG's presentation-only collector suffixes into card names. TCGdex and Scrydex agree on every directly comparable name except spacing in Nidoran's gender symbol at #96. One raw JustTCG payload also says 124/127 for Grass Cube 01 while the other providers say 124/147; this should remain a provider contradiction, with 124 as the identity numerator. No rows have been promoted from this analysis: a controller must still curate and later lock them explicitly.

Canonical identity

A ULID is an identifier, not a deduplication rule. Canonical collections therefore use two records:

Collection owns the Toko ULID, versioned canonical key, and display name.
CollectionSource maps (provider, provider_key) to that ULID and retains the provider's complete category record.

The curated registry assigns a namespace-stable Toko ULID to every supported collection. The current 18 JustTCG category records map to 17 canonical collections because pokemon and pokemon-japan both identify Pokémon. one-piece-card-game maps to the broader One Piece collection while retaining the provider's name in its source record. Every canonical ID is identical in every feed canister and is not derived from a provider ID.

There is deliberately no uncurated or locally generated collection state. A provider category absent from the registry makes collection ingestion fail before any collection rows are written. Supporting it requires an explicit canonical name, key, and Toko ULID plus tests. game and game_id remain raw JustTCG vocabulary; they are not canonical Toko concepts.

Set ingestion applies the same boundary: a set whose provider game_id has no curated collection-source mapping is rejected before its provider source row is written. A valid provider set is retained as evidence but is never promoted automatically.

Sets and cards use the same explicit reconciliation pattern. Source rows own generated local ULIDs and keep (provider, provider_key) unique. Canonical rows also own generated Toko ULIDs, but they are not created until a curator accepts a source. Multiple provider source rows can map to the same canonical row.

PokemonCard is fully normalized: its first field and primary key is id, and its only set identity is the required pokemon_set_id relation. Provider set, set_name, card IDs, and other free-form values remain exclusively in PokemonCardSource. Likewise, PokemonSet stores one required canonical collection_id; provider game and game_id remain exclusively in PokemonSetSource.

Unopened products follow the same shape in PokemonSealed and PokemonSealedSource. Their conservative versioned match key uses set name and product name. Product kind is a query facet inferred from the normalized provider name; an unfamiliar package remains Other instead of being rejected or forcing a schema change.

The embedded schema enforces its single-target links. Collection, set, card, job, run, source-observation, price-observation, and operational-log references are strong IcyDB relations with restricted target deletion. The cross-kind SourceObservation.source_id remains as an indexed lookup key and is accompanied by exactly one typed relation to a collection, set source, card source, or sealed product. Provider keys remain ordinary evidence fields and are never treated as foreign keys.

When a card source has not been seen before, its conservative comparison key is the normalized Pokémon set name, collector number, and card name. This lets a future provider attach to an existing printing without pretending that unrelated provider IDs are comparable. The key is explicitly versioned so a future reconciliation migration can improve matching without changing public ULIDs. A match key is evidence for a curator, never an automatic merge rule.

Canister interface

The authoritative interface is toko-feed.did.

Method	Access	Purpose
`toko_feed_configure`	Controller update	Store or rotate JustTCG and optional paired Scrydex credentials, and configure automatic card polling.
`toko_feed_provider_cache_page`	Controller query	Export one byte-bounded page of current raw provider JSON and its canister-validated set partition.
`toko_feed_replay_provider_cache`	Controller update	Validate the provider format, set partition, and source class, then replay one byte-bounded batch without making a provider request.
`toko_feed_ingest_collections`	Controller update	Refresh canonical collections from JustTCG source mappings.
`toko_feed_ingest_sets`	Controller update	Fetch and upsert the next bounded batch of current JustTCG sets into provider source storage.
`toko_feed_ingest_tcgdex_set`	Controller update	Fetch and stage one TCGdex v2 English set resource.
`toko_feed_ingest_tcgdex_card`	Controller update	Fetch and stage exactly one card selected by offset from a stored TCGdex set.
`toko_feed_ingest_scrydex_set`	Controller update	Fetch and stage one exact Scrydex Pokémon v1 English expansion.
`toko_feed_ingest_scrydex_cards`	Controller update	Fetch and stage one expansion-scoped Scrydex page of at most 10 cards.
`toko_feed_set_sources` / `toko_feed_set_source`	Public query	Review provider set records and their pending, mapped, or rejected resolution.
`toko_feed_curate_set`	Controller update	Create or update verified canonical set truth from one provider source.
`toko_feed_reject_set_source`	Controller update	Reject one unresolved provider set source.
`toko_feed_lock_set`	Controller update	Irreversibly lock a verified canonical set using its expected revision.
`toko_feed_ingest`	Controller update	Fetch, route, source-stage, and checkpoint up to 10 card-feed records.
`toko_feed_ingest_set_cards`	Controller update	Fetch up to 10 card-feed records for one canonical set from an explicit offset.
`toko_feed_ingest_provider_set_cards`	Controller update	Fetch a bounded explicit-offset card range by provider set ID for pre-curation acquisition.
`toko_feed_status`	Public query	Read credential-free cursor and health state.
`toko_feed_scheduler`	Public query	Read every durable job's cadence, next run, failure count, and active run.
`toko_feed_runs`	Public query	Read append-only ingestion attempts using a descending compound cursor.
`toko_feed_logs`	Public query	Read sanitized operational events using a descending compound cursor.
`toko_feed_collections`	Public query	Read up to 100 canonical collections using an optional ULID continuation.
`toko_feed_collection`	Public query	Read one canonical collection and all known provider mappings.
`toko_feed_sets`	Public query	Read up to 100 canonical set truth rows using an optional ULID continuation.
`toko_feed_set`	Public query	Read one canonical set together with its latest mapped provider evidence.
`toko_feed_card_sources` / `toko_feed_card_source`	Public query	Review provider card records and their resolution.
`toko_feed_curate_card`	Controller update	Create or update normalized verified card truth from one provider source.
`toko_feed_reject_card_source`	Controller update	Reject one unresolved provider card source.
`toko_feed_lock_card`	Controller update	Irreversibly lock a verified canonical card using its expected revision.
`toko_feed_cards`	Public query	Read up to 100 canonical card rows using an optional ULID continuation.
`toko_feed_card`	Public query	Read one canonical card and current prices derived from mapped sources.
`toko_feed_card_prices`	Public query	Read attributed, immutable price observations through mapped card sources.
`toko_feed_sealed`	Public query	Read up to 100 sealed-product summaries using an optional ULID continuation.
`toko_feed_sealed_product`	Public query	Read one sealed-product projection and its latest price variants.
`toko_feed_sealed_prices`	Public query	Read attributed, immutable price observations for one sealed-product identity.

The HTTPS transform query is part of the IC management-canister outcall protocol and is not an application endpoint.

Query pagination

Start a listing with toko_feed_collections(null, limit), toko_feed_set_sources(null, limit), toko_feed_sets(null, limit), toko_feed_card_sources(null, limit), toko_feed_cards(null, limit), or toko_feed_sealed(null, limit), then pass the returned next_after value into the corresponding next call. Results are ordered by ascending local ULID and limit must be between 1 and 100. Missing records are returned as Ok(null) by their singular query; malformed IDs and invalid limits return a bounded FeedError.

Run, log, and price-history pages are newest first. Begin with a null TimeCursor, then pass the returned next_before record—both its timestamp and ULID—to the next call. The compound cursor prevents equal timestamps from dropping or repeating rows. These limits are also between 1 and 100.

Operating the canister

Build and install the generated Wasm with the checked-in Candid interface. The service constructor takes the empty Candid argument ().
Fund the canister with enough cycles for HTTPS outcalls. One ingestion can make up to three JustTCG requests. Each TCGdex method makes exactly one provider request.
Have a controller invoke toko_feed_configure with the API key and desired polling settings through a secure deployment or administration path. The canister creates and starts all three durable jobs; Toko does not need to run a separate scheduler or database.
Use the controller-only ingestion methods for an immediate seed, repair, or operator-requested refresh. They use exactly the same run, evidence, checkpoint, logging, and retry bookkeeping as timer execution.
Monitor toko_feed_scheduler, toko_feed_runs, and toko_feed_logs. Fund enough cycles for the configured request rate and choose a cadence that fits the provider plan's rate and quota limits. Review every warning provider_integrity_warning promptly. The fixed message identifies the suspicious Unicode or encoding category without copying raw provider content. The evidence remains stored and the job continues normally.
Review pending set and card sources, explicitly curate or reject each one, and lock only canonical identities and metadata that should never change.
Let consumers query canonical collections, sets, and cards. Treat prices as attributed observations rather than canonical values.

There is no deployable hub canister yet. A future Toko or Canic fleet can coordinate multiple copies through this public interface while each feed canister continues to own its database, provider integration, and retry loop. The proposed topology, current Canic compatibility gap, opaque-role contract, and secret/configuration boundaries are documented in Canic fleet integration.

Build and installation

Install the pinned Rust toolchain, its Wasm target, ICP CLI, and ic-wasm, then build the complete canister package:

rustup target add wasm32-unknown-unknown
make canister

The installable artifacts are:

target/wasm32-unknown-unknown/release/toko_feed.wasm
crates/toko-feed/toko-feed.did

The full interface gate also expects didc and candid-extractor on PATH so the checked-in file can be compared structurally with the compiled Wasm.

make canister builds and validates the canister but does not install or deploy it. The checked-in icp.yaml uses the official pinned Rust recipe for ICP lifecycle builds, including Candid metadata and Wasm shrinking. Use make wasm when only the compiler Wasm artifact is needed and interface validation is not required.

Install or replace the checkout's operator CLI with:

make install
toko-feed --help

This follows the same package split and local-development pattern as ic-query/ic-query-cli: Make performs the Cargo installation, then toko-feed is invoked directly as a normal executable.

Operators who also need direct database inspection can install both CLIs with:

make install-tools

This installs the checkout's toko-feed binary and icydb-cli 0.221.6, matching the embedded IcyDB dependency requirement. It deliberately does not install an unconstrained latest pre-1.0 CLI.

On install and post-upgrade, the canister applies its embedded IcyDB schema. Keep the API key out of shell history, logs, committed argument files, and fixtures. The credential is never returned by status, data, debug, or error surfaces.

Local deployment

The checked-in icp.yaml defines a project-local managed network on port 8003 and the standalone canister build. Create an ignored local credential file, restrict its permissions, then start, deploy, configure, and seed the canister:

cp .env.example .env.local
chmod 600 .env.local
# Edit .env.local and set JUST_TCG_API_KEY, SCRYDEX_API_KEY, and SCRYDEX_TEAM_ID.

make local-ready

make local-ready first ensures the managed local replica is healthy, then builds and upgrades or installs the canister, configures it from .env.local, keeps automatic polling disabled, and invokes the data-aware CLI bootstrap. The first run imports the JustTCG provider category catalog, every current Pokémon set, and all cards and sealed products from Expedition, Aquapolis, and Skyridge. It also imports the complete paced TCGdex and Scrydex Aquapolis comparison datasets. The local evidence is written under source/providers/just-tcg/v1/ and source/providers/tcgdex/v2/en/ or source/providers/scrydex/pokemon-v1/en/. Later runs replay every file without provider requests. The complete source/ payload is ignored and is reacquired through the canister on a fresh clone. Set TOKO_FEED_SOURCE, TOKO_FEED_TCGDEX_SOURCE, or TOKO_FEED_SCRYDEX_SOURCE to move a cache, or use the corresponding CLI --refresh option to update set metadata and fill missing card records without re-downloading retained cards. Reviewed canonical JSON snapshots belong in the tracked data/ directory instead.

Local polling stays off to avoid repeatedly paying for provider responses during schema and curation work. When price-history collection is intentional, enable it explicitly with TOKO_FEED_POLLING_ENABLED=true bash scripts/dev/configure-local.sh; disabling it again uses the same command with false.

Run toko-feed backup after curation to atomically replace data/canonical.json with the current canonical collections, Pokémon sets, and Pokémon cards. The snapshot deliberately excludes raw sources, observations, prices, scheduler state, and operational logs. It is a migration safety copy, not an automatic restore format while the schema is still changing.

make local-start remains available when only the replica is needed. The ignored .env.local file contains the provider credential only; scheduler policy is non-secret control-plane state and does not belong there. If an intentionally clean local database is required, make local-reset also starts the replica when necessary, then performs a reinstall and therefore erases all existing local Toko Feed data before replaying the local provider cache. Removing the relevant source/providers/ directory makes the next reset perform and save one fresh live bootstrap.

Direct local database inspection

Toko Feed exports IcyDB's controller-only readonly SQL and accepted-schema inspection endpoints. It does not export SQL mutation or DDL endpoints. After make local-ready, open the interactive database shell with:

icydb sql --environment local --canister toko-feed

Terminate interactive statements with ;. For one-shot inspection:

icydb sql -e local -c toko-feed --sql "SHOW ENTITIES"
icydb sql -e local -c toko-feed --sql "SELECT * FROM PokemonSetSource LIMIT 20"
icydb sql -e local -c toko-feed --sql "SELECT * FROM PokemonSet LIMIT 20"
icydb schema show toko-feed --environment local

IcyDB 0.221.6 derives the SHOW ENTITIES relation summary from the accepted catalog. Inspect the authoritative constraint details with:

icydb sql -e local -c toko-feed --sql "SHOW CONSTRAINTS FROM PokemonCard"
icydb sql -e local -c toko-feed --sql "SHOW CONSTRAINTS FROM SourceObservation"

Accepted links appear as validated relation constraints with restrict deletion semantics.

The ICP identity used by icydb must control the canister. The default local workflow installs under the anonymous identity; use the same explicit identity for deployment and inspection when overriding LOCAL_IDENTITY.

This unreleased schema deliberately replaces the earlier mixed set/card model. PokemonSetSource and PokemonCardSource now own provider records; PokemonSet and PokemonCard now own normalized canonical truth. Observation relations and receipt fields changed with that split. Every entity remains at schema version 1 while the data model is pre-migration and disposable. A canister installed from 0.1.6 or an earlier unreleased build needs make local-reset; no migration is claimed for these development layouts. Do not upgrade a persistent production database from them until an explicit IcyDB migration has been prepared and tested.

Local setup and CLI calls default to ICP's anonymous identity, which the managed development network funds and which becomes the local canister controller. Override setup with make local-ready LOCAL_IDENTITY=<name> when testing another identity. Protected identities can be used by the installed CLI with --identity and --identity-password-file.

Use the installed CLI for every operational and query endpoint. It targets the local ICP environment and toko-feed canister by default, passes raw typed arguments to ICP CLI internally, and prints clean typed JSON:

toko-feed bootstrap
toko-feed bootstrap --source
toko-feed bootstrap --source --refresh
toko-feed backup
toko-feed provider set tcgdex ecard2
toko-feed provider set tcgdex ecard2 --refresh --cards 10
toko-feed provider set tcgdex ecard2 --refresh --all
toko-feed provider set scrydex ecard2
toko-feed provider set scrydex ecard2 --refresh --cards 10
toko-feed provider set scrydex ecard2 --refresh --all
toko-feed status
toko-feed scheduler
toko-feed runs list --limit 100
toko-feed logs list --all
toko-feed collections ingest
toko-feed collections list --all
toko-feed collections get 00000000000000000000000001
toko-feed sets ingest
toko-feed sets sources list --all
toko-feed sets sources get 01KZ9GFKW3SY1G000000000001
toko-feed sets list --limit 10
toko-feed sets list --limit 100 --all
toko-feed sets list --limit 10 --after 01KZ9GFKW3SY1G000000000001
toko-feed sets curate 01KZ9GFKW3SY1G000000000001 --name "Base Set" --release-date 1999-01-09
toko-feed sets get 01KZ9GFKW3SY1G000000000002
toko-feed sets lock 01KZ9GFKW3SY1G000000000002 --revision 0
toko-feed sets sources reject 01KZ9GFKW3SY1G000000000003
toko-feed cards ingest
toko-feed cards ingest --set 01KZ9GFKW3SY1G000000000002 --limit 10
toko-feed cards ingest --set 01KZ9GFKW3SY1G000000000002 --offset 10 --limit 10
toko-feed cards sources list --all
toko-feed cards sources get 01KZ9GFKW3SY1G000000000004
toko-feed cards curate 01KZ9GFKW3SY1G000000000004 --set 01KZ9GFKW3SY1G000000000002 --name "Alakazam" --collector-number "1/102" --rarity "Rare Holo"
toko-feed cards list --limit 100 --all
toko-feed cards get 01KZ9GFKW3SY1G000000000005
toko-feed cards lock 01KZ9GFKW3SY1G000000000005 --revision 0
toko-feed cards prices 01KZ9GFKW3SY1G000000000005 --limit 100
toko-feed sealed list --all
toko-feed sealed get 01KZ9GFKW3SY1G000000000002
toko-feed sealed prices 01KZ9GFKW3SY1G000000000002 --limit 100

Without --all, a list call returns one page and its next_after continuation, which can be passed back through --after. With --all, the CLI follows that cursor, combines the records, and reports its page and record counts. Automatic pagination stops on a repeated cursor and defaults to at most 1,000 pages; use --max-pages to choose a lower operational bound.

History commands use next_before rather than next_after. For a manual next page, copy both returned fields into --before-time and --before-id. Use --all to have the CLI follow compound cursors and enforce the same repeated- cursor and maximum-page safeguards automatically.

Targeted card ingestion uses a canonical Toko set ULID, never a provider ID. It starts at offset zero unless --offset is supplied and defaults to the bounded maximum of 10 provider records. Copy the returned next_offset into the next command; a null continuation with provider_complete: true means that set is complete. The receipt reports card and sealed counts separately. Repeating an offset safely updates existing provider source rows and deduplicates unchanged immutable source and price observations. Sealed products use the same provider feed and therefore have no separate ingest command.

Set and card curation is controller-only. Begin with sets sources get or cards sources get; the positional ULID passed to curate is always that provider source ID. Omit --set/--card and --revision to create new canonical truth. Supply the existing canonical ID and its current revision as a pair to update it or attach another provider source. Set curation requires either --release-date YYYY-MM-DD or --no-release-date; card curation requires either --rarity VALUE or --no-rarity. Read the canonical row again before lock and pass its current revision. Locking is final.

Run toko-feed --help for the complete syntax. Global options must precede the command. --environment, --canister, and --identity select another ICP deployment; --project-root overrides icp.yaml discovery; --identity-password-file supports protected non-interactive identities; and --compact emits one-line JSON. For example:

toko-feed --environment ic --canister <principal> --identity operator status

The CLI deliberately has no configuration or API-key argument. Configuration continues to read the ignored .env.local file so credentials are not copied into command history.

ICP CLI keeps ephemeral local network state under ignored .icp/cache/ paths. Connected-network mappings under .icp/data/ are durable deployment records and may be committed when an IC canister is deployed.

make ic-deploy builds and deploys the same canister to the IC using the active ICP identity. It intentionally does not copy a local credential or trigger a paid provider import; configure those production operations through an appropriately protected controller workflow.

API access does not itself grant permission to cache or redistribute provider data. JustTCG's current commercial-use guidance and terms distinguish displaying or combining provider-derived information from redistributing raw data or offering a substitute bulk pricing API. Toko Feed therefore keeps raw payload evidence internal and exposes bounded, attributed product views. Review the active provider plan and terms before exposing any production feed publicly.

Layout

Path	Responsibility
`crates/toko-feed/src/api/just_tcg`	JustTCG request construction, wire models, pagination, compatibility, and sanitized tests.
`crates/toko-feed/src/api/tcgdex`	Credential-free TCGdex v2 request construction, provider wire models, and sanitized failures.
`crates/toko-feed/src/api/scrydex`	Authenticated Scrydex Pokémon v1 request construction, provider wire models, bounded pages, and redacted failures.
`crates/toko-feed/src/canister`	Candid endpoints, IC HTTPS adapter, durable timer/retry orchestration, bounded ingestion, and IcyDB access.
`crates/toko-feed/src/collection.rs`	Curated collectible collections, stable Toko IDs, provider aliases, and fallback matching policy.
`crates/toko-feed/src/pokemon`	Provider comparison projections, canonical identity keys, broad sealed-kind classification, and price normalization.
`crates/toko-feed/src/schema`	Embedded IcyDB canister plus job, run, log, observation, collection, Pokémon set/card/sealed, provider-source, Magic set/card/face templates, and future hub declarations. No hub actor is built yet.
`crates/toko-feed-cli`	Host-only installed CLI, typed ICP process boundary, lookup commands, and safe manual or automatic pagination.
`icp.yaml`	Official Rust recipe and local/IC project lifecycle definition.

Future APIs belong under api/<provider>. Their wire types should remain provider-specific; only explicitly curated, normalized identity fields belong in canonical set and card rows.

Development

The repository is pinned to Rust 1.97.1. Run every required native, Candid, and Wasm check with:

make check

Useful individual targets include make fmt, make clippy, make test, make docs, make wasm, and make candid-check. Tests use sanitized fixtures and never call the live provider.

make check retains Cargo artifacts for fast local iteration. CI runs make ci, which starts clean, redirects temporary files into target/tmp, and always removes Cargo plus Toko Feed-owned temporary artifacts on success, failure, or interruption. make clean performs the same scoped cleanup manually; it never removes unrelated entries from /tmp.

Releases and publication

Release notes live in CHANGELOG.md. Prepare its dated version entry and commit all non-version work before invoking a release target. cargo-edit is required for version management.

make release-patch
make release-minor
make release-major

These guarded targets run the full checks, bump the workspace version, create an annotated v<version> tag, and atomically push the release commit and tag. Minor and major releases require typed confirmation. Patch, minor, and major bump workflows use the same always-run cleanup, while the final push repeats cleanup defensively.

After the release is pushed, validate and publish the canister package followed by the CLI package with:

make publish-dry-run
make publish

License

Toko Feed is licensed under the MIT License.

toko-feed-cli 0.2.1