# Toko Feed
Toko Feed is a standalone Internet Computer canister for building a durable,
provider-corroborated source of collectible identities, metadata, prices, and
eventually real-time market feeds. It currently starts with Pokémon evidence
from JustTCG, TCGdex, and Scrydex, plus a schema-only Magic: The Gathering
template, but its authority comes from retained evidence and explicit Toko
identities—not from treating any API as truth. It owns its HTTPS integration,
polling engine, canonical identities, historical observations, pagination
checkpoints, and embedded IcyDB database. Consumers see only its Candid
endpoints; whether and how it stores the data is an internal implementation
detail.
The workspace has two deliberately separated packages. `toko-feed` builds as
an `rlib` for shared protocol types and native tests plus a `cdylib` canister.
`toko-feed-cli` is the host-only operator package and installs the `toko-feed`
executable. There is no separate `canisters/` tree, schema crate, Canic
dependency, or Toko dependency. A future Canic fleet can install many copies
of the compiled Wasm as opaque feed roles.
## Current behavior
- A controller configures the canister's JustTCG API key, optional paired
Scrydex API key and team ID, whether polling is enabled, and an optional card
cadence. Polling defaults to every 15 minutes; the accepted range is one
minute through seven days. Credentials are stored together in a versioned,
private configuration payload and never returned by public status methods.
- `toko_feed_ingest_collections` fetches JustTCG's complete bounded
[game catalog](https://www.justtcg.com/docs/api/games), interprets those
provider-specific game categories as Toko-owned collectible collections,
and stores each provider ID as a source mapping.
- `toko_feed_ingest_sets` fetches all Pokémon sets in one bounded
[JustTCG v1 sets request](https://justtcg.com/docs/api/sets) and upserts their
complete current metadata into `PokemonSetSource`. Ingestion never creates
or updates canonical `PokemonSet` rows. Each new source starts as `pending`
until a controller maps, rejects, or curates it. Provider fields include
release date, total set value,
card/variant/sealed counts, and 7/30/90-day value changes. Complete raw set
objects are retained so newly introduced provider fields are not discarded.
- `toko_feed_ingest_tcgdex_set` fetches one credential-free
[TCGdex v2 English set resource](https://tcgdex.dev/rest/sets).
`toko_feed_ingest_tcgdex_card` then consumes exactly one card
ID from that stored set response per call. Both write only provider source
rows and immutable observations; neither promotes data into canonical truth.
The current local Aquapolis evidence contains its complete 185-card index
and 177 usable detail responses. Eight indexed `Cube 01` cards currently
return provider-data failures and remain explicit acquisition holes rather
than disappearing from the comparison. TCGdex reports 147 official cards,
while the current JustTCG evidence reports 186 total printings, immediately
demonstrating why provider totals are evidence rather than authority.
- `toko_feed_ingest_scrydex_set` fetches one exact English Scrydex expansion;
`toko_feed_ingest_scrydex_cards` fetches one expansion-scoped page of at most
10 cards. Both calls run through the canister, omit price expansions, retain
provider wire records, and stage only source evidence. The default Aquapolis
comparison uses Scrydex's `ecard2` expansion ID. A full acquisition follows
19 rate-paced pages and retains all 182 card responses. Scrydex reports 147
printed cards, disagreeing with both other providers' total-printing models
while matching TCGdex's official count.
- Each bounded product run fetches and checkpoints at most 10 JustTCG records,
staying below both IcyDB's per-request execution budget and
[JustTCG's documented free-tier page limit](https://justtcg.com/docs/api/cards).
Repeated calls resume from the stored offset; reaching the provider's end
resets the next pass to offset zero.
- JustTCG returns unopened products through its card feed. Records with a
nonempty variant list whose conditions are all explicitly `Sealed` are
routed to `PokemonSealed`; they are never inserted into `PokemonCard`.
Packaging remains one entity with the broad `BoosterBox`, `BoosterPack`,
`Blister`, `Box`, `Bundle`, `Collection`, `Deck`, `Tin`, `Kit`, `Case`, and
`Other` enum rather than one table per product format.
- A controller can instead target one canonical set with
`toko_feed_ingest_set_cards`. Toko Feed resolves the set's JustTCG identifier
through the provider translation table, applies the provider's exact `set`
filter, and returns an explicit continuation without advancing the global
Pokémon cursor. Page evidence is partitioned by provider set and offset.
- The CLI's `bootstrap` operation builds the development corpus from provider
collections, all current Pokémon set metadata, and the complete Expedition,
Aquapolis, and Skyridge card feeds. With `--source`, retained records are
replayed before a requested refresh. Refreshing then updates only catalogs
and set metadata and resumes each card feed at its retained count; complete
local card sets make no card API requests. The operation composes ordinary
bounded canister calls with hard page bounds and no-progress detection.
- The CLI never connects to JustTCG and contains no provider HTTP transport.
Live bootstrap invokes typed controller-only canister methods through `icp`;
only the canister's IC HTTPS adapter constructs and sends provider requests.
The canister also owns provider decoding, card/sealed classification, set
partitioning, and replay validation. The CLI only writes the returned raw
JSON evidence to local files or sends those files back for canister replay.
- The same canister-only rule applies to TCGdex. `toko-feed provider set tcgdex
ecard2` replays `source/providers/tcgdex/v2/en` when present; `--refresh`
first replays that cache, performs one set-metadata request, and requests
only card IDs not already retained. The default acquisition target is a
10-card comparison sample; `--all` follows the complete ordered set index.
Full acquisition checkpoints every ten new details and reports missing or
unprojectable indexed cards without preventing later records from being
acquired. A gap behind a later retained index position is treated as a
previously attempted provider hole and is not retried on every refresh.
Removing the TCGdex source directory is the explicit full-refetch path. This
command takes TCGdex's exact set ID; after ingestion,
operator-visible names can be inspected with `toko-feed sets sources list
--all` but do not replace the provider identity.
- `toko-feed provider set scrydex ecard2` follows the same cache-first flow for
Scrydex. With no local cache it makes one expansion request and one bounded
10-card page request through the canister, then exports the retained
evidence. `--refresh --all` follows the provider's continuation through
rate-paced pages with page-continuity and no-progress checks. A refresh
replays and validates the cached contiguous provider prefix first, then
starts at the first uncached page; a complete cache makes no card-page
requests.
- The CLI persists the resulting provider evidence as neutral, ignored source
data rather than as a canonical backup. The file
`source/providers/just-tcg/v1/collections.json` contains provider categories,
`sets.json` contains every set, and `cards/<provider-set-id>.json` contains
each fetched set's card and sealed-product records. TCGdex and Scrydex use
equivalent provider-versioned trees under `source/providers/`. `toko-feed
bootstrap --source` replays every file present without HTTPS outcalls,
preserving original observation times. If the directory is absent, it
performs the live bootstrap and writes the split dataset; `--refresh`
refreshes provider catalogs and set metadata but retains immutable card
payloads and fetches only missing card positions. This is provider evidence
replay, not a canonical snapshot or an IcyDB physical backup. Tracked
canonical JSON belongs under `data/`.
- Three durable jobs poll collections, Pokémon sets, and Pokémon cards. IC
one-shot timers execute only the next due job, so intervals cannot overlap.
Successful collection, set, and card runs recur after 24 hours, six hours,
and the configured card cadence respectively. Failed runs retry from their
durable checkpoint with exponential backoff beginning at 30 seconds,
deterministic jitter, and a six-hour cap.
- Timer handles are recreated after upgrade. A run is written before its HTTPS
await; an upgrade that interrupts it records an `interrupted` failure and
schedules a retry instead of silently losing the attempt.
- Complete current provider records remain in their source tables. Separately,
content-addressed `SourceObservation` rows keep
each distinct provider payload, while `PriceObservation` rows retain each
distinct provider/variant/timestamp/price sample. Repeated identical payloads
are deduplicated without erasing the `IngestionRun` that proved the poll took
place. These rows use strong IcyDB relations to their run and typed provider
source rather than relying on unvalidated ULID-shaped values.
- Public methods return bounded, attributed projections and price observations
rather than raw provider JSON.
- Card and sealed-product identity rows never store a price, a lowest-price
summary, or a provider price timestamp. Current detail prices are derived
from the latest provider source payload and explicitly identify that provider
and its item key; historical queries read append-only observations instead.
- Re-importing a provider card updates only its existing `PokemonCardSource`
row and observations. A canonical `PokemonCard` exists only after explicit
curation. Sealed products retain their current source-plus-identity flow.
- Card ingestion requires an initial provider set import. JustTCG can
occasionally return a valid card whose set is absent from its sets endpoint;
that card is retained with its provider set ID and name instead of failing
the entire bounded ingestion job. When the provider set source is mapped,
the card source also carries a validated relation to its Toko canonical set;
otherwise that optional relation remains empty until a later refresh can
reconcile it.
- Detail queries expose source-attributed condition, printing, language,
current price, provider timestamp, and recent percentage changes for each
provider price variant. Price-history queries preserve different prices from
different providers rather than collapsing them into one canonical value.
- Every manual or timed attempt creates an append-only `IngestionRun` and
sanitized operational events. Logs contain fixed messages and bounded error
codes, never credentials, upstream bodies, or raw transport diagnostics.
- Every live response and logical cache replay is screened at ingress. The
non-destructive screen detects invalid
UTF-8; invisible, directional, private-use, tag, variation-selector, or
excessive combining Unicode; confusable mixed-script tokens; and suspicious
encoded or repeated-text patterns. It accepts visible multilingual text such
as accented Latin and Japanese rather than treating all non-ASCII data as
hostile.
- A finding appends a sanitized `warning` operational event with a bounded
`ProviderIntegrity` code, while the original valid provider bytes continue
through decoding and persistence unchanged. The warning identifies only the
category; suspect provider text is never copied into the alert. Disagreement
with canonical data is expected source evidence and is not an integrity
finding, even when the related canonical row is locked.
- The initial Magic template is canonical-schema only: `MagicSet`, `MagicCard`,
and ordered `MagicCardFace` rows are related to the existing Toko collection
identity for Magic: The Gathering. Face-specific mana cost, type line, rules
text, colors, power, toughness, loyalty, and defense remain separate from
Pokémon HP and attacks. No Magic provider, ingestion endpoint, or automatic
canonical write path is enabled yet.
## Evidence, truth, and API versions
Provider data is evidence, not authority. That boundary is physical for both
sets and cards:
1. `PokemonSetSource` and `PokemonCardSource` retain replaceable current
provider records, provider keys, free-form provider vocabulary, raw payloads,
and a `pending`, `mapped`, or `rejected` resolution state.
2. `SourceObservation` and `PriceObservation` retain immutable historical
evidence and point to those provider source rows.
3. `PokemonSet` and `PokemonCard` contain only explicitly curated Toko truth.
Provider polling has no write path into either table.
Curating a pending source creates a canonical row directly in `verified`
state, or maps it to an existing canonical row using that row's expected
revision. A verified row remains correctable by a controller. It can then be
moved to `locked`, after which no canonical field can ever be changed through
the canister. There is deliberately no unlock operation. Additional provider
sources can still be mapped to an already locked row when the supplied
canonical values match it exactly.
Every canonical operator write includes the revision returned by the latest
read. A stale revision returns `CanonicalConflict`, preventing two curators
from silently overwriting one another. `sets sources get` and
`cards sources get` expose unresolved evidence for review. Canonical detail
queries return only mapped evidence and derived pricing data, so pending or
rejected provider rows never masquerade as truth.
Every job, run, source observation, and price observation records its provider
API version. This lets v1 and v2—or several unrelated providers—coexist without
forcing their wire formats into one premature universal schema.
Toko-owned closed domains are variants throughout storage and Candid: ingestion
operations, manual versus timer triggers, observation target kinds, log
severity, and persisted failure categories cannot acquire accidental new
spellings. Normalized prices currently support the deliberately small currency
set `Usd`, `Cad`, `Eur`, `Gbp`, `Jpy`, and `Aud`. Toko does not assign an
inferred market to a price; any provider-specific market context stays in the
attributed raw evidence. Raw condition, printing, rarity, and other wire labels
also remain provider-owned text.
A price has no canonical value. `PriceObservation` is an append-only fact that
links a card source (or current sealed-product identity) to a provider,
provider item, provider variant, API version, currency, provider
timestamp, observed timestamp, and integer-micro price. Two providers quoting
the same card create two rows; a later quote creates another row. Nothing
updates a price on the identity record, and prices in different currencies are
not compared implicitly. Any future “best price” or aggregate is a
derived query or cache, not canonical card data.
The price-history layer samples current provider variants whenever the source,
provider item, variant, timestamp, or price changes. Each immutable row retains
the provider, API version, currency, source relation, and provider
keys. The full provider payload is also retained internally, so future
reconciliation can derive richer statistics without rewriting the original
evidence. Future providers should add their own source records and mapping
rules; agreement, disagreement, freshness, currency, and confidence remain
visible rather than being flattened into an unexplained “correct” value.
JustTCG currently exposes card identity and external IDs, variants, current
prices, statistics, and optional price history through its
[card](https://justtcg.com/docs/schema/card) and
[variant](https://justtcg.com/docs/schema/variant) schemas. Its v2 card API is
[documented as beta](https://justtcg.com/docs/api/cards-v2), so this release
records its active integration explicitly as `v1` and does not silently mix v2
responses into v1 observations.
TCGdex is integrated as an independent Pokémon evidence source using its
[v2 English REST resources](https://tcgdex.dev/rest). It
[requires no API key](https://tcgdex.dev/fr/faq), so no new secret is stored.
The adapter still uses bounded calls and the local ignored cache rather
than treating the absence of a published hard rate limit as permission to
send bursts. Its richer card payload—including HP and attacks—is retained raw
in `PokemonCardSource`. `category` describes a card kind such as `Pokemon`,
`Trainer`, or `Energy`; all are valid Pokémon TCG evidence. Those game-specific
details will be normalized only after the cross-provider canonical model is
chosen.
Scrydex is a third independent Pokémon evidence source. Its adapter sends the
required
[API-key and team-ID headers](https://scrydex.com/docs/getting-started/authentication)
only from the canister, marks both as sensitive, and requests the English
Pokémon v1 [expansion](https://scrydex.com/docs/pokemon/expansions) and
[card](https://scrydex.com/docs/pokemon/cards) resources without price
includes. Full acquisition remains deliberately paced at one bounded 10-card
page per call. Raw responses remain ignored local evidence; their normalized
match keys let operators compare them with JustTCG and TCGdex without silently
merging provider identities.
### Aquapolis evidence baseline
The ignored local capture completed on 2026-08-07. These are provider claims
and acquisition results, not canonical rows:
| JustTCG v1 | `aquapolis-pokemon` | 186 | 186 | Also exposes one sealed product. |
| TCGdex v2 English | `ecard2` | 185 | 177 | Omits #128 from its index; eight `Cube 01` details fail provider decoding. |
| Scrydex Pokémon v1 English | `ecard2` | 182 | 182 | Collapses four `a`/`b` artwork pairs into unsuffixed numbers. |
After removing harmless zero-padding from collector numbers, JustTCG contains
all 185 identities indexed by TCGdex plus Memory Berry #128. JustTCG and
TCGdex both distinguish `50a/50b`, `74a/74b`, `95a/95b`, and `103a/103b`;
Scrydex represents each pair as one unsuffixed card. Scrydex independently
contains #128. The eight missing TCGdex details are #119, #121, #122, #124,
#127, #129, #132, and #140; both other providers contain them.
The strongest current canonical candidate is therefore 186 distinct English
printings: retain the four `a`/`b` pairs and #128, but do not copy JustTCG's
presentation-only collector suffixes into card names. TCGdex and Scrydex agree
on every directly comparable name except spacing in Nidoran's gender symbol at
#96. One raw JustTCG payload also says `124/127` for Grass Cube 01 while the
other providers say `124/147`; this should remain a provider contradiction,
with `124` as the identity numerator. No rows have been promoted from this
analysis: a controller must still curate and later lock them explicitly.
## Canonical identity
A ULID is an identifier, not a deduplication rule. Canonical collections
therefore use two records:
1. `Collection` owns the Toko ULID, versioned canonical key, and display name.
2. `CollectionSource` maps `(provider, provider_key)` to that ULID and retains
the provider's complete category record.
The curated registry assigns a namespace-stable Toko ULID to every supported
collection. The current 18 JustTCG category records map to 17 canonical
collections because `pokemon` and `pokemon-japan` both identify Pokémon.
`one-piece-card-game` maps to the broader One Piece collection while retaining
the provider's name in its source record. Every canonical ID is identical in
every feed canister and is not derived from a provider ID.
There is deliberately no uncurated or locally generated collection state. A
provider category absent from the registry makes collection ingestion fail
before any collection rows are written. Supporting it requires an explicit
canonical name, key, and Toko ULID plus tests. `game` and `game_id` remain raw
JustTCG vocabulary; they are not canonical Toko concepts.
Set ingestion applies the same boundary: a set whose provider `game_id` has no
curated collection-source mapping is rejected before its provider source row
is written. A valid provider set is retained as evidence but is never promoted
automatically.
Sets and cards use the same explicit reconciliation pattern. Source rows own
generated local ULIDs and keep `(provider, provider_key)` unique. Canonical rows
also own generated Toko ULIDs, but they are not created until a curator accepts
a source. Multiple provider source rows can map to the same canonical row.
`PokemonCard` is fully normalized: its first field and primary key is `id`, and
its only set identity is the required `pokemon_set_id` relation. Provider
`set`, `set_name`, card IDs, and other free-form values remain exclusively in
`PokemonCardSource`. Likewise, `PokemonSet` stores one required canonical
`collection_id`; provider `game` and `game_id` remain exclusively in
`PokemonSetSource`.
Unopened products follow the same shape in `PokemonSealed` and
`PokemonSealedSource`. Their conservative versioned match key uses set name
and product name. Product kind is a query facet inferred from the normalized
provider name; an unfamiliar package remains `Other` instead of being rejected
or forcing a schema change.
The embedded schema enforces its single-target links. Collection, set, card,
job, run, source-observation, price-observation, and operational-log references
are strong IcyDB relations with restricted target deletion. The cross-kind
`SourceObservation.source_id` remains as an indexed lookup key and is
accompanied by exactly one typed relation to a collection, set source, card
source, or sealed product. Provider keys remain ordinary evidence fields and
are never treated as foreign keys.
When a card source has not been seen before, its conservative comparison key is
the normalized Pokémon set name, collector number, and card name. This lets a
future provider attach to an existing printing without pretending that
unrelated provider IDs are comparable. The key is explicitly versioned so a
future reconciliation migration can improve matching without changing public
ULIDs. A match key is evidence for a curator, never an automatic merge rule.
## Canister interface
The authoritative interface is
[toko-feed.did](crates/toko-feed/toko-feed.did).
| `toko_feed_configure` | Controller update | Store or rotate JustTCG and optional paired Scrydex credentials, and configure automatic card polling. |
| `toko_feed_provider_cache_page` | Controller query | Export one byte-bounded page of current raw provider JSON and its canister-validated set partition. |
| `toko_feed_replay_provider_cache` | Controller update | Validate the provider format, set partition, and source class, then replay one byte-bounded batch without making a provider request. |
| `toko_feed_ingest_collections` | Controller update | Refresh canonical collections from JustTCG source mappings. |
| `toko_feed_ingest_sets` | Controller update | Fetch and upsert the next bounded batch of current JustTCG sets into provider source storage. |
| `toko_feed_ingest_tcgdex_set` | Controller update | Fetch and stage one TCGdex v2 English set resource. |
| `toko_feed_ingest_tcgdex_card` | Controller update | Fetch and stage exactly one card selected by offset from a stored TCGdex set. |
| `toko_feed_ingest_scrydex_set` | Controller update | Fetch and stage one exact Scrydex Pokémon v1 English expansion. |
| `toko_feed_ingest_scrydex_cards` | Controller update | Fetch and stage one expansion-scoped Scrydex page of at most 10 cards. |
| `toko_feed_set_sources` / `toko_feed_set_source` | Public query | Review provider set records and their pending, mapped, or rejected resolution. |
| `toko_feed_curate_set` | Controller update | Create or update verified canonical set truth from one provider source. |
| `toko_feed_reject_set_source` | Controller update | Reject one unresolved provider set source. |
| `toko_feed_lock_set` | Controller update | Irreversibly lock a verified canonical set using its expected revision. |
| `toko_feed_ingest` | Controller update | Fetch, route, source-stage, and checkpoint up to 10 card-feed records. |
| `toko_feed_ingest_set_cards` | Controller update | Fetch up to 10 card-feed records for one canonical set from an explicit offset. |
| `toko_feed_ingest_provider_set_cards` | Controller update | Fetch a bounded explicit-offset card range by provider set ID for pre-curation acquisition. |
| `toko_feed_status` | Public query | Read credential-free cursor and health state. |
| `toko_feed_scheduler` | Public query | Read every durable job's cadence, next run, failure count, and active run. |
| `toko_feed_runs` | Public query | Read append-only ingestion attempts using a descending compound cursor. |
| `toko_feed_logs` | Public query | Read sanitized operational events using a descending compound cursor. |
| `toko_feed_collections` | Public query | Read up to 100 canonical collections using an optional ULID continuation. |
| `toko_feed_collection` | Public query | Read one canonical collection and all known provider mappings. |
| `toko_feed_sets` | Public query | Read up to 100 canonical set truth rows using an optional ULID continuation. |
| `toko_feed_set` | Public query | Read one canonical set together with its latest mapped provider evidence. |
| `toko_feed_card_sources` / `toko_feed_card_source` | Public query | Review provider card records and their resolution. |
| `toko_feed_curate_card` | Controller update | Create or update normalized verified card truth from one provider source. |
| `toko_feed_reject_card_source` | Controller update | Reject one unresolved provider card source. |
| `toko_feed_lock_card` | Controller update | Irreversibly lock a verified canonical card using its expected revision. |
| `toko_feed_cards` | Public query | Read up to 100 canonical card rows using an optional ULID continuation. |
| `toko_feed_card` | Public query | Read one canonical card and current prices derived from mapped sources. |
| `toko_feed_card_prices` | Public query | Read attributed, immutable price observations through mapped card sources. |
| `toko_feed_sealed` | Public query | Read up to 100 sealed-product summaries using an optional ULID continuation. |
| `toko_feed_sealed_product` | Public query | Read one sealed-product projection and its latest price variants. |
| `toko_feed_sealed_prices` | Public query | Read attributed, immutable price observations for one sealed-product identity. |
The HTTPS transform query is part of the IC management-canister outcall
protocol and is not an application endpoint.
### Query pagination
Start a listing with `toko_feed_collections(null, limit)`,
`toko_feed_set_sources(null, limit)`, `toko_feed_sets(null, limit)`,
`toko_feed_card_sources(null, limit)`, `toko_feed_cards(null, limit)`, or
`toko_feed_sealed(null, limit)`, then pass the returned `next_after` value into
the corresponding next call. Results are ordered by ascending local ULID and
`limit` must be between 1 and 100.
Missing records are returned as `Ok(null)` by their singular query; malformed
IDs and invalid limits return a bounded `FeedError`.
Run, log, and price-history pages are newest first. Begin with a null
`TimeCursor`, then pass the returned `next_before` record—both its timestamp and
ULID—to the next call. The compound cursor prevents equal timestamps from
dropping or repeating rows. These limits are also between 1 and 100.
## Operating the canister
1. Build and install the generated Wasm with the checked-in Candid interface.
The service constructor takes the empty Candid argument `()`.
2. Fund the canister with enough cycles for HTTPS outcalls. One ingestion can
make up to three JustTCG requests. Each TCGdex method makes exactly one
provider request.
3. Have a controller invoke `toko_feed_configure` with the API key and desired
polling settings through a secure deployment or administration path. The
canister creates and starts all three durable jobs; Toko does not need to
run a separate scheduler or database.
4. Use the controller-only ingestion methods for an immediate seed, repair, or
operator-requested refresh. They use exactly the same run, evidence,
checkpoint, logging, and retry bookkeeping as timer execution.
5. Monitor `toko_feed_scheduler`, `toko_feed_runs`, and `toko_feed_logs`.
Fund enough cycles for the configured request rate and choose a cadence that
fits the provider plan's [rate and quota limits](https://justtcg.com/docs/rate-limits).
Review every `warning` `provider_integrity_warning` promptly. The fixed
message identifies the suspicious Unicode or encoding category without
copying raw provider content. The evidence remains stored and the job
continues normally.
6. Review pending set and card sources, explicitly curate or reject each one,
and lock only canonical identities and metadata that should never change.
7. Let consumers query canonical collections, sets, and cards. Treat prices as
attributed observations rather than canonical values.
There is no deployable hub canister yet. A future Toko or Canic fleet can
coordinate multiple copies through this public interface while each feed
canister continues to own its database, provider integration, and retry loop.
The proposed topology, current Canic compatibility gap, opaque-role contract,
and secret/configuration boundaries are documented in
[Canic fleet integration](docs/architecture/canic-fleet.md).
## Build and installation
Install the pinned Rust toolchain, its Wasm target, ICP CLI, and `ic-wasm`, then
build the complete canister package:
```bash
rustup target add wasm32-unknown-unknown
make canister
```
The installable artifacts are:
- `target/wasm32-unknown-unknown/release/toko_feed.wasm`
- `crates/toko-feed/toko-feed.did`
The full interface gate also expects `didc` and `candid-extractor` on `PATH` so
the checked-in file can be compared structurally with the compiled Wasm.
`make canister` builds and validates the canister but does not install or
deploy it. The checked-in `icp.yaml` uses the official pinned Rust recipe for
ICP lifecycle builds, including Candid metadata and Wasm shrinking. Use
`make wasm` when only the compiler Wasm artifact is needed and interface
validation is not required.
Install or replace the checkout's operator CLI with:
```bash
make install
toko-feed --help
```
This follows the same package split and local-development pattern as
`ic-query`/`ic-query-cli`: Make performs the Cargo installation, then
`toko-feed` is invoked directly as a normal executable.
Operators who also need direct database inspection can install both CLIs with:
```bash
make install-tools
```
This installs the checkout's `toko-feed` binary and `icydb-cli` 0.221.6,
matching the embedded IcyDB dependency requirement. It
deliberately does not install an unconstrained latest pre-1.0 CLI.
On install and post-upgrade, the canister applies its embedded IcyDB schema.
Keep the API key out of shell history, logs, committed argument files, and
fixtures. The credential is never returned by status, data, debug, or error
surfaces.
### Local deployment
The checked-in `icp.yaml` defines a project-local managed network on port 8003
and the standalone canister build. Create an ignored local credential file,
restrict its permissions, then start, deploy, configure, and seed the canister:
```bash
cp .env.example .env.local
chmod 600 .env.local
# Edit .env.local and set JUST_TCG_API_KEY, SCRYDEX_API_KEY, and SCRYDEX_TEAM_ID.
make local-ready
```
`make local-ready` first ensures the managed local replica is healthy, then
builds and upgrades or installs the canister, configures it from `.env.local`,
keeps automatic polling disabled, and invokes the data-aware CLI bootstrap.
The first run imports the JustTCG provider category catalog, every current
Pokémon set, and all cards and sealed products from Expedition, Aquapolis, and
Skyridge. It also imports the complete paced TCGdex and Scrydex Aquapolis
comparison datasets. The local evidence is written under `source/providers/just-tcg/v1/` and
`source/providers/tcgdex/v2/en/` or `source/providers/scrydex/pokemon-v1/en/`.
Later runs replay every file without provider
requests. The complete `source/` payload is ignored and is reacquired through
the canister on a fresh clone. Set `TOKO_FEED_SOURCE`, `TOKO_FEED_TCGDEX_SOURCE`,
or `TOKO_FEED_SCRYDEX_SOURCE` to move a cache, or use the corresponding CLI
`--refresh` option to update set metadata and fill missing card records without
re-downloading retained cards. Reviewed canonical JSON snapshots belong in the
tracked `data/` directory instead.
Local polling stays off to avoid repeatedly paying for provider responses
during schema and curation work. When price-history collection is intentional,
enable it explicitly with
`TOKO_FEED_POLLING_ENABLED=true bash scripts/dev/configure-local.sh`; disabling
it again uses the same command with `false`.
Run `toko-feed backup` after curation to atomically replace
`data/canonical.json` with the current canonical collections, Pokémon sets, and
Pokémon cards. The snapshot deliberately excludes raw sources, observations,
prices, scheduler state, and operational logs. It is a migration safety copy,
not an automatic restore format while the schema is still changing.
`make local-start` remains available when only the replica is needed. The
ignored `.env.local` file contains the provider credential only; scheduler
policy is non-secret control-plane state and does not belong there.
If an intentionally clean local database is required,
`make local-reset` also starts the replica when necessary, then performs a
reinstall and therefore erases all existing local Toko Feed data before
replaying the local provider cache. Removing the relevant `source/providers/`
directory makes the next reset perform and save one fresh live bootstrap.
### Direct local database inspection
Toko Feed exports IcyDB's controller-only readonly SQL and accepted-schema
inspection endpoints. It does not export SQL mutation or DDL endpoints. After
`make local-ready`, open the interactive database shell with:
```bash
icydb sql --environment local --canister toko-feed
```
Terminate interactive statements with `;`. For one-shot inspection:
```bash
icydb sql -e local -c toko-feed --sql "SHOW ENTITIES"
icydb sql -e local -c toko-feed --sql "SELECT * FROM PokemonSetSource LIMIT 20"
icydb sql -e local -c toko-feed --sql "SELECT * FROM PokemonSet LIMIT 20"
icydb schema show toko-feed --environment local
```
IcyDB 0.221.6 derives the `SHOW ENTITIES` relation summary from the accepted
catalog. Inspect the authoritative constraint details with:
```bash
icydb sql -e local -c toko-feed --sql "SHOW CONSTRAINTS FROM PokemonCard"
icydb sql -e local -c toko-feed --sql "SHOW CONSTRAINTS FROM SourceObservation"
```
Accepted links appear as validated `relation` constraints with `restrict`
deletion semantics.
The ICP identity used by `icydb` must control the canister. The default local
workflow installs under the `anonymous` identity; use the same explicit
identity for deployment and inspection when overriding `LOCAL_IDENTITY`.
This unreleased schema deliberately replaces the earlier mixed set/card model.
`PokemonSetSource` and `PokemonCardSource` now own provider records;
`PokemonSet` and `PokemonCard` now own normalized canonical truth. Observation
relations and receipt fields changed with that split. Every entity remains at
schema version 1 while the data model is pre-migration and disposable. A
canister installed from 0.1.6 or an earlier unreleased build needs
`make local-reset`; no migration is claimed for these development layouts. Do
not upgrade a persistent production database from them until an explicit IcyDB
migration has been prepared and tested.
Local setup and CLI calls default to ICP's `anonymous` identity, which the
managed development network funds and which becomes the local canister
controller. Override setup with `make local-ready LOCAL_IDENTITY=<name>` when
testing another identity. Protected identities can be used by the installed
CLI with `--identity` and `--identity-password-file`.
Use the installed CLI for every operational and query endpoint. It targets the
local ICP environment and `toko-feed` canister by default, passes raw typed
arguments to ICP CLI internally, and prints clean typed JSON:
```bash
toko-feed bootstrap
toko-feed bootstrap --source
toko-feed bootstrap --source --refresh
toko-feed backup
toko-feed provider set tcgdex ecard2
toko-feed provider set tcgdex ecard2 --refresh --cards 10
toko-feed provider set tcgdex ecard2 --refresh --all
toko-feed provider set scrydex ecard2
toko-feed provider set scrydex ecard2 --refresh --cards 10
toko-feed provider set scrydex ecard2 --refresh --all
toko-feed status
toko-feed scheduler
toko-feed runs list --limit 100
toko-feed logs list --all
toko-feed collections ingest
toko-feed collections list --all
toko-feed collections get 00000000000000000000000001
toko-feed sets ingest
toko-feed sets sources list --all
toko-feed sets sources get 01KZ9GFKW3SY1G000000000001
toko-feed sets list --limit 10
toko-feed sets list --limit 100 --all
toko-feed sets list --limit 10 --after 01KZ9GFKW3SY1G000000000001
toko-feed sets curate 01KZ9GFKW3SY1G000000000001 --name "Base Set" --release-date 1999-01-09
toko-feed sets get 01KZ9GFKW3SY1G000000000002
toko-feed sets lock 01KZ9GFKW3SY1G000000000002 --revision 0
toko-feed sets sources reject 01KZ9GFKW3SY1G000000000003
toko-feed cards ingest
toko-feed cards ingest --set 01KZ9GFKW3SY1G000000000002 --limit 10
toko-feed cards ingest --set 01KZ9GFKW3SY1G000000000002 --offset 10 --limit 10
toko-feed cards sources list --all
toko-feed cards sources get 01KZ9GFKW3SY1G000000000004
toko-feed cards curate 01KZ9GFKW3SY1G000000000004 --set 01KZ9GFKW3SY1G000000000002 --name "Alakazam" --collector-number "1/102" --rarity "Rare Holo"
toko-feed cards list --limit 100 --all
toko-feed cards get 01KZ9GFKW3SY1G000000000005
toko-feed cards lock 01KZ9GFKW3SY1G000000000005 --revision 0
toko-feed cards prices 01KZ9GFKW3SY1G000000000005 --limit 100
toko-feed sealed list --all
toko-feed sealed get 01KZ9GFKW3SY1G000000000002
toko-feed sealed prices 01KZ9GFKW3SY1G000000000002 --limit 100
```
Without `--all`, a list call returns one page and its `next_after` continuation,
which can be passed back through `--after`. With `--all`, the CLI follows that
cursor, combines the records, and reports its page and record counts. Automatic
pagination stops on a repeated cursor and defaults to at most 1,000 pages; use
`--max-pages` to choose a lower operational bound.
History commands use `next_before` rather than `next_after`. For a manual next
page, copy both returned fields into `--before-time` and `--before-id`. Use
`--all` to have the CLI follow compound cursors and enforce the same repeated-
cursor and maximum-page safeguards automatically.
Targeted card ingestion uses a canonical Toko set ULID, never a provider ID.
It starts at offset zero unless `--offset` is supplied and defaults to the
bounded maximum of 10 provider records. Copy the returned `next_offset` into the next
command; a null continuation with `provider_complete: true` means that set is
complete. The receipt reports card and sealed counts separately. Repeating an
offset safely updates existing provider source rows and deduplicates unchanged
immutable source and price observations. Sealed products use the same provider
feed and therefore have no separate ingest command.
Set and card curation is controller-only. Begin with `sets sources get` or
`cards sources get`; the positional ULID passed to `curate` is always that
provider source ID. Omit `--set`/`--card` and `--revision` to create new
canonical truth. Supply the existing canonical ID and its current revision as
a pair to update it or attach another provider source. Set curation requires
either `--release-date YYYY-MM-DD` or `--no-release-date`; card curation
requires either `--rarity VALUE` or `--no-rarity`. Read the canonical row again
before `lock` and pass its current revision. Locking is final.
Run `toko-feed --help` for the complete syntax. Global options must precede the
command. `--environment`, `--canister`, and `--identity` select another ICP
deployment; `--project-root` overrides `icp.yaml` discovery;
`--identity-password-file` supports protected non-interactive identities; and
`--compact` emits one-line JSON. For example:
```bash
toko-feed --environment ic --canister <principal> --identity operator status
```
The CLI deliberately has no configuration or API-key argument. Configuration
continues to read the ignored `.env.local` file so credentials are not copied
into command history.
ICP CLI keeps ephemeral local network state under ignored `.icp/cache/` paths.
Connected-network mappings under `.icp/data/` are durable deployment records
and may be committed when an IC canister is deployed.
`make ic-deploy` builds and deploys the same canister to the IC using the active
ICP identity. It intentionally does not copy a local credential or trigger a
paid provider import; configure those production operations through an
appropriately protected controller workflow.
API access does not itself grant permission to cache or redistribute provider
data. JustTCG's current [commercial-use guidance](https://justtcg.com/docs/commercial-use)
and [terms](https://justtcg.com/terms) distinguish displaying or combining
provider-derived information from redistributing raw data or offering a
substitute bulk pricing API. Toko Feed therefore keeps raw payload evidence
internal and exposes bounded, attributed product views. Review the active
provider plan and terms before exposing any production feed publicly.
## Layout
| `crates/toko-feed/src/api/just_tcg` | JustTCG request construction, wire models, pagination, compatibility, and sanitized tests. |
| `crates/toko-feed/src/api/tcgdex` | Credential-free TCGdex v2 request construction, provider wire models, and sanitized failures. |
| `crates/toko-feed/src/api/scrydex` | Authenticated Scrydex Pokémon v1 request construction, provider wire models, bounded pages, and redacted failures. |
| `crates/toko-feed/src/canister` | Candid endpoints, IC HTTPS adapter, durable timer/retry orchestration, bounded ingestion, and IcyDB access. |
| `crates/toko-feed/src/collection.rs` | Curated collectible collections, stable Toko IDs, provider aliases, and fallback matching policy. |
| `crates/toko-feed/src/pokemon` | Provider comparison projections, canonical identity keys, broad sealed-kind classification, and price normalization. |
| `crates/toko-feed/src/schema` | Embedded IcyDB canister plus job, run, log, observation, collection, Pokémon set/card/sealed, provider-source, Magic set/card/face templates, and future hub declarations. No hub actor is built yet. |
| `crates/toko-feed-cli` | Host-only installed CLI, typed ICP process boundary, lookup commands, and safe manual or automatic pagination. |
| `icp.yaml` | Official Rust recipe and local/IC project lifecycle definition. |
Future APIs belong under `api/<provider>`. Their wire types should remain
provider-specific; only explicitly curated, normalized identity fields belong
in canonical set and card rows.
## Development
The repository is pinned to Rust 1.97.1. Run every required native, Candid, and
Wasm check with:
```bash
make check
```
Useful individual targets include `make fmt`, `make clippy`, `make test`,
`make docs`, `make wasm`, and `make candid-check`. Tests use sanitized fixtures
and never call the live provider.
`make check` retains Cargo artifacts for fast local iteration. CI runs
`make ci`, which starts clean, redirects temporary files into `target/tmp`, and
always removes Cargo plus Toko Feed-owned temporary artifacts on success,
failure, or interruption. `make clean` performs the same scoped cleanup
manually; it never removes unrelated entries from `/tmp`.
## Releases and publication
Release notes live in [CHANGELOG.md](CHANGELOG.md). Prepare its dated version
entry and commit all non-version work before invoking a release target.
[`cargo-edit`](https://github.com/killercup/cargo-edit) is required for version
management.
```bash
make release-patch
make release-minor
make release-major
```
These guarded targets run the full checks, bump the workspace version, create
an annotated `v<version>` tag, and atomically push the release commit and tag.
Minor and major releases require typed confirmation. Patch, minor, and major
bump workflows use the same always-run cleanup, while the final push repeats
cleanup defensively.
After the release is pushed, validate and publish the canister package followed
by the CLI package with:
```bash
make publish-dry-run
make publish
```
## License
Toko Feed is licensed under the [MIT License](LICENSE).