Toko Feed
Durable, provider-corroborated collectible data on the Internet Computer.
Toko Feed is a standalone canister that collects provider evidence, preserves price history, and turns reviewed records into stable Toko identities. It currently supports Pokémon data from JustTCG, TCGdex, and Scrydex, with an embedded IcyDB database and a typed operator CLI.
Provider data is evidence, not truth. Ingestion never silently creates or overwrites canonical Pokémon sets or cards.
✨ Highlights
- Self-contained canister — HTTPS outcalls, scheduling, retries, checkpoints, and storage live together.
- Multi-provider evidence — provider wire formats stay isolated and are retained without being flattened into a universal schema.
- Explicit reconciliation — corroborated records can become provisional truth; ambiguous records remain queued for human review.
- Queryable game mechanics — Pokémon types, HP, stages, retreat costs, attacks, costs, abilities, weaknesses, resistances, and rules are normalized into typed relations rather than buried in JSON.
- Attributed prices — every price keeps its provider, item, variant, currency, API version, and observation time.
- Upgrade-safe ingestion — durable jobs resume from bounded checkpoints and recover interrupted runs.
- Safe operations — credentials, response bodies, and raw transport errors never appear on public or operational surfaces.
🚀 Quick start
The repository pins Rust 1.97.1. The full build also expects the
wasm32-unknown-unknown target, the current icp CLI, didc, and
candid-extractor.
Build artifacts:
target/wasm32-unknown-unknown/release/toko_feed.wasmcrates/toko-feed/toko-feed.did
Run locally
Create an ignored credential file. JustTCG is required by canister configuration; Scrydex credentials are optional but must be supplied as a key/team pair. Bootstrap itself is cache-only and makes no provider requests.
# Edit .env.local.
make local-ready starts the managed local network, deploys the canister,
disables automatic polling, and replays every retained development provider
cache without network access. Cached evidence lives under ignored
source/providers/ directories; reviewed canonical snapshots live under
tracked data/.
Caution:
make local-resetreinstalls the local canister and erases its existing data before replaying the provider cache.
🧭 How the data flows
Provider APIs ──> current source rows ──> immutable observations
│
▼
reconciliation ──> canonical Toko rows ──> public queries
The boundaries are deliberate:
- Collection, set, card, and sealed source rows retain current, replaceable
provider records. Set and card evidence also carries its reconciliation
state:
pending,mapped, orrejected. SourceObservationandPriceObservationretain immutable historical evidence.PokemonSet,PokemonCard,PokemonCardPrinting, and the normalized Pokémon metadata entities contain curated Toko truth. Their lifecycle isprovisional→verified→locked.
Safe reconciliation accepts only corroborated, non-conflicting candidates and
creates provisional rows. Explicit reconciliation resolves single-source
claims, conflicts, and physical printing suffixes such as 50a/50b. Locking
is irreversible; additional matching provider evidence may still be attached
to a locked row. Sealed products remain provider-derived projections until a
dedicated canonical reconciliation model is introduced.
Canonical IDs are generated ULIDs. Collection IDs use explicit, namespace-stable assignments; unknown provider categories are rejected before storage. Provider keys are mappings, never canonical identities.
Prices are never canonical values. Current prices are derived from mapped sources, while history remains a sequence of attributed observations. Values in different currencies are not compared implicitly.
Gameplay metadata follows the same evidence boundary. Each provider adapter
preserves its wire model and owns its normalization into shared Pokémon
vocabulary. Canonical metadata is stored as relations: card types, attacks,
typed attack-cost counts, abilities, type modifiers, and rules. Printed damage
such as 30+, 30×, and 40- is represented by a numeric amount and an enum
operation, so it remains searchable without discarding its rules meaning.
IcyDB 0.221 does not yet expose typed continuation cursors, so type lookup uses
a deterministic materialized search key derived from the type and metadata
ULID; the enum remains the canonical value.
Collections share only the generic Collection catalogue root. Their storage
branches remain typed and separate: Pokémon uses PokemonSet, PokemonCard,
printings, sealed products, and Pokémon mechanics; Magic uses MagicSet, a
rules-level MagicCard, MagicCardPrinting, and ordered MagicCardFace rows.
No Magic row can be stored in a Pokémon table. Cross-collection search will be
a read model over these branches rather than a universal table full of
collection-specific nullable columns. The extension rules are documented in
the canonical data model.
🔌 Providers
| Provider | Integration | Credentials | Bounds |
|---|---|---|---|
| JustTCG v1 | Collections, Pokémon sets, cards, sealed products, and prices | API key | Card pages contain at most 10 records |
| TCGdex v2 English | Pokémon set indexes and card details | None | One provider request per canister call |
| Scrydex Pokémon v1 English | Pokémon expansions and cards, without price expansions | API key + team ID | Card pages contain at most 10 records |
| Scrydex Magic v1 | Magic expansions and cards, preserving mechanics and print evidence | API key + team ID | Card pages contain at most 10 records |
Each adapter owns request construction, wire models, compatibility aliases,
sanitized fixtures, and provider-local normalization under
crates/toko-feed/src/api/<provider>/. Tests never call live providers.
Provider acquisition has no adapter-named public methods. The canister exposes
toko_feed_ingest_provider_set and toko_feed_ingest_provider_cards; both take
a text provider field and dispatch internally. Canonical names are
just-tcg, tcgdex, and scrydex; ASCII case, hyphens, underscores, and
spaces are normalized, and just, jt, and scry are accepted operator
aliases. Unknown or suspicious names fail before an HTTP request is made.
The current development corpus uses Expedition, Aquapolis, and Skyridge for JustTCG and an Aquapolis comparison across all three providers. Disagreement is retained as normal curation input instead of being hidden or rewritten.
Magic: The Gathering evidence can be acquired from JustTCG and Scrydex without
entering Pokémon tables or prematurely creating canonical Magic truth. Current
set and card payloads retain each provider's wire format in collection-scoped
ProviderRecord rows plus immutable observations; future Magic reconciliation
will project those records into the separate rules, printing, face, and set
hierarchy.
🧑💻 Operator workflow
1. Acquire or replay evidence
bootstrap replays all retained provider data and never calls an API. Use a
provider command with --refresh to update metadata and fetch missing card
positions. JustTCG selectors may be exact provider IDs or unambiguous display
names. Collection arguments use the same centralized, Unicode-safe lookup
policy as provider arguments, so magic, mtg, magic-the-gathering, and
Magic: The Gathering all resolve to the canonical slug. Scrydex uses its
provider expansion ID (DRK for The Dark). Complete cached card records are
not downloaded again.
2. Inspect the canister
3. Reconcile canonical truth
Safe card acceptance processes bounded pages and leaves conflicts,
single-source candidates, and suffixed printings pending. Use the explicit
accept and accept-printing commands for reviewed exceptions; run the
relevant --help command for their required fields.
Reviewed exceptions use the same reconciliation boundary:
Every canonical write uses the latest revision. A stale revision returns
CanonicalConflict instead of overwriting another curator's work.
Metadata acceptance currently promotes one reviewed, mapped TCGdex or Scrydex
payload into a provisional normalized aggregate and records the exact source
row and normalizer version. Another provider can be attached when its complete
normalized projection agrees exactly; contradictions remain source evidence
and return CanonicalConflict. Field-by-field consensus is a later
reconciliation step, and raw evidence is always retained unchanged.
4. Query canonical data
Without --all, list commands return one page and a continuation. Entity
lists use next_after; run, log, and price history use the compound
next_before cursor. Automatic pagination detects repeated cursors and is
bounded by --max-pages.
Global connection options must precede the command:
Use --identity-password-file for protected non-interactive identities and
--compact for one-line JSON. The CLI deliberately has no API-key argument;
local credentials are read by the configuration script from .env.local.
CLI hierarchy
toko-feed
├── bootstrap
├── backup
├── status
├── scheduler
├── runs [pagination]
├── logs [pagination]
├── collections list|get
├── sets list|get|lock
├── cards list|get|by-type|lock|prices
├── sealed list|get|prices
├── sources
│ ├── collections list|get
│ ├── sets list|get|reject
│ ├── cards list|get|reject
│ └── sealed list|get
├── provider set <PROVIDER> <SET> [--collection <COLLECTION>]
└── reconcile
├── sets plan|accept-safe|accept
└── cards plan|accept-safe|accept|accept-printing|accept-metadata
Canonical resources, provider evidence, acquisition, and reconciliation each have one top-level home. Collection and sealed sources are read-only because they do not yet have a pending/rejected reconciliation lifecycle.
⏱️ Scheduling and reliability
The canister maintains three durable jobs: collections, Pokémon sets, and Pokémon cards. Successful runs recur after 24 hours, six hours, and the configured card cadence respectively; the default card cadence is 15 minutes.
Jobs use one-shot timers so intervals cannot overlap. A run is recorded before its first HTTPS await, each page is checkpointed, and interrupted work is recovered explicitly after upgrade. Failures use capped exponential backoff with jitter, separate from the normal success cadence.
Every response and replayed payload passes a non-destructive integrity screen for unsafe Unicode, mixed-script confusables, encoded payload markers, and excessive combining or repetition patterns. Findings produce fixed, credential-free warning categories while the original valid evidence remains unchanged.
🔐 Security notes
- Provider HTTPS requests are made only by the canister; the CLI has no provider transport.
- Credentials are controller-only configuration and are never returned by status, query, debug, or error surfaces.
- Logs contain typed codes and fixed bounded messages—not request headers, provider bodies, credentials, or raw transport diagnostics.
- Public methods expose bounded, attributed projections rather than raw provider JSON.
- Provider access does not grant redistribution rights. Review each active provider plan and terms before exposing a production feed.
🗄️ Database inspection
Install the operator CLI and the matching IcyDB 0.221.6 CLI:
Toko Feed exposes controller-only, read-only SQL and schema inspection:
The ICP identity used by icydb must control the canister. SQL mutation and
DDL endpoints are not exposed.
🏗️ Repository layout
| Path | Responsibility |
|---|---|
crates/toko-feed |
Standalone canister, provider adapters, canonical rules, public protocol DTOs, and embedded schema |
crates/toko-feed/src/canister |
Authorization, HTTPS execution, bounded ingestion, durable jobs, and Candid endpoints |
crates/toko-feed/src/schema |
Stable IcyDB entities for sources, observations, canonical data, jobs, and logs |
crates/toko-feed-cli |
Host-only operator executable and typed icp process boundary |
crates/toko-feed/toko-feed.did |
Authoritative public Candid interface |
data/ |
Tracked curated canonical snapshots |
source/providers/ |
Ignored replayable provider evidence |
icp.yaml |
Local and IC project lifecycle configuration |
The canister is the product boundary. There is no separate schema canister, Canic dependency, or Toko repository dependency. A future fleet can consume the compiled Wasm as an opaque role; the compatibility boundary is documented in Canic fleet integration.
🧪 Development
Run the complete required check suite:
This runs formatting, Clippy, tests, rustdoc, the release Wasm build, and strict
equality between the compiled interface and
toko-feed.did.
Useful focused targets:
CI uses make ci, which starts with a clean target directory and removes only
Toko Feed-owned generated artifacts. Run make help for all targets.
🚢 Deployment and releases
Deploy the same standalone canister to the IC with the active ICP identity:
Deployment does not copy local credentials or trigger paid imports. Configure production credentials through a protected controller workflow and ensure the canister has enough cycles for HTTPS outcalls.
Release notes live in CHANGELOG.md. From a clean branch, after committing non-release work:
Release targets run the full checks, update workspace versions, create an
annotated tag, and push the release atomically. Publication uploads
toko-feed before toko-feed-cli.
📄 License
Licensed under the MIT License.