Toko Feed
Durable, provider-corroborated collectible data on the Internet Computer.
Toko Feed is a standalone canister that collects provider evidence, preserves price history, and turns reviewed records into stable Toko identities. It currently supports Pokémon data from JustTCG, TCGdex, and Scrydex, with an embedded IcyDB database and a typed operator CLI.
Provider data is evidence, not truth. Ingestion never silently creates or overwrites canonical Pokémon sets or cards.
✨ Highlights
- Self-contained canister — HTTPS outcalls, scheduling, retries, checkpoints, and storage live together.
- Multi-provider evidence — provider wire formats stay isolated and are retained without being flattened into a universal schema.
- Explicit reconciliation — corroborated records can become provisional truth; ambiguous records remain queued for human review.
- Queryable game mechanics — Pokémon types, HP, stages, retreat costs, attacks, costs, abilities, weaknesses, resistances, and rules are normalized into typed relations rather than buried in JSON.
- Attributed prices — every price keeps its provider, item, variant, currency, API version, and observation time.
- Upgrade-safe ingestion — durable jobs resume from bounded checkpoints and recover interrupted runs.
- Safe operations — credentials, response bodies, and raw transport errors never appear on public or operational surfaces.
🚀 Quick start
The repository pins Rust 1.97.1. The full build also expects the
wasm32-unknown-unknown target, the current icp CLI, didc, and
candid-extractor.
Build artifacts:
target/wasm32-unknown-unknown/release/toko_feed.wasmcrates/toko-feed/toko-feed.did
Run locally
Create an ignored credential file. JustTCG is required by canister configuration; Scrydex credentials are optional but must be supplied as a key/team pair. Bootstrap itself is cache-only and makes no provider requests.
# Edit .env.local.
make local-ready starts the managed local network, deploys the canister,
disables automatic polling, and replays every retained development provider
cache without network access. Cached evidence lives under ignored
source/providers/ directories; reviewed canonical snapshots live under
tracked data/.
Caution:
make local-resetreinstalls the local canister and erases its existing data before replaying the provider cache.
🧭 How the data flows
Provider APIs ──> ProviderRecord ──> indexed source projections
│ │
├──> ProviderObservation ▼
└──> ProviderListing ──> reconciliation ──> canonical catalogue
│ │
└──> PriceObservation └──> public queries
The boundaries are deliberate:
ProviderRecordstores each latest provider payload once. Pokémon source rows are indexed projections related to that raw record and carry their reconciliation state:pending,mapped, orrejected.ProviderObservationretains immutable payload versions. Current market identities live inProviderListing;PriceObservationretains their immutable attributed history.Set,Item, andItemVariantform the one canonical catalogue shared by every collection. Their lifecycle isprovisional→verified→locked.- Pokémon and Magic mechanics are queryable extension relations rooted at the
shared
Item; provider JSON remains linked evidence and is never rewritten as though it were canonical.
Safe reconciliation accepts only corroborated, non-conflicting candidates and
creates provisional rows. Explicit reconciliation resolves single-source
claims, conflicts, and physical printing suffixes such as 50a/50b. Locking
protects canonical fields while additional matching provider evidence may still
be attached. A controller can revision-safely unlock a row back to verified;
stronger administrator-priority policy is deferred. A sealed product is a
normal canonical Item(kind = sealed product) with a small Pokémon packaging
classification extension; provider evidence remains separate.
Canonical IDs are Toko-owned ULIDs generated before persistence, so a snapshot can restore them exactly instead of accepting replacement database IDs. Collection IDs use explicit, namespace-stable assignments; unknown provider categories are rejected before storage. Provider keys are mappings, never canonical identities. The ULID is the sole stored identity on a canonical row: there is no parallel canonical key and no provider, source, or external ID in canonical truth. Scoped domain uniqueness and accepted source mappings make reconciliation idempotent.
Prices are never canonical values. Current prices are derived from mapped sources, while history remains a sequence of attributed observations. Values in different currencies are not compared implicitly.
Gameplay metadata follows the same evidence boundary. Its existing relational
model is intentionally unchanged by this structural hard cut while its
long-term vocabulary is reviewed separately. Each provider adapter
preserves its wire model and owns its normalization into the relevant
collection vocabulary. Pokémon metadata is stored as relations: card types,
attacks, typed attack-cost counts, abilities, type modifiers, and rules. Printed damage
such as 30+, 30×, and 40- is represented by a numeric amount and an enum
operation, so it remains searchable without discarding its rules meaning.
IcyDB does not yet expose typed continuation cursors for this lookup, so type
search uses a deterministic materialized key derived from the type and metadata
ULID; the enum remains the canonical value.
Magic metadata is likewise relational rather than a JSON blob. Card faces own
ordered mana symbols, typed power/toughness expressions, color and color-
identity membership, deduplicated type terms, classified rules paragraphs, and
deduplicated mechanic terms such as Banding and First strike. Expressions such
as 2+* retain their star coefficient and numeric offset as separately
queryable fields. Format legality is deliberately left in provider evidence
because it changes over time.
Collections share the normalized Collection → Set → Item → ItemVariant
catalogue spine. Common identities and relations are never copied into
Pokemon* or Magic* roots. Collection-specific fields remain queryable in
extension entities such as PokemonCardMetadata and MagicCardFace, avoiding
both a universal row full of nullable mechanics and opaque canonical JSON.
The extension rules are documented in
the canonical data model.
🔌 Providers
| Provider | Integration | Credentials | Bounds |
|---|---|---|---|
| JustTCG v1 | Collections, Pokémon sets, cards, sealed products, and prices | API key | Card pages contain at most 10 records |
| TCGdex v2 English | Pokémon set indexes and card details | None | One provider request per canister call |
| Scrydex Pokémon v1 English | Pokémon expansions and cards, without price expansions | API key + team ID | Card pages contain at most 10 records |
| Scrydex Magic v1 | Magic expansions and cards, preserving mechanics and print evidence | API key + team ID | Card pages contain at most 10 records |
Each adapter owns request construction, wire models, compatibility aliases,
sanitized fixtures, and provider-local normalization under
crates/toko-feed/src/api/<provider>/. Tests never call live providers.
Provider acquisition has no adapter-named public methods. The canister exposes
toko_feed_ingest_provider_set and toko_feed_ingest_provider_cards; both take
a text provider field and dispatch internally. Canonical names are
just-tcg, tcgdex, and scrydex; ASCII case, hyphens, underscores, and
spaces are normalized. Unique leading fragments such as just and scry are
accepted without an alias table. Unknown, ambiguous, or suspicious names fail
before an HTTP request is made.
The current development corpus uses Expedition, Aquapolis, and Skyridge for JustTCG and an Aquapolis comparison across all three providers. Disagreement is retained as normal curation input instead of being hidden or rewritten.
Magic: The Gathering evidence can be acquired from JustTCG and Scrydex without
entering Pokémon tables. Set and card payloads retain each provider's wire
format in collection-scoped ProviderRecord rows plus immutable observations.
A controller may explicitly normalize one cached Scrydex set into the separate
Magic* canonical branch; all rows enter as provisional, never as verified
truth.
🧑💻 Operator workflow
1. Acquire or replay evidence
bootstrap replays all retained provider data and never calls an API. Use a
provider command with --refresh to update metadata and fetch missing card
positions. JustTCG selectors may be exact provider IDs or unambiguous display
names. Collection arguments use centralized, Unicode-safe matching:
magic is an unambiguous leading fragment of Magic: The Gathering; the full
slug and display name also work. Every accepted form resolves to the canonical
magic-the-gathering slug. There is no collection alias table to maintain. An
ambiguous fragment is rejected and must be made more specific. Scrydex uses
its provider expansion ID (DRK for The Dark). Complete cached card records
are not downloaded again.
Operator matching folds Latin diacritics and Unicode compatibility forms to
ASCII, so poke, Pokemon, and Pokémon all resolve to the canonical
pokemon collection. Invisible and non-Latin confusable characters remain
invalid; provider response bodies retain their original bytes.
Provider arguments follow the same rule: scry and just work as unique
leading fragments of scrydex and just-tcg, without a provider alias table.
2. Inspect the canister
status is the bounded reconciliation dashboard. Each collection/set row
separates canonical card totals and lock percentage from provider set/card
evidence, reports per-provider coverage, and supplies the highest-priority
typed next action. Collection and set filters use the same exact-or-unique-
prefix matching as acquisition. Provider-reported totals remain attributed
evidence and are never used as the denominator for canonical lock progress.
Human-readable text tables are the default. Add --json to any command for
the complete script-friendly JSON representation.
3. Reconcile canonical truth
Safe card acceptance processes bounded pages and leaves conflicts,
single-source candidates, and suffixed printings pending. Use the explicit
accept and accept-printing commands for reviewed exceptions; run the
relevant --help command for their required fields.
Reviewed exceptions use the same reconciliation boundary:
Every canonical write uses the latest revision. A stale revision returns
CanonicalConflict instead of overwriting another curator's work.
Metadata acceptance currently promotes one reviewed, mapped TCGdex or Scrydex
payload into a provisional normalized aggregate and records the exact source
row and normalizer version. Another provider can be attached when its complete
normalized projection agrees exactly; contradictions remain source evidence
and return CanonicalConflict. Field-by-field consensus is a later
reconciliation step, and raw evidence is always retained unchanged.
accept-source is the initial bounded Magic normalization path. It reads only
previously retained provider records, creates stable Toko-owned set, rules-card,
printing, face, and mechanic identities, and links each accepted source record
to the resulting canonical row. Repeating it is idempotent. The current adapter
is Scrydex Magic; the command remains provider-neutral so later adapters can
implement the same contract without adding provider-specific commands.
Magic verification and locking are deliberately not exposed yet, so these rows
remain in the reviewable provisional queue rather than implying approval.
4. Query canonical data
Without --all, list commands return one page and a continuation. Entity
lists use next_after; run, log, and price history use the compound
next_before cursor. Automatic pagination detects repeated cursors and is
bounded by --max-pages.
backup writes provider-independent canonical truth to
data/canonical.json; provider evidence, observations, scheduler state, and
credentials are deliberately excluded. restore validates that snapshot and
replays it in bounded dependency order. Repeating a matching restore is a
no-op, while an ID or scoped canonical-field conflict fails rather than
overwriting curated truth. make local-ready and make local-reset restore
canonical data before replaying the retained provider cache.
The move to application-owned canonical ULIDs changes the schema-v1 insert
policy for canonical entities. Existing development deployments created before
this change should run make local-reset once; the tracked snapshot restores
their canonical IDs and fields after reinstall. This is intentionally not
presented as an in-place production migration while the schema is still in its
reinstall-first development phase.
Global connection options must precede the command:
Use --identity-password-file for protected non-interactive identities.
Human-readable text is the default; --json selects raw JSON and
--json --compact puts it on one line. Output flags may appear after the
command. The CLI deliberately has no API-key argument; local credentials are
read by the configuration script from .env.local.
CLI hierarchy
toko-feed
├── bootstrap
├── backup
├── restore
├── status [--collection <COLLECTION>] [--set <SET>] [pagination]
├── scheduler
├── runs [pagination]
├── logs [pagination]
├── collections list|get
├── sets list|get|verify|lock|unlock
├── cards list|get|by-type|verify|lock|unlock|prices
├── sealed list|get|prices
├── sources
│ ├── collections list|get
│ ├── sets list|get|reject
│ ├── cards list|get|reject
│ └── sealed list|get
├── provider set <PROVIDER> <SET> [--collection <COLLECTION>] [--records <COUNT>|--all]
└── reconcile
├── sets plan|accept-source|accept-safe|accept
└── cards plan|accept-safe|accept|accept-printing|accept-metadata
Canonical resources, provider evidence, acquisition, and reconciliation each have one top-level home. Collection and sealed sources are read-only because they do not yet have a pending/rejected reconciliation lifecycle.
⏱️ Scheduling and reliability
The canister maintains three durable jobs: collections, Pokémon sets, and Pokémon cards. Successful runs recur after 24 hours, six hours, and the configured card cadence respectively; the default card cadence is 15 minutes.
Jobs use one-shot timers so intervals cannot overlap. A run is recorded before its first HTTPS await, each page is checkpointed, and interrupted work is recovered explicitly after upgrade. Failures use capped exponential backoff with jitter, separate from the normal success cadence.
Every response and replayed payload passes a non-destructive integrity screen for unsafe Unicode, mixed-script confusables, encoded payload markers, and excessive combining or repetition patterns. Findings produce fixed, credential-free warning categories while the original valid evidence remains unchanged.
🔐 Security notes
- Provider HTTPS requests are made only by the canister; the CLI has no provider transport.
- Credentials are controller-only configuration and are never returned by status, query, debug, or error surfaces.
- Logs contain typed codes and fixed bounded messages—not request headers, provider bodies, credentials, or raw transport diagnostics.
- Public methods expose bounded, attributed projections rather than raw provider JSON.
- Provider access does not grant redistribution rights. Review each active provider plan and terms before exposing a production feed.
🗄️ Database inspection
Install the operator CLI and the matching IcyDB CLI:
Toko Feed exposes controller-only, read-only SQL and schema inspection:
The ICP identity used by icydb must control the canister. SQL mutation and
DDL endpoints are not exposed.
🏗️ Repository layout
| Path | Responsibility |
|---|---|
crates/toko-feed |
Standalone canister, provider adapters, canonical rules, public protocol DTOs, and embedded schema |
crates/toko-feed/src/canister |
Authorization, HTTPS execution, bounded ingestion, durable jobs, and Candid endpoints |
crates/toko-feed/src/schema |
Stable IcyDB entities for sources, observations, canonical data, jobs, and logs |
crates/toko-feed-cli |
Host-only operator executable and typed icp process boundary |
crates/toko-feed/toko-feed.did |
Authoritative public Candid interface |
data/ |
Tracked curated canonical snapshots |
source/providers/ |
Ignored replayable provider evidence |
icp.yaml |
Local and IC project lifecycle configuration |
The canister is the product boundary. There is no separate schema canister, Canic dependency, or Toko repository dependency. A future fleet can consume the compiled Wasm as an opaque role; the compatibility boundary is documented in Canic fleet integration.
🧪 Development
Run the complete required check suite:
This runs formatting, Clippy, tests, rustdoc, the release Wasm build, and strict
equality between the compiled interface and
toko-feed.did.
Useful focused targets:
CI uses make ci, which starts with a clean target directory and removes only
Toko Feed-owned generated artifacts. Run make help for all targets.
🚢 Deployment and releases
Deploy the same standalone canister to the IC with the active ICP identity:
Deployment does not copy local credentials or trigger paid imports. Configure production credentials through a protected controller workflow and ensure the canister has enough cycles for HTTPS outcalls.
Release notes live in CHANGELOG.md. From a clean branch, after committing non-release work:
Release targets run the full checks, update workspace versions, create an
annotated tag, and push the release atomically. Publication uploads
toko-feed before toko-feed-cli.
📄 License
Licensed under the MIT License.