# Toko Feed
> Durable, provider-corroborated collectible data on the Internet Computer.
Toko Feed is a standalone canister that collects provider evidence, preserves
price history, and turns reviewed records into stable Toko identities. It
currently supports Pokémon data from JustTCG, TCGdex, and Scrydex, with an
embedded IcyDB database and a typed operator CLI.
Provider data is **evidence, not truth**. Ingestion never silently creates or
overwrites canonical Pokémon sets or cards.
## ✨ Highlights
- **Self-contained canister** — HTTPS outcalls, scheduling, retries,
checkpoints, and storage live together.
- **Multi-provider evidence** — provider wire formats stay isolated and are
retained without being flattened into a universal schema.
- **Explicit reconciliation** — corroborated records can become provisional
truth; ambiguous records remain queued for human review.
- **Queryable game mechanics** — Pokémon types, HP, stages, retreat costs,
attacks, costs, abilities, weaknesses, resistances, and rules are normalized
into typed relations rather than buried in JSON.
- **Attributed prices** — every price keeps its provider, item, variant,
currency, API version, and observation time.
- **Upgrade-safe ingestion** — durable jobs resume from bounded checkpoints
and recover interrupted runs.
- **Safe operations** — credentials, response bodies, and raw transport errors
never appear on public or operational surfaces.
## 🚀 Quick start
The repository pins Rust 1.97.1. The full build also expects the
`wasm32-unknown-unknown` target, the current `icp` CLI, `didc`, and
`candid-extractor`.
```bash
rustup target add wasm32-unknown-unknown
make canister
make install
toko-feed --help
```
Build artifacts:
- `target/wasm32-unknown-unknown/release/toko_feed.wasm`
- `crates/toko-feed/toko-feed.did`
### Run locally
Create an ignored credential file. JustTCG is required by canister
configuration; Scrydex credentials are optional but must be supplied as a
key/team pair. Bootstrap itself is cache-only and makes no provider requests.
```bash
cp .env.example .env.local
chmod 600 .env.local
# Edit .env.local.
make local-ready
toko-feed status
```
`make local-ready` starts the managed local network, deploys the canister,
disables automatic polling, and replays every retained development provider
cache without network access. Cached evidence lives under ignored
`source/providers/` directories; reviewed canonical snapshots live under
tracked `data/`.
> **Caution:** `make local-reset` reinstalls the local canister and erases its
> existing data before replaying the provider cache.
## 🧭 How the data flows
```text
Provider APIs ──> current source rows ──> immutable observations
│
▼
reconciliation ──> canonical Toko rows ──> public queries
```
The boundaries are deliberate:
1. Collection, set, card, and sealed source rows retain current, replaceable
provider records. Set and card evidence also carries its reconciliation
state: `pending`, `mapped`, or `rejected`.
2. `SourceObservation` and `PriceObservation` retain immutable historical
evidence.
3. `PokemonSet`, `PokemonCard`, `PokemonCardPrinting`, and the normalized
Pokémon metadata entities contain curated Toko truth. Their lifecycle is
`provisional` → `verified` → `locked`.
Safe reconciliation accepts only corroborated, non-conflicting candidates and
creates provisional rows. Explicit reconciliation resolves single-source
claims, conflicts, and physical printing suffixes such as `50a`/`50b`. Locking
is irreversible; additional matching provider evidence may still be attached
to a locked row. Sealed products remain provider-derived projections until a
dedicated canonical reconciliation model is introduced.
Canonical IDs are Toko-owned ULIDs generated before persistence, so a snapshot
can restore them exactly instead of accepting replacement database IDs.
Collection IDs use explicit, namespace-stable assignments; unknown provider
categories are rejected before storage. Provider keys are mappings, never
canonical identities.
Prices are never canonical values. Current prices are derived from mapped
sources, while history remains a sequence of attributed observations. Values
in different currencies are not compared implicitly.
Gameplay metadata follows the same evidence boundary. Each provider adapter
preserves its wire model and owns its normalization into shared Pokémon
vocabulary. Canonical metadata is stored as relations: card types, attacks,
typed attack-cost counts, abilities, type modifiers, and rules. Printed damage
such as `30+`, `30×`, and `40-` is represented by a numeric amount and an enum
operation, so it remains searchable without discarding its rules meaning.
IcyDB 0.221 does not yet expose typed continuation cursors, so type lookup uses
a deterministic materialized search key derived from the type and metadata
ULID; the enum remains the canonical value.
Collections share only the generic `Collection` catalogue root. Their storage
branches remain typed and separate: Pokémon uses `PokemonSet`, `PokemonCard`,
printings, sealed products, and Pokémon mechanics; Magic uses `MagicSet`, a
rules-level `MagicCard`, `MagicCardPrinting`, and ordered `MagicCardFace` rows.
No Magic row can be stored in a Pokémon table. Cross-collection search will be
a read model over these branches rather than a universal table full of
collection-specific nullable columns. The extension rules are documented in
[the canonical data model](docs/architecture/data-model.md).
## 🔌 Providers
| JustTCG v1 | Collections, Pokémon sets, cards, sealed products, and prices | API key | Card pages contain at most 10 records |
| TCGdex v2 English | Pokémon set indexes and card details | None | One provider request per canister call |
| Scrydex Pokémon v1 English | Pokémon expansions and cards, without price expansions | API key + team ID | Card pages contain at most 10 records |
| Scrydex Magic v1 | Magic expansions and cards, preserving mechanics and print evidence | API key + team ID | Card pages contain at most 10 records |
Each adapter owns request construction, wire models, compatibility aliases,
sanitized fixtures, and provider-local normalization under
`crates/toko-feed/src/api/<provider>/`. Tests never call live providers.
Provider acquisition has no adapter-named public methods. The canister exposes
`toko_feed_ingest_provider_set` and `toko_feed_ingest_provider_cards`; both take
a text `provider` field and dispatch internally. Canonical names are
`just-tcg`, `tcgdex`, and `scrydex`; ASCII case, hyphens, underscores, and
spaces are normalized. Unique leading fragments such as `just` and `scry` are
accepted without an alias table. Unknown, ambiguous, or suspicious names fail
before an HTTP request is made.
The current development corpus uses Expedition, Aquapolis, and Skyridge for
JustTCG and an Aquapolis comparison across all three providers. Disagreement
is retained as normal curation input instead of being hidden or rewritten.
Magic: The Gathering evidence can be acquired from JustTCG and Scrydex without
entering Pokémon tables or prematurely creating canonical Magic truth. Current
set and card payloads retain each provider's wire format in collection-scoped
`ProviderRecord` rows plus immutable observations; future Magic reconciliation
will project those records into the separate rules, printing, face, and set
hierarchy.
## 🧑💻 Operator workflow
### 1. Acquire or replay evidence
```bash
toko-feed bootstrap
toko-feed provider set just-tcg aquapolis-pokemon --all
toko-feed provider set tcgdex ecard2 --all
toko-feed provider set scry ecard2 --all
toko-feed provider set just-tcg "The Dark" \
--collection magic --all
toko-feed provider set scry DRK \
--collection magic --all
```
`bootstrap` replays all retained provider data and never calls an API. Use a
provider command with `--refresh` to update metadata and fetch missing card
positions. JustTCG selectors may be exact provider IDs or unambiguous display
names. Collection arguments use centralized, Unicode-safe matching:
`magic` is an unambiguous leading fragment of `Magic: The Gathering`; the full
slug and display name also work. Every accepted form resolves to the canonical
`magic-the-gathering` slug. There is no collection alias table to maintain. An
ambiguous fragment is rejected and must be made more specific. Scrydex uses
its provider expansion ID (`DRK` for The Dark). Complete cached card records
are not downloaded again.
Operator matching folds Latin diacritics and Unicode compatibility forms to
ASCII, so `poke`, `Pokemon`, and `Pokémon` all resolve to the canonical
`pokemon` collection. Invisible and non-Latin confusable characters remain
invalid; provider response bodies retain their original bytes.
Provider arguments follow the same rule: `scry` and `just` work as unique
leading fragments of `scrydex` and `just-tcg`, without a provider alias table.
### 2. Inspect the canister
```bash
toko-feed status
toko-feed status --collection pokemon --all
toko-feed status --collection magic --set the
toko-feed status --collection pokemon --set aquapolis --json
toko-feed scheduler
toko-feed runs --limit 100
toko-feed logs --all
toko-feed sources collections list --all
toko-feed sources sets list --all
toko-feed sources cards list --all
toko-feed sources sealed list --all
```
`status` is the bounded reconciliation dashboard. Each collection/set row
separates canonical card totals and lock percentage from provider set/card
evidence, reports per-provider coverage, and supplies the highest-priority
typed next action. Collection and set filters use the same exact-or-unique-
prefix matching as acquisition. Provider-reported totals remain attributed
evidence and are never used as the denominator for canonical lock progress.
Human-readable text tables are the default. Add `--json` to any command for
the complete script-friendly JSON representation.
### 3. Reconcile canonical truth
```bash
toko-feed reconcile sets plan --all
toko-feed reconcile sets accept-safe \
pokemon-set:v1:pokemon:aquapolis:2003-01-15
toko-feed reconcile cards plan --set <SET_ULID> --all
toko-feed reconcile cards accept-safe --set <SET_ULID> --all
```
Safe card acceptance processes bounded pages and leaves conflicts,
single-source candidates, and suffixed printings pending. Use the explicit
`accept` and `accept-printing` commands for reviewed exceptions; run the
relevant `--help` command for their required fields.
Reviewed exceptions use the same reconciliation boundary:
```bash
toko-feed sources sets get <SOURCE_ULID>
toko-feed reconcile sets accept --source <SOURCE_ULID> \
--name "Base Set" --release-date 1999-01-09 --active
toko-feed sets verify <SET_ULID> --revision <REVISION>
toko-feed sets lock <SET_ULID> --revision <REVISION>
toko-feed sources cards get <SOURCE_ULID>
toko-feed reconcile cards accept --source <SOURCE_ULID> --set <SET_ULID> \
--name "Alakazam" --collector-number "1/102" --rarity "Rare Holo"
toko-feed reconcile cards accept-metadata \
--card <CARD_ULID> --source <SOURCE_ULID>
toko-feed cards verify <CARD_ULID> --revision <REVISION>
toko-feed cards lock <CARD_ULID> --revision <REVISION>
```
Every canonical write uses the latest revision. A stale revision returns
`CanonicalConflict` instead of overwriting another curator's work.
Metadata acceptance currently promotes one reviewed, mapped TCGdex or Scrydex
payload into a provisional normalized aggregate and records the exact source
row and normalizer version. Another provider can be attached when its complete
normalized projection agrees exactly; contradictions remain source evidence
and return `CanonicalConflict`. Field-by-field consensus is a later
reconciliation step, and raw evidence is always retained unchanged.
### 4. Query canonical data
```bash
toko-feed collections list --all
toko-feed sets list --all
toko-feed sets get <SET_ULID>
toko-feed cards list --all
toko-feed cards get <CARD_ULID>
toko-feed cards by-type fire --all
toko-feed cards prices <CARD_ULID> --limit 100
toko-feed sealed list --all
toko-feed backup
toko-feed restore
```
Without `--all`, list commands return one page and a continuation. Entity
lists use `next_after`; run, log, and price history use the compound
`next_before` cursor. Automatic pagination detects repeated cursors and is
bounded by `--max-pages`.
`backup` writes provider-independent canonical truth to
`data/canonical.json`; provider evidence, observations, scheduler state, and
credentials are deliberately excluded. `restore` validates that snapshot and
replays it in bounded dependency order. Repeating a matching restore is a
no-op, while a conflicting ID or canonical key fails rather than overwriting
curated truth. `make local-ready` and `make local-reset` restore canonical data
before replaying the retained provider cache.
The move to application-owned canonical ULIDs changes the schema-v1 insert
policy for canonical entities. Existing development deployments created before
this change should run `make local-reset` once; the tracked snapshot restores
their canonical IDs and fields after reinstall. This is intentionally not
presented as an in-place production migration while the schema is still in its
reinstall-first development phase.
Global connection options must precede the command:
```bash
toko-feed --environment ic --canister <PRINCIPAL> \
--identity operator status
```
Use `--identity-password-file` for protected non-interactive identities.
Human-readable text is the default; `--json` selects raw JSON and
`--json --compact` puts it on one line. Output flags may appear after the
command. The CLI deliberately has no API-key argument; local credentials are
read by the configuration script from `.env.local`.
### CLI hierarchy
```text
toko-feed
├── bootstrap
├── backup
├── restore
├── status [--collection <COLLECTION>] [--set <SET>] [pagination]
├── scheduler
├── runs [pagination]
├── logs [pagination]
├── cards list|get|by-type|verify|lock|prices
├── sealed list|get|prices
├── sources
│ ├── collections list|get
│ ├── sets list|get|reject
│ ├── cards list|get|reject
│ └── sealed list|get
├── provider set <PROVIDER> <SET> [--collection <COLLECTION>] [--records <COUNT>|--all]
└── reconcile
├── sets plan|accept-safe|accept
└── cards plan|accept-safe|accept|accept-printing|accept-metadata
```
Canonical resources, provider evidence, acquisition, and reconciliation each
have one top-level home. Collection and sealed sources are read-only because
they do not yet have a pending/rejected reconciliation lifecycle.
## ⏱️ Scheduling and reliability
The canister maintains three durable jobs: collections, Pokémon sets, and
Pokémon cards. Successful runs recur after 24 hours, six hours, and the
configured card cadence respectively; the default card cadence is 15 minutes.
Jobs use one-shot timers so intervals cannot overlap. A run is recorded before
its first HTTPS await, each page is checkpointed, and interrupted work is
recovered explicitly after upgrade. Failures use capped exponential backoff
with jitter, separate from the normal success cadence.
Every response and replayed payload passes a non-destructive integrity screen
for unsafe Unicode, mixed-script confusables, encoded payload markers, and
excessive combining or repetition patterns. Findings produce fixed,
credential-free warning categories while the original valid evidence remains
unchanged.
## 🔐 Security notes
- Provider HTTPS requests are made only by the canister; the CLI has no
provider transport.
- Credentials are controller-only configuration and are never returned by
status, query, debug, or error surfaces.
- Logs contain typed codes and fixed bounded messages—not request headers,
provider bodies, credentials, or raw transport diagnostics.
- Public methods expose bounded, attributed projections rather than raw
provider JSON.
- Provider access does not grant redistribution rights. Review each active
provider plan and terms before exposing a production feed.
## 🗄️ Database inspection
Install the operator CLI and the matching IcyDB 0.221.6 CLI:
```bash
make install-tools
```
Toko Feed exposes controller-only, read-only SQL and schema inspection:
```bash
icydb sql --environment local --canister toko-feed
icydb sql -e local -c toko-feed --sql "SHOW ENTITIES"
icydb sql -e local -c toko-feed \
--sql "SELECT * FROM PokemonSetSource LIMIT 20"
icydb sql -e local -c toko-feed \
--sql "SELECT * FROM PokemonCardType LIMIT 20"
icydb schema show toko-feed --environment local
```
The ICP identity used by `icydb` must control the canister. SQL mutation and
DDL endpoints are not exposed.
## 🏗️ Repository layout
| `crates/toko-feed` | Standalone canister, provider adapters, canonical rules, public protocol DTOs, and embedded schema |
| `crates/toko-feed/src/canister` | Authorization, HTTPS execution, bounded ingestion, durable jobs, and Candid endpoints |
| `crates/toko-feed/src/schema` | Stable IcyDB entities for sources, observations, canonical data, jobs, and logs |
| `crates/toko-feed-cli` | Host-only operator executable and typed `icp` process boundary |
| `crates/toko-feed/toko-feed.did` | Authoritative public Candid interface |
| `data/` | Tracked curated canonical snapshots |
| `source/providers/` | Ignored replayable provider evidence |
| `icp.yaml` | Local and IC project lifecycle configuration |
The canister is the product boundary. There is no separate schema canister,
Canic dependency, or Toko repository dependency. A future fleet can consume
the compiled Wasm as an opaque role; the compatibility boundary is documented
in [Canic fleet integration](docs/architecture/canic-fleet.md).
## 🧪 Development
Run the complete required check suite:
```bash
make check
```
This runs formatting, Clippy, tests, rustdoc, the release Wasm build, and strict
equality between the compiled interface and
[`toko-feed.did`](crates/toko-feed/toko-feed.did).
Useful focused targets:
```bash
make fmt
make clippy
make test
make docs
make wasm
make candid-check
```
CI uses `make ci`, which starts with a clean target directory and removes only
Toko Feed-owned generated artifacts. Run `make help` for all targets.
## 🚢 Deployment and releases
Deploy the same standalone canister to the IC with the active ICP identity:
```bash
make ic-deploy
```
Deployment does not copy local credentials or trigger paid imports. Configure
production credentials through a protected controller workflow and ensure the
canister has enough cycles for HTTPS outcalls.
Release notes live in [CHANGELOG.md](CHANGELOG.md). From a clean branch, after
committing non-release work:
```bash
make release-patch # or release-minor / release-major
make publish-dry-run
make publish
```
Release targets run the full checks, update workspace versions, create an
annotated tag, and push the release atomically. Publication uploads
`toko-feed` before `toko-feed-cli`.
## 📄 License
Licensed under the [MIT License](LICENSE).