# Toko Feed
Toko Feed is a standalone Internet Computer canister for building a durable,
provider-corroborated source of collectible identities, metadata, prices, and
eventually real-time market feeds. It currently starts with Pokémon and
JustTCG, but its authority comes from retained evidence and explicit Toko
identities—not from treating any API as truth. It owns its HTTPS integration,
polling engine, canonical identities, historical observations, pagination
checkpoints, and embedded IcyDB database. Consumers see only its Candid
endpoints; whether and how it stores the data is an internal implementation
detail.
The workspace has two deliberately separated packages. `toko-feed` builds as
an `rlib` for shared protocol types and native tests plus a `cdylib` canister.
`toko-feed-cli` is the host-only operator package and installs the `toko-feed`
executable. There is no separate `canisters/` tree, schema crate, Canic
dependency, or Toko dependency. A future Canic fleet can install many copies
of the compiled Wasm as opaque feed roles.
## Current behavior
- A controller configures the canister's JustTCG API key, whether polling is
enabled, and an optional card cadence. Polling defaults to every 15 minutes;
the accepted range is one minute through seven days.
- `toko_feed_ingest_collections` fetches JustTCG's complete bounded
[game catalog](https://www.justtcg.com/docs/api/games), interprets those
provider-specific game categories as Toko-owned collectible collections,
and stores each provider ID as a source mapping.
- `toko_feed_ingest_sets` fetches all Pokémon sets in one bounded
[JustTCG v1 sets request](https://justtcg.com/docs/api/sets) and upserts their
complete current metadata before card ingestion. A newly discovered set
also seeds a separate Toko-owned canonical row in `candidate` state.
The current provider fields include release date, total set value,
card/variant/sealed counts, and 7/30/90-day value changes. Complete raw set
objects are retained so newly introduced provider fields are not discarded.
- Each bounded card run fetches and checkpoints at most 50 Pokémon cards.
The requests are split into 20, 20, and 10 cards so they remain compatible
with [JustTCG's documented free-tier page limit](https://justtcg.com/docs/api/cards).
Repeated calls resume from the stored offset; reaching the provider's end
resets the next pass to offset zero.
- Three durable jobs poll collections, Pokémon sets, and Pokémon cards. IC
one-shot timers execute only the next due job, so intervals cannot overlap.
Successful collection, set, and card runs recur after 24 hours, six hours,
and the configured card cadence respectively. Failed runs retry from their
durable checkpoint with exponential backoff beginning at 30 seconds,
deterministic jitter, and a six-hour cap.
- Timer handles are recreated after upgrade. A run is written before its HTTPS
await; an upgrade that interrupts it records an `interrupted` failure and
schedules a retry instead of silently losing the attempt.
- Complete current provider records remain available to the internal
projection path. Separately, content-addressed `SourceObservation` rows keep
each distinct provider payload, while `PriceObservation` rows retain each
distinct provider/variant/timestamp/price sample. Repeated identical payloads
are deduplicated without erasing the `IngestionRun` that proved the poll took
place. These rows use strong IcyDB relations to their run and typed canonical
target rather than relying on unvalidated ULID-shaped values.
- Public methods return bounded, attributed projections and price observations
rather than raw provider JSON.
- Re-importing a provider card updates its existing row. Each local card
receives a Toko-owned generated ULID; provider IDs are stored in a separate
source mapping rather than used as the public identity.
- Card ingestion requires an initial provider set import. JustTCG can
occasionally return a valid card whose set is absent from its sets endpoint;
that card is retained with its provider set ID and name instead of failing
the entire bounded ingestion job. When the provider set is known, the card
also carries a validated relation to its Toko canonical set; otherwise that
optional relation remains empty until a later card refresh can reconcile it.
- Summary queries expose the lowest current USD price. Detail queries expose
condition, printing, language, current price, provider timestamp, and recent
percentage changes for each market variant.
- Every manual or timed attempt creates an append-only `IngestionRun` and
sanitized operational events. Logs contain fixed messages and bounded error
codes, never credentials, upstream bodies, or raw transport diagnostics.
## Evidence, truth, and API versions
Provider data is evidence, not authority. The set layer makes that boundary
physical: `PokemonSet` is a replaceable normalized provider projection,
`PokemonSetSource` retains the provider record and identifier mapping,
`SourceObservation` retains distinct historical payloads, and
`CanonicalPokemonSet` contains only Toko-owned truth.
A new canonical set begins as `candidate`, where later evidence may still
refresh its proposed name and release date. A controller can explicitly curate
the complete editable row to `verified`; provider ingestion then continues to
record and refresh evidence but cannot alter the canonical row. A verified row
can be moved to `locked`, after which no canonical field—including its active
or retired lifecycle state—can ever be changed through the canister. There is
deliberately no unlock operation.
Every canonical operator write includes the revision returned by the latest
read. A stale revision returns `CanonicalConflict`, preventing two curators
from silently overwriting one another. `sets get` returns the canonical row and
the latest normalized evidence together so a curator can compare them before
accepting anything.
Every job, run, source observation, and price observation records its provider
API version. This lets v1 and v2—or several unrelated providers—coexist without
forcing their wire formats into one premature universal schema.
The first price-history layer samples current JustTCG variant prices whenever
the provider timestamp or price changes. The full provider payload is retained
internally, so future reconciliation can derive richer statistics without
rewriting the original evidence. Future providers should add their own source
records and mapping rules; agreement, disagreement, freshness, market,
currency, and confidence must remain visible rather than being flattened into
an unexplained “correct” value.
JustTCG currently exposes card identity and external IDs, variants, current
prices, statistics, and optional price history through its
[card](https://justtcg.com/docs/schema/card) and
[variant](https://justtcg.com/docs/schema/variant) schemas. Its v2 card API is
[documented as beta](https://justtcg.com/docs/api/cards-v2), so this release
records its active integration explicitly as `v1` and does not silently mix v2
responses into v1 observations.
## Canonical identity
A ULID is an identifier, not a deduplication rule. Canonical collections
therefore use two records:
1. `Collection` owns the Toko ULID, versioned canonical key, and display name.
2. `CollectionSource` maps `(provider, provider_key)` to that ULID and retains
the provider's complete category record.
The curated registry assigns a namespace-stable Toko ULID to every supported
collection. The current 18 JustTCG category records map to 17 canonical
collections because `pokemon` and `pokemon-japan` both identify Pokémon.
`one-piece-card-game` maps to the broader One Piece collection while retaining
the provider's name in its source record. Every canonical ID is identical in
every feed canister and is not derived from a provider ID.
There is deliberately no uncurated or locally generated collection state. A
provider category absent from the registry makes collection ingestion fail
before any collection rows are written. Supporting it requires an explicit
canonical name, key, and Toko ULID plus tests. `game` and `game_id` remain raw
JustTCG vocabulary; they are not canonical Toko concepts.
Set ingestion applies the same boundary: a set whose provider `game_id` has no
curated source mapping is rejected before its provider projection, source
record, or canonical candidate is written.
The current card layer keeps two related records:
1. `PokemonCard` owns the Toko ULID and current provider-derived query
projection.
2. `PokemonCardSource` maps `(provider, provider_key)` to that ULID and
retains the complete provider record.
The embedded schema enforces its single-target links. Collection, set, card,
job, run, source-observation, price-observation, and operational-log references
are strong IcyDB relations with restricted target deletion. The cross-kind
`SourceObservation.canonical_id` remains as an indexed lookup key, accompanied
by exactly one application-selected typed relation to `Collection`,
`CanonicalPokemonSet`, or `PokemonCard`. Provider keys remain ordinary evidence
fields and are never treated as foreign keys.
Canonical curation is intentionally implemented for sets first. Card rows are
still provisional provider-derived projections and must not yet be treated as
reviewed Toko truth. Their local ULIDs are generated and remain the public
identity. Source and page rows also have locally generated ULID primary keys,
while provider-owned keys and cursors are stored separately and uniquely
indexed. Re-imports locate the row by its provider key and then update it
through its stable local ID.
When a source has not been seen before, the initial conservative match key is
the normalized Pokémon set name, collector number, and card name. This lets a
future provider attach to an existing printing without pretending that
unrelated provider IDs are comparable. The key is explicitly versioned so a
future reconciliation migration can improve matching without changing public
ULIDs.
## Canister interface
The authoritative interface is
[toko-feed.did](crates/toko-feed/toko-feed.did).
| `toko_feed_configure` | Controller update | Store or rotate the JustTCG credential and configure automatic card polling. |
| `toko_feed_ingest_collections` | Controller update | Refresh canonical collections from JustTCG source mappings. |
| `toko_feed_ingest_sets` | Controller update | Fetch and upsert every current JustTCG Pokémon set. |
| `toko_feed_curate_set` | Controller update | Verify a candidate or correct a verified canonical set using its expected revision. |
| `toko_feed_lock_set` | Controller update | Irreversibly lock a verified canonical set using its expected revision. |
| `toko_feed_ingest` | Controller update | Fetch, upsert, and checkpoint up to 50 cards. |
| `toko_feed_status` | Public query | Read credential-free cursor and health state. |
| `toko_feed_scheduler` | Public query | Read every durable job's cadence, next run, failure count, and active run. |
| `toko_feed_runs` | Public query | Read append-only ingestion attempts using a descending compound cursor. |
| `toko_feed_logs` | Public query | Read sanitized operational events using a descending compound cursor. |
| `toko_feed_collections` | Public query | Read up to 100 canonical collections using an optional ULID continuation. |
| `toko_feed_collection` | Public query | Read one canonical collection and all known provider mappings. |
| `toko_feed_sets` | Public query | Read up to 100 canonical set truth rows using an optional ULID continuation. |
| `toko_feed_set` | Public query | Read one canonical set together with its latest replaceable provider evidence. |
| `toko_feed_cards` | Public query | Read up to 100 summaries using an optional ULID continuation. |
| `toko_feed_card` | Public query | Read one provider-derived local card projection and its latest price variants. |
| `toko_feed_card_prices` | Public query | Read attributed, immutable price observations for one local card identity. |
The HTTPS transform query is part of the IC management-canister outcall
protocol and is not an application endpoint.
### Query pagination
Start a listing with `toko_feed_collections(null, limit)`,
`toko_feed_sets(null, limit)`, or `toko_feed_cards(null, limit)`, then pass the
returned `next_after` value into the corresponding next call. Results are
ordered by ascending canonical ULID and `limit` must be between 1 and 100.
Missing records are returned as `Ok(null)` by their singular query; malformed
IDs and invalid limits return a bounded `FeedError`.
Run, log, and price-history pages are newest first. Begin with a null
`TimeCursor`, then pass the returned `next_before` record—both its timestamp and
ULID—to the next call. The compound cursor prevents equal timestamps from
dropping or repeating rows. These limits are also between 1 and 100.
## Operating the canister
1. Build and install the generated Wasm with the checked-in Candid interface.
The service constructor takes the empty Candid argument `()`.
2. Fund the canister with enough cycles for HTTPS outcalls. One ingestion can
make up to three JustTCG requests.
3. Have a controller invoke `toko_feed_configure` with the API key and desired
polling settings through a secure deployment or administration path. The
canister creates and starts all three durable jobs; Toko does not need to
run a separate scheduler or database.
4. Use the controller-only ingestion methods for an immediate seed, repair, or
operator-requested refresh. They use exactly the same run, evidence,
checkpoint, logging, and retry bookkeeping as timer execution.
5. Monitor `toko_feed_scheduler`, `toko_feed_runs`, and `toko_feed_logs`.
Fund enough cycles for the configured request rate and choose a cadence that
fits the provider plan's [rate and quota limits](https://justtcg.com/docs/rate-limits).
6. Curate set candidates deliberately, then lock only identities and metadata
that should never change again.
7. Let consumers query canonical collections and sets. Treat current card and
price surfaces as provider-derived evidence until their own curation model
is introduced.
There is no deployable hub canister yet. A future Toko or Canic fleet can
coordinate multiple copies through this public interface while each feed
canister continues to own its database, provider integration, and retry loop.
The proposed topology, current Canic compatibility gap, opaque-role contract,
and secret/configuration boundaries are documented in
[Canic fleet integration](docs/architecture/canic-fleet.md).
## Build and installation
Install the pinned Rust toolchain, its Wasm target, ICP CLI, and `ic-wasm`, then
build the complete canister package:
```bash
rustup target add wasm32-unknown-unknown
make canister
```
The installable artifacts are:
- `target/wasm32-unknown-unknown/release/toko_feed.wasm`
- `crates/toko-feed/toko-feed.did`
The full interface gate also expects `didc` and `candid-extractor` on `PATH` so
the checked-in file can be compared structurally with the compiled Wasm.
`make canister` builds and validates the canister but does not install or
deploy it. The checked-in `icp.yaml` uses the official pinned Rust recipe for
ICP lifecycle builds, including Candid metadata and Wasm shrinking. Use
`make wasm` when only the compiler Wasm artifact is needed and interface
validation is not required.
Install or replace the checkout's operator CLI with:
```bash
make install
toko-feed --help
```
This follows the same package split and local-development pattern as
`ic-query`/`ic-query-cli`: Make performs the Cargo installation, then
`toko-feed` is invoked directly as a normal executable.
Operators who also need direct database inspection can install both CLIs with:
```bash
make install-tools
```
This installs the checkout's `toko-feed` binary and exactly `icydb-cli`
0.220.1, matching the embedded IcyDB dependency. It deliberately does not
install an unconstrained latest pre-1.0 CLI.
On install and post-upgrade, the canister applies its embedded IcyDB schema.
Keep the API key out of shell history, logs, committed argument files, and
fixtures. The credential is never returned by status, data, debug, or error
surfaces.
### Local deployment
The checked-in `icp.yaml` defines a project-local managed network on port 8003
and the standalone canister build. Create an ignored local credential file,
restrict its permissions, then start, deploy, configure, and seed the canister:
```bash
cp .env.example .env.local
chmod 600 .env.local
# Edit .env.local and set JUST_TCG_API_KEY without committing it.
make local-start
make local-ready
```
`make local-ready` builds and upgrades or installs the canister, configures it
from `.env.local`, starts its default polling jobs, imports the provider
category catalog as collections, and then imports every Pokémon set. The
ignored file contains the provider credential only; scheduler policy is
non-secret control-plane state and does not belong there. If an intentionally
clean local database is required,
`make local-reset` performs a reinstall first and therefore erases all existing
local Toko Feed data.
### Direct local database inspection
Toko Feed exports IcyDB's controller-only readonly SQL and accepted-schema
inspection endpoints. It does not export SQL mutation or DDL endpoints. After
`make local-ready`, open the interactive database shell with:
```bash
icydb sql --environment local --canister toko-feed
```
Terminate interactive statements with `;`. For one-shot inspection:
```bash
icydb sql -e local -c toko-feed --sql "SHOW ENTITIES"
icydb sql -e local -c toko-feed --sql "SELECT * FROM canonical_pokemon_set LIMIT 20"
icydb schema show toko-feed --environment local
```
IcyDB 0.220.1 currently reports an incorrect zero in the `SHOW ENTITIES`
`relations` summary even when accepted relation constraints are active. Inspect
the authoritative constraints instead:
```bash
icydb sql -e local -c toko-feed --sql "SHOW CONSTRAINTS FROM PokemonCard"
icydb sql -e local -c toko-feed --sql "SHOW CONSTRAINTS FROM SourceObservation"
```
Accepted links appear as validated `relation` constraints with `restrict`
deletion semantics.
The ICP identity used by `icydb` must control the canister. The default local
workflow installs under the `anonymous` identity; use the same explicit
identity for deployment and inspection when overriding `LOCAL_IDENTITY`.
The observation, scheduler, run, log, and canonical-set truth entities are
newer than the 0.1.5 database layout. `CanonicalPokemonSet` is an additive
version-1 entity; `PokemonCard` and the observability entities now declare
version 2 for their new relation contracts. Existing provider projection
tables are intentionally preserved instead of being renamed or rewritten. A
local canister last installed from 0.1.5 or from an earlier unreleased build
needs one `make local-reset`; no migration artifact has yet been exercised for
these pre-release schema changes. Do not upgrade a persistent production
database from that layout until the complete
IcyDB migration path has been prepared and tested.
Local setup and CLI calls default to ICP's `anonymous` identity, which the
managed development network funds and which becomes the local canister
controller. Override setup with `make local-ready LOCAL_IDENTITY=<name>` when
testing another identity. Protected identities can be used by the installed
CLI with `--identity` and `--identity-password-file`.
Use the installed CLI for every operational and query endpoint. It targets the
local ICP environment and `toko-feed` canister by default, passes raw typed
arguments to ICP CLI internally, and prints clean typed JSON:
```bash
toko-feed status
toko-feed scheduler
toko-feed runs list --limit 100
toko-feed logs list --all
toko-feed collections ingest
toko-feed collections list --all
toko-feed collections get 00000000000000000000000001
toko-feed sets ingest
toko-feed sets list --limit 10
toko-feed sets list --limit 100 --all
toko-feed sets list --limit 10 --after 01KZ9GFKW3SY1G000000000001
toko-feed sets get 01KZ9GFKW3SY1G000000000001
toko-feed sets curate 01KZ9GFKW3SY1G000000000001 --revision 0 --name "Base Set" --release-date 1999-01-09
toko-feed sets curate 01KZ9GFKW3SY1G000000000001 --revision 1 --name "Base Set" --release-date 1999-01-09 --retired
toko-feed sets lock 01KZ9GFKW3SY1G000000000001 --revision 2
toko-feed cards ingest
toko-feed cards list --limit 100 --all
toko-feed cards get 01KZ9GFKW3SY1G000000000001
toko-feed cards prices 01KZ9GFKW3SY1G000000000001 --limit 100
```
Without `--all`, a list call returns one page and its `next_after` continuation,
which can be passed back through `--after`. With `--all`, the CLI follows that
cursor, combines the records, and reports its page and record counts. Automatic
pagination stops on a repeated cursor and defaults to at most 1,000 pages; use
`--max-pages` to choose a lower operational bound.
History commands use `next_before` rather than `next_after`. For a manual next
page, copy both returned fields into `--before-time` and `--before-id`. Use
`--all` to have the CLI follow compound cursors and enforce the same repeated-
cursor and maximum-page safeguards automatically.
Set curation is controller-only. Read the row with `sets get`, compare its
`set` and `latest_evidence` objects, then pass the returned `set.revision` to
`sets curate`. The command requires either `--release-date YYYY-MM-DD` or
`--no-release-date`, so an omitted option cannot accidentally erase a known
date. It defaults to `--active`; use `--retired` deliberately. Read the row
again before `sets lock` and pass its new revision. Locking is final.
Run `toko-feed --help` for the complete syntax. Global options must precede the
command. `--environment`, `--canister`, and `--identity` select another ICP
deployment; `--project-root` overrides `icp.yaml` discovery;
`--identity-password-file` supports protected non-interactive identities; and
`--compact` emits one-line JSON. For example:
```bash
toko-feed --environment ic --canister <principal> --identity operator status
```
The CLI deliberately has no configuration or API-key argument. Configuration
continues to read the ignored `.env.local` file so credentials are not copied
into command history.
ICP CLI keeps ephemeral local network state under ignored `.icp/cache/` paths.
Connected-network mappings under `.icp/data/` are durable deployment records
and may be committed when an IC canister is deployed.
`make ic-deploy` builds and deploys the same canister to the IC using the active
ICP identity. It intentionally does not copy a local credential or trigger a
paid provider import; configure those production operations through an
appropriately protected controller workflow.
API access does not itself grant permission to cache or redistribute provider
data. JustTCG's current [commercial-use guidance](https://justtcg.com/docs/commercial-use)
and [terms](https://justtcg.com/terms) distinguish displaying or combining
provider-derived information from redistributing raw data or offering a
substitute bulk pricing API. Toko Feed therefore keeps raw payload evidence
internal and exposes bounded, attributed product views. Review the active
provider plan and terms before exposing any production feed publicly.
## Layout
| `crates/toko-feed/src/api/just_tcg` | JustTCG request construction, wire models, pagination, compatibility, and sanitized tests. |
| `crates/toko-feed/src/canister` | Candid endpoints, IC HTTPS adapter, durable timer/retry orchestration, bounded ingestion, and IcyDB access. |
| `crates/toko-feed/src/collection.rs` | Curated collectible collections, stable Toko IDs, provider aliases, and fallback matching policy. |
| `crates/toko-feed/src/pokemon` | Canonical set/card match keys, set projections, and card price-summary rules. |
| `crates/toko-feed/src/schema` | Embedded IcyDB canister plus job, run, log, observation, collection, set/card, provider-source, and future hub declarations. No hub actor is built yet. |
| `crates/toko-feed-cli` | Host-only installed CLI, typed ICP process boundary, lookup commands, and safe manual or automatic pagination. |
| `icp.yaml` | Official Rust recipe and local/IC project lifecycle definition. |
Future APIs belong under `api/<provider>`. Their wire types should remain
provider-specific; only deliberately selected Pokémon identity and price fields
belong in the canonical projection.
## Development
The repository is pinned to Rust 1.97.1. Run every required native, Candid, and
Wasm check with:
```bash
make check
```
Useful individual targets include `make fmt`, `make clippy`, `make test`,
`make docs`, `make wasm`, and `make candid-check`. Tests use sanitized fixtures
and never call the live provider.
## Releases and publication
Release notes live in [CHANGELOG.md](CHANGELOG.md). Prepare its dated version
entry and commit all non-version work before invoking a release target.
[`cargo-edit`](https://github.com/killercup/cargo-edit) is required for version
management.
```bash
make release-patch
make release-minor
make release-major
```
These guarded targets run the full checks, bump the workspace version, create
an annotated `v<version>` tag, and atomically push the release commit and tag.
Minor and major releases require typed confirmation.
After the release is pushed, validate and publish the canister package followed
by the CLI package with:
```bash
make publish-dry-run
make publish
```
## License
Toko Feed is licensed under the [MIT License](LICENSE).