# Toko Feed
Toko Feed is a standalone Internet Computer canister that builds and serves a
durable Pokémon set and card-price catalog. It owns its JustTCG HTTPS
integration, canonical identities, pagination checkpoint, and embedded IcyDB
database. Consumers see only its Candid endpoints; whether and how it stores
the data is an internal implementation detail.
The workspace has two deliberately separated packages. `toko-feed` builds as
an `rlib` for shared protocol types and native tests plus a `cdylib` canister.
`toko-feed-cli` is the host-only operator package and installs the `toko-feed`
executable. There is no separate `canisters/` tree, schema crate, Canic
dependency, or Toko dependency. A future Canic fleet can install many copies
of the compiled Wasm as opaque feed roles.
## Current behavior
- A controller configures the canister's JustTCG API key.
- `toko_feed_ingest_collections` fetches JustTCG's complete bounded
[game catalog](https://www.justtcg.com/docs/api/games), interprets those
provider-specific game categories as Toko-owned collectible collections,
and stores each provider ID as a source mapping.
- `toko_feed_ingest_sets` fetches all Pokémon sets in one bounded
[JustTCG v1 sets request](https://justtcg.com/docs/api/sets) and upserts their
complete current metadata before card ingestion.
The current provider fields include release date, total set value,
card/variant/sealed counts, and 7/30/90-day value changes. Complete raw set
objects are retained so newly introduced provider fields are not discarded.
- Each `toko_feed_ingest` call fetches and checkpoints at most 50 Pokémon cards.
The requests are split into 20, 20, and 10 cards so they remain compatible
with [JustTCG's documented free-tier page limit](https://justtcg.com/docs/api/cards).
Repeated calls resume from the stored offset; reaching the provider's end
resets the next manually triggered pass to offset zero. This release does not
run an internal refresh timer.
- Raw successful pages and complete provider card records are retained in
IcyDB. Public methods return bounded projections rather than raw provider
JSON.
- Re-importing a provider card updates its existing row. Each canonical card
receives a Toko-owned generated ULID; provider IDs are stored in a separate
source mapping rather than used as the public identity.
- Card ingestion requires an initial provider set import. JustTCG can
occasionally return a valid card whose set is absent from its sets endpoint;
that card is retained with its provider set ID and name instead of failing
the entire bounded ingestion job.
- Summary queries expose the lowest current USD price. Detail queries expose
condition, printing, language, current price, provider timestamp, and recent
percentage changes for each market variant.
## Canonical identity
A ULID is an identifier, not a deduplication rule. Canonical collections
therefore use two records:
1. `Collection` owns the Toko ULID, versioned canonical key, and display name.
2. `CollectionSource` maps `(provider, provider_key)` to that ULID and retains
the provider's complete category record.
The curated registry assigns a namespace-stable Toko ULID to every supported
collection. The current 18 JustTCG category records map to 17 canonical
collections because `pokemon` and `pokemon-japan` both identify Pokémon.
`one-piece-card-game` maps to the broader One Piece collection while retaining
the provider's name in its source record. Every canonical ID is identical in
every feed canister and is not derived from a provider ID.
There is deliberately no uncurated or locally generated collection state. A
provider category absent from the registry makes collection ingestion fail
before any collection rows are written. Supporting it requires an explicit
canonical name, key, and Toko ULID plus tests. `game` and `game_id` remain raw
JustTCG vocabulary; they are not canonical Toko concepts.
Cards likewise keep two related records:
1. `PokemonCard` owns the Toko ULID and canonical query projection.
2. `PokemonCardSource` maps `(provider, provider_key)` to that ULID and
retains the complete provider record.
Canonical card ULIDs are generated and remain the public identity. Source and
page rows also have locally generated ULID primary keys, while provider-owned
keys and cursors are stored separately and uniquely indexed. Re-imports locate
the row by its provider key and then update it through its stable local ID.
When a source has not been seen before, the initial conservative match key is
the normalized Pokémon set name, collector number, and card name. This lets a
future provider attach to an existing printing without pretending that
unrelated provider IDs are comparable. The key is explicitly versioned so a
future reconciliation migration can improve matching without changing public
ULIDs.
## Canister interface
The authoritative interface is
[toko-feed.did](crates/toko-feed/toko-feed.did).
| `toko_feed_configure` | Controller update | Store or rotate the JustTCG credential. |
| `toko_feed_ingest_collections` | Controller update | Refresh canonical collections from JustTCG source mappings. |
| `toko_feed_ingest_sets` | Controller update | Fetch and upsert every current JustTCG Pokémon set. |
| `toko_feed_ingest` | Controller update | Fetch, upsert, and checkpoint up to 50 cards. |
| `toko_feed_status` | Public query | Read credential-free cursor and health state. |
| `toko_feed_collections` | Public query | Read up to 100 canonical collections using an optional ULID continuation. |
| `toko_feed_collection` | Public query | Read one canonical collection and all known provider mappings. |
| `toko_feed_sets` | Public query | Read up to 100 set records using an optional ULID continuation. |
| `toko_feed_set` | Public query | Read one complete projected set record by its local ULID. |
| `toko_feed_cards` | Public query | Read up to 100 summaries using an optional ULID continuation. |
| `toko_feed_card` | Public query | Read one canonical card and its latest price variants. |
The HTTPS transform query is part of the IC management-canister outcall
protocol and is not an application endpoint.
### Query pagination
Start a listing with `toko_feed_collections(null, limit)`,
`toko_feed_sets(null, limit)`, or `toko_feed_cards(null, limit)`, then pass the
returned `next_after` value into the corresponding next call. Results are
ordered by ascending canonical ULID and `limit` must be between 1 and 100.
Missing records are returned as `Ok(null)` by their singular query; malformed
IDs and invalid limits return a bounded `FeedError`.
## Operating the canister
1. Build and install the generated Wasm with the checked-in Candid interface.
The service constructor takes the empty Candid argument `()`.
2. Fund the canister with enough cycles for HTTPS outcalls. One ingestion can
make up to three JustTCG requests.
3. Have a controller invoke `toko_feed_configure` with the API key through a
secure deployment or administration path.
4. Have a controller invoke `toko_feed_ingest_collections` to refresh canonical
collection identities and provider mappings.
5. Have a controller invoke `toko_feed_ingest_sets` to establish or refresh the
Pokémon set catalog. Set results include the mapped
`canonical_collection_id`.
6. Have a controller invoke `toko_feed_ingest` whenever a card refresh is required.
Its receipt reports `next_offset` and `provider_complete`; when completion is
reported, the following invocation begins a new pass at offset zero.
7. Let consumers use the public status, collection, set, card-list, and
card-detail queries.
There is no deployable hub canister or automatic scheduler yet. A future Toko
or Canic fleet can coordinate multiple copies through this public interface
while the feed canister continues to own its database and provider integration.
## Build and installation
Install the pinned Rust toolchain, its Wasm target, ICP CLI, and `ic-wasm`, then
build the complete canister package:
```bash
rustup target add wasm32-unknown-unknown
make canister
```
The installable artifacts are:
- `target/wasm32-unknown-unknown/release/toko_feed.wasm`
- `crates/toko-feed/toko-feed.did`
The full interface gate also expects `didc` and `candid-extractor` on `PATH` so
the checked-in file can be compared structurally with the compiled Wasm.
`make canister` builds and validates the canister but does not install or
deploy it. The checked-in `icp.yaml` uses the official pinned Rust recipe for
ICP lifecycle builds, including Candid metadata and Wasm shrinking. Use
`make wasm` when only the compiler Wasm artifact is needed and interface
validation is not required.
Install or replace the checkout's operator CLI with:
```bash
make install
toko-feed --help
```
This follows the same package split and local-development pattern as
`ic-query`/`ic-query-cli`: Make performs the Cargo installation, then
`toko-feed` is invoked directly as a normal executable.
On install and post-upgrade, the canister applies its embedded IcyDB schema.
Keep the API key out of shell history, logs, committed argument files, and
fixtures. The credential is never returned by status, data, debug, or error
surfaces.
### Local deployment
The checked-in `icp.yaml` defines a project-local managed network on port 8003
and the standalone canister build. Create an ignored local credential file,
restrict its permissions, then start, deploy, configure, and seed the canister:
```bash
cp .env.example .env.local
chmod 600 .env.local
# Edit .env.local and set JUST_TCG_API_KEY without committing it.
make local-start
make local-ready
```
`make local-ready` builds and upgrades or installs the canister, configures it
from `.env.local`, imports the provider category catalog as collections, and
then imports every Pokémon set. If an intentionally clean local database is
required,
`make local-reset` performs a reinstall first and therefore erases all existing
local Toko Feed data.
The collection and set entities are newer than the 0.1.4 database layout. A
local canister last installed from that layout needs one `make local-reset`.
Do not upgrade a persistent production database from the pre-catalog layout
until an explicit IcyDB migration artifact has been prepared and exercised.
Local setup and CLI calls default to ICP's `anonymous` identity, which the
managed development network funds and which becomes the local canister
controller. Override setup with `make local-ready LOCAL_IDENTITY=<name>` when
testing another identity. Protected identities can be used by the installed
CLI with `--identity` and `--identity-password-file`.
Use the installed CLI for every operational and query endpoint. It targets the
local ICP environment and `toko-feed` canister by default, passes raw typed
arguments to ICP CLI internally, and prints clean typed JSON:
```bash
toko-feed status
toko-feed collections ingest
toko-feed collections list --all
toko-feed collections get 00000000000000000000000001
toko-feed sets ingest
toko-feed sets list --limit 10
toko-feed sets list --limit 100 --all
toko-feed sets list --limit 10 --after 01KZ9GFKW3SY1G000000000001
toko-feed sets get 01KZ9GFKW3SY1G000000000001
toko-feed cards ingest
toko-feed cards list --limit 100 --all
toko-feed cards get 01KZ9GFKW3SY1G000000000001
```
Without `--all`, a list call returns one page and its `next_after` continuation,
which can be passed back through `--after`. With `--all`, the CLI follows that
cursor, combines the records, and reports its page and record counts. Automatic
pagination stops on a repeated cursor and defaults to at most 1,000 pages; use
`--max-pages` to choose a lower operational bound.
Run `toko-feed --help` for the complete syntax. Global options must precede the
command. `--environment`, `--canister`, and `--identity` select another ICP
deployment; `--project-root` overrides `icp.yaml` discovery;
`--identity-password-file` supports protected non-interactive identities; and
`--compact` emits one-line JSON. For example:
```bash
toko-feed --environment ic --canister <principal> --identity operator status
```
The CLI deliberately has no configuration or API-key argument. Configuration
continues to read the ignored `.env.local` file so credentials are not copied
into command history.
ICP CLI keeps ephemeral local network state under ignored `.icp/cache/` paths.
Connected-network mappings under `.icp/data/` are durable deployment records
and may be committed when an IC canister is deployed.
`make ic-deploy` builds and deploys the same canister to the IC using the active
ICP identity. It intentionally does not copy a local credential or trigger a
paid provider import; configure those production operations through an
appropriately protected controller workflow.
API access does not itself grant permission to cache or redistribute provider
data. Review the provider's current subscription, attribution, caching, and
redistribution terms before exposing a production feed publicly.
## Layout
| `crates/toko-feed/src/api/just_tcg` | JustTCG request construction, wire models, pagination, compatibility, and sanitized tests. |
| `crates/toko-feed/src/canister` | Candid endpoints, IC HTTPS adapter, bounded ingestion, and IcyDB access. |
| `crates/toko-feed/src/collection.rs` | Curated collectible collections, stable Toko IDs, provider aliases, and fallback matching policy. |
| `crates/toko-feed/src/pokemon` | Canonical set/card match keys, set projections, and card price-summary rules. |
| `crates/toko-feed/src/schema` | Embedded IcyDB canister plus operational, collection, set/card, provider-source, and future hub declarations. No hub actor is built yet. |
| `crates/toko-feed-cli` | Host-only installed CLI, typed ICP process boundary, lookup commands, and safe manual or automatic pagination. |
| `icp.yaml` | Official Rust recipe and local/IC project lifecycle definition. |
Future APIs belong under `api/<provider>`. Their wire types should remain
provider-specific; only deliberately selected Pokémon identity and price fields
belong in the canonical projection.
## Development
The repository is pinned to Rust 1.97.1. Run every required native, Candid, and
Wasm check with:
```bash
make check
```
Useful individual targets include `make fmt`, `make clippy`, `make test`,
`make docs`, `make wasm`, and `make candid-check`. Tests use sanitized fixtures
and never call the live provider.
## Releases and publication
Release notes live in [CHANGELOG.md](CHANGELOG.md). Prepare its dated version
entry and commit all non-version work before invoking a release target.
[`cargo-edit`](https://github.com/killercup/cargo-edit) is required for version
management.
```bash
make release-patch
make release-minor
make release-major
```
These guarded targets run the full checks, bump the workspace version, create
an annotated `v<version>` tag, and atomically push the release commit and tag.
Minor and major releases require typed confirmation.
After the release is pushed, validate and publish the canister package followed
by the CLI package with:
```bash
make publish-dry-run
make publish
```
## License
Toko Feed is licensed under the [MIT License](LICENSE).