Expand description
decdn bundle pull — fetch every blob a bundle manifest references into an
output directory over the paid cdn/client/v1 path (issue #391).
The manifest comes from a local file (-i) or is fetched first by its own
BLAKE3 hash (--hash); either way entries are then fetched per distinct
blob hash. Entries naming the same blob (one file at two paths) are fetched —
and paid for — once and hard-linked (or copied) to each path (#1306). Node selection is
per blob (#936): with an explicit --node-id every entry is pulled from that
one node, otherwise each distinct blob discovers its own holder among the
region-nearest active nodes. Every group’s fetch — a plain whole-file blob,
or a hint-carrying entry’s pay-now-range fetch — fans out concurrently,
bounded by one global --jobs cap (PullCtx.gate). An entry that only
waits for a sibling’s donor chunks holds no slot while it waits. A manifest chunks
entry is never fetched or stored as its own blob: its hints only let a
byte range shared with another entry be recognized and spliced from disk
instead of paid for again.
Shared chunks are fetched once. When a chunk hint hash appears in two or
more entries, the run assigns that chunk to one entry — its smallest holder,
computed up front from the whole manifest by build_fetch_plan — and only
that entry pays to fetch it. Groups are scheduled smallest-first, so a holder
starts before the larger entries that share its chunks. Each entry drives its
own ranges (unique chunks, chunks assigned to it, and the partial groups at
the two ends of each spliced run) up front and defers the rest; at its tail it splices each deferred range as the
assigned holder registers it, waiting on the holder — not a clock — and driving
a deferred range itself only if that holder finishes without producing it.
One shared pool. The whole bundle pulls from the caller’s single
PaymentPool deposit (ADR 003) — opened once and reused across every
provider the manifest touches. Two concurrency guards follow from that: the
run’s shared LaneLedgers (ADR 039) give each (pool_id, signer, provider)
lane one monotonic voucher issuer, so concurrent fetches sharing a lane —
including every leg of a multi-source entry’s admitted provider set — draw
from the same watermark instead of racing it, while their transfers still
run concurrently; and a single global mutex serializes every open-or-reuse
call — the pool’s on-chain state (deposit, allowance) is one shared resource,
regardless of which provider an entry is bound for.
Incremental re-runs. A run reads <out_root>/.decdn-manifest.json
before fetching and folds each completed file’s record back into it as the
pull progresses — rewriting the merged cache atomically whenever a batch of
files or a gigabyte of new bytes has landed, and once more when the pull
ends. An interrupted pull therefore leaves the files it already landed
recorded, so the next run skips them rather than re-hashing the whole tree.
Only a file that completed successfully this run is recorded; a still-in-flight
or not-yet-fetched entry is never written from bytes this run has not landed.
An in-scope path is
skipped when the saved record’s hash, size, and mtime match the new
manifest. A path with no matching saved record is still skipped when
re-hashing its on-disk bytes matches the new manifest hash. A path whose
on-disk content no longer matches the new manifest hash is re-fetched, not
silently kept. Every skipped or freshly written path also seeds its
on-disk byte ranges as chunk donors, so a later run’s or bundle’s
complement-range fetch can splice an unchanged range from disk instead of
paying for it again.
Whole-root content reuse. A run also reuses a byte-identical file
already anywhere in the output root, not only at its own manifest path. It
consults a whole-root content index built from .decdn-manifest.json’s
saved records: target blob hash → an on-disk regular file the manifest
records with that hash. When a blob’s hash hits that index, the candidate
is confirmed by a whole-file re-hash, then the destination is materialized
by hard link (or copy) with no download and no payment. A file that only
partially matches an on-disk candidate still splices its common byte
ranges from disk the same way, through the donor mechanism described
above.
Functions§
- bundle_
pull - Entry point for
decdn bundle pull.