ptox
A language-neutral SDK for building file-search clients across many networks (clearnet APIs, anonymous overlays, peer-to-peer networks, content-addressed storage, …).
It provides the contracts, engines, and tooling a client builds on:
- Contracts —
FileQuery,FileResult,Address,SearchProvider,Fetcher,Capabilities. - Engines —
Engine(search + retrieve),fanout,merge(deduplicate + rank),FetchRegistry. - Bridge — the NDJSON wire format and
SubprocessProvider, so providers can be written in any language.
# async
SearchReport::metrics reports per-provider result/failure counts and elapsed
time (no query or result content); fanout_with_metrics exposes the same
ptox::Metrics collector to clients that drive providers directly. The fan-out
also enforces the result contract: a structurally invalid result (no addresses,
or an address with an empty scheme/uri) is reported as a malformed failure
instead of being delivered, while the provider's other results still pass.
Capabilities are probed before every search. Engine::with_capability_cache()
memoizes each provider's declarations (off by default) and
Engine::invalidate_capabilities() drops the cache and asks each provider to
re-probe, so a client with an expensive capabilities() pays it once.
Engine::provider_capabilities() reports each provider's declared facets,
requirements, and address schemes without running a query — a
providers/diagnostics view; a provider that does not answer in time falls back
to the universal default (answered: false).
FileQuery selectors can be built fluently: FileQuery::new("annual report") .with_mimetype("application/pdf").with_extension(".pdf") .with_size_range(0, 10_000_000).with_offset(20).with_limit(20). The extension
helpers normalize case and leading dots; empty collections clear their field.
offset is the per-provider counterpart to limit (skip then cap), so
with_offset(page * size).with_limit(size) requests one page.
Retrieval (the fetch feature, on by default) turns a result's Address into
bytes: Engine::fetch(&result, "out.bin").await. The opt-in ipfs feature adds
an ipfs:// gateway fetcher, IpfsGatewayFetcher, which resolves CIDs through a
configurable HTTP gateway and delegates to the HTTP fetcher. The
off-by-default file feature adds FileFetcher for file:// addresses; it
copies a local file atomically with sha256 verification and can be restricted to
a data directory with FileFetcher::with_allowed_roots (a file:// address is a
filesystem read capability, so enabling the feature is a deliberate choice).
HttpFetcher::with_retry can replay transient GET failures (5xx, 408, 429,
connection/stream errors) with exponential backoff; retries are off by default.
HttpFetcher::with_resume(true) continues an interrupted .part download with a
Range request instead of restarting it, and with_connect_timeout /
with_read_timeout bound a stalled connect or transfer (both off by default).
Other schemes (e.g. magnet) can be delegated to an external command, and
multiple mirrors of a result are tried in preference order. See
docs/fetching.md in the repository for the end-to-end retrieval guide.
Providers (network plugins) and clients (frontends) are separate software. The contract, JSON Schema, and conformance suite live in the repository: https://github.com/despulii-bit/ptox.
MergedResult is a persistence contract: its serialized shape only gains fields
within a major version, so a client can store search results between a search and
a later fetch and still read them after an SDK upgrade. For on-disk client state,
prefer the smaller versioned ResultRef (ResultRef::from_merged / to_merged),
which keeps only the fetch-essential fields and carries its own schema version.
suggest_filename(&result)
suggests a safe name for the file (the title, else the preferred address, with
illegal characters replaced).
Client wiring has a canonical, versioned TOML schema in ptox::config (opt-in
feature config): Config::load / from_toml_str parse and validate it,
and Config::providers() / fetch_registry() / merge_options() build the
pieces a client needs. See docs/configuration.md in the repository.
The repository ships a complete, runnable client — an in-process provider, a
ranked search, and a fetch to a temp file through a fake Fetcher, all offline:
Writing a provider
Implement the SearchProvider trait and pass it to fanout or Engine. The
repository ships a complete, runnable example — trait impl, Capabilities, error
codes, and an offline test with a fake transport:
A provider in any other language can speak the NDJSON wire format and be wrapped
with SubprocessProvider. Validate either kind with the ptox-conformance
suite. See docs/writing-a-provider.md in the repository for the end-to-end
guide.
Building a client
A client assembles the SDK: providers in, ranked results and fetched files out.
See docs/building-a-client.md in the repository for the end-to-end guide —
engine setup, streaming vs ranked search, partial failures, persistence,
retrieval, and offline testing.
Responsible use
ptox is neutral infrastructure — a contract and engines, with no bundled network, provider, client, or index. What you search for and retrieve, and the providers and networks you configure, are your responsibility. Provided "as is", without warranty (see the license).
License
Licensed under either of Apache License, Version 2.0 or MIT license at your option.