Expand description
Agentic client for the Spider Cloud API.
Three things this adds over a plain API binding:
- it picks the cheapest request settings likely to work, before spending a call
- it reads both status planes and escalates on its own when a fetch is blocked
- it returns only what the caller asked for, so an LLM reading the result pays for less
The two status planes are the thing to understand first. A call to Spider
Cloud has an HTTP status, and the page it fetched has its own status inside
the body. A 403 in the first means your key is wrong. A 403 in the second
means the site blocked the fetch. See status.
Re-exports§
pub use client::RunBudget;pub use client::RunSpend;pub use client::Spider;pub use client::SpiderBuilder;pub use client::Transport;pub use credits::Credits;pub use credits::Usd;pub use credits::WholeCredits;pub use credits::CREDITS_PER_USD;pub use error::AuthCause;pub use error::BudgetKind;pub use error::Error;pub use error::Recovery;pub use memory::SiteMemoryStore;pub use params::RequestParams;pub use params::ReturnFormat;pub use params::SearchParams;pub use params::WaitFor;pub use policy::Budget;pub use policy::Ladder;pub use policy::Policy;pub use policy::Rung;pub use policy::Step;pub use record::JsonlRecorder;pub use record::Recorder;pub use response::Attempt;pub use response::Body;pub use response::FailedPage;pub use response::Outcome;pub use response::Page;pub use response::PageResult;pub use response::Pages;pub use routing::Explorer;pub use status::ApiClass;pub use status::ApiStatus;pub use status::PageClass;pub use status::PageStatus;pub use thrift::Need;pub use thrift::ThriftReport;pub use thrift::TokenBudget;
Modules§
- auth
- Where the API key comes from, and how to get one.
- client
- The way in.
- credits
- Spend, in credits, in dollars, and in the whole credits a cap has to be.
- error
- What can go wrong, and how to tell the cases apart.
- memory
- What this process remembers about the sites it has fetched.
- ops
- One builder per endpoint.
- optimize
optimize - The parameter optimizer, on the first attempt of an operation.
- params
- The request body sent to the API.
- policy
- When to try again, and what to change before you do.
- record
- The training tap.
- response
- What the API gives back, typed.
- routing
- Where the router meets the request.
- status
- The two status planes, kept apart on purpose.
- thrift
- Paying for less of what comes back.
Structs§
- Action
- The settings one attempt uses.
- Attempt
Outcome - How one attempt ended.
- Caller
Pins - Settings the caller fixed itself.
- Country
- A two letter ISO 3166-1 country code, checked when it is built.
- Heuristic
Router - The rules router.
- Route
Decision - A routing answer.
- Route
Input - One request, as the router sees it.
- Site
Memory - What the caller knows about this site from its own history.
Enums§
- Declared
Need - What the caller wants back, coarsely.
- Proxy
Pool - Which pool the outbound address comes from.
- Request
Mode - How the page is fetched.
- Route
Source - Where a decision came from.
- Router
Version - Which layer of the router answered, and at what version.
- Status
Class - How the last attempt against a site ended, in the coarsest terms that still change what to do next.
- Wait
- What has to happen before the page is read.
Traits§
- Router
- Picks the settings for a first attempt.
Functions§
- featurize
- Turn a request into the vector the router reads.
Type Aliases§
- Result
- Result alias used across the crate.