Skip to main content

Crate spider_cloud_agent

Crate spider_cloud_agent 

Source
Expand description

Agentic client for the Spider Cloud API.

Three things this adds over a plain API binding:

  • it picks the cheapest request settings likely to work, before spending a call
  • it reads both status planes and escalates on its own when a fetch is blocked
  • it returns only what the caller asked for, so an LLM reading the result pays for less

The two status planes are the thing to understand first. A call to Spider Cloud has an HTTP status, and the page it fetched has its own status inside the body. A 403 in the first means your key is wrong. A 403 in the second means the site blocked the fetch. See status.

Re-exports§

pub use client::RunBudget;
pub use client::RunSpend;
pub use client::Spider;
pub use client::SpiderBuilder;
pub use client::Transport;
pub use credits::Credits;
pub use credits::Usd;
pub use credits::WholeCredits;
pub use credits::CREDITS_PER_USD;
pub use error::AuthCause;
pub use error::BudgetKind;
pub use error::Error;
pub use error::Recovery;
pub use memory::SiteMemoryStore;
pub use params::RequestParams;
pub use params::ReturnFormat;
pub use params::SearchParams;
pub use params::WaitFor;
pub use policy::Budget;
pub use policy::Ladder;
pub use policy::Policy;
pub use policy::Rung;
pub use policy::Step;
pub use record::JsonlRecorder;
pub use record::Recorder;
pub use response::Attempt;
pub use response::Body;
pub use response::FailedPage;
pub use response::Outcome;
pub use response::Page;
pub use response::PageResult;
pub use response::Pages;
pub use routing::Explorer;
pub use status::ApiClass;
pub use status::ApiStatus;
pub use status::PageClass;
pub use status::PageStatus;
pub use thrift::Need;
pub use thrift::ThriftReport;
pub use thrift::TokenBudget;

Modules§

auth
Where the API key comes from, and how to get one.
client
The way in.
credits
Spend, in credits, in dollars, and in the whole credits a cap has to be.
error
What can go wrong, and how to tell the cases apart.
memory
What this process remembers about the sites it has fetched.
ops
One builder per endpoint.
optimizeoptimize
The parameter optimizer, on the first attempt of an operation.
params
The request body sent to the API.
policy
When to try again, and what to change before you do.
record
The training tap.
response
What the API gives back, typed.
routing
Where the router meets the request.
status
The two status planes, kept apart on purpose.
thrift
Paying for less of what comes back.

Structs§

Action
The settings one attempt uses.
AttemptOutcome
How one attempt ended.
CallerPins
Settings the caller fixed itself.
Country
A two letter ISO 3166-1 country code, checked when it is built.
HeuristicRouter
The rules router.
RouteDecision
A routing answer.
RouteInput
One request, as the router sees it.
SiteMemory
What the caller knows about this site from its own history.

Enums§

DeclaredNeed
What the caller wants back, coarsely.
ProxyPool
Which pool the outbound address comes from.
RequestMode
How the page is fetched.
RouteSource
Where a decision came from.
RouterVersion
Which layer of the router answered, and at what version.
StatusClass
How the last attempt against a site ended, in the coarsest terms that still change what to do next.
Wait
What has to happen before the page is read.

Traits§

Router
Picks the settings for a first attempt.

Functions§

featurize
Turn a request into the vector the router reads.

Type Aliases§

Result
Result alias used across the crate.