Crate spider_agent 

Spider Agent

A concurrent-safe multimodal agent for web automation and research.

Features

• Concurrent-safe: Designed to be wrapped in Arc for multi-task access
• Feature-gated: Only include dependencies you need
• Multiple LLM providers: OpenAI, OpenAI-compatible APIs
• Multiple search providers: Serper, Brave, Bing, Tavily
• Browser automation: Chrome support via chromiumoxide

Quick Start

use spider_agent::{Agent, AgentConfig};
use std::sync::Arc;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let agent = Arc::new(Agent::builder()
        .with_openai("sk-...", "gpt-4o-mini")
        .with_search_serper("serper-key")
        .build()?);

    // Search
    let results = agent.search("rust web frameworks").await?;
    println!("Found {} results", results.len());

    // Extract from first result
    let html = agent.fetch(&results.results[0].url).await?.html;
    let data = agent.extract(&html, "Extract framework name and features").await?;
    println!("{}", serde_json::to_string_pretty(&data)?);

    Ok(())
}

Concurrent Execution

use spider_agent::Agent;
use std::sync::Arc;

let agent = Arc::new(Agent::builder()
    .with_openai("sk-...", "gpt-4o")
    .with_search_serper("serper-key")
    .with_max_concurrent_llm_calls(10)
    .build()?);

// Execute multiple searches concurrently
let queries = vec!["rust async", "rust web frameworks", "rust databases"];
let mut handles = Vec::new();

for query in queries {
    let agent = agent.clone();
    let query = query.to_string();
    handles.push(tokio::spawn(async move {
        agent.search(&query).await
    }));
}

// Collect results
for handle in handles {
    let result = handle.await??;
    println!("Found {} results", result.results.len());
}

Feature Flags

• openai - OpenAI/OpenAI-compatible LLM provider
• chrome - Browser automation via chromiumoxide
• search - Base search functionality
• search_serper - Serper.dev search provider
• search_brave - Brave Search provider
• search_bing - Bing Search provider
• search_tavily - Tavily AI Search provider
• full - All features

Re-exports

pub use tools::AuthConfig;

pub use tools::CustomTool;

pub use tools::CustomToolRegistry;

pub use tools::CustomToolResult;

pub use tools::HttpMethod;

pub use automation::ActionRecord;

pub use automation::ActionResult;

pub use automation::ActionType;

pub use automation::ActResult;

pub use automation::AutomationConfig;

pub use automation::AutomationResult;

pub use automation::AutomationUsage;

pub use automation::CaptureProfile;

pub use automation::ChainBuilder;

pub use automation::ChainCondition;

pub use automation::ChainContext;

pub use automation::ChainResult;

pub use automation::ChainStep;

pub use automation::ChainStepResult;

pub use automation::CleaningIntent;

pub use automation::ClipViewport;

pub use automation::ContentAnalysis;

pub use automation::CostTier;

pub use automation::ExtractionSchema;

pub use automation::FormField;

pub use automation::FormInfo;

pub use automation::HtmlCleaningProfile;

pub use automation::InteractiveElement;

pub use automation::ModelEndpoint;

pub use automation::ModelPolicy;

pub use automation::NavigationOption;

pub use automation::PageObservation;

pub use automation::PromptUrlGate;

pub use automation::RecoveryStrategy;

pub use automation::RetryPolicy;

pub use automation::SelectorCache;

pub use automation::SelectorCacheEntry;

pub use automation::StructuredOutputConfig;

pub use automation::VisionRouteMode;

pub use automation::RemoteMultimodalConfig;

pub use automation::RemoteMultimodalConfigs;

pub use automation::RemoteMultimodalEngine;

pub use automation::EngineError;

pub use automation::EngineResult;

pub use automation::best_effort_parse_json_object;

pub use automation::extract_assistant_content;

pub use automation::extract_last_code_block;

pub use automation::extract_last_json_array;

pub use automation::extract_last_json_boundaries;

pub use automation::extract_last_json_object;

pub use automation::extract_usage;

pub use automation::fnv1a64;

pub use automation::truncate_utf8_tail;

pub use automation::clean_html;

pub use automation::clean_html_base;

pub use automation::clean_html_full;

pub use automation::clean_html_raw;

pub use automation::clean_html_slim;

pub use automation::clean_html_with_profile;

pub use automation::clean_html_with_profile_and_intent;

pub use automation::smart_clean_html;

pub use automation::categories;

pub use automation::DiscoveredUrl;

pub use automation::MapResult;

pub use automation::AutomationMemory;

pub use automation::MemoryOperation;

pub use automation::ACT_SYSTEM_PROMPT;

pub use automation::CONFIGURATION_SYSTEM_PROMPT;

pub use automation::DEFAULT_SYSTEM_PROMPT;

pub use automation::EXTRACT_SYSTEM_PROMPT;

pub use automation::MAP_SYSTEM_PROMPT;

pub use automation::OBSERVE_SYSTEM_PROMPT;

pub use automation::ConcurrentChainConfig;

pub use automation::ConcurrentChainResult;

pub use automation::DependencyGraph;

pub use automation::DependentStep;

pub use automation::StepResult;

pub use automation::Alternative;

pub use automation::ConfidenceRetryStrategy;

pub use automation::ConfidenceSummary;

pub use automation::ConfidenceTracker;

pub use automation::ConfidentStep;

pub use automation::Verification;

pub use automation::VerificationType;

pub use automation::ActionToolSchemas;

pub use automation::FunctionCall;

pub use automation::FunctionDefinition;

pub use automation::ToolCall;

pub use automation::ToolCallingMode;

pub use automation::ToolDefinition;

pub use automation::ChangeType;

pub use automation::DiffStats;

pub use automation::ElementChange;

pub use automation::HtmlDiffMode;

pub use automation::HtmlDiffResult;

pub use automation::PageStateDiff;

pub use automation::Checkpoint;

pub use automation::CheckpointResult;

pub use automation::CheckpointType;

pub use automation::ExecutionPlan;

pub use automation::PageState;

pub use automation::PlanExecutionState;

pub use automation::PlanningModeConfig;

pub use automation::PlannedStep;

pub use automation::ReplanContext;

pub use automation::HealedSelectorCache;

pub use automation::HealingDiagnosis;

pub use automation::HealingRequest;

pub use automation::HealingResult;

pub use automation::HealingStats;

pub use automation::SelectorIssueType;

pub use automation::SelfHealingConfig;

pub use automation::MultiPageContext;

pub use automation::PageContext;

pub use automation::PageContribution;

pub use automation::SynthesisConfig;

pub use automation::SynthesisResult;

pub use automation::build_schema_generation_prompt;

pub use automation::generate_schema;

pub use automation::infer_schema;

pub use automation::infer_schema_from_examples;

pub use automation::refine_schema;

pub use automation::GeneratedSchema;

pub use automation::SchemaCache;

pub use automation::SchemaGenerationRequest;

pub use automation::extract_html_context;

pub use automation::parse_tool_calls;

pub use automation::tool_calls_to_steps;

pub use automation::execute_graph;

pub use automation::cache::CacheStats;

pub use automation::cache::CacheValue;

pub use automation::cache::SmartCache;

pub use automation::executor::BatchExecutor;

pub use automation::executor::ChainExecutor;

pub use automation::executor::PrefetchManager;

pub use automation::router::ModelRouter;

pub use automation::router::RoutingDecision;

pub use automation::router::TaskAnalysis;

pub use automation::router::TaskCategory;

Modules

automation

Automation module for spider_agent.

tools

Custom tool support for external API calls.

Structs

Agent

Multimodal agent for web automation and research.

AgentBuilder

Agent builder for configuring and creating agents.

AgentConfig

Agent configuration.

AgentMemory

Session memory for storing state across operations.

CompletionOptions

Options for completion requests.

CompletionResponse

Response from a completion request.

FetchResult

Result from fetching a URL.

Message

A message in a conversation.

PageExtraction

Extraction from a single page.

ResearchOptions

Research options for research tasks.

RetryConfig

Retry configuration.

SearchOptions

Search options for web search.

TokenUsage

Token usage from a completion.

UsageLimits

Usage limits for controlling agent resource consumption.

UsageSnapshot

Snapshot of usage statistics.

UsageStats

Usage statistics for tracking agent operations.

Enums

AgentError

Agent error types.

HtmlCleaningMode

HTML cleaning mode.

LimitType

Type of limit that was exceeded.

MessageContent

Message content - either text or multi-part.

SearchError

Search-specific error types.

TimeRange

Time range for filtering search results.

Traits

LLMProvider

LLM provider trait for abstracting different LLM APIs.

Type Aliases

AgentResult

Result type for agent operations.