Kowalski Web Agent

Initial Proof of Concept Agent

A specialized AI agent for web research and online information retrieval, built on the Kowalski framework. The Web Agent provides intelligent, conversational access to web search and web scraping, enabling users to find, summarize, and analyze online content.

What is the Web Agent?

The Web Agent is an AI-powered assistant that combines large language models with web search and scraping tools. It helps users discover, extract, and summarize information from the internet, supporting research, fact-checking, and content synthesis.

Core Capabilities

Web Search: Query the internet using multiple search providers (DuckDuckGo, Google, Bing)
Web Scraping: Extract content from web pages for further analysis
ReAct-Style Tool Calling: Intelligent tool usage with reasoning and iteration
Summarization: Generate human-readable summaries of web content
Conversational Q&A: Ask follow-up questions and get iterative, context-aware answers
Role-based Summaries: Tailor explanations for different audiences (e.g., technical, family-friendly)
Streaming AI Analysis: Real-time, conversational web research

What Does It Do?

Search: Finds relevant web pages for a given query
Extract: Scrapes and processes content from web pages
Reason & Act: Uses ReAct-style tool calling to intelligently chain tools
Summarize: Provides concise, audience-tailored summaries of online information
Interactive Research: Supports follow-up questions and iterative exploration
Conversation History: Maintains context for multi-step research tasks

ReAct-Style Tool Calling

The Web Agent implements a ReAct (Reasoning and Acting) loop that enables intelligent tool usage:

Reason: The agent analyzes the user's query and decides which tools to use
Act: The agent executes the appropriate tool (web_search or web_scrape)
Observe: The agent processes the tool's results
Iterate: The agent continues the loop until a final answer is reached

Tool Usage Examples

The agent can intelligently chain tools:

// The agent will automatically:
// 1. Use web_search to find relevant URLs
// 2. Use web_scrape to extract content from promising URLs
// 3. Synthesize the information into a comprehensive answer
let response = agent.chat_with_tools(&conv_id, "What are the latest developments in AI?").await?;

Example Usage

Basic Usage

use kowalski_web_agent::WebAgent;
use kowalski_core::config::Config;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = Config::default();
    let mut web_agent = WebAgent::new(config).await?;

    // Perform a web search
    let results = web_agent.search("AI").await?;
    for result in &results {
        println!("Title: {}", result.title);
        println!("URL: {}", result.url);
        println!("Snippet: {}", result.snippet);
    }

    // Fetch and summarize a web page
    if let Some(first) = results.first() {
        let page = web_agent.fetch_page(&first.url).await?;
        println!("Page Title: {}", page.title);
        println!("Content: {}", &page.content[..100]);
    }

    Ok(())
}

Advanced Tool-Calling Usage

use kowalski_web_agent::WebAgent;
use kowalski_core::config::Config;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = Config::default();
    let mut agent = WebAgent::new(config).await?;
    
    // Start a conversation
    let conv_id = agent.start_conversation("llama3.2");
    
    // Use ReAct-style tool calling for complex queries
    let response = agent.chat_with_tools(&conv_id, "What's the latest news about AI?").await?;
    println!("Response: {}", response);
    
    // The agent will automatically:
    // 1. Search for recent AI news
    // 2. Scrape relevant articles
    // 3. Synthesize the information
    // 4. Provide a comprehensive answer
    
    Ok(())
}

How Could It Be Extended?

Additional Search Providers: Integrate with more APIs (Google, Bing, academic search, etc.)
Deeper Scraping: Support for dynamic content, login-protected pages, or API endpoints
Content Classification: Detect news, blogs, forums, or scientific content
Fact Checking: Cross-reference multiple sources for verification
Sentiment and Topic Analysis: Extract opinions, topics, and trends from web content
Automated Alerts: Notify users of new or changing information on topics of interest
Integration: Embed in chatbots, research assistants, or browser extensions
Advanced Tool Chaining: More sophisticated reasoning and tool selection logic

Potential Benefits

For Researchers

Faster Discovery: Rapidly find and summarize relevant online information
Contextual Summaries: Get explanations tailored to your audience or expertise
Iterative Exploration: Ask follow-up questions and dig deeper
Intelligent Tool Usage: The agent automatically chooses the right tools for each task

For Developers

API-First: Integrate web research into your own tools and workflows
Customizable: Extend with new providers, scrapers, or analysis modules
ReAct Integration: Leverage the reasoning and acting capabilities in your applications

For Organizations

Knowledge Aggregation: Gather and synthesize information from across the web
Monitoring: Track topics, competitors, or trends in real time
Automated Research: Reduce manual effort in information gathering and analysis

Example Output

The example successfully:

Performs a web search and lists results with titles, URLs, and snippets
Fetches and processes the content of a selected web page
Generates a simplified summary for a non-technical audience
Handles streaming responses and conversation context
Uses ReAct-style tool calling for intelligent information gathering

Running the web research example:

🤖 Starting web agent...
Web Agent Conversation ID: 12345678-90ab-cdef-1234-567890abcdef

🔍 Searching: AI

📑 Result:
Title: Artificial Intelligence - Wikipedia
URL: https://en.wikipedia.org/wiki/Artificial_intelligence
Snippet: Artificial intelligence (AI) is intelligence demonstrated by machines, in contrast to the natural intelligence displayed by humans and animals...

📑 Result:
Title: What is AI? | IBM
URL: https://www.ibm.com/topics/artificial-intelligence
Snippet: Artificial intelligence leverages computers and machines to mimic the problem-solving and decision-making capabilities of the human mind...

🌐 Processing first result: https://en.wikipedia.org/wiki/Artificial_intelligence

📝 Generating summary...
Artificial intelligence (AI) is when computers or machines are designed to think and learn like humans. It helps solve problems, make decisions, and can be found in things like voice assistants or self-driving cars.

✅ Summary complete!

Running the tool-calling demo:

Kowalski Web Agent Tool-Calling Demo
=====================================
Conversation started with ID: abc123

--- Query: What's the latest news about AI? ---
Processing with ReAct-style tool calling...
[DEBUG] Iteration 1: What's the latest news about AI?
[DEBUG] Tool call detected: web_search with input: latest AI news 2024
[DEBUG] Tool web_search executed successfully: [search results...]
[DEBUG] Iteration 2: Based on the tool result: [search results...]
[DEBUG] Tool call detected: web_scrape with input: https://example-news-site.com
[DEBUG] Tool web_scrape executed successfully: [article content...]
Final Response: Based on recent news, AI developments include...

--- End Query ---

Note: This is an initial proof of concept agent. Features, reliability, and coverage will expand in future versions.

kowalski-web-agent 0.5.0