Struct LlmWeb 

pub struct LlmWeb { /* private fields */ }

The main struct for interacting with web pages and LLMs.

It holds the client for the LLM and provides methods to perform completions on web content.

Implementations

impl LlmWeb

pub fn new(name: &str) -> Self

Creates a new LlmWeb instance.

Arguments

• name - The name of the LLM model to use (e.g., “gemini-1.5-flash”).

Examples found in repository

examples/v2ex.rs (line 29)

async fn main() {
    let schema_str = include_str!("../schemas/v2ex_schema.json");

    let llmweb = LlmWeb::new("gemini-2.0-flash");
    let structed_value: Vec<VXNA> = llmweb
        .exec_from_schema_str("https://v2ex.com/go/vxna", schema_str)
        .await
        .unwrap();
    println!("{:#?}", structed_value);
}

examples/v2ex_stream.rs (line 32)

async fn main() {
    // Load the schema from an external file as a string.
    let schema_str = include_str!("../schemas/v2ex_schema.json");
    let schema: Value = serde_json::from_str(schema_str).unwrap();

    let structed_value: Vec<VXNA> = LlmWeb::new("gemini-2.0-flash")
        .stream("https://v2ex.com/go/vxna", schema)
        .await
        .unwrap();
    println!("{:#?}", structed_value);
}

examples/x.rs (line 18)

async fn main() {
    let schema_json = json!({
        "type": "object",
        "properties": {
            "tweet_text": {
            "type": "string",
            },
        },
        "required": ["tweet_text"]
    });

    let llmweb = LlmWeb::new("gemini-2.0-flash");
    let structed_value: Value = llmweb
        .exec(
            "https://x.com/ztgx5/status/1942242787317133452",
            schema_json,
        )
        .await
        .unwrap();
    println!("{:#?}", structed_value);
}

examples/hn.rs (line 17)

async fn main() {
    // Load the schema from an external file as a string.
    let schema_str = include_str!("../schemas/hn_schema.json");

    let llmweb = LlmWeb::new("gemini-2.0-flash");
    eprintln!("Fetching from Hacker News and extracting stories...");

    // Use the convenience method `exec_from_schema_str` which handles
    // parsing the schema string internally.
    let structed_value: Vec<Story> = llmweb
        .exec_from_schema_str("https://news.ycombinator.com", schema_str)
        .await
        .unwrap();
    println!("{:#?}", structed_value);
}

pub async fn exec<R>(&self, url: &str, scheme: Value) -> Result<R>

where R: DeserializeOwned + Debug,

Fetches content from a URL, sends it to an LLM for processing based on a schema, and returns the structured data.

This function performs the following steps:

1. Launches a headless browser.
2. Navigates to the specified URL.
3. Extracts the HTML content of the page.
4. Sends the content and a JSON schema to the configured LLM.
5. Parses the LLM’s JSON response into the specified Rust type R.

Arguments

• url - The URL of the web page to process.
• scheme - A serializable object representing the JSON schema for data extraction. This is typically a serde_json::Value.

Errors

This function can return an LlmWebError if any of the steps fail, such as browser errors, network issues, LLM API errors, or JSON deserialization errors.

Examples found in repository

examples/x.rs (lines 20-23)

async fn main() {
    let schema_json = json!({
        "type": "object",
        "properties": {
            "tweet_text": {
            "type": "string",
            },
        },
        "required": ["tweet_text"]
    });

    let llmweb = LlmWeb::new("gemini-2.0-flash");
    let structed_value: Value = llmweb
        .exec(
            "https://x.com/ztgx5/status/1942242787317133452",
            schema_json,
        )
        .await
        .unwrap();
    println!("{:#?}", structed_value);
}

pub async fn exec_from_schema_str<R>( &self, url: &str, schema_str: &str, ) -> Result<R>

where R: DeserializeOwned + Debug,

A convenience method that accepts a schema as a string slice.

This method is useful when loading a schema from a file. It parses the string into a serde_json::Value and then calls the main completion method.

Arguments

• url - The URL of the web page to process.
• schema_str - A string slice containing the JSON schema.

Errors

Returns an error if the schema_str is not valid JSON, or if any of the underlying operations in completion fail.

Examples found in repository

examples/v2ex.rs (line 31)

async fn main() {
    let schema_str = include_str!("../schemas/v2ex_schema.json");

    let llmweb = LlmWeb::new("gemini-2.0-flash");
    let structed_value: Vec<VXNA> = llmweb
        .exec_from_schema_str("https://v2ex.com/go/vxna", schema_str)
        .await
        .unwrap();
    println!("{:#?}", structed_value);
}

examples/hn.rs (line 23)

async fn main() {
    // Load the schema from an external file as a string.
    let schema_str = include_str!("../schemas/hn_schema.json");

    let llmweb = LlmWeb::new("gemini-2.0-flash");
    eprintln!("Fetching from Hacker News and extracting stories...");

    // Use the convenience method `exec_from_schema_str` which handles
    // parsing the schema string internally.
    let structed_value: Vec<Story> = llmweb
        .exec_from_schema_str("https://news.ycombinator.com", schema_str)
        .await
        .unwrap();
    println!("{:#?}", structed_value);
}

pub async fn stream<R>(&self, url: &str, scheme: Value) -> Result<R>

where R: DeserializeOwned + Debug,

Fetches content from a URL, sends it to an LLM for processing based on a schema, and returns the structured data.

This function performs the following steps:

1. Launches a headless browser.
2. Navigates to the specified URL.
3. Extracts the HTML content of the page.
4. Sends the content and a JSON schema to the configured LLM.
5. Parses the LLM’s JSON response into the specified Rust type R.

This method is intended for streaming responses.

Arguments

• url - The URL of the web page to process.
• scheme - A serializable object representing the JSON schema for data extraction. This is typically a serde_json::Value.

Errors

This function can return an LlmWebError if any of the steps fail, such as browser errors, network issues, LLM API errors, or JSON deserialization errors.

Examples found in repository

examples/v2ex_stream.rs (line 33)

async fn main() {
    // Load the schema from an external file as a string.
    let schema_str = include_str!("../schemas/v2ex_schema.json");
    let schema: Value = serde_json::from_str(schema_str).unwrap();

    let structed_value: Vec<VXNA> = LlmWeb::new("gemini-2.0-flash")
        .stream("https://v2ex.com/go/vxna", schema)
        .await
        .unwrap();
    println!("{:#?}", structed_value);
}