headless-engine 1.0.0

Ultra-lightweight (<30MB RAM), detection-free pure-Rust headless browser engine built specifically for AI agents, web scraping, and Go-based MCP servers.

Unlike traditional headless browsers (Puppeteer, Playwright, Selenium) that spawn resource-heavy Chromium processes consuming 300MB–800MB of RAM per instance, Headless Engine is written entirely in pure Rust. It operates with an astonishing <30MB RAM footprint, allowing you to run 20+ concurrent browser sessions seamlessly on a single 512MB RAM server.

Key Highlights

• 🪶 Ultra-Lightweight (<30MB RAM): Zero GPU overhead, zero compositor, zero native V8 instantiation. Pure data extraction and structural analysis using Rust's scraper and isolated boa_engine runtimes.
• 🛡️ Zero-Detection WAF Bypass: Employs a revolutionary Offline Rendering Engine. The raw HTML payload is fetched via an impersonated pure-Rust HTTP/2 client. A local <base href="target"> is injected, and Chromium renders file:///...html. Because Chromium never negotiates TLS with the target server, Cloudflare and Datadome traps are bypassed completely.
• 📝 Native HTML-to-Markdown: Strips noise (<script>, <style>, <nav>, <footer>, ads) and outputs dense, LLM-ready Markdown, cutting LLM token usage by 75–80% natively inside the engine.
• 🔍 Multi-Modal SERP Extractor: Built-in parsers for Google AI Overview (SGE), Knowledge Panels, Google Images, YouTube Videos, Google News, and People Also Ask.
• 📱 Multi-Device Fingerprint Rotator: Seamlessly switch between Windows Chrome, Linux Chrome, macOS Safari, iOS Safari (iPhone 16), and Android Chrome (Pixel 8) with deep JS BOM profile spoofing.
• 🔌 Universal Multi-Language Support: First-class SDKs for Rust, Python, Node.js / TypeScript, Go, and Docker.
• 🗂️ Multi-Tab Concurrency: Built-in arena-allocated tab manager (BrowserEngine) for concurrent, isolated multi-tab scraping.

Universal Installation & Dependency Setup

1. 🦀 Rust Crate

cargo add headless-engine

2. 🐍 Python Package (pip)

pip install headless-engine

from headless_engine import HeadlessBrowser

with HeadlessBrowser() as browser:
    report = browser.navigate("https://en.wikipedia.org/wiki/Artificial_intelligence")
    markdown = browser.extract_markdown()
    print("LLM Markdown:\n", markdown)

3. 🟢 Node.js / TypeScript (npm)

npm install headless-engine

import { HeadlessBrowser } from 'headless-engine';

const browser = new HeadlessBrowser();
const report = await browser.navigate('https://news.ycombinator.com');
const markdown = await browser.extractMarkdown();
console.log('Markdown:', markdown);
browser.close();

4. 🐹 Go Module (go get)

go get github.com/yutuknown/headless-engine/sdk/go

package main

import (
    "fmt"
    "github.com/yutuknown/headless-engine/sdk/go"
)

func main() {
    client, _ := headless.NewClient("")
    defer client.Close()

    report, _ := client.Navigate("https://en.wikipedia.org/wiki/Quantum_computing")
    markdown, _ := client.ExtractMarkdown("")
    fmt.Println("Page Title:", report.PageTitle)
    fmt.Println("Markdown Length:", len(markdown))
}

5. 🐳 Docker Container (<20MB Image)

docker run -d --name headless-engine -p 9222:9222 ghcr.io/yutuknown/headless-engine:latest

6. ⚡ 1-Line Standalone Binary Installer

• Linux & macOS:

  curl -fsSL https://raw.githubusercontent.com/yutuknown/headless-engine/master/install.sh | bash
  

• Windows PowerShell:

  iwr -useb https://raw.githubusercontent.com/yutuknown/headless-engine/master/install.ps1 | iex
  

Benchmark Comparison: Headless Engine vs. Lightpanda

We built Headless Engine after observing the architectural friction and lack of native cross-platform support in Zig-based alternatives like Lightpanda. Here is how we stack up:

💻 Rust SDK Usage

Single-Tab Example

use headless_engine::{BrowserTab, DeviceProfile};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    // Launch with Windows Chrome profile
    let mut tab = BrowserTab::with_profile(DeviceProfile::ChromeWindows)?;

    // Navigate to page
    let report = tab.navigate("https://news.ycombinator.com/").await?;
    println!("Title: {} (Status: {})", report.page_title, report.status);

    // Extract clean LLM Markdown
    let markdown = tab.extract_markdown(None).unwrap();
    println!("Markdown Content:\n{}", markdown);

    // Extract actionable links
    for link in tab.extract_links().iter().take(5) {
        println!("Link: [{}] -> {}", link.text, link.href);
    }

    Ok(())
}

Multi-Tab Concurrency (<50MB RAM)

use headless_engine::{BrowserEngine, DeviceProfile};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let mut engine = BrowserEngine::new()?;

    // Create 2 isolated tabs
    let tab1 = engine.create_tab(Some(DeviceProfile::ChromeWindows))?;
    let tab2 = engine.create_tab(Some(DeviceProfile::SafariIos))?;

    // Concurrent navigation
    engine.get_tab_mut(&tab1).unwrap().navigate("https://google.com/").await?;
    engine.get_tab_mut(&tab2).unwrap().navigate("https://github.com/trending").await?;

    println!("Active Tabs: {}", engine.list_tabs().len());

    Ok(())
}

🤖 Go MCP Server Integration Example

package main

import (
    "bufio"
    "encoding/json"
    "fmt"
    "os/exec"
)

func main() {
    cmd := exec.Command("headless-engine", "--stdio")
    stdin, _ := cmd.StdinPipe()
    stdout, _ := cmd.StdoutPipe()
    cmd.Start()
    scanner := bufio.NewScanner(stdout)

    // 1. Send Navigate Command
    req, _ := json.Marshal(map[string]interface{}{
        "jsonrpc": "2.0",
        "id":      1,
        "method":  "tab.navigate",
        "params":  map[string]string{"url": "https://en.wikipedia.org/wiki/Artificial_intelligence"},
    })
    stdin.Write(append(req, '\n'))
    scanner.Scan()
    fmt.Println("Navigation Response:", scanner.Text())

    // 2. Extract LLM Markdown
    req2, _ := json.Marshal(map[string]interface{}{
        "jsonrpc": "2.0",
        "id":      2,
        "method":  "tab.extractMarkdown",
    })
    stdin.Write(append(req2, '\n'))
    scanner.Scan()
    fmt.Println("LLM Markdown:", scanner.Text())

    // 3. Close
    cmd.Process.Kill()
}

📚 JSON-RPC 2.0 API Reference

All methods can be called over standard I/O in --stdio mode:

📜 License

Dual-licensed under either of:

• Apache License, Version 2.0 (LICENSE-APACHE)
• MIT license (LICENSE-MIT)

at your option.