headless-engine 1.0.2

Traditional headless browsers (such as Puppeteer or Playwright) orchestrate full multi-process Chromium instances that typically consume 350MB–800MB of RAM per instance with significant initialization overhead.

Headless Engine is designed as a purpose-built alternative: a fast, pure-Rust headless browser runtime that operates with a <30MB RSS footprint and sub-5ms initialization. It parses DOM trees directly, applies client fingerprint emulation to avoid automated bot triggers, and converts pages into clean, token-efficient Markdown optimized for LLM context windows.

Key Features

• 🪶 Low Memory Overhead: Sub-10MB idle RSS and ~15–35MB under active browsing, enabling high-density concurrency on resource-constrained servers.
• 🛡️ Fingerprint Emulation & Anti-Detection: Injects realistic client profiles (navigator.webdriver = false, full window.chrome hierarchy, WebGL ANGLE vendor/renderer masking, Client Hints, and session persistence).
• 🤖 Search & Knowledge Extraction: Built-in extraction for Google AI Mode (udm=50), AI Overviews (SGE), Oxford Languages Knowledge Cards, YouTube video timestamps, and organic results.
• 📝 LLM-Ready Markdown Distillation: Natively converts HTML DOM trees into clean Markdown by stripping navigation menus, ads, headers/footers, and boilerplate scripts (~85% token reduction).
• ⚡ Offline Ingestion & DOM Injection: Ingest raw HTML strings or local files into active tabs via browser.set_content(html) or file:// URIs for offline parsing and deterministic testing.
• 📱 Multi-Device Profiles: Switch between Windows Chrome, Linux Chrome, macOS Safari, iOS Safari, and Android Chrome.
• 🔌 Universal Multi-Language Support: First-class SDKs and bindings for Rust, Python, Node.js / TypeScript, Go, and Docker.
• 🗂️ Multi-Tab Isolation: Built-in arena-allocated tab manager (BrowserEngine) for concurrent, isolated multi-tab automation.
• 📡 Standard JSON-RPC 2.0 Interface: Connect over standard I/O for direct integration with AI agents and Model Context Protocol (MCP) servers.

Installation & Quick Start

1. 🦀 Rust Crate

cargo add headless-engine

use headless_engine::{BrowserTab, DeviceProfile};

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    let mut tab = BrowserTab::with_profile(DeviceProfile::ChromeWindows)?;
    let report = tab.navigate("https://news.ycombinator.com/").await?;
    
    println!("Title: {} (Status: {})", report.page_title, report.status);
    let markdown = tab.extract_markdown(None).unwrap();
    println!("Markdown:\n{}", markdown);
    
    Ok(())
}

2. 🐍 Python SDK (pip)

pip install headless-engine

from headless_engine import HeadlessBrowser

with HeadlessBrowser() as browser:
    # Navigate with automated fingerprint emulation
    report = browser.navigate("https://www.google.com/search?q=quantum+computing&udm=50")
    
    # Extract token-efficient LLM Markdown
    markdown = browser.extract_markdown()
    print("Markdown:\n", markdown)
    
    # Extract structured search entities (AI Overviews, PAA, Organic Results)
    results = browser.extract_results()

3. 🟢 Node.js / TypeScript SDK (npm)

npm install headless-engine

import { HeadlessBrowser } from 'headless-engine';

const browser = new HeadlessBrowser();
const report = await browser.navigate('https://news.ycombinator.com');
const markdown = await browser.extractMarkdown();
console.log('Markdown:', markdown);
browser.close();

4. 🐹 Go Module

go get github.com/yutuknown/headless-engine/sdk/go

package main

import (
    "fmt"
    "github.com/yutuknown/headless-engine/sdk/go"
)

func main() {
    client, _ := headless.NewClient("")
    defer client.Close()

    report, _ := client.Navigate("https://en.wikipedia.org/wiki/Quantum_computing")
    markdown, _ := client.ExtractMarkdown("")
    fmt.Printf("Page Title: %s\nContent Length: %d\n", report.PageTitle, len(markdown))
}

5. 🐳 Docker Container

docker run -d --name headless-engine -p 9222:9222 ghcr.io/yutuknown/headless-engine:latest

6. ⚡ Standalone Binary Installers

• Linux & macOS:

  curl -fsSL https://raw.githubusercontent.com/yutuknown/headless-engine/master/scripts/install.sh | bash
  

• Windows PowerShell:

  iwr -useb https://raw.githubusercontent.com/yutuknown/headless-engine/master/scripts/install.ps1 | iex
  

Architectural Comparison

🤖 MCP Server & CLI Automation Example

Headless Engine provides a standard JSON-RPC 2.0 interface over Stdio:

package main

import (
    "bufio"
    "encoding/json"
    "fmt"
    "os/exec"
)

func main() {
    cmd := exec.Command("headless-engine", "--stdio")
    stdin, _ := cmd.StdinPipe()
    stdout, _ := cmd.StdoutPipe()
    cmd.Start()
    scanner := bufio.NewScanner(stdout)

    // 1. Send Navigate Command
    req, _ := json.Marshal(map[string]interface{}{
        "jsonrpc": "2.0",
        "id":      1,
        "method":  "tab.navigate",
        "params":  map[string]string{"url": "https://en.wikipedia.org/wiki/Artificial_intelligence"},
    })
    stdin.Write(append(req, '\n'))
    scanner.Scan()
    fmt.Println("Navigation Response:", scanner.Text())

    // 2. Extract LLM Markdown
    req2, _ := json.Marshal(map[string]interface{}{
        "jsonrpc": "2.0",
        "id":      2,
        "method":  "tab.extractMarkdown",
    })
    stdin.Write(append(req2, '\n'))
    scanner.Scan()
    fmt.Println("LLM Markdown:", scanner.Text())

    cmd.Process.Kill()
}

📚 JSON-RPC 2.0 API Reference

📜 License

Licensed under either of:

• MIT License (LICENSE-MIT or opensource.org/licenses/MIT)
• Apache License, Version 2.0 (LICENSE-APACHE or apache.org/licenses/LICENSE-2.0)

at your option.

Contribution

Unless you explicitly state otherwise, any contribution intentionally submitted for inclusion in this project shall be dual-licensed as above, without any additional terms or conditions.