huginn-net-http

HTTP fingerprinting and browser detection for Huginn Net.

Overview

This crate provides HTTP-based passive fingerprinting capabilities. It analyzes HTTP/1.x and HTTP/2 headers to identify browsers, web servers, and detect preferred languages.

Why choose huginn-net-http?

• No third-party tools - No tshark, wireshark, or external tools required
• Comprehensive analysis - Browser, server, and language detection
• Pure Rust implementation - No system libraries required
• High performance - 562.1K pps for full analysis, 200M pps detection
• HTTP/1.x & HTTP/2 - Support for both major protocol versions
• Type-safe architecture - Prevents entire classes of bugs at compile time
• Typed observable data access - Access to typed HTTP headers, header ordering, language preferences, and other observable signals for custom fingerprinting and analysis
• Extensible fingerprinting - Build custom fingerprints using typed observable data (ObservableHttpRequest, ObservableHttpResponse) without being limited to predefined p0f signatures

Features

• Browser Detection - Identify browsers from HTTP request headers
• Web Server Detection - Identify servers from HTTP response headers
• Language Detection - Extract preferred languages from Accept-Language headers
• HTTP/1.x & HTTP/2 - Support for both major HTTP versions
• Quality Scoring - Confidence metrics for all matches
• Parallel Processing - Multi-threaded worker pool for live network capture (high-throughput scenarios)
• Sequential Mode - Single-threaded processing (for PCAP files and low-resource environments)
• Akamai HTTP/2 Fingerprinting - Extract Akamai fingerprints from HTTP/2 ClientHello frames (see Akamai Fingerprinting section)

Akamai HTTP/2 Fingerprinting

This crate includes an Akamai HTTP/2 fingerprint parser that extracts fingerprints from HTTP/2 connection frames (SETTINGS, WINDOW_UPDATE, PRIORITY, HEADERS) following the Blackhat EU 2017 specification.

Important Design Consideration:

Unlike p0f HTTP fingerprinting (which normalizes header order), Akamai fingerprinting requires preserving the original header order from the HTTP/2 frames. This is because:

1. The pseudo-header order (:method, :path, :authority, :scheme) is a critical component of the Akamai fingerprint
2. The order of SETTINGS parameters matters for fingerprint accuracy
3. This original ordering is essential when using Akamai fingerprints in TLS termination scenarios where headers must be reconstructed exactly as they appeared in the original connection

Why it's not integrated into the main processing pipeline:

Due to this requirement for preserving original header order, the Akamai fingerprint extractor is provided as a standalone utility (Http2FingerprintExtractor) rather than being integrated into the main HTTP processing pipeline. The main pipeline normalizes and processes headers for p0f-style fingerprinting, which would corrupt the original ordering needed for Akamai fingerprints.

Usage:

use huginn_net_http::http2_fingerprint_extractor::Http2FingerprintExtractor;

let mut extractor = Http2FingerprintExtractor::new();

// Add HTTP/2 data incrementally (handles connection preface automatically)
extractor.add_bytes(&http2_data)?;

if let Some(fingerprint) = extractor.get_fingerprint() {
    println!("Akamai fingerprint: {}", fingerprint.fingerprint);
    println!("Fingerprint hash: {}", fingerprint.hash);
}

This design allows you to extract Akamai fingerprints before TLS termination or in scenarios where you need to preserve the exact original frame structure, while still using the main pipeline for standard HTTP/1.x and HTTP/2 analysis with p0f-style fingerprinting.

Quick Start

Note: Live packet capture requires libpcap (usually pre-installed on Linux/macOS).

Installation

Add this to your Cargo.toml:

[dependencies]
huginn-net-http = "1.7.1"
huginn-net-db = "1.7.1"

Basic Usage

use huginn_net_db::Database;
use huginn_net_http::{FilterConfig, HuginnNetHttp, HuginnNetHttpError, IpFilter, PortFilter, HttpAnalysisResult};
use std::sync::{Arc, mpsc};

fn main() -> Result<(), HuginnNetHttpError> {
    // Load database for browser/server fingerprinting
    let db = match Database::load_default() {
        Ok(db) => Arc::new(db),
        Err(e) => {
            eprintln!("Failed to load database: {e}");
            return Err(HuginnNetHttpError::Parse(format!("Database error: {e}")));
        }
    };
    
    // Create analyzer
    let mut analyzer = match HuginnNetHttp::new(Some(db), 1000) {
        Ok(analyzer) => analyzer,
        Err(e) => {
            eprintln!("Failed to create analyzer: {e}");
            return Err(e);
        }
    };
    
    // Optional: Configure filters (can be combined)
    if let Ok(ip_filter) = IpFilter::new().allow("192.168.1.0/24") {
        let filter = FilterConfig::new()
            .with_port_filter(PortFilter::new().destination(80))
            .with_ip_filter(ip_filter);
        analyzer = analyzer.with_filter(filter);
    }
    
    let (sender, receiver) = mpsc::channel::<HttpAnalysisResult>();
    
    // Live capture (use parallel mode for high throughput)
    std::thread::spawn(move || {
        if let Err(e) = analyzer.analyze_network("eth0", sender, None) {
            eprintln!("Analysis error: {e}");
        }
    });
    
    // Or PCAP analysis (always use sequential mode)
    // std::thread::spawn(move || {
    //     if let Err(e) = analyzer.analyze_pcap("capture.pcap", sender, None) {
    //         eprintln!("Analysis error: {e}");
    //     }
    // });
    
    for result in receiver {
        if let Some(http_request) = result.http_request { println!("{http_request}"); }
        if let Some(http_response) = result.http_response { println!("{http_response}"); }
    }
    
    Ok(())
}

For a complete working example with signal handling, error management, and CLI options, see examples/capture-http.rs.

Filtering

The library supports packet filtering to reduce processing overhead and focus on specific traffic. Filters can be combined using AND logic (all conditions must match):

Filter Types:

• Port Filter: Filter by TCP source/destination ports (supports single ports, lists, and ranges)
• IP Filter: Filter by specific IPv4/IPv6 addresses (supports source-only, destination-only, or both)
• Subnet Filter: Filter by CIDR subnets (supports IPv4 and IPv6)

All filters support both Allow (allowlist) and Deny (denylist) modes. See the filter documentation for complete details.

Example Output

[HTTP Request] 1.2.3.4:1524 → 4.3.2.1:80
  Browser: Firefox:10.x or newer
  Lang:    English
  Params:  none
  Sig:     1:Host,User-Agent,Accept=[,*/*;q=],?Accept-Language=[;q=],Accept-Encoding=[gzip, deflate],?DNT=[1],Connection=[keep-alive],?Referer:Accept-Charset,Keep-Alive:Firefox/

[HTTP Response] 192.168.1.22:58494 → 91.189.91.21:80
  Server:  nginx/1.14.0 (Ubuntu)
  Params:  anonymous
  Sig:     server=[nginx/1.14.0 (Ubuntu)],date=[Tue, 17 Dec 2024 13:54:16 GMT],x-cache-status=[from content-cache-1ss/0],connection=[close]:Server,Date,X-Cache-Status,Connection:

Huginn Net Ecosystem

This crate is part of the Huginn Net ecosystem. For multi-protocol analysis, see huginn-net. For protocol-specific analysis:

• huginn-net-tcp - OS fingerprinting, MTU detection, uptime estimation
• huginn-net-tls - JA4 fingerprinting, TLS version detection

Documentation

For complete documentation, examples, and integration guides, see the main huginn-net README.

License

Dual-licensed under MIT or Apache 2.0.