Crate pcapsql_core 

pcapsql-core

Engine-agnostic PCAP protocol parsing library.

This crate provides the core parsing functionality for pcapsql, without any SQL engine dependencies. It can be used standalone for protocol analysis or as the foundation for SQL integrations (DataFusion, DuckDB).

Features

• Protocol Parsing: 17 built-in protocol parsers (Ethernet, IP, TCP, UDP, DNS, HTTP, TLS, DHCP, NTP, and more)
• PCAP Reading: Support for PCAP and PCAPNG formats, including gzip/zstd compression
• Memory-Mapped I/O: Efficient reading of large capture files
• Parse Caching: LRU cache to avoid redundant parsing during JOINs
• TCP Stream Reassembly: Connection tracking and application-layer parsing

Quick Start

use pcapsql_core::prelude::*;
use pcapsql_core::io::FilePacketSource;

// Create a protocol registry with all built-in parsers
let registry = default_registry();

// Open a PCAP file
let source = FilePacketSource::open("capture.pcap").unwrap();
let mut reader = source.reader(None).unwrap();

// Read and parse packets using callback pattern
reader.process_packets(1000, |packet| {
    let results = pcapsql_core::parse_packet(
        &registry,
        packet.link_type as u16,
        &packet.data,
    );

    for (protocol_name, result) in results {
        println!("{}: {} fields", protocol_name, result.fields.len());
    }
    Ok(())
}).unwrap();

Architecture

+---------------------------------------------------------------------+
|                        pcapsql-core                                 |
+---------------------------------------------------------------------+
|  schema/     - FieldDescriptor, DataKind (engine-agnostic)          |
|  protocol/   - Protocol trait, 17 parsers, FieldValue               |
|  io/         - PacketSource, PacketReader, mmap support             |
|  pcap/       - PCAP/PCAPNG reading, compression                     |
|  cache/      - LRU parse cache                                      |
|  stream/     - TCP reassembly, HTTP/TLS stream parsing              |
|  format/     - Address formatting utilities                         |
|  error/      - Error types                                          |
+---------------------------------------------------------------------+

Crate Features

• default - Gzip and Zstd compression enabled
• compress-gzip - Gzip decompression support
• compress-zstd - Zstd decompression support
• compress-lz4 - LZ4 decompression support
• compress-bzip2 - Bzip2 decompression support
• compress-xz - XZ decompression support
• compress-all - All compression formats

Supported Protocols

Layer	Protocols
Link	Ethernet, VLAN (802.1Q)
Network	IPv4, IPv6, ARP, ICMP, ICMPv6
Transport	TCP, UDP
Application	DNS, DHCP, NTP, HTTP, TLS, SSH, QUIC

Re-exports

pub use cache::CacheStats;

pub use cache::CachedParse;

pub use cache::LruParseCache;

pub use cache::NoCache;

pub use cache::OwnedParseResult;

pub use cache::ParseCache;

pub use error::Error;

pub use error::PcapError;

pub use error::ProtocolError;

pub use error::Result;

pub use format::detect_address_column;

pub use format::format_ipv4;

pub use format::format_ipv6;

pub use format::format_mac;

pub use format::AddressKind;

pub use io::FilePacketReader;

pub use io::FilePacketSource;

pub use io::PacketReader;

pub use io::PacketSource;

pub use io::RawPacket;

pub use io::MmapPacketReader;

pub use io::MmapPacketSource;

pub use pcap::PcapReader;

pub use protocol::OwnedFieldValue;

pub use protocol::chain_fields_for_protocol;

pub use protocol::compute_required_protocols;

pub use protocol::default_registry;

pub use protocol::merge_with_chain_fields;

pub use protocol::parse_packet;

pub use protocol::parse_packet_projected;

pub use protocol::parse_packet_pruned;

pub use protocol::parse_packet_pruned_projected;

pub use protocol::should_continue_parsing;

pub use protocol::should_run_parser;

pub use protocol::BuiltinProtocol;

pub use protocol::FieldValue;

pub use protocol::ParseContext;

pub use protocol::ParseResult;

pub use protocol::PayloadMode;

pub use protocol::ProjectionConfig;

pub use protocol::Protocol;

pub use protocol::ProtocolRegistry;

pub use protocol::TunnelLayer;

pub use protocol::TunnelType;

pub use schema::DataKind;

pub use schema::FieldDescriptor;

pub use schema::ProtocolSchema;

pub use stream::Connection;

pub use stream::ConnectionState;

pub use stream::ConnectionTracker;

pub use stream::Direction;

pub use stream::ParsedMessage;

pub use stream::StreamConfig;

pub use stream::StreamContext;

pub use stream::StreamManager;

pub use stream::StreamParseResult;

pub use stream::StreamParser;

pub use stream::StreamRegistry;

pub use stream::TcpFlags;

pub use tls::derive_tls12_keys;

pub use tls::derive_tls13_keys;

pub use tls::extract_tls13_inner_content_type;

pub use tls::hash_for_cipher_suite;

pub use tls::tls12_prf;

pub use tls::AeadAlgorithm;

pub use tls::DecryptionContext;

pub use tls::DecryptionError;

pub use tls::Direction as TlsDirection;

pub use tls::HandshakeData;

pub use tls::HashAlgorithm;

pub use tls::KeyDerivationError;

pub use tls::KeyLog;

pub use tls::KeyLogEntries;

pub use tls::KeyLogEntry;

pub use tls::KeyLogError;

pub use tls::SessionError;

pub use tls::SessionState;

pub use tls::Tls12KeyMaterial;

pub use tls::Tls13KeyMaterial;

pub use tls::TlsSession;

pub use tls::TlsVersion;

Modules

cache

Parse cache for avoiding redundant protocol parsing.

error

Error types for pcapsql-core.

format

Value formatting utilities for network addresses.

io

Packet I/O abstractions.

pcap

PCAP file reading module.

prelude

Convenient re-exports for common usage.

protocol

Protocol parsing framework.

schema

Engine-agnostic schema types.

stream

TCP stream processing and application-layer parsing.

tls

TLS decryption support for pcapsql.

Constants

VERSION

Library version.