IPFIX Parser

A dependency-free Rust library for parsing IP Flow Information Export (IPFIX) messages, as specified in RFC 7011.

Implementation Status

The IPFIX protocol, the full IANA Information Element registry, and the structured data types are implemented.

Each implemented wire structure is validated against the figures and field definitions of its RFC, and the Information Element table is generated from, and validated against, the IANA IPFIX registry.

Status Legend:

• ✅ Implemented
• ⬜ Not implemented
• 🪦 Deprecated

Specifications

Note: see RFC Errata for corrections to published specifications. The following technical errata are implemented:

Message Structures

Abstract Data Types

All data types of the IANA registry, with reduced-size encoding (RFC 7011 §6.2) where allowed.

Information Elements

All 502 Information Elements of the IANA registry are known (17 of them 🪦 deprecated), with their name, data type, semantics, status and units: see src/registry/information_elements.rs.

The table is generated from the IANA registry with make registry-update. The weekly CI run fails when IANA assigns or changes an element, which is the signal to regenerate it.

Values that cannot be decoded with their Information Element's type are kept as FieldValue::Raw instead of failing the whole Message. This covers enterprise-specific elements (their types are unknown), lengths not allowed for the type, invalid UTF-8, and lists referencing an unknown Template.

Planned: enterprise-specific Information Elements. Two sources of type information will share one per-session table:

1. Definitions registered by the caller (e.g. from vendor documentation). They are the Collecting Process's "own internal definition" and always take precedence (RFC 5610 §3.9).
2. RFC 5610 type records sent by the Exporter, under the RFC's rules: they never override internal definitions, conflicting records disable the element, invalid type/semantics combinations and names with NUL characters are ignored, and the table is bounded.

Caller-registered definitions come first because most vendors (none of the 13 vendor samples in the test suite send type records) document their elements out-of-band.

Usage

IPFIX is template-based: Data Records can only be decoded with a Template that the Exporter sent earlier. The Decoder keeps the Template cache of one Transport Session, so create one per Exporter (TCP connection, or UDP source address) and keep it alive:

use ipfix_parser::{Decoder, FieldValue};

let mut decoder = Decoder::new();

// For each UDP datagram / IPFIX Message received from this Exporter
let message = decoder.decode(&data)?;
for record in message.data_records() {
    if let Some(FieldValue::Ipv4Address(src)) = record.get("sourceIPv4Address") {
        println!("{} {:?}", src, record.get("octetDeltaCount"));
    }
}

Template management depends on the transport (RFC 7011 §8.4), so tell the decoder which one the session uses. UDP is the default:

use ipfix_parser::{Decoder, DecoderConfig, Transport};

// UDP (default): Template Withdrawals are ignored, as the RFC requires (this
// also protects the cache from spoofed withdrawals), and Templates not
// refreshed within the lifetime are discarded
let udp = Decoder::with_config(DecoderConfig {
    template_lifetime: Some(1800), // at least 3x the Exporter's refresh interval
    ..DecoderConfig::default()
});

// TCP, SCTP or IPFIX File: Template Withdrawals apply, Templates never expire
let tcp = Decoder::with_config(DecoderConfig {
    transport: Transport::Tcp,
    ..DecoderConfig::default()
});

Time is the Export Time of the Message Headers, not the system clock, so decoding is deterministic. Decoder::template_last_received exposes when each Template was last refreshed, and Decoder::expire_templates expires Templates with your own clock (e.g. when an Exporter goes silent).

Data Sets whose Template is not known yet are returned as Set::UndecodedData, so the caller can buffer or drop them (RFC 7011 §8). The Template cache is bounded (DecoderConfig::max_templates, 4096 by default), as is the nesting depth of structured data (DecoderConfig::max_list_depth).

For self-contained buffers, parse_message decodes a single Message (UDP semantics), and parse_messages reads a buffer of consecutive Messages as an IPFIX File (Transport::File semantics: withdrawals apply, no expiry). A compressed file must be decompressed first.

Serialization / Deserialization Support

Enables optional serialization and deserialization support for all IPFIX data structures using serde. This allows you to easily convert parsed IPFIX data to and from JSON, YAML, or any other format supported by serde.

[dependencies]
ipfix-parser = { version = "0.1", features = ["serde"] }
serde_json = "1.0"

Example usage:

use ipfix_parser::{parse_message, Message};

let message = parse_message(&data)?;

// Serialize to JSON
let json = serde_json::to_string(&message)?;

// Deserialize from JSON
let message: Message = serde_json::from_str(&json)?;

Testing

Unit & Integration Tests

Run the comprehensive test suite:

make test              # Run all tests
make test-unit         # Run unit tests only
make test-integration  # Run integration tests only
make test-vendors      # Run integration tests against vendor samples (network)

The integration tests decode a capture generated with softflowd, and real exports from 13 vendors and exporters (YAF, VMware VDS, Citrix NetScaler, Juniper, MikroTik, Nokia, OpenBSD pflow, ...) published by logstash-codec-netflow under the Apache-2.0 license. To keep this repository MIT-only, these samples are downloaded at a pinned commit rather than vendored.

Fuzz Tests

The project includes comprehensive fuzz testing using cargo-fuzz:

make fuzz-install     # Install fuzzing tools (requires nightly Rust)
make fuzz-single      # Fuzz single Message parsing (60s)
make fuzz-session     # Fuzz a Transport Session: Templates across Messages (60s)
make fuzz-structured  # Fuzz with valid framing and Templates over every data type (60s)
make fuzz-all         # Run all fuzzers (5 minutes each)

Specifications Validation

The project includes comprehensive validation against the IANA IPFIX registries and the RFC documents, using the syn crate to parse Rust source files and extract struct metadata:

make specs-validate   # Validate against IANA and the RFCs
make registry-update  # Regenerate the Information Element table from IANA

This validation allowed us to identify additional specification issues that are not yet part of the official errata.

Benchmarks

Performance benchmarks using Criterion:

make bench

Results: ~7.3µs to decode the test capture (2 Messages, 31 Data Records of 14 to 16 fields), i.e. ~235ns per Data Record (~230 MiB/s throughput), on an Apple Silicon laptop. Like sflow-parser, the parser is not zero-copy (at least for now) and does not use any unsafe code.

License

This project is licensed under the MIT License.