Crate dbn

The official crate for working with Databento Binary Encoding (DBN), an extremely fast message encoding and storage format for normalized market data. The DBN specification includes a simple, self-describing metadata header and a fixed set of struct definitions, which enforce a standardized way to normalize market data.

All official Databento client libraries use DBN under the hood, both as a data interchange format and for in-memory representation of data. DBN is also the default encoding for all Databento APIs, including live data streaming, historical data streaming, and batch flat files. For more information about the encoding, read our introduction to DBN.

The crate supports reading and writing DBN files and streams, as well as converting them to other Encodings. It can also be used to update legacy DBZ files to DBN.

This crate provides:

• Decoders for DBN and DBZ (the precursor to DBN), both sync and async, with the async feature flag
• Encoders for CSV, DBN, and JSON, both sync and async, with the async feature flag
• Normalized market data struct definitions corresponding to the different market data schemas offered by Databento
• A wrapper type for holding a reference to a record struct of a dynamic type
• Helper functions and macros for common tasks

Feature flags

• async: enables async decoding and encoding
• python: enables pyo3 bindings
• serde: enables deriving serde traits for types
• trivial_copy: enables deriving the Copy trait for records

Re-exports

pub use crate::enums::rtype;

pub use crate::enums::Action;

pub use crate::enums::Compression;

pub use crate::enums::Encoding;

pub use crate::enums::ErrorCode;

pub use crate::enums::InstrumentClass;

pub use crate::enums::MatchAlgorithm;

pub use crate::enums::RType;

pub use crate::enums::SType;

pub use crate::enums::Schema;

pub use crate::enums::SecurityUpdateAction;

pub use crate::enums::Side;

pub use crate::enums::StatType;

pub use crate::enums::StatUpdateAction;

pub use crate::enums::StatusAction;

pub use crate::enums::StatusReason;

pub use crate::enums::SystemCode;

pub use crate::enums::TradingEvent;

pub use crate::enums::TriState;

pub use crate::enums::UserDefinedInstrument;

pub use crate::enums::VersionUpgradePolicy;

pub use crate::error::Error;

pub use crate::error::Result;

pub use crate::flags::FlagSet;

pub use crate::metadata::MappingInterval;

pub use crate::metadata::Metadata;

pub use crate::metadata::MetadataBuilder;

pub use crate::metadata::SymbolMapping;

pub use crate::publishers::Dataset;

pub use crate::publishers::Publisher;

pub use crate::publishers::Venue;

pub use crate::record::Bbo1MMsg;

pub use crate::record::Bbo1SMsg;

pub use crate::record::BboMsg;

pub use crate::record::BidAskPair;

pub use crate::record::Cbbo1MMsg;

pub use crate::record::Cbbo1SMsg;

pub use crate::record::CbboMsg;

pub use crate::record::Cmbp1Msg;

pub use crate::record::ConsolidatedBidAskPair;

pub use crate::record::ErrorMsg;

pub use crate::record::HasRType;

pub use crate::record::ImbalanceMsg;

pub use crate::record::InstrumentDefMsg;

pub use crate::record::MboMsg;

pub use crate::record::Mbp10Msg;

pub use crate::record::Mbp1Msg;

pub use crate::record::OhlcvMsg;

pub use crate::record::Record;

pub use crate::record::RecordHeader;

pub use crate::record::RecordMut;

pub use crate::record::StatMsg;

pub use crate::record::StatusMsg;

pub use crate::record::SymbolMappingMsg;

pub use crate::record::SystemMsg;

pub use crate::record::TbboMsg;

pub use crate::record::TradeMsg;

pub use crate::record::WithTsOut;

pub use crate::record_ref::RecordRef;

pub use crate::symbol_map::PitSymbolMap;

pub use crate::symbol_map::SymbolIndex;

pub use crate::symbol_map::TsSymbolMap;

Modules

compat

Compatibility shims for different DBN versions.

decode

Decoding DBN and Zstd-compressed DBN files and streams. Sync decoders implement

encode

Encoding DBN and Zstd-compressed DBN files and streams. Encoders implement the EncodeDbn trait.

enums

Enums used in Databento APIs.

error

Types for errors that can occur while working with DBN.

flags

Bit set flags used in Databento market data.

macros

Helper macros for working with multiple RTypes, Schemas, and types of records.

metadata

Contains Metadata struct which comes at the beginning of any DBN file or stream and MetadataBuilder for creating a Metadata with defaults.

pretty

Contains new types for pretty-printing values found in DBN records.

publishers

Enumerations for different data sources, venues, and publishers.

pythonpython

Python wrappers around dbn functions. These are implemented here instead of in python/ to be able to implement pyo3 traits for DBN types.

record

Market data types for encoding different Databento Schemas in the most recent DBN version, as well as conversion functions.

record_ref

The RecordRef struct for non-owning dynamically-typed references to DBN records.

symbol_map

Maps for mapping instrument IDs to human-readable symbols.

v1

Record data types for encoding different Databento Schemas in DBN version 1.

v2

Record data types for encoding different Databento Schemas in DBN version 2.

v3

Record data types for encoding different Databento Schemas in the upcoming DBN version 3.

Macros

rtype_async_dispatchDeprecated

Specializes a generic async function to all record types and dispatches based rtype.

rtype_dispatch

Dispatches to a generic function or method based on $rtype and optionally $ts_out.

rtype_dispatch_with_ts_outDeprecated

Specializes a generic function to all record types wrapped in WithTsOut and dispatches based on the rtype.

rtype_method_dispatchDeprecated

Specializes a generic method to all record types and dispatches based rtype.

rtype_ts_out_async_dispatchDeprecated

Specializes a generic async function to all record types and dispatches based rtype and ts_out.

rtype_ts_out_async_method_dispatchDeprecated

Specializes a generic async method to all record types and dispatches based rtype and ts_out.

rtype_ts_out_dispatchDeprecated

Specializes a generic function to all record types and dispatches based on the rtype and ts_out.

rtype_ts_out_method_dispatchDeprecated

Specializes a generic method to all record types and dispatches based on the rtype and ts_out.

schema_async_method_dispatchDeprecated

Specializes a generic async method to all record types with an associated schema.

schema_dispatch

Dispatches to a generic function or method based on $schema and optionally $ts_out.

schema_method_dispatchDeprecated

Specializes a generic method to all record types with an associated schema.

schema_ts_out_method_dispatchDeprecated

Specializes a generic method to all record types based on the associated type for schema and ts_out.

Enums

RecordEnum

An owned DBN record type of flexible type. Unlike RecordRef, this type allows matching.

RecordRefEnum

An immutable reference to a DBN record of flexible type. Unlike RecordRef, this type allows matching.

Constants

DBN_VERSION

The current version of the DBN encoding, which is different from the crate version.

FIXED_PRICE_SCALE

The denominator of fixed prices in DBN.

MAX_RECORD_LEN

The length in bytes of the largest record type.

SYMBOL_CSTR_LEN

The length of fixed-length symbol strings.

UNDEF_ORDER_SIZE

The sentinel value for an unset or null order quantity.

UNDEF_PRICE

The sentinel value for an unset or null price.

UNDEF_STAT_QUANTITY

The sentinel value for an unset or null stat quantity.

UNDEF_TIMESTAMP

The sentinel value for an unset or null timestamp.