Crate http_streams_core 

Framework-neutral building blocks for streaming HTTP bodies as sequences of items.

This crate holds the parts that are the same whichever HTTP library is in use: the wire formats (encoding and decoding), the error type, and the progress/observability state machine. It knows nothing about any specific client or server.

You are unlikely to depend on it directly. It backs:

• axum-streams: server side
• reqwest-streams: client side

Both re-export the types they expose, so downstream code names them through those crates rather than here.

Why this crate exists

It was extracted from those two, which had grown up as a pair: one encoding response bodies on the server, the other decoding them on the client. Adding streaming request bodies meant each needed what the other already had, and the two had by then also grown near-identical progress and error handling independently. The shared parts moved here so a body encoded by one is decoded by the same code in the other.

Features

Note: the default features do not include any formats.

• json: JSON array and JSON Lines (JSONL)
• csv: CSV
• protobuf: length-prefixed Protobuf
• arrow: Apache Arrow IPC
• text: raw UTF-8 text (encode only, see below)
• tracing: report progress and errors through tracing

Directionality

Every format can encode. All but text can decode: text framing writes raw bytes with no delimiter, so ["ab", "c"] and ["a", "bc"] are byte-identical on the wire and splitting them back into items is not merely unimplemented but impossible.

Re-exports

pub use buffer::buffer_bytes;

pub use buffer::buffer_ready_items;

pub use content_type::ContentType;

pub use envelope::StreamFormatEnvelope;

pub use error::StreamError;

pub use error::StreamErrorKind;

pub use format::DecodeOptions;

pub use format::DefaultFormat;

pub use format::FrameParser;

pub use format::IdentityParser;

pub use format::ItemEncoder;

pub use format::StreamFormat;

pub use format::StreamFormatDecode;

pub use format::StreamFormatEncode;

pub use format::DEFAULT_BUF_CAPACITY;

pub use progress::count_bytes;

pub use progress::count_items;

pub use progress::instrument;

pub use progress::Counting;

pub use progress::Direction;

pub use progress::ErrorInfo;

pub use progress::Progress;

pub use progress::ProgressItem;

pub use progress::ProgressOptions;

pub use progress::Side;

pub use progress::StreamContext;

pub use progress::StreamErrorHandler;

pub use progress::StreamOutcome;

pub use progress::StreamProgress;

pub use progress::StreamProgressHandler;

pub use progress::DEFAULT_PROGRESS_INTERVAL;

pub use stream::decode_stream;

pub use stream::encode_stream;

Modules

buffer

Coalescing small encoded chunks into larger ones.

content_type

A deliberately tiny media-type parser.

envelope

Wrapping a streamed array in a surrounding JSON object.

error

The error type shared by every streaming format, in both directions.

format

The symmetric format abstraction: one trait for identity, one per direction.

progress

Accounting and observability for one streamed body, in either direction.

stream

Driving an ItemEncoder or a Decoder over a stream.

Structs

ArrowIpcCodecarrow

A Decoder that yields one RecordBatch per IPC message.

ArrowIpcEncoderarrow

Per-stream state for ArrowRecordBatchIpcStreamFormat.

ArrowRecordBatchIpcStreamFormatarrow

Arrow record batches in IPC stream framing.

CsvEncodercsv

Per-stream state for CsvStreamFormat.

CsvFrameConfigcsv

How to frame CSV records.

CsvParsercsv

Deserialises one framed CSV record.

CsvRecordCodeccsv

A Decoder that yields one csv::ByteRecord per CSV record.

CsvStreamFormatcsv

CSV rows, with an optional header row.

JsonArrayCodecjson

A Decoder that yields the elements of a JSON array.

JsonArrayEncoderjson

Per-stream state for JsonArrayStreamFormat.

JsonArrayStreamFormatjson

A JSON array: [item, item, …], optionally wrapped in an envelope object.

JsonNewLineCodecjson

A Decoder that yields one deserialised item per line.

JsonNewLineEncoderjson

Per-stream state for JsonNewLineStreamFormat. There is none: the framing is per-item.

JsonNewLineStreamFormatjson

JSON Lines: one JSON value per line, newline-terminated.

ProtobufEncoderprotobuf

Per-stream state for ProtobufStreamFormat. There is none: every frame is self-describing.

ProtobufLenPrefixCodecprotobuf

A Decoder that yields one message per length-prefixed frame.

ProtobufStreamFormatprotobuf

Protobuf messages, each preceded by its length as a LEB128 varint.

TextEncodertext

Per-stream state for TextStreamFormat. There is none.

TextStreamFormattext

Raw UTF-8 text, undelimited.

Enums

QuoteStylecsv

Re-exported so callers can configure CsvStreamFormat without depending on csv. The quoting style to use when writing CSV data.

Terminatorcsv

Re-exported so callers can configure CsvStreamFormat without depending on csv. A record terminator.

Type Aliases

StreamResult

Alias for the Result type produced by streaming a body in either direction.