Struct Buffer

pub struct Buffer { /* private fields */ }

A reusable buffer to prepare a batch of ILP messages.

Example

use questdb::ingress::{Buffer, TimestampMicros, TimestampNanos};
let mut buffer = sender.new_buffer();

// first row
buffer
    .table("table1")?
    .symbol("bar", "baz")?
    .column_bool("a", false)?
    .column_i64("b", 42)?
    .column_f64("c", 3.14)?
    .column_str("d", "hello")?
    .column_ts("e", TimestampMicros::now())?
    .at(TimestampNanos::now())?;

// second row
buffer
    .table("table2")?
    .symbol("foo", "bar")?
    .at(TimestampNanos::now())?;

Send the buffer to QuestDB using sender.flush(&mut buffer).

Sequential Coupling

The Buffer API is sequentially coupled:

• A row always starts with table.
• A row must contain at least one symbol or column ( column_bool, column_i64, column_f64, column_str, column_arr, column_ts).
• Symbols must appear before columns.
• A row must be terminated with either at or at_now.

This diagram visualizes the sequence:

Buffer method calls, Serialized ILP types and QuestDB types

Buffer Method	Serialized as ILP type (Click on link to see possible casts)
symbol	SYMBOL
column_bool	BOOLEAN
column_i64	INTEGER
column_f64	FLOAT
column_str	STRING
column_arr	ARRAY
column_ts	TIMESTAMP

QuestDB supports both STRING and SYMBOL column types.

To understand the difference, refer to the QuestDB documentation. In a nutshell, symbols are interned strings, most suitable for identifiers that are repeated many times throughout the column. They offer an advantage in storage space and query performance.

Inserting NULL values

To insert a NULL value, skip the symbol or column for that row.

Recovering from validation errors

If you want to recover from potential validation errors, call buffer.set_marker() to track the last known good state, append as many rows or parts of rows as you like, and then call buffer.clear_marker() on success.

If there was an error in one of the rows, use buffer.rewind_to_marker() to go back to the marked last known good state.

Implementations

impl Buffer

pub fn new(protocol_version: ProtocolVersion) -> Self

Creates a new Buffer with default parameters.

• Uses the specified protocol version
• Sets maximum name length to 127 characters (QuestDB server default)

This is equivalent to [Sender::new_buffer] when using the [Sender::protocol_version] and [Sender::max_name_len] is 127.

For custom name lengths, use Self::with_max_name_len

pub fn with_max_name_len( protocol_version: ProtocolVersion, max_name_len: usize, ) -> Self

Creates a new Buffer with a custom maximum name length.

• max_name_len: Maximum allowed length for table/column names, match your QuestDB server’s cairo.max.file.name.length configuration
• protocol_version: Protocol version to use

This is equivalent to [Sender::new_buffer] when using the [Sender::protocol_version] and [Sender::max_name_len].

For the default max name length limit (127), use Self::new.

pub fn protocol_version(&self) -> ProtocolVersion

pub fn reserve(&mut self, additional: usize)

Pre-allocate to ensure the buffer has enough capacity for at least the specified additional byte count. This may be rounded up. This does not allocate if such additional capacity is already satisfied. See: capacity.

pub fn len(&self) -> usize

The number of bytes accumulated in the buffer.

pub fn row_count(&self) -> usize

The number of rows accumulated in the buffer.

pub fn transactional(&self) -> bool

Tells whether the buffer is transactional. It is transactional iff it contains data for at most one table. Additionally, you must send the buffer over HTTP to get transactional behavior.

pub fn is_empty(&self) -> bool

pub fn capacity(&self) -> usize

The total number of bytes the buffer can hold before it needs to resize.

pub fn as_bytes(&self) -> &[u8]

pub fn set_marker(&mut self) -> Result<()>

Mark a rewind point. This allows undoing accumulated changes to the buffer for one or more rows by calling rewind_to_marker. Any previous marker will be discarded. Once the marker is no longer needed, call clear_marker.

pub fn rewind_to_marker(&mut self) -> Result<()>

Undo all changes since the last set_marker call.

As a side effect, this also clears the marker.

pub fn clear_marker(&mut self)

Discard any marker as may have been set by set_marker.

Idempotent.

pub fn clear(&mut self)

Reset the buffer and clear contents whilst retaining capacity.

pub fn check_can_flush(&self) -> Result<()>

Checks if this buffer is ready to be flushed to a sender via one of the [Sender::flush] functions. An Ok value indicates that the buffer is ready to be flushed via a [Sender] while an Err will contain a message indicating why this Buffer cannot be flushed at the moment.

pub fn table<'a, N>(&mut self, name: N) -> Result<&mut Self>

where N: TryInto<TableName<'a>>, Error: From<N::Error>,

Begin recording a new row for the given table.

buffer.table("table_name")?;

or

use questdb::ingress::TableName;

let table_name = TableName::new("table_name")?;
buffer.table(table_name)?;

pub fn symbol<'a, N, S>(&mut self, name: N, value: S) -> Result<&mut Self>

where N: TryInto<ColumnName<'a>>, S: AsRef<str>, Error: From<N::Error>,

Record a symbol for the given column. Make sure you record all symbol columns before any other column type.

buffer.symbol("col_name", "value")?;

or

let value: String = "value".to_owned();
buffer.symbol("col_name", value)?;

or

use questdb::ingress::ColumnName;

let col_name = ColumnName::new("col_name")?;
buffer.symbol(col_name, "value")?;

pub fn column_bool<'a, N>(&mut self, name: N, value: bool) -> Result<&mut Self>

where N: TryInto<ColumnName<'a>>, Error: From<N::Error>,

Record a boolean value for the given column.

buffer.column_bool("col_name", true)?;

or

use questdb::ingress::ColumnName;

let col_name = ColumnName::new("col_name")?;
buffer.column_bool(col_name, true)?;

pub fn column_i64<'a, N>(&mut self, name: N, value: i64) -> Result<&mut Self>

where N: TryInto<ColumnName<'a>>, Error: From<N::Error>,

Record an integer value for the given column.

buffer.column_i64("col_name", 42)?;

or

use questdb::ingress::ColumnName;

let col_name = ColumnName::new("col_name")?;
buffer.column_i64(col_name, 42)?;

pub fn column_f64<'a, N>(&mut self, name: N, value: f64) -> Result<&mut Self>

where N: TryInto<ColumnName<'a>>, Error: From<N::Error>,

Record a floating point value for the given column.

buffer.column_f64("col_name", 3.14)?;

or

use questdb::ingress::ColumnName;

let col_name = ColumnName::new("col_name")?;
buffer.column_f64(col_name, 3.14)?;

pub fn column_str<'a, N, S>(&mut self, name: N, value: S) -> Result<&mut Self>

where N: TryInto<ColumnName<'a>>, S: AsRef<str>, Error: From<N::Error>,

Record a string value for the given column.

buffer.column_str("col_name", "value")?;

or

let value: String = "value".to_owned();
buffer.column_str("col_name", value)?;

or

use questdb::ingress::ColumnName;

let col_name = ColumnName::new("col_name")?;
buffer.column_str(col_name, "value")?;

pub fn column_arr<'a, N, T, D>( &mut self, name: N, view: &T, ) -> Result<&mut Self>

where N: TryInto<ColumnName<'a>>, T: NdArrayView<D>, D: ArrayElement + ArrayElementSealed, Error: From<N::Error>,

Record a multidimensional array value for the given column.

Supports arrays with up to MAX_ARRAY_DIMS dimensions. The array elements must be of type f64, which is currently the only supported data type.

Note: QuestDB server version 9.0.0 or later is required for array support.

Examples

Recording a 2D array using slices:

let array_2d = vec![vec![1.1, 2.2], vec![3.3, 4.4]];
buffer.column_arr("array_col", &array_2d)?;

Recording a 3D array using vectors:

let array_3d = vec![vec![vec![42.0; 4]; 3]; 2];
let col_name = ColumnName::new("col1")?;
buffer.column_arr(col_name, &array_3d)?;

Errors

Returns Error if:

• Array dimensions exceed MAX_ARRAY_DIMS
• Failed to get dimension sizes
• Column name validation fails
• Protocol version v1 is used (arrays require v2+)

pub fn column_ts<'a, N, T>(&mut self, name: N, value: T) -> Result<&mut Self>

where N: TryInto<ColumnName<'a>>, T: TryInto<Timestamp>, Error: From<N::Error> + From<T::Error>,

Record a timestamp value for the given column.

use questdb::ingress::TimestampMicros;
buffer.column_ts("col_name", TimestampMicros::now())?;

or

use questdb::ingress::TimestampMicros;

buffer.column_ts("col_name", TimestampMicros::new(1659548204354448))?;

or

use questdb::ingress::TimestampMicros;
use questdb::ingress::ColumnName;

let col_name = ColumnName::new("col_name")?;
buffer.column_ts(col_name, TimestampMicros::now())?;

or you can also pass in a TimestampNanos.

Note that both TimestampMicros and TimestampNanos can be constructed easily from either std::time::SystemTime or chrono::DateTime.

This last option requires the chrono_timestamp feature.

pub fn at<T>(&mut self, timestamp: T) -> Result<()>

where T: TryInto<Timestamp>, Error: From<T::Error>,

Complete the current row with the designated timestamp. After this call, you can start recording the next row by calling Buffer::table again, or you can send the accumulated batch by calling [Sender::flush] or one of its variants.

use questdb::ingress::TimestampNanos;
buffer.at(TimestampNanos::now())?;

or

use questdb::ingress::TimestampNanos;

buffer.at(TimestampNanos::new(1659548315647406592))?;

You can also pass in a TimestampMicros.

Note that both TimestampMicros and TimestampNanos can be constructed easily from either std::time::SystemTime or chrono::DateTime.

pub fn at_now(&mut self) -> Result<()>

Complete the current row without providing a timestamp. The QuestDB instance will insert its own timestamp.

Letting the server assign the timestamp can be faster since it reliably avoids out-of-order operations in the database for maximum ingestion throughput. However, it removes the ability to deduplicate rows.

This is NOT equivalent to calling Buffer::at with the current time: the QuestDB server will set the timestamp only after receiving the row. If you’re flushing infrequently, the server-assigned timestamp may be significantly behind the time the data was recorded in the buffer.

In almost all cases, you should prefer the Buffer::at function.

After this call, you can start recording the next row by calling Buffer::table again, or you can send the accumulated batch by calling [Sender::flush] or one of its variants.

buffer.at_now()?;

Trait Implementations

impl Clone for Buffer

fn clone(&self) -> Buffer

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Buffer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.