pub struct Buffer { /* private fields */ }
Expand description
A reusable buffer to prepare a batch of ILP messages.
§Example
use questdb::ingress::{Buffer, TimestampMicros, TimestampNanos};
let mut buffer = sender.new_buffer();
// first row
buffer
.table("table1")?
.symbol("bar", "baz")?
.column_bool("a", false)?
.column_i64("b", 42)?
.column_f64("c", 3.14)?
.column_str("d", "hello")?
.column_ts("e", TimestampMicros::now())?
.at(TimestampNanos::now())?;
// second row
buffer
.table("table2")?
.symbol("foo", "bar")?
.at(TimestampNanos::now())?;
Send the buffer to QuestDB using sender.flush(&mut buffer)
.
§Sequential Coupling
The Buffer API is sequentially coupled:
- A row always starts with
table
. - A row must contain at least one
symbol
or column (column_bool
,column_i64
,column_f64
,column_str
,column_arr
,column_ts
). - Symbols must appear before columns.
- A row must be terminated with either
at
orat_now
.
This diagram visualizes the sequence:
§Buffer method calls, Serialized ILP types and QuestDB types
Buffer Method | Serialized as ILP type (Click on link to see possible casts) |
---|---|
symbol | SYMBOL |
column_bool | BOOLEAN |
column_i64 | INTEGER |
column_f64 | FLOAT |
column_str | STRING |
column_arr | ARRAY |
column_ts | TIMESTAMP |
QuestDB supports both STRING
and SYMBOL
column types.
To understand the difference, refer to the QuestDB documentation. In a nutshell, symbols are interned strings, most suitable for identifiers that are repeated many times throughout the column. They offer an advantage in storage space and query performance.
§Inserting NULL values
To insert a NULL value, skip the symbol or column for that row.
§Recovering from validation errors
If you want to recover from potential validation errors, call
buffer.set_marker()
to track the last known good state,
append as many rows or parts of rows as you like, and then call
buffer.clear_marker()
on success.
If there was an error in one of the rows, use
buffer.rewind_to_marker()
to go back to the
marked last known good state.
Implementations§
Source§impl Buffer
impl Buffer
Sourcepub fn new(protocol_version: ProtocolVersion) -> Self
pub fn new(protocol_version: ProtocolVersion) -> Self
Creates a new Buffer
with default parameters.
- Uses the specified protocol version
- Sets maximum name length to 127 characters (QuestDB server default)
This is equivalent to [Sender::new_buffer
] when using the [Sender::protocol_version
]
and [Sender::max_name_len
] is 127.
For custom name lengths, use Self::with_max_name_len
Sourcepub fn with_max_name_len(
protocol_version: ProtocolVersion,
max_name_len: usize,
) -> Self
pub fn with_max_name_len( protocol_version: ProtocolVersion, max_name_len: usize, ) -> Self
Creates a new Buffer
with a custom maximum name length.
max_name_len
: Maximum allowed length for table/column names, match your QuestDB server’scairo.max.file.name.length
configurationprotocol_version
: Protocol version to use
This is equivalent to [Sender::new_buffer
] when using the [Sender::protocol_version
]
and [Sender::max_name_len
].
For the default max name length limit (127), use Self::new
.
pub fn protocol_version(&self) -> ProtocolVersion
Sourcepub fn reserve(&mut self, additional: usize)
pub fn reserve(&mut self, additional: usize)
Pre-allocate to ensure the buffer has enough capacity for at least the
specified additional byte count. This may be rounded up.
This does not allocate if such additional capacity is already satisfied.
See: capacity
.
Sourcepub fn transactional(&self) -> bool
pub fn transactional(&self) -> bool
Tells whether the buffer is transactional. It is transactional iff it contains data for at most one table. Additionally, you must send the buffer over HTTP to get transactional behavior.
pub fn is_empty(&self) -> bool
Sourcepub fn capacity(&self) -> usize
pub fn capacity(&self) -> usize
The total number of bytes the buffer can hold before it needs to resize.
pub fn as_bytes(&self) -> &[u8] ⓘ
Sourcepub fn set_marker(&mut self) -> Result<()>
pub fn set_marker(&mut self) -> Result<()>
Mark a rewind point.
This allows undoing accumulated changes to the buffer for one or more
rows by calling rewind_to_marker
.
Any previous marker will be discarded.
Once the marker is no longer needed, call
clear_marker
.
Sourcepub fn rewind_to_marker(&mut self) -> Result<()>
pub fn rewind_to_marker(&mut self) -> Result<()>
Undo all changes since the last set_marker
call.
As a side effect, this also clears the marker.
Sourcepub fn clear_marker(&mut self)
pub fn clear_marker(&mut self)
Discard any marker as may have been set by
set_marker
.
Idempotent.
Sourcepub fn check_can_flush(&self) -> Result<()>
pub fn check_can_flush(&self) -> Result<()>
Sourcepub fn table<'a, N>(&mut self, name: N) -> Result<&mut Self>
pub fn table<'a, N>(&mut self, name: N) -> Result<&mut Self>
Begin recording a new row for the given table.
buffer.table("table_name")?;
or
use questdb::ingress::TableName;
let table_name = TableName::new("table_name")?;
buffer.table(table_name)?;
Sourcepub fn symbol<'a, N, S>(&mut self, name: N, value: S) -> Result<&mut Self>
pub fn symbol<'a, N, S>(&mut self, name: N, value: S) -> Result<&mut Self>
Record a symbol for the given column. Make sure you record all symbol columns before any other column type.
buffer.symbol("col_name", "value")?;
or
let value: String = "value".to_owned();
buffer.symbol("col_name", value)?;
or
use questdb::ingress::ColumnName;
let col_name = ColumnName::new("col_name")?;
buffer.symbol(col_name, "value")?;
Sourcepub fn column_bool<'a, N>(&mut self, name: N, value: bool) -> Result<&mut Self>
pub fn column_bool<'a, N>(&mut self, name: N, value: bool) -> Result<&mut Self>
Record a boolean value for the given column.
buffer.column_bool("col_name", true)?;
or
use questdb::ingress::ColumnName;
let col_name = ColumnName::new("col_name")?;
buffer.column_bool(col_name, true)?;
Sourcepub fn column_i64<'a, N>(&mut self, name: N, value: i64) -> Result<&mut Self>
pub fn column_i64<'a, N>(&mut self, name: N, value: i64) -> Result<&mut Self>
Record an integer value for the given column.
buffer.column_i64("col_name", 42)?;
or
use questdb::ingress::ColumnName;
let col_name = ColumnName::new("col_name")?;
buffer.column_i64(col_name, 42)?;
Sourcepub fn column_f64<'a, N>(&mut self, name: N, value: f64) -> Result<&mut Self>
pub fn column_f64<'a, N>(&mut self, name: N, value: f64) -> Result<&mut Self>
Record a floating point value for the given column.
buffer.column_f64("col_name", 3.14)?;
or
use questdb::ingress::ColumnName;
let col_name = ColumnName::new("col_name")?;
buffer.column_f64(col_name, 3.14)?;
Sourcepub fn column_str<'a, N, S>(&mut self, name: N, value: S) -> Result<&mut Self>
pub fn column_str<'a, N, S>(&mut self, name: N, value: S) -> Result<&mut Self>
Record a string value for the given column.
buffer.column_str("col_name", "value")?;
or
let value: String = "value".to_owned();
buffer.column_str("col_name", value)?;
or
use questdb::ingress::ColumnName;
let col_name = ColumnName::new("col_name")?;
buffer.column_str(col_name, "value")?;
Sourcepub fn column_arr<'a, N, T, D>(
&mut self,
name: N,
view: &T,
) -> Result<&mut Self>where
N: TryInto<ColumnName<'a>>,
T: NdArrayView<D>,
D: ArrayElement + ArrayElementSealed,
Error: From<N::Error>,
pub fn column_arr<'a, N, T, D>(
&mut self,
name: N,
view: &T,
) -> Result<&mut Self>where
N: TryInto<ColumnName<'a>>,
T: NdArrayView<D>,
D: ArrayElement + ArrayElementSealed,
Error: From<N::Error>,
Record a multidimensional array value for the given column.
Supports arrays with up to MAX_ARRAY_DIMS
dimensions. The array elements must
be of type f64
, which is currently the only supported data type.
Note: QuestDB server version 9.0.0 or later is required for array support.
§Examples
Recording a 2D array using slices:
let array_2d = vec![vec![1.1, 2.2], vec![3.3, 4.4]];
buffer.column_arr("array_col", &array_2d)?;
Recording a 3D array using vectors:
let array_3d = vec![vec![vec![42.0; 4]; 3]; 2];
let col_name = ColumnName::new("col1")?;
buffer.column_arr(col_name, &array_3d)?;
§Errors
Returns Error
if:
- Array dimensions exceed
MAX_ARRAY_DIMS
- Failed to get dimension sizes
- Column name validation fails
- Protocol version v1 is used (arrays require v2+)
Sourcepub fn column_ts<'a, N, T>(&mut self, name: N, value: T) -> Result<&mut Self>
pub fn column_ts<'a, N, T>(&mut self, name: N, value: T) -> Result<&mut Self>
Record a timestamp value for the given column.
use questdb::ingress::TimestampMicros;
buffer.column_ts("col_name", TimestampMicros::now())?;
or
use questdb::ingress::TimestampMicros;
buffer.column_ts("col_name", TimestampMicros::new(1659548204354448))?;
or
use questdb::ingress::TimestampMicros;
use questdb::ingress::ColumnName;
let col_name = ColumnName::new("col_name")?;
buffer.column_ts(col_name, TimestampMicros::now())?;
or you can also pass in a TimestampNanos
.
Note that both TimestampMicros
and TimestampNanos
can be constructed
easily from either std::time::SystemTime
or chrono::DateTime
.
This last option requires the chrono_timestamp
feature.
Sourcepub fn at<T>(&mut self, timestamp: T) -> Result<()>
pub fn at<T>(&mut self, timestamp: T) -> Result<()>
Complete the current row with the designated timestamp. After this call, you can start recording the next row by calling Buffer::table again, or you can send the accumulated batch by calling [Sender::flush] or one of its variants.
use questdb::ingress::TimestampNanos;
buffer.at(TimestampNanos::now())?;
or
use questdb::ingress::TimestampNanos;
buffer.at(TimestampNanos::new(1659548315647406592))?;
You can also pass in a TimestampMicros
.
Note that both TimestampMicros
and TimestampNanos
can be constructed
easily from either std::time::SystemTime
or chrono::DateTime
.
Sourcepub fn at_now(&mut self) -> Result<()>
pub fn at_now(&mut self) -> Result<()>
Complete the current row without providing a timestamp. The QuestDB instance will insert its own timestamp.
Letting the server assign the timestamp can be faster since it reliably avoids out-of-order operations in the database for maximum ingestion throughput. However, it removes the ability to deduplicate rows.
This is NOT equivalent to calling Buffer::at with the current time: the QuestDB server will set the timestamp only after receiving the row. If you’re flushing infrequently, the server-assigned timestamp may be significantly behind the time the data was recorded in the buffer.
In almost all cases, you should prefer the Buffer::at function.
After this call, you can start recording the next row by calling Buffer::table again, or you can send the accumulated batch by calling [Sender::flush] or one of its variants.
buffer.at_now()?;