Module questdb::ingress

Fast Ingestion of data into QuestDB

The ingress module implements QuestDB’s variant of the InfluxDB Line Protocol (ILP) over TCP.

To get started:

• Connect to QuestDB by creating a Sender object.
• Populate a Buffer with one or more rows of data.
• Send the buffer via the Sender’s flush method.

use questdb::{
    Result,
    ingress::{
        Sender,
        Buffer,
        SenderBuilder}};
 
fn main() -> Result<()> {
   let mut sender = SenderBuilder::new("localhost", 9009).connect()?;
   let mut buffer = Buffer::new();
   buffer
       .table("sensors")?
       .symbol("id", "toronto1")?
       .column_f64("temperature", 20.0)?
       .column_i64("humidity", 50)?
       .at_now()?;
   sender.flush(&mut buffer)?;
   Ok(())
}

Flushing

The Sender’s flush method will clear the buffer which is then reusable for another batch of rows.

Dropping the sender will close the connection to QuestDB and any unflushed messages will be lost: In other words, do not forget to flush before closing the connection!

A common technique is to flush periodically on a timer and/or once the buffer exceeds a certain size. You can check the buffer’s size by the calling Buffer’s len method.

Note that flushing will automatically clear the buffer’s contents. If you’d rather preserve the contents (for example, to send the same data to multiple QuestDB instances), you can call flush_and_keep instead.

Connection Security Options

To establish an authenticated and TLS-encrypted connection, call the SenderBuilder’s auth and tls methods.

Here’s an example that uses full security:

use questdb::ingress::{SenderBuilder, Tls, CertificateAuthority};
 
// See: https://questdb.io/docs/reference/api/ilp/authenticate
let mut sender = SenderBuilder::new("localhost", 9009)
    .auth(
        "testUser1",                                    // kid
        "5UjEMuA0Pj5pjK8a-fa24dyIf-Es5mYny3oE_Wmus48",  // d
        "fLKYEaoEb9lrn3nkwLDA-M_xnuFOdSt9y0Z7_vWSHLU",  // x
        "Dt5tbS1dEDMSYfym3fgMv0B99szno-dFc1rYF9t0aac")  // y
    .tls(Tls::Enabled(CertificateAuthority::WebpkiRoots))
    .connect()?;

Note that as of writing QuestDB does not natively support TLS encryption. To use TLS use a TLS proxy such as HAProxy.

For testing, you can use a self-signed certificate and key.

See our notes on how to generate keys that this library will accept.

From the API, you can then point to a custom CA file:

use std::path::PathBuf;
use questdb::ingress::{SenderBuilder, Tls, CertificateAuthority};
 
let mut sender = SenderBuilder::new("localhost", 9009)
    .tls(Tls::Enabled(CertificateAuthority::File(
        PathBuf::from("/path/to/server_rootCA.pem"))))
    .connect()?;

Avoiding revalidating names

To avoid re-validating table and column names, consider re-using them across rows.

use questdb::ingress::{
    TableName,
    ColumnName,
    Buffer};
 
let mut buffer = Buffer::new();
let tide_name = TableName::new("tide")?;
let water_level_name = ColumnName::new("water_level")?;
buffer.table(tide_name)?.column_f64(water_level_name, 20.4)?.at_now()?;
buffer.table(tide_name)?.column_f64(water_level_name, 17.2)?.at_now()?;

Buffer API sequential coupling

Symbols must always be written before rows. See the Buffer documentation for details. Each row must be terminated with a call to either at or at_now.

Data quality and threading considerations

• Library considerations documentation.

Troubleshooting Common Issues

Production-optimized QuestDB configuration

If you can’t initially see your data through a select SQL query straight away, this is normal: by default the database will only commit data it receives though the line protocol periodically to maximize throughput.

For dev/testing you may want to tune the following database configuration parameters as so in server.conf:

cairo.max.uncommitted.rows=1
line.tcp.maintenance.job.interval=100

Infrequent Flushing

You may not see data appear in a timely manner because you’re not calling the flush method often enough.

Debugging disconnects and inspecting errors

The ILP protocol does not send errors back to the client. Instead, on error, the QuestDB server will disconnect and any error messages will be present in the server logs.

If you want to inspect or log a buffer’s contents before it is sent, you can call its as_str method.

Structs

Buffer

A reusable buffer to prepare ILP messages.

ColumnName

A validated column name.

Sender

Connects to a QuestDB instance and inserts data via the ILP protocol.

SenderBuilder

Accumulate parameters for a new Sender instance.

Service

A u16 port number or String port service name as is registered with /etc/services or equivalent.

TableName

A validated table name.

TimestampMicros

A i64 timestamp expressed as microseconds since the UNIX epoch (UTC).

TimestampNanos

A i64 timestamp expressed as nanoseconds since the UNIX epoch (UTC).

Enums

CertificateAuthority

Root used to determine how to validate the server’s TLS certificate.

Tls

Options for full-connection encryption via TLS.