Module questdb::ingress

Fast Ingestion of data into QuestDB

The ingress module implements QuestDB’s variant of the InfluxDB Line Protocol (ILP) over TCP.

To get started:

• Connect to QuestDB by creating a Sender object, usually via Sender::from_conf.
• Populate a Buffer with one or more rows of data.
• Send the buffer via the Sender’s flush method.

use questdb::{
    Result,
    ingress::{
        Sender,
        Buffer,
        TimestampNanos}};

fn main() -> Result<()> {
   let mut sender = Sender::from_conf("http::addr=localhost:9000;")?;
   let mut buffer = Buffer::new();
   buffer
       .table("sensors")?
       .symbol("id", "toronto1")?
       .column_f64("temperature", 20.0)?
       .column_i64("humidity", 50)?
       .at(TimestampNanos::now())?;
   sender.flush(&mut buffer)?;
   Ok(())
}

Flushing

The Sender’s flush method will clear the buffer which is then reusable for another batch of rows.

Dropping the sender will close the connection to QuestDB and any unflushed messages will be lost: In other words, do not forget to flush before closing the connection!

A common technique is to flush periodically on a timer and/or once the buffer exceeds a certain size. You can check the buffer’s size by the calling Buffer’s len method.

Note that flushing will automatically clear the buffer’s contents. If you’d rather preserve the contents (for example, to send the same data to multiple QuestDB instances), you can call flush_and_keep instead.

Connection Security Options

To establish an authenticated and TLS-encrypted connection, call the SenderBuilder’s authentication and tls methods.

Here’s an example that uses full security with TCP:

use questdb::ingress::{Protocol, SenderBuilder};

// See: https://questdb.io/docs/reference/api/ilp/authenticate
let mut sender = SenderBuilder::new(Protocol::Tcps, "localhost", 9009)
    .username("testUser1")? // kid
    .token("5UjEMuA0Pj5pjK8a-fa24dyIf-Es5mYny3oE_Wmus48")? // d
    .token_x("fLKYEaoEb9lrn3nkwLDA-M_xnuFOdSt9y0Z7_vWSHLU")? // x
    .token_y("Dt5tbS1dEDMSYfym3fgMv0B99szno-dFc1rYF9t0aac")? // y
    .build()?;

Note that Open Source QuestDB does not natively support TLS encryption (this is a QuestDB enterprise feature).

To use TLS with QuestDB open source, use a TLS proxy such as HAProxy.

For testing, you can use a self-signed certificate and key.

See our notes on how to generate keys that this library will accept.

From the API, you can then point to a custom CA file:

use std::path::PathBuf;
use questdb::ingress::{SenderBuilder, Protocol, CertificateAuthority};

let mut sender = SenderBuilder::new(Protocol::Tcps, "localhost", 9009)
    .tls_roots("/path/to/server_rootCA.pem")?
    .build()?;

Avoiding revalidating names

To avoid re-validating table and column names, consider re-using them across rows.

use questdb::ingress::{
    TableName,
    ColumnName,
    Buffer,
    TimestampNanos};

let mut buffer = Buffer::new();
let tide_name = TableName::new("tide")?;
let water_level_name = ColumnName::new("water_level")?;
buffer.table(tide_name)?.column_f64(water_level_name, 20.4)?.at(TimestampNanos::now())?;
buffer.table(tide_name)?.column_f64(water_level_name, 17.2)?.at(TimestampNanos::now())?;

Buffer API sequential coupling

Symbols must always be written before rows. See the Buffer documentation for details. Each row must be terminated with a call to either at or at_now.

Considerations

The Library considerations documentation goes through:

• Threading.
• Differences between ILP vs QuestDB Data Types.
• Data Quality
• Client-side checks and server errors
• Flushing
• Disconnections, data errors and troubleshooting

Troubleshooting Common Issues

Infrequent Flushing

You may not see data appear in a timely manner because you’re not calling the flush method often enough.

Debugging disconnects and inspecting errors

The ILP protocol does not send errors back to the client. Instead, on error, the QuestDB server will disconnect and any error messages will be present in the server logs.

If you want to inspect or log a buffer’s contents before it is sent, you can call its as_str method.

Structs

• Buffer
  A reusable buffer to prepare ILP messages.
• ColumnName
  A validated column name.
• Port
  A u16 port number or String port service name as is registered with /etc/services or equivalent.
• Sender
  Connects to a QuestDB instance and inserts data via the ILP protocol.
• SenderBuilder
  Accumulate parameters for a new Sender instance.
• TableName
  A validated table name.
• TimestampMicros
  A i64 timestamp expressed as microseconds since the UNIX epoch (UTC).
• TimestampNanos
  A i64 timestamp expressed as nanoseconds since the UNIX epoch (UTC).

Enums

• CertificateAuthority
  Possible sources of the root certificates used to validate the server’s TLS certificate.
• Protocol
  Protocol used to communicate with the QuestDB server.
• Timestamp
  A timestamp expressed as micros or nanos. You should seldom use this directly. Instead use one of: