Crate rusty_chunkenc

A Rust implementation of Prometheus’ chunkenc library.

Features

• Parse Prometheus XOR-encoded chunks (that are heavily inspired by Gorilla).
• Serialise time series to Prometheus XOR-encoded chunks.
• Read Prometheus’ cold data directly from the disk.
• Also comes with utilities to read and write varint, uvarint, varbit, varbit_ts, and varbit_xor numbers.

Why?

Prometheus uses XOR Chunks in its remote read API, and I wanted to understand how they work in detail. This crate enables SensApp to stream data to Prometheus. SensApp is written in Rust, and I wanted a chunkenc Rust implementation.

Also, writing a parser and a serialiser did sound fun.

Acknowledgements

This project is ported from Prometheus’ chunkenc, that used go-tzs, that is based on the Gorilla paper. The parsing heavily relies on nom.

The project supports the Smart Building Hub research infrastructure project, which is funded by the Norwegian Research Council.

Example

let chunk_disk_format = rusty_chunkenc::ChunksDiskFormat::new(
vec![
    rusty_chunkenc::Chunk::new_xor(vec![
        rusty_chunkenc::XORSample {
            timestamp: 7200000,
            value: 12000.0,
        },
        rusty_chunkenc::XORSample {
            timestamp: 7201000,
            value: 12001.0,
        },
    ]),
    rusty_chunkenc::Chunk::new_xor(vec![
        rusty_chunkenc::XORSample {
            timestamp: 7200000,
            value: 123.45,
        },
        rusty_chunkenc::XORSample {
            timestamp: 7201000,
            value: 123.46,
        },
    ]),
],
None,
);

// Serialise the chunks
let mut buffer: Vec<u8> = Vec::new();
chunk_disk_format.write(&mut buffer).unwrap();

// Parse a chunk from a buffer
let (_, parsed_chunk_disk_format) = rusty_chunkenc::read_chunks(&buffer, None).unwrap();
println!("parsed_chunks: {:?}", parsed_chunk_disk_format);
assert_eq!(parsed_chunk_disk_format, chunk_disk_format);

Or for a single chunk:

let chunk = rusty_chunkenc::Chunk::new_xor(vec![
    rusty_chunkenc::XORSample {
        timestamp: 7200000,
        value: 12000.0,
    },
    rusty_chunkenc::XORSample {
        timestamp: 7201000,
        value: 12001.0,
    },
]);

// Serialise the chunk
let mut buffer: Vec<u8> = Vec::new();
chunk.write(&mut buffer).unwrap();

assert_eq!(
    buffer,
    [
        0x12, 0x01, 0x00, 0x02, 0x80, 0xF4, 0xEE, 0x06, 0x40, 0xC7, 0x70, 0x00, 0x00, 0x00,
        0x00, 0x00, 0xE8, 0x07, 0xF0, 0x0C, 0x1F, 0xCE, 0x4F, 0xA7
    ]
);

// Parse a chunk from a buffer
let (_, parsed_chunk) = rusty_chunkenc::read_chunk(&buffer).unwrap();
println!("parsed_chunk: {:?}", parsed_chunk);

Re-exports

• pub use chunk::read_chunk;
• pub use chunk::Chunk;
• pub use chunks::read_chunks;
• pub use chunks::ChunksDiskFormat;
• pub use xor::XORSample;

Modules

• chunk
  Single Prometheus chunk.
• chunks
  Prometheus chunks disk format.
• crc32c
  CRC32 Castagnoli Checksum.
• folder
  WIP: Parse all prometheus data from the prometheus folder.
• histogram
  Histogram and Float Histogram chunks, not implemented yet.
• index
  WIP: Prometheus index files
• uvarint
  Golang’s uvarint.
• varbit
  Prometheus’s varbit encoding. Prometheus’ varbit encoding.
• varbit_ts
  Prometheus’s varbit timestamp encoding.
• varbit_xor
  Prometheus’s varbit xor encoding.
• varint
  Golang’s varint.
• xor
  XOR chunk.