Crate kafko 

In-process log with Kafka-like semantics for Rust.

kafko exists for use cases where your data never needs to leave the process: embedded event sourcing, edge buffers, durable in-process pub/sub, deterministic integration tests without Docker or a broker, single-binary services that want a real log instead of a VecDeque<T> under a mutex.

Quickstart

use bytes::Bytes;
use kafko::Kafko;

#[tokio::main]
async fn main() -> kafko::Result<()> {
    let broker = Kafko::open("./data").await?;
    broker.create_topic("orders").await?;

    // Produce one record
    let producer = broker.producer_for("orders").await?;
    let offset = producer.send(None, Bytes::from("order-1")).await?;
    println!("appended at offset {offset}");

    // Produce many records atomically in one round-trip
    let offsets = producer
        .send_batch(vec![
            (None, Bytes::from("order-2")),
            (None, Bytes::from("order-3")),
        ])
        .await?;
    println!("batch offsets: {:?}", offsets);

    // Consume from the beginning
    let mut consumer = broker.consumer_for("orders").await?;
    consumer.seek(0);
    let record = consumer.next_record().await?;
    println!("read: {:?}", record.value());

    broker.shutdown().await?;
    Ok(())
}

Durability

Producer::send().await resolves once the record is in the OS file (page cache) — the same contract as Kafka acks=1. Records survive process crashes (panic, SIGKILL, OOM) but may be lost on OS panic or power loss until the kernel writes back. Call Kafko::shutdown for a hard durability boundary, or Partition::sync for mid-life fsync.

The recovery path on next Kafko::open CRC-scans the active segment and truncates any torn writes at the tail.

See the project README for benchmarks, the full architecture diagram, and the v0.2 roadmap.

Structs

Consumer

Cursor over the records of a single topic partition.

Kafko

In-process log broker — owns topic registry, segment storage, and writer tasks.

Log

Append-only segment log backing one Partition.

LogConfig

Per-topic configuration for a partition’s on-disk log.

Partition

Single-partition log with its own writer task.

Producer

Cheap handle that appends records to a single topic partition.

Record

A single log entry: timestamp, optional key, value bytes.

Segment

One on-disk .log segment file plus its in-memory append cursor.

SparseIndex

On-disk sparse offset index mapping record offset → file position.

Enums

Compression

Per-topic value compression.

KafkoError

All errors surfaced by kafko’s public API.

Type Aliases

Result

Result alias for kafko operations.