Crate kafko 

In-process log with Kafka-like semantics for Rust.

kafko exists for use cases where your data never needs to leave the process: embedded event sourcing, edge buffers, durable in-process pub/sub, deterministic integration tests without Docker or a broker, single-binary services that want a real log instead of a VecDeque<T> under a mutex.

Quickstart

use bytes::Bytes;
use kafko::Kafko;

#[tokio::main]
async fn main() -> kafko::Result<()> {
    let broker = Kafko::open("./data").await?;
    broker.create_topic("orders").await?;

    // Produce one record
    let producer = broker.producer_for("orders").await?;
    let pos = producer.send(None, Bytes::from("order-1")).await?;
    println!("appended at partition {} offset {}", pos.partition(), pos.offset());

    // Produce many records atomically in one round-trip
    let positions = producer
        .send_batch(vec![
            (None, Bytes::from("order-2")),
            (None, Bytes::from("order-3")),
        ])
        .await?;
    println!("batch positions: {:?}", positions);

    // Consume from the beginning
    let mut consumer = broker.consumer_for("orders").await?;
    consumer.seek_all(0);
    let record = consumer.next_record().await?;
    println!("read: {:?}", record.value());

    broker.shutdown().await?;
    Ok(())
}

Durability

Producer::send().await resolves once the record is in the OS file (page cache) — the same contract as Kafka acks=1. Records survive process crashes (panic, SIGKILL, OOM) but may be lost on OS panic or power loss until the kernel writes back. Call Kafko::shutdown for a hard durability boundary, or Partition::sync for mid-life fsync.

The recovery path on next Kafko::open CRC-scans the active segment and truncates any torn writes at the tail.

See the project README for benchmarks, the full architecture diagram, and the v0.2 roadmap.

Structs

Consumer

Cursor over the records of a topic, merging all its partitions into one stream.

Kafko

In-process log broker — owns topic registry, segment storage, and writer tasks.

Log

Append-only segment log backing one Partition.

LogConfig

Per-topic configuration for a partition’s on-disk log.

Partition

Single-partition log with its own writer task.

Producer

Cheap handle that appends records to a topic, routing each to a partition.

Record

A single log entry: timestamp, optional key, value bytes.

RecordPosition

Where a record landed: the partition it was routed to and its offset within that partition.

Segment

One on-disk .log segment file plus its in-memory append cursor.

SparseIndex

On-disk sparse offset index mapping record offset → file position.

Topic

A named stream split into one or more Partitions.

Enums

Compression

Per-topic value compression.

KafkoError

All errors surfaced by kafko’s public API.

Type Aliases

Result

Result alias for kafko operations.