Struct FileLog 

pub struct FileLog { /* private fields */ }

A commit log implementation that uses the file system as its backing store.

Considerations:

1. Partition values cannot be non-zero.
2. The number of subscriptions of a topic will translate to the number of tasks that are spawned, along with their associated resources.
3. Only one process can produce to a specific topic. There is no process-wide locking considered. Multiple processes can read a topic though.

Implementations

impl FileLog

pub fn new<P>(root_path: P) -> Self

where P: Into<PathBuf>,

Construct a new file log that will also spawn a task for each topic being produced.

pub fn with_config<P>( root_path: P, compaction_threshold_size: u64, read_buffer_size: usize, compaction_write_buffer_size: usize, write_buffer_size: usize, max_record_size: usize, ) -> Self

where P: Into<PathBuf>,

Construct a new file log that will also spawn a task for each topic being produced. The compaction_threshold_size is the size of the active file that the compactor looks at before deciding to perform a compaction (in bytes). This typically equates to the blocksize on disk i.e. 64KB for flash based storage. 64KB is still small enough that scans over a topic are relatively fast, working on the principle of having roughly 2,000 records. We also require a read and write buffer sizes to reduce system calls. When writing, either the buffer reaches capacity or a flush of the buffer occurs in the absence of another write to perform.

pub fn close_topic(&mut self, topic: &Topic)

Frees resources associated with a topic, but not any associated compaction. Invoking the method is benign in that if consuming or producing occurs on this post closing, resources will be re-established.

pub async fn register_compaction<CS>( &mut self, topic: Topic, compaction_strategy: CS, ) -> Result<(), CompactionRegistrationError>

where CS: CompactionStrategy + Send + Sync + 'static,

Register compaction for a given topic. Any previously registered compaction is replaced. A new task for compaction will be created in the background.

Compaction’s memory can be controlled somewhat through compaction_write_buffer_size when creating this file commit log. This buffer size is selected to minimize writing to flash and will be allocated once per topic compaction registered here.

pub fn unregister_compaction(&mut self, topic: &Topic)

Unregister compaction for a given topic

Trait Implementations

impl Clone for FileLog

fn clone(&self) -> FileLog

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl CommitLog for FileLog

fn offsets<'life0, 'async_trait>( &'life0 self, topic: Topic, _partition: Partition, ) -> Pin<Box<dyn Future<Output = Option<PartitionOffsets>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Retrieve the current offsets of a topic if they are present.

fn produce<'life0, 'async_trait>( &'life0 self, record: ProducerRecord, ) -> Pin<Box<dyn Future<Output = Result<ProducedOffset, ProducerError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait,

Publish a record and return the offset that was assigned.

fn scoped_subscribe<'a>( &'a self, _consumer_group_name: &str, offsets: Vec<ConsumerOffset>, subscriptions: Vec<Subscription>, idle_timeout: Option<Duration>, ) -> Pin<Box<dyn Stream<Item = ConsumerRecord> + Send + 'a>>

Subscribe to one or more topics for a given consumer group having committed zero or more topics. The records are streamed back indefinitely unless an idle timeout argument is provided. In the case of an idle timeout, if no record is received within that period, None is returned to end the stream.