Struct TrackedReader 

pub struct TrackedReader { /* private fields */ }

Structure that implements Read, ReadBuf and Seek while working with persistent offset in underlying logrotated files. External file is used to persist offset across restarts.

Usage

Instantiate TrackedReader by passing it a path to logfile intended for reading as well as a path to file used as a registry for persistent offset storage. Created structure can be used where implementation of Read or BufRead is expected. Additionally, limited Seek implementation is provided (see Limitations for more info).

// running this script will fetch and print new lines on each execution
let mut reader = TrackedReader::new("examples/file.txt", "examples/registry")?;
let mut input = String::new();
loop {
    match reader.read_line(&mut input)? {
        0 => break Ok(()),
        _ => println!("read line: `{}`", input.trim_end()),
    };
}

Cleanup

There are two distinct ways to perform cleanup for this structure:

• explicit by calling .close(). This will allow you to handle any errors that may happen in the process
• implicitly by relying on Drop. Note that errors generated while working with the filesystem cannot be handled and will cause a panic in this case.

Working principles

To maintain offset in a file across restarts, separate “registry” file is used for persistence. Inode is stored additionally to offset, which allows to keep reading log file in case it was logrotate’d. During intialization, inode of file to be read is compared to previously known and if it differs, it means that file was rotated and a search for original file is performed by checking a file identified by path appended by .1 (eg. mail.log and mail.log.1) and so on. After that you are given a file-like structure that allows buffered reading and seeking in up to specified number of files files.

Limitations

• You can only expect this to work if logrotation happened not more than the number you specified as search_depth. This means that if you are creating a log processor for example, it should be run frequently enough to keep up with logs that are written and rotated.

• Due to simple scheme of persistence, we cannot seek back into rotated file version after saving state while reading from current log file. This means that if your program must do some conditional seeking in a file, you should perform any pointer rollback before performing final save (done by .close() or Drop). Overall, this library is intended to be used for mostly forward reading of log files.

Implementations

impl TrackedReader

pub fn new( filepath: impl AsRef<Path>, registry: impl AsRef<Path>, ) -> Result<Self, TrackedReaderError>

Creates a new TrackedReader possibly loading current offset from a registry file. On a first execution registry file most likely will not exist and in that case it will be created with zero offset.

Arguments

• filepath - a path to log file to be read. TrackedReader will additionally search for logrotated file under {filepath}.1.
• registry - path to registry file used to persist offset and other metadata.

pub fn with_search_depth( filepath: impl AsRef<Path>, registry: impl AsRef<Path>, search_depth: usize, ) -> Result<Self, TrackedReaderError>

Like ::new but allows specifying how many rotated items to check.

search_depth of 2 means that apart from log file we will check for log.1 and log.2.

pub fn persist(&mut self) -> Result<()>

Explicitly save current state into registry file and return any errors generated.

pub fn close(self) -> Result<()>

Explicitly finalize structure, returning any errors that were produced in the process. Alternative to relying on Drop.

pub fn get_persistent_state(&self) -> State

Get current state for possible manual handling.

Methods from Deref<Target = InodeAwareReader>

pub fn get_persistent_offset(&self) -> InodeAwareOffset

Get offset that can be used across restarts and log rotations.

pub fn seek_persistent(&mut self, offset: InodeAwareOffset) -> Result<()>

Seek by persistent offset.

Will return NotFound io error if file with given inode was not found.

pub fn get_inodes(&self) -> &[u64]

Get slice of inodes for current execution.

pub fn get_current_inode(&self) -> u64

Get inode of an item that is currently read.

pub fn get_item_index_by_inode(&self, inode: u64) -> Option<usize>

Search for item index by given inode.

pub fn compare_offsets( &self, first: InodeAwareOffset, second: InodeAwareOffset, ) -> Option<Ordering>

Compare two offsets as if they were pointing into one large buffer. Returns None if any of the offsets do not belong to underlying files.

Methods from Deref<Target = Multireader<BufReader<File>>>

pub fn get_global_offset(&self) -> u64

Offset amoung all underlying items.

pub fn get_local_offset(&self) -> u64

Offset inside current item.

pub fn len(&self) -> usize

Number of underlying items.

pub fn get_current_item_index(&self) -> usize

index of an item that is currently read.

pub fn get_total_size(&mut self) -> Result<u64>

Get total size of underlying items.

Computes total size of underlying items. This method requires mut ref and returns io::Result because we need to seek inside last item to determine its size at the moment of call.

pub fn seek_current_item(&mut self, pos: SeekFrom) -> Result<u64>

Seek current underlying reader properly updating any internal state.

Returns current local offset after seek.

pub fn seek_to_item_start(&mut self, item_index: usize) -> Result<u64>

Perform seek to 0 offset in item identified by item_index.

pub fn seek_by_local_index( &mut self, item_index: usize, pos: SeekFrom, ) -> Result<u64>

Seek globally by providing local pos inside item at index item_index.

Provided pos must be inside indexed item. Returns current local offset.

pub fn get_current_item_size(&self) -> Option<u64>

Returns item size of item. If it is last, returns None instead.

To determine size of last item, use get_last_item_size.

pub fn get_bytes_before_current_item(&self) -> u64

Computes global offset from which current item starts.

pub fn get_last_item_size(&mut self) -> Result<u64>

Computes last item size.

Last file in this reader may still be written into, so this number may soon become invalid.

Trait Implementations

impl Deref for TrackedReader

type Target = InodeAwareReader

The resulting type after dereferencing.

fn deref(&self) -> &Self::Target

Dereferences the value.

impl DerefMut for TrackedReader

fn deref_mut(&mut self) -> &mut Self::Target

Mutably dereferences the value.

impl Drop for TrackedReader

Executes destructor. If .close() was not called previously, will write state to disk, possibly panicking if any error happens. If panic is not what you want, use .close() and handle errors manually instead.

fn drop(&mut self)

Executes the destructor for this type.