Module misc_utils::fs

Expand description

This module contains functions related to filesystem operations.

`file_open_read` / `file_open_read_with_capacity`

These functions are convenience wrappers around file I/O. They allow reading of compressed files in a transparent manner.

Reading compressed files works for .bz2/.gz/.xz files. The support for for the different file formats is optional. By default .gz and .xz are enabled. The file-* features enable support for the corresponding file extensions.

The example shows how to read a file into a string:

let mut reader = file_open_read("./text.txt").unwrap();
let mut content = String::new();
reader.read_to_string(&mut content).unwrap();

`file_write`

Similar to file_open_read this functions is a convenience wrapper but for writing compressed files. The function always requires an argument as the filetype has to be specified. The function detects the correct filetype from the file extension. The detected choice can be overwritten using WriteBuilder::filetype.

There are two modes the file can be opened, in either the truncate or the append mode.

let mut writer = file_write("./text.txt").truncate()?;
writer.write_all("Hello World".as_bytes())?;

let mut writer = file_write("./text.txt").append()?;
writer.write_all("Hello World".as_bytes())?;

`parse_jsonl_multi_threaded`

Create multiple thread reading and parsing a JSONL file.

This function is especially useful if the file is compressed with a high compression (such as xz2) and the parsing overhead is non-negligible. The inter-thread communication is batched to reduce overhead.