Module fs

This module contains functions related to filesystem operations.

file_open_read / file_open_read_with_capacity

These functions are convenience wrappers around file I/O. They allow reading of compressed files in a transparent manner.

Reading compressed files works for .bz2/.gz/.xz files. The support for for the different file formats is optional. By default .gz and .xz are enabled. The file-* features enable support for the corresponding file extensions.

The example shows how to read a file into a string:

let mut reader = file_open_read("./text.txt").unwrap();
let mut content = String::new();
reader.read_to_string(&mut content).unwrap();

file_write

Similar to file_open_read this functions is a convenience wrapper but for writing compressed files. The function always requires an argument as the filetype has to be specified. The function detects the correct filetype from the file extension. The detected choice can be overwritten using WriteBuilder::filetype.

There are two modes the file can be opened, in either the truncate or the append mode.

let mut writer = file_write("./text.txt").truncate()?;
writer.write_all("Hello World".as_bytes())?;

let mut writer = file_write("./text.txt").append()?;
writer.write_all("Hello World".as_bytes())?;

parse_jsonl_multi_threaded

Create multiple thread reading and parsing a JSONL file.

This function is especially useful if the file is compressed with a high compression (such as xz2) and the parsing overhead is non-negligible. The inter-thread communication is batched to reduce overhead.

Structs

MtJsonl

An iterator over deserialized JSON objects

WriteBuilder

Builder to control how the writeable file will be opened.

Enums

Compression

Specify the compression level used.

FileType

Specify the output filetype.

Functions

append

Append the content to the file.

file_open_read

Create reader for uncompressed or compressed files transparently.

file_open_read_with_capacity

Create reader for uncompressed or compressed files transparently.

file_write

Create writers for plaintext or compressed files.

parse_jsonl_multi_threaded

Create a multi-threaded JSONL parser.

read

Read the entire contents of a file into a bytes vector.

read_to_string

Read the entire contents of a file into a string.

write

Write a slice as the entire contents of a file.