OneIO
OneIO is a Rust library that provides a unified simple IO interface for reading and writing to and from data files from different sources and compressions.
Usage and Feature Flags
Enable all compression algorithms and handle remote files (default)
= "0.16"
Select from supported feature flags
= { = "0.16", = false, = ["remote", "gz"] }
Default flags include lib-core
and rustls
.
Core features: lib-core
lib-core
core features include:
remote
: allow reading from remote files, including http(s) and ftpcompressions
: support all compression algorithmsgz
: supportgzip
files usingflate2
cratebz
: supportbzip2
files usingbzip2
cratelz
: supportlz4
files usinglz4
cratexz
: supportxz
files usingxz2
crate (requires xz library installed)
json
: allow reading JSON content into structs withserde
andserde_json
TLS choice: rustls
or native-tls
Users can choose between rustls
or native-tls
as their TLS library. We use rustls
as the basic library.
Optional features: cli
, s3
, digest
s3
: allow reading from AWS S3 compatible bucketscli
: build commandline programoneio
, uses the following featureslib-core
,rustls
,s3
for core functionalitiesclap
,tracing
for CLI basics
digest
for generating SHA256 digest string
Selecting some compression algorithms
Users can also manually opt-in to specific compression algorithms. For example, to work with only local gzip
and bzip2
files:
= { = "0.16", = false, = ["gz", "bz"] }
Use oneio
commandline tool
OneIO comes with a commandline tool, oneio
, that opens and reads local/remote files
to terminal and handles decompression automatically. This can be useful if you want to
read some compressed plain-text files from a local or remote source.
oneio attempts to read files from local or remote locations with any compression
Usage: oneio [OPTIONS] [FILE] [COMMAND]
Commands:
s3 S3-related subcommands
help Print this message or the help of the given subcommand(s)
Arguments:
[FILE] file to open, remote or local
Options:
-d, --download download the file to current directory, similar to run `wget`
-o, --outfile <OUTFILE> output file path
--cache-dir <CACHE_DIR> cache reading to specified directory
--cache-force force re-caching if local cache already exists
--cache-file <CACHE_FILE> specify cache file name
-s, --stats read through file and only print out stats
-h, --help Print help
-V, --version Print version
You can specify a data file location after oneio
. The following command
prints out the raw HTML file from https://bgpkit.com.
Here is another example of using oneio
to read a remote compressed JSON file,
pipe it to jq
and count the number of JSON objects in the array.
|
You can also directly download a file with the --download
(or -d
) flag.
|
|||||||||||||
|||||||||||||
|||||||||||||
|||||||||||||
|||||||||||||
Use OneIO Reader as Library
The returned reader implements BufRead, and handles decompression from the following types:
gzip
: files ending withgz
orgzip
bzip2
: files ending withbz
orbz2
lz4
: files ending withlz4
orlz
xz
: files ending withxz
orxz2
It also handles reading from remote or local files transparently.
Examples
Read all into string:
const TEST_TEXT: &str = "OneIO test file.
This is a test.";
Read into lines:
use BufRead;
const TEST_TEXT: &str = "OneIO test file.
This is a test.";
Use OneIO Writer as a Library
[get_writer] returns a generic writer that implements [Write], and handles decompression from the following types:
gzip
: files ending withgz
orgzip
bzip2
: files ending withbz
orbz2
Note: lz4 writer is not currently supported.
Example
Common IO operations
Read remote content with custom headers
use HashMap;
Download remote file to local directory
S3-related operations (needs s3
feature flag)