Expand description
OneIO creates readers and writers for files that just works, local or remote, compressed or raw.
§Usage and Feature Flags
Enable all compression algorithms and handle remote files (default)
oneio = "0.16"
Select from supported feature flags
oneio = {version = "0.16", features = ["remote", "gz"]}
Supported feature flags:
all
(default): all flags (["gz", "bz", "lz", "remote", "json"]
)remote
: allow reading from remote files, including http(s) and ftpremote-rustls
: userustls
instead ofnative-tls
for remote files via httpsrustls
: userustls
instead ofnative-tls
for remote files via https or S3, if either are enabledgz
: supportgzip
filesbz
: supportbzip2
fileslz
: supportlz4
filesjson
: allow reading JSON content into structs directly
§Reader
The returned reader implements BufRead, and handles decompression from the following types:
gzip
: files ending withgz
orgzip
bzip2
: files ending withbz
orbz2
lz4
: files ending withlz4
orlz
It also handles reading from remote or local files transparently.
§Examples
Read all into string:
const TEST_TEXT: &str = "OneIO test file.
This is a test.";
let mut reader = oneio::get_reader("https://spaces.bgpkit.org/oneio/test_data.txt.gz").unwrap();
let mut text = "".to_string();
reader.read_to_string(&mut text).unwrap();
assert_eq!(text.as_str(), TEST_TEXT);
Read into lines:
use std::io::BufRead;
use std::io::BufReader;
const TEST_TEXT: &str = "OneIO test file.
This is a test.";
let lines = oneio::read_lines("https://spaces.bgpkit.org/oneio/test_data.txt.gz").unwrap()
.map(|line| line.unwrap()).collect::<Vec<String>>();
assert_eq!(lines.len(), 2);
assert_eq!(lines[0].as_str(), "OneIO test file.");
assert_eq!(lines[1].as_str(), "This is a test.");
Cache data first, then read
use std::io::BufRead;
use std::io::BufReader;
const TEST_TEXT: &str = "OneIO test file.
This is a test.";
let reader = BufReader::new(oneio::get_cache_reader("https://spaces.bgpkit.org/oneio/test_data.txt.gz", "/tmp/oneio/cache/", None, false).unwrap());
let lines = reader.lines().into_iter().map(|line| line.unwrap()).collect::<Vec<String>>();
assert_eq!(lines.len(), 2);
assert_eq!(lines[0].as_str(), "OneIO test file.");
assert_eq!(lines[1].as_str(), "This is a test.");
let mut reader = oneio::get_reader("/tmp/oneio/cache/test_data.txt.gz").unwrap();
let mut text = "".to_string();
reader.read_to_string(&mut text).unwrap();
assert_eq!(text.as_str(), TEST_TEXT);
Read remote content with custom headers
use std::collections::HashMap;
let mut reader = oneio::get_remote_reader(
"https://SOME_REMOTE_RESOURCE_PROTECTED_BY_ACCESS_TOKEN",
HashMap::from([("X-Custom-Auth-Key".to_string(), "TOKEN".to_string())])
).unwrap();
let mut text = "".to_string();
reader.read_to_string(&mut text).unwrap();
println!("{}", text);
§Writer
get_writer returns a generic writer that implements std::io::Write, and handles decompression from the following types:
gzip
: files ending withgz
orgzip
bzip2
: files ending withbz
orbz2
Note: lz4 writer is not currently supported.
§Example
let to_read_file = "https://spaces.bgpkit.org/oneio/test_data.txt.gz";
let to_write_file = "/tmp/test_write.txt.bz2";
// read text from remote gzip file
let mut text = "".to_string();
oneio::get_reader(to_read_file).unwrap().read_to_string(&mut text).unwrap();
// write the same text to a local bz2 file
let mut writer = oneio::get_writer(to_write_file).unwrap();
writer.write_all(text.as_ref()).unwrap();
drop(writer);
// read from the newly-generated bz2 file
let mut new_text = "".to_string();
oneio::get_reader(to_write_file).unwrap().read_to_string(&mut new_text).unwrap();
// compare the decompressed content of the remote and local files
assert_eq!(text.as_str(), new_text.as_str());
std::fs::remove_file(to_write_file).unwrap();
Modules§
Enums§
Functions§
- Downloads a file from a remote location to a local path.
- Downloads a file from a remote path and saves it locally with retry mechanism.
- Check if a file or directory exists.
- get file reader with local cache.
- Gets a reader for the given file path.
- Get a reader for remote content with the capability to specify headers.
- Returns a writer for the given file path with the corresponding compression.
- Reads a JSON file and deserializes it into the specified struct.
- Reads lines from a file specified by the given path.
- Reads the contents of a file to a string.