S3Reader
A Rust
library to read from S3 object as if they were files on a local filesystem (almost). The S3Reader
adds both Read
and Seek
traits, allowing to place the cursor anywhere within the S3 object and read from any byte offset. This allows random access to bytes within S3 objects.
Usage
Add this to your Cargo.toml
:
[dependencies]
s3reader = "1.0.0"
Use BufRead
to read line by line
use ;
use S3Reader;
use S3ObjectUri;
Use Seek
to jump to positions
use ;
use S3Reader;
use S3ObjectUri;
Q/A
Does this library really provide random access to S3 objects?
According to this StackOverflow answer, yes.
Are the reads sync or async?
The S3-SDK uses mostly async operations, but the Read
and Seek
traits require sync methods. Due to this, I'm using a blocking tokio runtime to wrap the async calls. This might not be the best solution, but works well for me. Any improvement suggestions are very welcome
Why is this useful?
Depends on your use-cases. If you need to access random bytes in the middle of large files/S3 object, this library is useful. For example, you can read it to stream mp4 files. It's also quite useful for some bioinformatic applications, where you might have a huge, several GB reference genome, but only need to access data of a few genes, accounting to only a few MB.