Struct s3reader::S3Reader

pub struct S3Reader { /* private fields */ }

A Reader for S3 objects that implements the Read and Seek traits

This reader allows byte-offset acces to any S3 objects

Example

use std::io::{Read, Seek};
use s3reader::S3Reader;
use s3reader::S3ObjectUri;

let uri = S3ObjectUri::new("s3://my-bucket/path/to/huge/file").unwrap();
let mut reader = S3Reader::open(uri).unwrap();

reader.seek(std::io::SeekFrom::Start(100)).unwrap();

let mut buf: Vec<u8> = [0; 1024].to_vec();
reader.read(&mut buf).expect("Error reading from S3");

Implementations

impl S3Reader

pub fn new(uri: S3ObjectUri) -> S3Reader

Creates a new S3Reader.

This method does not check for presence of an actual object in S3 or for connectivity. Use S3Reader::open instead to ensure that the S3 object actually exists.

pub fn open(uri: S3ObjectUri) -> Result<S3Reader, S3ReaderError>

Creates a new S3Reader and checks for presence of the S3 object

This method is the preferred way to create a Reader. It has a minor overhead because it fetches the object’s header from S3, but this ensures that the object is actually available and thus prevents possible runtime errors.

pub fn from_config(config: &SdkConfig, uri: S3ObjectUri) -> S3Reader

Creates a new S3Reader with a custom AWS SdkConfig

This method is useful if you don’t want to use the default configbuilder using the environment. It does not check for correctness, connectivity to the S3 bucket or presence of the S3 object.

pub async fn read_range(
    &mut self,
    from: u64,
    to: u64
) -> Result<AggregatedBytes, S3ReaderError>

Returns A Future for the bytes read from the S3 object for the specified byte-range

This method does not update the internal cursor position. To maintain an internal state, use S3Reader::seek and S3Reader::read instead.

The byte ranges from and to are both inclusive, see https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35

Example

use tokio::runtime::Runtime;

use s3reader::S3Reader;
use s3reader::S3ObjectUri;

let uri = S3ObjectUri::new("s3://my-bucket/path/to/huge/file").unwrap();
let mut reader = S3Reader::open(uri).unwrap();

// `read_range` is an async function, we must wrap it in a runtime in the doctest
let bytes = Runtime::new().unwrap().block_on(
    reader.read_range(100, 249)
).unwrap().into_bytes();
assert_eq!(bytes.len(), 150);

pub fn read_range_sync(
    &mut self,
    from: u64,
    to: u64
) -> Result<AggregatedBytes, S3ReaderError>

Returns the bytes read from the S3 object for the specified byte-range

This method does not update the internal cursor position. To maintain an internal state, use S3Reader::seek and S3Reader::read instead.

The byte ranges from and to are both inclusive, see https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.35

This method also exists as an async method: S3Reader::read_range

Example

use s3reader::S3Reader;
use s3reader::S3ObjectUri;

let uri = S3ObjectUri::new("s3://my-bucket/path/to/huge/file").unwrap();
let mut reader = S3Reader::open(uri).unwrap();

let bytes = reader.read_range_sync(100, 249).unwrap().into_bytes();
assert_eq!(bytes.len(), 150);

pub async fn fetch_header(&mut self) -> Result<(), SdkError<HeadObjectError>>

Fetches the object’s header from S3

Example

use tokio::runtime::Runtime;

use s3reader::S3Reader;
use s3reader::S3ObjectUri;

let uri = S3ObjectUri::new("s3://my-bucket/path/to/huge/file").unwrap();
let mut reader = S3Reader::open(uri).unwrap();

// `fetch_header` is an async function, we must wrap it in a runtime in the doctest
Runtime::new().unwrap().block_on(
    reader.fetch_header()
).unwrap();
assert_eq!(reader.len(), 150);

pub fn len(&mut self) -> u64

Returns the content_length of the S3 object

Panics

This method can panic if the header cannot be fetched (e.g. due to network issues, wrong URI etc). This can be prevented by using S3Reader::open which guarantees that the header is present.

pub fn pos(&self) -> u64

Trait Implementations

impl Read for S3Reader

fn read_to_end(&mut self, buf: &mut Vec<u8>) -> Result<usize, Error>

Custom implementation to avoid too many read calls. The default trait reads in 32 bytes blocks that grow over time. However, the IO for S3 has way more latency so S3Reader tries to fetch all data in a single call

fn read_to_string(&mut self, buf: &mut String) -> Result<usize, Error>

Custom implementation to avoid too many read calls. The default trait reads in 32 bytes blocks that grow over time. However, the IO for S3 has way more latency so S3Reader tries to fetch all data in a single call

fn read(&mut self, buf: &mut [u8]) -> Result<usize, Error>

Pull some bytes from this source into the specified buffer, returning how many bytes were read.

fn read_vectored(&mut self, bufs: &mut [IoSliceMut<'_>]) -> Result<usize, Error>

Like read, except that it reads into a slice of buffers.

fn is_read_vectored(&self) -> bool

🔬This is a nightly-only experimental API. (can_vector)

Determines if this Reader has an efficient read_vectored implementation.

fn read_exact(&mut self, buf: &mut [u8]) -> Result<(), Error>

Read the exact number of bytes required to fill buf.

fn read_buf(&mut self, buf: BorrowedCursor<'_>) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Pull some bytes from this source into the specified buffer.

fn read_buf_exact(&mut self, cursor: BorrowedCursor<'_>) -> Result<(), Error>

🔬This is a nightly-only experimental API. (read_buf)

Read the exact number of bytes required to fill cursor.

fn by_ref(&mut self) -> &mut Self

Creates a “by reference” adaptor for this instance of Read.

fn bytes(self) -> Bytes<Self>

Transforms this Read instance to an Iterator over its bytes.

fn chain<R>(self, next: R) -> Chain<Self, R>where
    R: Read,

Creates an adapter which will chain this stream with another.

fn take(self, limit: u64) -> Take<Self>

Creates an adapter which will read at most limit bytes from it.

impl Seek for S3Reader

fn seek(&mut self, pos: SeekFrom) -> Result<u64, Error>

Seek to an offset, in bytes, in a stream.

fn rewind(&mut self) -> Result<(), Error>

Rewind to the beginning of a stream.

fn stream_len(&mut self) -> Result<u64, Error>

🔬This is a nightly-only experimental API. (seek_stream_len)

Returns the length of this stream (in bytes).

fn stream_position(&mut self) -> Result<u64, Error>

Returns the current seek position from the start of the stream.