Struct blobd_client_rs::BlobdClient

source ·

pub struct BlobdClient { /* private fields */ }

Implementations§

source §

impl BlobdClient

source

pub fn new(url_prefix: String, token_secret: [u8; 32]) -> BlobdClient

source

pub fn generate_presigned_url( &self, key: &str, action: AuthTokenAction, expires_in_seconds: u64 ) -> String

source

pub async fn batch_create_objects<DS, Objects>( &self, objects: Objects, transfer_byte_counter: Option<UnboundedSender<usize>> ) -> Result<BatchCreatedObjects>where DS: 'static + Stream<Item = Result<Bytes, Box<dyn Error + Send + Sync>>> + Send + Sync, Objects: 'static + Stream<Item = BatchCreateObjectEntry<DS>> + Send + Sync,

Parts to this method:

The list of objects. It’s an infallible stream to allow for scenarios where the list is not known ahead of time or is buffered asynchronously.
Each object’s data. It’s a fallable stream of chunks of bytes, using the bytes::Bytes type as that’s what reqwest requires. Approaches to the list of objects:
If it’s all in memory, use futures::stream::iter(my_list_of_objects).
If it can be derived from an existing stream, use the StreamExt methods to transform items into BatchCreateObjectEntry.
If it is dynamically built from another thread/async function, use futures::channel::mpsc::unbounded; the receiver already implements Stream and can be provided as the list. Approaches to object data stream:
If it’s all in memory, use futures::stream::once(Ok(bytes::Bytes::from(my_object_data))).
If it’s a synchronous Read, wrap in a Stream that reads chunks into a Bytes, preferably on a different thread (e.g. spawn_blocking).
If it’s an AsyncRead, read in chunks and wrap each chunk with Bytes::from. Approaches to error handling:
The blobd server will never return a bad status, but will bail as soon as an error occurs.
However, reqwest will bail if the request body fails, which can occur if a data_stream of a BatchCreateObjectEntry yields an Err chunk.
If reqwest bails, the response will be unavailable and the amount of successfully committed objects will not be returned.
Therefore, there are some optional approaches worth considering:
- Filter out BatchCreateObjectEntry items that have a data_stream that will definitely fail, as once an entry starts being transferred, there’s no way to skip over it midway.
- When a data_stream chunk is about to yield Err, return an Ok instead with empty data and stop the object list stream (as the current data transferred is midway in an object and there’s no way to skip over it). The server will notice that the object was not completely uploaded, decide to bail, and return the successful count. Provide an optional async channel sender for transfer_byte_counter to get data stream chunk sizes as they’re about to be uploaded, which can be useful for calculating transfer progress or rate.