Please check the build logs for more information.
See Builds for ideas on how to fix a failed build, or Metadata for how to configure docs.rs builds.
If you believe this is docs.rs' fault, open an issue.
conditional-s3-fetch
File container struct that fetches and parses S3 files, with conditional fetching.
Often you'll need to implement a background file fetcher on a longer-live process. To avoid unnecessary fetches, you can use the conditional-s3-fetch
crate to fetch files from S3 only when they have been modified.
It will use the If-None-Match
header with the provided AWS S3 ETag
response header, to avoid fetching and parsing the file if it hasn't been modified.
This crate provides a File
struct which adds metadata to the parsed content, so it can be reused in future fetch calls.
Installation
Add the following to your Cargo.toml
file:
[]
= "0.1.0"
Provided file parses are:
- [
String
] - [
Vec<u8>
] - [
bytes::Bytes
]
Additional schemaless file format parses provided on this crate:
simd-json
(default) orjson
: Provides theJson
parser to help read files into structure.cbor
(default): Provides theCbor
parser to help read files into structure.
You can customize which built-in additional parser is provided by disabling the default features and enabling the desired one.
[]
= { = "0.1.0", = false, = ["json"] }
Example
You can start with a [File::unloaded
] instance which doesn't have any data, and then fetch it using the fetch
method later, such as a background process loop.
use File;;
let mut file = unloaded;
for x in 1..10
Adding shared mutable-access, such as Arc
's are left as an exercise to each project to better fit their needs.
Implementing a custom parser
You can implement your own parser by implementing the [Parse
] trait with your custom parser logic.
It then can be called with a File::<MyParser>
turbofish syntax.
use ;
;
let file = unloaded;
Local development
There is an example binary that can be used to test the crate locally, using a minio
container locally.