docs.rs failed to build tfrecord-0.2.0
Please check the
build logs for more information.
See
Builds for ideas on how to fix a failed build,
or
Metadata for how to configure docs.rs builds.
If you believe this is docs.rs' fault,
open an issue.
tfrecord-rust
The crate provides the functionality to serialize and deserialize TFRecord data format from TensorFlow.
Features
- Provide both high level
Example
type as well as low level Vec<u8>
bytes {,de}serialization.
- Support async/await syntax. It's easy to work with futures-rs.
- Interoperability with serde, image, ndarray and tch.
- TensorBoard support.
Usage
Use this crate in your project
Append this line to your Cargo.toml
.
tfrecord = "0.2.0"
Due to a bug in async-std 1.6.0, it cannot read file properly sometimes. Please add the patch at the end of Cargo.toml
unless the issue is fixed in future release.
[patch.crates-io]
async-std = { git = "https://github.com/async-rs/async-std", branch = "master" }
Notice on TensorFlow updates
The crate compiles the pre-generated ProtocolBuffer code from TensorFlow. In case of TensorFlow updates or custom patches, please run the code generation manually, see Generate ProtocolBuffer code from TensorFlow section for details.
Available Cargo features
Module features
full
: Enable all features.
async_
: Enable async/await feature.
dataset
: Enable the dataset API that can load records from multiple TFRecord files.
summary
: Enable the summary and event types and writters, mainly for TensorBoard.
Third-party crate support features
serde
: Enable support with serde crate.
image
: Enable support with image crate.
ndarray
: Enable support with ndarray crate.
tch
: Enable support with tch crate.
Documentation
See docs.rs for the API.
Example
File reading example
This is a snipplet copied from examples/tfrecord_info.rs.
use tfrecord::{Error, ExampleReader, Feature, RecordReaderInit};
fn main() -> Result<(), Error> {
let reader: ExampleReader<_> = RecordReaderInit::default().open(&*INPUT_TFRECORD_PATH)?;
println!("example_no\tfeature_no\tname\ttype\tsize");
for (example_index, result) in reader.enumerate() {
let example = result?;
for (feature_index, (name, feature)) in example.into_iter().enumerate() {
print!("{}\t{}\t{}\t", example_index, feature_index, name);
match feature {
Feature::BytesList(list) => {
println!("bytes\t{}", list.len());
}
Feature::FloatList(list) => {
println!("float\t{}", list.len());
}
Feature::Int64List(list) => {
println!("int64\t{}", list.len());
}
Feature::None => {
println!("none");
}
}
}
}
Ok(())
}
Work with async/await syntax
The snipplet from examples/tfrecord_info_async.rs demonstrates the integration with async-std.
use futures::stream::TryStreamExt;
use std::{fs::File, io::BufWriter, path::PathBuf};
use tfrecord::{Error, Feature, RecordStreamInit};
pub async fn _main() -> Result<(), Error> {
let stream = RecordStreamInit::default()
.examples_open(&*INPUT_TFRECORD_PATH)
.await?;
println!("example_no\tfeature_no\tname\ttype\tsize");
stream
.try_fold(0, |example_index, example| {
async move {
for (feature_index, (name, feature)) in example.into_iter().enumerate() {
print!("{}\t{}\t{}\t", example_index, feature_index, name);
match feature {
Feature::BytesList(list) => {
println!("bytes\t{}", list.len());
}
Feature::FloatList(list) => {
println!("float\t{}", list.len());
}
Feature::Int64List(list) => {
println!("int64\t{}", list.len());
}
Feature::None => {
println!("none");
}
}
}
Ok(example_index + 1)
}
})
.await?;
Ok(())
}
Work with TensorBoard
This is a simplified example of examples/tensorboard.rs that sends summary data to log_dir
directory. After running the example, launch tensorboard --logdir log_dir
to watch the outcome in TensorBoard.
use super::*;
use rand::seq::SliceRandom;
use rand_distr::{Distribution, Normal};
use std::{f32::consts::PI, thread, time::Duration};
use tfrecord::{EventInit, EventWriterInit};
pub fn _main() -> Fallible<()> {
let prefix = "log_dir/my_prefix";
println!("downloading images...");
let images = IMAGE_URLS
.iter()
.cloned()
.map(|url| {
let bytes = reqwest::blocking::get(url)?.bytes()?;
let image = image::load_from_memory(bytes.as_ref())?;
Ok(image)
})
.collect::<Fallible<Vec<_>>>()?;
let mut writer = EventWriterInit::from_prefix(prefix, None)?;
let mut rng = rand::thread_rng();
for step in 0..30 {
println!("step: {}", step);
{
let value: f32 = (step as f32 * PI / 8.0).sin();
writer.write_scalar("scalar", EventInit::with_step(step), value)?;
}
{
let normal = Normal::new(-20.0, 50.0).unwrap();
let values = normal
.sample_iter(&mut rng)
.take(1024)
.collect::<Vec<f32>>();
writer.write_histogram("histogram", EventInit::with_step(step), values)?;
}
{
let image = images.choose(&mut rng).unwrap();
writer.write_image("image", EventInit::with_step(step), image)?;
}
thread::sleep(Duration::from_millis(100));
}
Ok(())
}
More examples
You can visit the examples and tests directories to see more verbose examples.
Generate ProtocolBuffer code from TensorFlow
The crate relies on ProtocolBuffer documents from TensorFlow. The crate ships pre-generated code from ProtocolBuffer documents by default. Most users don't need to bother with the code generation. The step is needed only in case of TensorFlow updates or your custom patch.
The build script accepts several ways to access the TensorFlow source code, controlled by the TFRECORD_BUILD_METHOD
environment variable. The generated code will be placed under prebuild_src
directory. See the examples below to understand the usage.
- Build from a source tarball
export TFRECORD_BUILD_METHOD="src_file:///home/myname/tensorflow-2.2.0.tar.gz"
cargo build --release --features serde,generate_protobuf_src cargo build --release --features generate_protobuf_src
- Build from a source directory
export TFRECORD_BUILD_METHOD="src_dir:///home/myname/tensorflow-2.2.0"
cargo build --release --features serde,generate_protobuf_src cargo build --release --features generate_protobuf_src
export TFRECORD_BUILD_METHOD="url://https://github.com/tensorflow/tensorflow/archive/v2.2.0.tar.gz"
cargo build --release --features serde,generate_protobuf_src cargo build --release --features generate_protobuf_src
- Build from installed TensorFlow on system. The build script will search
${install_prefix}/include/tensorflow
directory for protobuf documents.
export TFRECORD_BUILD_METHOD="install_prefix:///usr"
cargo build --release --features serde,generate_protobuf_src cargo build --release --features generate_protobuf_src
License
MIT license. See LICENSE file for full license.