Crate serde_pipe

Turn serde+bincode into a pipe: push Ts and pull u8s, or vice versa.

Crates.io │ Repo

This library gives you a Serializer pipe, into which you can push Ts and pull u8s; and a Deserializer pipe, into which you can push u8s and pull Ts.

Both are bounded in their memory usage, i.e. they do not simply allocate a vector for serde+bincode to serialize into. Instead, libfringe is leveraged to turn serde+bincode into a Generator from which u8s can be pulled from/pushed to on demand.

This is useful for example if you have 10GiB memory available, and want to serialize+send or receive+deserialize an 8GiB vector. Note, this is perfectly possible with serde+bincode normally, but you'd need to dedicate a thread that blocks until completion to the task.

Example

use serde_pipe::Serializer;

let large_vector = (0..1u64<<30).collect::<Vec<_>>();
let mut serializer = Serializer::new();
serializer.push().unwrap()(large_vector);

while let Some(pull) = serializer.pull() {
	let byte = pull();
	println!("byte! {}", byte);
}

Note

This crate currently depends on libfringe, and as such inherits these limitations:

• Rust nightly is required for the asm and naked_functions features;
• The architectures currently supported are: x86, x86_64, aarch64, or1k;
• The platforms currently supported are: bare metal, Linux (any libc), FreeBSD, DragonFly BSD, macOS. Windows is not supported.

Structs

Deserializer	Deserializer pipe: push u8; pull T.
Serializer	Serializer pipe: push T; pull u8.