# IO module
This document describes the overall design of this module.
## Rules:
- Each directory in this module corresponds to a specific format such as `csv` and `json`.
- directories that depend on external dependencies MUST be feature gated, with a feature named with
a prefix `io_`.
- modules MUST re-export any API of external dependencies they require as part of their public API.
E.g.
- if a module as an API `write(writer: &mut csv:Writer<W>, ...)`, it MUST contain
`pub use csv::Writer;`.
The rational is that adding this crate to `cargo.toml` must be sufficient to use it.
- Each directory SHOULD contain two directories, `read` and `write`, corresponding to functionality
about reading from the format and writing to the format respectively.
- The base module SHOULD contain `use pub read;` and `use pub write;`.
- Implementations SHOULD separate reading of "data" from reading of "metadata". Examples:
- schema read or inference SHOULD be a separate function
- functions that read "data" SHOULD consume a schema typically pre-read.
- Implementations SHOULD separate IO-bounded operations from CPU-bounded operations. I.e.
implementations SHOULD:
- contain functions that consume a `Read` implementor and output a "raw" struct, i.e. a struct
that is e.g. compressed and serialized
- contain functions that consume a "raw" struct and convert it into Arrow.
- offer each of these functions as independent public APIs, so that consumers can decide how to
balance CPU-bounds and IO-bounds.