## httpio – async HTTP/1.1 over any transport
**httpio** is a low-level, async HTTP/1.1 client library that is **agnostic to the underlying transport**.
You bring any `AsyncBufRead` / `AsyncWrite` streams (plain TCP, TLS-over-TCP, in‑memory pipes, custom tunnels) and `httpio` handles the HTTP protocol: request building, parsing, transfer encodings, compression and multipart bodies.
> Note: replace `httpio` with the actual crate name you publish to crates.io.
---
### Why this crate?
- **Transport-agnostic by design**: works with bare TCP, Rustls, custom TLS stacks, proxies, SSH tunnels, or test in‑memory streams – anything that implements `AsyncBufRead + Unpin` and `AsyncWrite + Unpin`.
- **Runtime-agnostic**: does not depend on any specific async runtime. You can use `async-std`, `tokio`, `async-foundation`, or your own executor.
- **Focused HTTP wire layer**: no cookie jars, redirect logic, or JSON helpers; this is a precise HTTP/1.1 core meant for building higher-level clients or specialized tools.
- **Robust body handling**: supports `Content-Encoding` (gzip/deflate), `Transfer-Encoding` (chunked, gzip/deflate) and multipart bodies.
- **Connection pooling**: built-in connection pool with idle timeouts for reusing HTTP/1.1 keep-alive connections.
- **Character set aware**: response bodies can be decoded to `String` according to the charset advertised by the server (with sensible defaults).
- **Structured logging**: pluggable logger so you can trace requests and responses without sprinkling manual `println!` calls.
If you want a small, composable HTTP core that you can plug into your own networking and TLS stack, this crate is for you.
> **Examples included**: Check the `examples/` directory to see how to integrate **Rustls** for TLS and **flate2** for Gzip/Deflate compression.
---
## Features
- **HTTP/1.1 client**
- Start-line and header construction via `HttpRequestBuilder`.
- `HttpConnection` abstraction over any async reader/writer pair.
- Connection reuse with `HttpConnectionPool`.
- **Transport flexibility**
- Generic over `R: AsyncBufRead + Unpin` and `W: AsyncWrite + Unpin`.
- Works with:
- plain `TcpStream` (from any runtime),
- TLS streams (e.g. Rustls, native-tls, custom TLS 1.2 implementation),
- any other async stream pair (e.g. in-memory for tests).
- **Body & encoding support**
- `HttpBody` enum for:
- empty bodies,
- raw bytes,
- multipart bodies (`HttpMultipartBody`).
- Handles:
- `Content-Length`,
- `Transfer-Encoding: chunked`,
- `Content-Encoding: gzip` / `deflate` (pluggable backend),
- combinations of transfer and content encodings (decoded in the correct order).
- **Compression Support**
- **Compression-agnostic**: No bundled compression library; you choose the implementation.
- **Pluggable**: Implement `CompressionDecoder` to support gzip, deflate, brotli, zstd, etc.
- **Zero default dependencies**: Keeps the crate small and compile times fast.
- **Customizable**: Register your provider via `set_compression_provider`.
- **Multipart support**
- Parses multipart bodies based on boundary and subtype.
- Exposes parts via `HttpMultipartBody` and `HttpBodyPart`.
- **Headers & utilities**
- `HttpHeaderList` with helpers for:
- content length,
- transfer encodings,
- content encodings,
- multipart boundary & subtype,
- charset detection.
- Constants for common header names (e.g. `HOST`, `USER_AGENT`, `ACCEPT`, `ACCEPT_ENCODING`, `TE`, …).
---
## Quick start
Add this to your `Cargo.toml`:
```toml
[dependencies]
httpio = "0.1" # replace with the actual crate name and version
futures = "0.3"
```
### Example: simple GET over plain TCP
This example uses a generic async TCP stream. It assumes you already have a connected stream that implements `AsyncRead + AsyncWrite`.
```rust
use futures::io::BufReader;
use futures::{AsyncReadExt, AsyncWriteExt};
use httpio::enums::char_set::CharSet;
use httpio::enums::http_body::HttpBody;
use httpio::enums::http_request_method::HttpRequestMethod;
use httpio::enums::http_version::HttpVersion;
use httpio::structures::connection::http_connection::HttpConnection;
use httpio::structures::header::header_list::HttpHeaderList;
use httpio::utils::http_header_field_name;
async fn fetch_example<R, W>(reader: R, writer: W) -> Result<String, Box<dyn std::error::Error>>
where
R: futures::AsyncBufRead + Unpin,
W: futures::AsyncWrite + Unpin,
{
let host = "example.com";
// Optional: add extra request headers
let mut headers = HttpHeaderList::default();
headers.insert(http_header_field_name::ACCEPT, "text/html,application/json,*/*;q=0.8");
let mut conn = HttpConnection::new(
BufReader::new(reader),
writer,
HttpVersion::Http11,
host,
headers,
/* your logger here */,
)?;
conn.send_request(
HttpRequestMethod::Get,
"/",
HttpHeaderList::default(),
HttpBody::None,
)
.await?;
let response = conn.read_response().await?;
let charset = response.headers.get_charset(CharSet::Iso88591);
let body = response.body.to_string(charset)?;
Ok(body)
}
```
You are free to choose how sockets are created and which runtime you use; the HTTP layer only cares about the async reader/writer.
---
## Example: plugging in TLS (Rustls)
Because the crate is transport-agnostic, TLS is just a matter of wrapping your TCP stream with a TLS stream that still implements `AsyncBufRead` / `AsyncWrite`, then passing it to `HttpConnection`.
> See `examples/client_tls/` for a full working example using `rustls`.
```rust
use futures::io::BufReader;
use httpio::structures::connection::http_connection::HttpConnection;
use httpio::enums::http_version::HttpVersion;
use httpio::structures::header::header_list::HttpHeaderList;
async fn connect_over_tls() -> Result<(), Box<dyn std::error::Error>> {
let url = "https://www.rust-lang.org/".parse::<url::Url>()?;
let host = url.host_str().unwrap().to_string();
// 1) Create a TCP connection using your runtime of choice
let tcp_stream = /* your async TcpStream connect here */;
// 2) Wrap it in TLS (e.g. with Rustls)
let tls_stream = /* perform Rustls handshake, returning an async TLS stream */;
// 3) Split into reader/writer and hand it to HttpConnection
let (reader, writer) = futures::io::split(tls_stream);
let reader = BufReader::new(reader);
let mut conn = HttpConnection::new(
reader,
writer,
HttpVersion::Http11,
&host,
HttpHeaderList::default(),
/* your logger here */,
)?;
// ... send requests & read responses ...
Ok(())
}
```
The library does **not** bundle a specific TLS stack; instead it embraces the async traits so you can choose Rustls, native-tls, a custom TLS 1.2 crate, or no TLS at all.
---
## Compression
This library is **compression-agnostic**. It does not bundle any compression library by default to keep the core lightweight and dependencies minimal.
> See `examples/client/main.rs` for a complete example of integrating `flate2`.
To support `Content-Encoding: gzip`, `deflate`, `br`, or `zstd`, you must provide an implementation of the `CompressionDecoder` trait. This design allows you to:
- Choose your preferred compression crates (e.g. `flate2`, `async-compression`, `brotli`).
- Support only the algorithms you need.
- Avoid compiling unused compression logic.
### Providing a compression implementation
You can implement the `CompressionDecoder` trait and register it globally at runtime start-up.
```rust
use httpio::compression::traits::CompressionDecoder;
use httpio::compression::set_compression_provider;
use httpio::enums::content_encoding::ContentEncoding;
use futures::future::{BoxFuture, FutureExt};
use httpio::enums::http_error::HttpError;
// Example using flate2 (you need to add flate2 to your dependencies)
// use flate2::read::{GzDecoder, ZlibDecoder};
// use std::io::Read;
struct MyCompression;
impl CompressionDecoder for MyCompression {
fn gzip_decode(&self, compressed: Vec<u8>) -> BoxFuture<'static, Result<Vec<u8>, HttpError>> {
async move {
// let mut decoder = GzDecoder::new(&compressed[..]);
// let mut decompressed = Vec::new();
// decoder.read_to_end(&mut decompressed)?;
// Ok(decompressed)
Ok(compressed) // placeholder
}.boxed()
}
// other methods (zlib_decode, brotli_decode, zstd_decode) have default implementations
// that return Unimplemented error, so you only need to override what you support.
fn supported_encodings(&self) -> Vec<ContentEncoding> {
vec![ContentEncoding::Gzip] // Advertises support for gzip only
}
}
// In your main initialization code:
fn main() {
set_compression_provider(Box::new(MyCompression))
.expect("Compression provider can only be set once");
}
```
---
## Design goals & non-goals
### Goals
- Be a **small, composable building block** for HTTP/1.1 over async transports.
- Make it easy to:
- plug in different runtimes,
- plug in different TLS implementations,
- build higher-level clients with their own policies (retries, redirects, auth, JSON, etc.).
- Provide **correct handling of encodings and multipart** without hiding the underlying protocol.
### Non-goals
- Compete with full-featured clients like `reqwest` or `surf`.
- Provide HTTP/2 / HTTP/3 support (HTTP/1.1 only for now).
- Implement high-level features such as:
- cookie management,
- redirects,
- authentication flows,
- automatic JSON (de)serialization.
You are expected to build those features on top of this crate, with full control.
---
## Status
- HTTP/1.1 client: **stable** for typical use (GET/POST, streaming bodies, gzip/deflate, chunked).
- TLS: **supported via external crates** (see examples).
- API: may still evolve before `1.0` as more real-world use cases are integrated.
Feedback and issues are very welcome.
---
## License
This project is licensed under either of:
- MIT license
- Apache License, Version 2.0
at your option.