httpio 0.2.4

A transport-agnostic, async HTTP/1.1 client library for any runtime.
Documentation

httpio – async HTTP/1.1 over any transport

httpio is a low-level, async HTTP/1.1 client and server library that is agnostic to the underlying transport.
You bring any AsyncBufRead / AsyncWrite streams (plain TCP, TLS-over-TCP, in‑memory pipes, custom tunnels) and httpio handles the HTTP protocol: request building, parsing, transfer encodings, compression and multipart bodies.

Note: replace httpio with the actual crate name you publish to crates.io.

Why this crate?

  • Transport-agnostic by design: works with bare TCP, Rustls, custom TLS stacks, proxies, SSH tunnels, or test in‑memory streams – anything that implements AsyncBufRead + Unpin and AsyncWrite + Unpin.
  • Runtime-agnostic: does not depend on any specific async runtime. You can use async-std, tokio, smol, async-foundation, or your own executor.
  • Client & Server: Provides low-level primitives for building both HTTP clients and servers.
  • Focused HTTP wire layer: no cookie jars, redirect logic, or JSON helpers; this is a precise HTTP/1.1 core meant for building higher-level clients or specialized tools.
  • Robust body handling: supports Content-Encoding (gzip/deflate), Transfer-Encoding (chunked, gzip/deflate) and multipart bodies.
  • Connection pooling: built-in connection pool with idle timeouts for reusing HTTP/1.1 keep-alive connections.
  • Character set aware: response bodies can be decoded to String according to the charset advertised by the server (with sensible defaults).
  • Structured logging: pluggable logger so you can trace requests and responses without sprinkling manual println! calls.

If you want a small, composable HTTP core that you can plug into your own networking and TLS stack, this crate is for you.

Examples included: Check the examples/ directory to see how to integrate Rustls for TLS and flate2 for Gzip/Deflate compression.

Features

  • HTTP/1.1 client

    • Start-line and header construction via HttpRequestBuilder.
    • HttpConnection abstraction over any async reader/writer pair.
    • Connection reuse with HttpConnectionPool.

  • HTTP/1.1 server

    • HttpServerConnection to read requests and send responses over any async stream.
    • Supports building simple HTTP/HTTPS servers.

  • Transport flexibility

    • Generic over R: AsyncBufRead + Unpin and W: AsyncWrite + Unpin.
    • Works with:
      • plain TcpStream (from any runtime),
      • TLS streams (e.g. Rustls, native-tls, custom TLS 1.2 implementation),
      • any other async stream pair (e.g. in-memory for tests).

  • Body & encoding support

    • HttpBody enum for:
      • empty bodies,
      • raw bytes,
      • multipart bodies (HttpMultipartBody).
    • Handles:
      • Content-Length,
      • Transfer-Encoding: chunked,
      • Content-Encoding: gzip / deflate (pluggable backend),
      • combinations of transfer and content encodings (decoded in the correct order).

  • Compression Support

    • Compression-agnostic: No bundled compression library; you choose the implementation.
    • Pluggable: Implement CompressionDecoder to support gzip, deflate, brotli, zstd, etc.
    • Zero default dependencies: Keeps the crate small and compile times fast.
    • Customizable: Register your provider via set_compression_provider.

  • Multipart support

    • Parses multipart bodies based on boundary and subtype.
    • Exposes parts via HttpMultipartBody and HttpBodyPart.

  • Headers & utilities

    • HttpHeaderList with helpers for:
      • content length,
      • transfer encodings,
      • content encodings,
      • multipart boundary & subtype,
      • charset detection.
    • Constants for common header names (e.g. HOST, USER_AGENT, ACCEPT, ACCEPT_ENCODING, TE, …).

Quick start

Add this to your Cargo.toml:

[dependencies]
httpio = "0.1" # replace with the actual crate name and version
futures = "0.3"

Example: simple GET over plain TCP

This example uses a generic async TCP stream. It assumes you already have a connected stream that implements AsyncRead + AsyncWrite.

use futures::io::BufReader;
use futures::{AsyncReadExt, AsyncWriteExt};

use httpio::enums::char_set::CharSet;
use httpio::enums::http_body::HttpBody;
use httpio::enums::http_request_method::HttpRequestMethod;
use httpio::enums::http_version::HttpVersion;
use httpio::structures::connection::http_connection::HttpConnection;
use httpio::structures::header::header_list::HttpHeaderList;
use httpio::utils::http_header_field_name;

async fn fetch_example<R, W>(reader: R, writer: W) -> Result<String, Box<dyn std::error::Error>>
where
    R: futures::AsyncBufRead + Unpin,
    W: futures::AsyncWrite + Unpin,
{
    let host = "example.com";

    // Optional: add extra request headers
    let mut headers = HttpHeaderList::default();
    headers.insert(http_header_field_name::ACCEPT, "text/html,application/json,*/*;q=0.8");

    let mut conn = HttpConnection::new(
        BufReader::new(reader),
        writer,
        HttpVersion::Http11,
        host,
        headers,
        /* your logger here */,
    )?;

    conn.send_request(
        HttpRequestMethod::Get,
        "/",
        HttpHeaderList::default(),
        HttpBody::None,
    )
    .await?;

    let response = conn.read_response().await?;
    let charset = response.headers.get_charset(CharSet::Iso88591);
    let body = response.body.to_string(charset)?;

    Ok(body)
}

You are free to choose how sockets are created and which runtime you use; the HTTP layer only cares about the async reader/writer.

Example: plugging in TLS (Rustls)

Because the crate is transport-agnostic, TLS is just a matter of wrapping your TCP stream with a TLS stream that still implements AsyncBufRead / AsyncWrite, then passing it to HttpConnection.

See examples/client_tls/ for a full working example using rustls and futures-rustls.

use futures::io::BufReader;
use httpio::structures::connection::http_connection::HttpConnection;
use httpio::enums::http_version::HttpVersion;
use httpio::structures::header::header_list::HttpHeaderList;

async fn connect_over_tls() -> Result<(), Box<dyn std::error::Error>> {
    let url = "https://www.rust-lang.org/".parse::<url::Url>()?;
    let host = url.host_str().unwrap().to_string();

    // 1) Create a TCP connection using your runtime of choice
    let tcp_stream = /* your async TcpStream connect here */;

    // 2) Wrap it in TLS (e.g. with Rustls)
    let tls_stream = /* perform Rustls handshake, returning an async TLS stream */;

    // 3) Split into reader/writer and hand it to HttpConnection
    // Note: Some TLS wrappers like futures-rustls take ownership, so split after handshake.
    let (reader, writer) = futures::io::split(tls_stream); 
    let reader = BufReader::new(reader);

    let mut conn = HttpConnection::new(
        reader,
        writer,
        HttpVersion::Http11,
        &host,
        HttpHeaderList::default(),
        /* your logger here */,
    )?;

    // ... send requests & read responses ...

    Ok(())
}

The library does not bundle a specific TLS stack; instead it embraces the async traits so you can choose Rustls, native-tls, a custom TLS 1.2 crate, or no TLS at all.

Server Support

The library now includes basic server-side functionality via HttpServerConnection. You can use it to build transport-agnostic HTTP servers.

Example: Simple Server

See examples/server/ for a complete example supporting both HTTP and HTTPS using smol and rustls.

use httpio::structures::connection::http_server_connection::HttpServerConnection;
use httpio::structures::http_response::HttpResponse;
use httpio::enums::http_status_code::HttpStatusCode;
use httpio::enums::http_body::HttpBody;

async fn handle_connection<R, W>(reader: R, writer: W) 
where R: AsyncBufRead + Unpin, W: AsyncWrite + Unpin 
{
    let mut connection = HttpServerConnection::new(reader, writer);

    if let Ok(request) = connection.read_request().await {
        let body = HttpBody::Content("Hello World".into());
        let response = HttpResponse::new(HttpStatusCode::Ok, body);
        connection.send_response(response).await.ok();
    }
}

Compression

This library is compression-agnostic. It does not bundle any compression library by default to keep the core lightweight and dependencies minimal.

See examples/client/main.rs for a complete example of integrating flate2.

To support Content-Encoding: gzip, deflate, br, or zstd, you must provide an implementation of the CompressionDecoder trait. This design allows you to:

  • Choose your preferred compression crates (e.g. flate2, async-compression, brotli).
  • Support only the algorithms you need.
  • Avoid compiling unused compression logic.

Providing a compression implementation

You can implement the CompressionDecoder trait and register it globally at runtime start-up.

use httpio::compression::traits::CompressionDecoder;
use httpio::compression::set_compression_provider;
use httpio::enums::content_encoding::ContentEncoding;
use futures::future::{BoxFuture, FutureExt};
use httpio::enums::http_error::HttpError;

// Example using flate2 (you need to add flate2 to your dependencies)
// use flate2::read::{GzDecoder, ZlibDecoder};
// use std::io::Read;

struct MyCompression;

impl CompressionDecoder for MyCompression {
    fn gzip_decode(&self, compressed: Vec<u8>) -> BoxFuture<'static, Result<Vec<u8>, HttpError>> {
        async move {
            // let mut decoder = GzDecoder::new(&compressed[..]);
            // let mut decompressed = Vec::new();
            // decoder.read_to_end(&mut decompressed)?;
            // Ok(decompressed)
            Ok(compressed) // placeholder
        }.boxed()
    }
    
    // other methods (zlib_decode, brotli_decode, zstd_decode) have default implementations 
    // that return Unimplemented error, so you only need to override what you support.
    
    fn supported_encodings(&self) -> Vec<ContentEncoding> {
        vec![ContentEncoding::Gzip] // Advertises support for gzip only
    }
}

// In your main initialization code:
fn main() {
    set_compression_provider(Box::new(MyCompression))
        .expect("Compression provider can only be set once");
}

Design goals & non-goals

Goals

  • Be a small, composable building block for HTTP/1.1 over async transports.
  • Make it easy to:
    • plug in different runtimes,
    • plug in different TLS implementations,
    • build higher-level clients with their own policies (retries, redirects, auth, JSON, etc.).
  • Provide correct handling of encodings and multipart without hiding the underlying protocol.

Non-goals

  • Compete with full-featured clients like reqwest or surf.
  • Provide HTTP/2 / HTTP/3 support (HTTP/1.1 only for now).
  • Implement high-level features such as:
    • cookie management,
    • redirects,
    • authentication flows,
    • automatic JSON (de)serialization.

You are expected to build those features on top of this crate, with full control.

Status

  • HTTP/1.1 client: stable for typical use (GET/POST, streaming bodies, gzip/deflate, chunked).
  • HTTP/1.1 server: basic support available.
  • TLS: supported via external crates (see examples).
  • API: may still evolve before 1.0 as more real-world use cases are integrated.

Feedback and issues are very welcome.

License

This project is licensed under either of:

  • MIT license
  • Apache License, Version 2.0

at your option.