gosub-sonar

Browser-agnostic priority-scheduled HTTP/HTTPS fetching library.

Overview

gosub-sonar provides two fetching APIs:

simple_get — one-shot GET for tools and scripts that just need bytes.
Fetcher — full priority scheduler with request coalescing, per-origin concurrency limits, and fan-out to multiple subscribers.

The library has no dependency on any browser engine and can be used standalone.

Usage

Add to your Cargo.toml (the scheduler API also uses these companion crates directly):

[dependencies]
gosub-sonar = "0.2"
http = "1"
tokio = { version = "1", features = ["rt", "macros"] }
tokio-util = "0.7"
url = "2"

One-shot GET

use gosub_sonar::net::simple::simple_get;
use url::Url;

let bytes = simple_get(&Url::parse("https://example.org")?).await?;

Priority scheduler

use std::sync::Arc;
use gosub_sonar::{FetchRequest, FetchResult, Fetcher, FetcherConfig, NullContext, Priority};
use http::Method;
use tokio_util::sync::CancellationToken;
use url::Url;

// NullContext ignores all lifecycle events; implement FetcherContext to receive them.
let fetcher = Arc::new(Fetcher::new(FetcherConfig::default(), Arc::new(NullContext))?);

let shutdown = CancellationToken::new();
let f = fetcher.clone();
let cancel = shutdown.clone();
tokio::spawn(async move { f.run(cancel).await });

let req = FetchRequest::builder(Method::GET, Url::parse("https://example.org")?)
    .with_priority(Priority::Normal)
    .with_auto_decode(true)
    .build();

match fetcher.fetch(req).await {
    FetchResult::Buffered { meta, body } => println!("{} — {} bytes", meta.status, body.len()),
    FetchResult::Stream { .. } => println!("streaming"),
    FetchResult::Error(e) => eprintln!("error: {e}"),
}

shutdown.cancel();

For per-subscriber cancellation use fetcher.fetch_with_cancel(req, token); for full control over the reply channel use fetcher.submit(req, cancel, reply_tx).

See the examples/ directory for runnable versions.

HSTS

HTTP Strict Transport Security (RFC 6797) is on by default: a site that sends Strict-Transport-Security over HTTPS is recorded, and later http:// requests to it are rewritten to https:// before any connection is opened. The default store is in-memory, so policies last for the life of the process and need no setup.

To persist across restarts, implement HstsStore — a host-keyed map. The crate handles max-age, includeSubDomains matching, and expiry, so the store interprets nothing:

use gosub_sonar::{HstsEntry, HstsStore, FetcherConfig};

struct ProfileStore { /* in-memory map + async write-through to disk */ }

impl HstsStore for ProfileStore {
    fn load(&self, host: &str) -> Option<HstsEntry> { /* ... */ }
    fn store(&self, host: &str, entry: HstsEntry) { /* ... */ }
    fn remove(&self, host: &str) { /* ... */ }
}

let cfg = FetcherConfig {
    hsts: Some(Arc::new(ProfileStore::open(&profile_dir)?)),
    ..Default::default()
};

load runs on every hop of every request, so it must not block — keep an in-memory map and persist in the background.

Set hsts: None to disable HSTS: nothing consulted, nothing recorded. This is what a private-browsing session wants.

There is no preload list. On wasm32 the browser's fetch() applies its own HSTS, so the field does not exist there.

Proxies

By default the fetcher reads HTTP_PROXY, HTTPS_PROXY, ALL_PROXY, and NO_PROXY from the environment. FetcherConfig::proxy replaces that with your own settings:

use gosub_sonar::{FetcherConfig, ProxyConfig, ProxyRule};

let cfg = FetcherConfig {
    proxy: ProxyConfig::Rules(vec![
        ProxyRule::all("http://proxy.corp:8080")
            .with_basic_auth("alice", "hunter2")
            .bypassing("localhost, 10.0.0.0/8, .internal.corp"),
    ]),
    ..Default::default()
};

Rules are matched in order; each carries a scope (http, https, or all), optional credentials, and an optional NO_PROXY-syntax bypass list. Anything other than ProxyConfig::System ignores the environment entirely, so ProxyConfig::Disabled guarantees a direct connection. socks4, socks5, and socks5h proxy URLs need the socks feature. On wasm32 the browser's fetch() uses the user's own proxy settings, so the field does not exist there.

Examples

cargo run --example simple_fetch -- https://example.org
cargo run --example fetcher -- https://example.org
cargo run --example fetcher_harness --features test-support

Documentation

API documentation is available on docs.rs. Design notes live in the docs/ directory:

architecture.md — overall structure of the fetch stack
net-design.md — scheduler design (coalescing, priorities, fan-out)
pump.md — how streamed bodies are pumped to subscribers

License

MIT — see LICENSE.

gosub-sonar 0.2.0