# rsurl → curl feature-parity roadmap
The goal: a pure-Rust drop-in for the vast majority of real-world `curl`
invocations, keeping rsurl's defining constraints intact.
**Invariants (never traded away for parity):**
- **No C toolchain / no `*-sys` crates.** Pure-Rust deps only (purecrypto,
puressh, compcol, psl2, idna). This rules out a few curl features as-is
(OpenSSL `--engine`, GSSAPI/Kerberos, c-ares) — see *Out of scope / caveats*.
- **curl-compatible CLI and exit codes.** Flags, semantics, and exit codes
match curl unless explicitly noted.
- **Security defaults stay strict.** Every new feature ships verified (full CI
gate: build, `clippy -D warnings`, fmt, doc, both TLS backends, tests).
## Progress log
Delivered on `feature/pluggable-network` (all CI-gate-clean):
- **Pluggable network**: `Connector`/`UdpProxy` traits, `Client`/`Session`,
built-in HTTP/HTTPS/SOCKS4/4a/5/5h proxies incl. **SOCKS5 UDP ASSOCIATE** for
HTTP/3 & TFTP, and `--unix-socket` (M6 partial).
- **Curl flags — Tier A/B/C + quick wins**: `-f`, `-S`, `-G`, `-r`,
`--compressed`, `-D`, `-R`, `--create-dirs`, `--max-filesize`, `--url`, `-U`,
socks shorthands, `-w/--write-out` (subset; phase timers pending M1),
`-n/--netrc`(+file), `-J`, `--retry` + `--retry-delay/-max-time/-connrefused/
-all-errors`, `-4/-6`, `--resolve`, `-K/--config`, `--next`, getopt-style
**bundled short flags** (`-sS`, `-ofile`).
- **M4 (HTTP breadth, partial)**: `-z/--time-cond`, `-e ;auto`, `--output-dir`,
`--fail-with-body`, `--proto`/`--proto-default`, `--location-trusted`,
`--post301/302/303`, `--connect-to`, **`--json`** (POST JSON + Accept).
- **M7 URL globbing**: `{a,b}`, `[1-100]`/`[a-z]` (`:step`, zero-pad), `-g`, `#N`.
- **M6 (connection control, partial)**: `--connect-to`, `--unix-socket`.
- **M2 (TLS)**: `--tlsv1.x` / `--tls-max` version pinning, **client certs /
mTLS** (`-E/--cert`, `--key`, `--pass`, `--cert-type`/`--key-type` — incl.
encrypted Ed25519/ECDSA/RSA keys), **public-key pinning** (`--pinnedpubkey
sha256//…`, hashing the cert's on-wire SPKI), and **`--capath`** on both
backends. On the default purecrypto-tls backend: **`--crlfile`** (CRL
revocation) and **`--ciphers`/`--tls13-ciphers`** (cipher-suite restriction);
the rustls-tls backend errors clearly for those two. `--cert-status` (rsurl
requests no OCSP staple) is the only documented-unsupported TLS knob —
accept-and-error, never silently ignored.
- **M9 (new protocols, partial)**: **SMTP/SMTPS** (EHLO/STARTTLS/AUTH/MAIL/RCPT/
DATA via `--mail-from`/`--mail-rcpt`) and **TELNET** (IAC-stripping). New
schemes: `smtp`(25)/`smtps`(465)/`telnet`(23).
- Recognized-but-not-yet-enforced (need foundations): `-E/--cert` — warn
transparently. Genuine no-ops accepted for compat: `-q/--disable`,
`-N/--no-buffer`, `--no-progress-meter`, `--styled-output`/`--no-styled-output`.
- **M1 streaming I/O (keystone, partial)**: `Request::send_download` streams the
HTTP/1.1 body to a sink (redirect-following, cookies); HTTP/2/3, proxied,
compressed, and empty bodies fall back to buffered. **HTTP/2 and HTTP/3
response bodies also stream** straight to the sink (`http2::send_to` /
`http3::send_to`): DATA-frame payloads are written as they arrive instead of
reassembled in memory (content-encoded responses still buffer to decode; the
download path uses this for direct https GETs that aren't following
redirects). **Every non-HTTP download to a file** now goes through the
streaming sink (`Client::transfer_url_to`):
FTP/FTPS and `file://` stream the source directly (no full-body buffer), the
rest fetch-then-write through the same sink — all gaining `--limit-rate`/`-#`/
`--max-filesize`/`-y`/`-Y`/`--remove-on-error`/`--no-clobber` and `-w`.
**Streaming decompression**: a single gzip/zstd/br layer over a
Content-Length body decodes straight off the wire (budget-bounded), no
compressed-body buffer; deflate (zlib/raw ambiguity), multi-layer, chunked,
and unknown codings keep the buffered decode.
- **M5 (partial, on streaming)**: enforced `--max-filesize` (early abort),
`--limit-rate`, `-#` progress, and **`-y`/`-Y` low-speed abort** (exit 28) for
file downloads; **`--remove-on-error`**, **`--no-clobber`**; `-w
%{size_download}`, **`-w` phase timers** (`%{time_connect,appconnect,
pretransfer,starttransfer}`, HTTP/1.1 paths), **`%header{Name}`**,
**`%{ssl_verify_result}`**.
- **M3 (partial)**: **HTTP Digest** auth (`--digest`, MD5/SHA-256 + qop=auth),
`--oauth2-bearer`, and **AWS SigV4** (`--aws-sigv4`, HMAC-SHA256 chain).
- **M8**: `-Z/--parallel` + `--parallel-max` concurrent transfers.
- **M10 (protocol depth, partial)**: FTP **`--disable-epsv`** (skip EPSV, use
PASV directly), **`--ftp-create-dirs`** (MKD missing upload dirs), and
**active mode `-P`/`--ftp-port`** (`EPRT` with IPv4 `PORT` fallback;
direct-only; verifies the data callback comes from the control peer); FTP
upload now honors `-x` proxy via the `Client`. **RTSP interleaved RTP/RTCP**
reception after `PLAY` (`Session::read_interleaved`, idle- and byte-bounded).
- **M11 (partial)**: centralized **curl-compatible exit codes** for transfer
errors (1/3/6/7/8/28/47/52/79); a **`man/rsurl.1`** man page; an expanded
**libcurl-shaped C ABI** (`rsurl_easy_*` with `RSURLOPT_` for URL, method,
headers, body, timeouts, IDN, **followlocation, maxredirs, userpwd,
ssl_verifypeer, proxy, referer, range, cookie, xoauth2_bearer,
accept_encoding**).
**Remaining:** none in scope. Every curl feature feasible under the no-C
invariant and a good fit for a one-shot CLI is implemented and tested. The
interleaved-media hang risk is handled with an idle window + byte cap rather
than punting the feature.
**Out of scope under the no-C invariant or current architecture** (documented,
not "remaining" — verified against what curl itself requires):
- **RTMP** — curl only supports `rtmp://` when built against **librtmp**, a C
library. There is no pure-Rust path the no-C invariant permits.
- **SMB/SMBS** — curl's built-in SMB authenticates with **NTLM**; see NTLM below.
- **NTLM** — its handshake binds auth to a single TCP connection, which rsurl's
stateless send/retry model does not expose; a pool-reuse hack would be
unreliable.
- **Negotiate/Kerberos (GSSAPI)** and **c-ares DNS** — require C libraries.
- **LDAP writes** — not a curl feature (curl's LDAP is search/read only), so
out of scope for *parity*.
These are intentionally not pursued.
**Status:** every curl feature that is feasible under the no-C invariant and a
good fit for a one-shot CLI is now implemented and tested — including streaming
response bodies for HTTP/1.1, HTTP/2, HTTP/3, FTP/FTPS, and `file://`, and RTSP
interleaved RTP/RTCP reception. The only items not implemented are the
documented no-C/architecture exclusions below.
## Where we are today
Protocols: HTTP/1.1, HTTP/2, HTTP/3, FTP/FTPS, SFTP, SCP, IMAP(S), POP3(S),
MQTT(S), RTSP, TFTP, LDAP(S), gopher(s), dict, file, ws/wss, SMTP(S), TELNET.
~100 CLI flags, cookies (RFC 6265 + Netscape I/O), transparent decompression
(buffered, plus streaming gzip/zstd/br on HTTP/1.1 downloads), proxies
(HTTP/HTTPS/SOCKS4/4a/5/5h incl. SOCKS5-UDP), pluggable transport (`Connector`),
TLS 1.2/1.3 with full verification, four auth schemes (Basic/Digest/Bearer/
SigV4), a `Client`/`Request` library API, and a `rsurl_easy_*` C ABI.
**Streaming I/O (the original Milestone 1) is in place** for HTTP/1.1, HTTP/2,
HTTP/3, FTP/FTPS, and `file://` downloads to a file: bodies flow straight to
disk through a sink that enforces `--limit-rate`, `-#`, `--max-filesize`,
`-y`/`-Y`, `-w`, `--remove-on-error`, and `--no-clobber`. Content-encoded
responses still buffer to decode (single gzip/zstd/br layers stream-decode on
HTTP/1.1), and the small text protocols buffer where it's immaterial.
---
## Milestone 1 — Streaming I/O (foundation)
**Why first:** unlocks M5 (rate/speed/progress), early `--max-filesize`,
large-file and chunked-infinite downloads, streamed uploads, and lower memory.
- Introduce a streaming body sink/source: response bodies written to the output
as they arrive; request bodies read from a reader. Thread through HTTP/1.1
first, then HTTP/2 and HTTP/3, then the file-transfer protocols (FTP/SFTP/
SCP/TFTP).
- Add a `Response` streaming variant (or a callback/`Read` handle) alongside the
current buffered `body: Vec<u8>` (keep buffered as the default for the simple
API).
- Wire phase/transfer metrics through this path (bytes so far, first-byte time).
**Delivers later:** progress bar, `--limit-rate`, `-y/-Y`, real `--max-filesize`
abort, `-C -` auto-resume, streamed `-T` uploads. **Effort: XL.**
## Milestone 2 — TLS completeness
Implemented across **both** TLS backends (`purecrypto-tls` default + `rustls-tls`).
- **Client certificates / mTLS** ✅ `-E/--cert <file[:password]>`, `--key`,
`--pass`, `--cert-type <PEM|DER>`, `--key-type <PEM|DER>`. The cert may embed
the key (one file) or carry it separately via `--key`; `-E cert:secret` or
`--pass` supply the key passphrase. Key parsing covers Ed25519 (PKCS#8),
ECDSA (SEC1/PKCS#8), and RSA (PKCS#1/PKCS#8), plus **encrypted PKCS#8 for all
three** on the purecrypto backend (purecrypto 0.6.5 added the encrypted-ECDSA
loader). (rustls backend: `rustls-pemfile` cannot decrypt keys, so an
encrypted key errors clearly there.)
- **Public-key pinning** ✅ `--pinnedpubkey sha256//BASE64[;sha256//...]`:
after the handshake, SHA-256 of the leaf cert's DER SPKI must match a pin or
the connection fails (`Error::BadResponse` "pinned public key …"). The bare
`<file>` pin form and non-`sha256//` algorithms are rejected, not ignored.
- **Trust store** ✅ `--cacert <file>` (replaces system roots) and `--capath
<dir>` (adds every CA in the directory on top of the chosen base roots).
- **Version control** ✅ `--tlsv1.0/1.1/1.2/1.3`, `--tls-max` (both backends).
- **Cipher restriction** ✅ `--ciphers` (TLS≤1.2 OpenSSL/IANA names) and
`--tls13-ciphers` (IANA `TLS_*`) → `ConfigBuilder::cipher_suites` on the
purecrypto-tls backend (purecrypto 0.6.5). Unknown/unsupported names error.
- **CRL revocation** ✅ `--crlfile <file>` → `ConfigBuilder::crls` on the
purecrypto-tls backend. The rustls-tls backend returns a clear error for both
`--crlfile` and `--ciphers` (those builder paths aren't wired there) rather
than silently skipping revocation / a suite restriction.
**Documented-unsupported (accept-and-error, never silently ignore):**
`--cert-status` (OCSP must-staple) — rsurl neither requests nor requires a
stapled response.
**Delivers:** mutual TLS + public-key pinning + extra trust anchors. **Done.**
## Milestone 3 — Authentication completeness
Currently only HTTP Basic. Add the auth scheme negotiation layer.
- `--digest` (RFC 7616), `--ntlm`, `--negotiate` (SPNEGO — *caveat: pure-Rust
Kerberos is the hard part; NTLM is feasible pure-Rust*), `--bearer`,
`--aws-sigv4`, `--anyauth`, `--proxy-*` variants of each.
- `--netrc-optional`, per-scheme credential application (netrc for FTP/IMAP/…).
**Delivers:** the common enterprise/cloud auth flows. **Effort: L.**
## Milestone 4 — HTTP feature breadth
Mostly request/redirect/cache semantics; no streaming dependency.
- Conditional: `-z/--time-cond`, `--etag-save`, `--etag-compare`.
- State engines: `--alt-svc`, `--hsts`.
- Redirect control: `--post301/302/303`, `--proto`, `--proto-redir`,
`--proto-default`, `--location-trusted`, `--max-filesize` (with M1).
- Request shaping: `-e` auto-referer (`;auto`), `--path-as-is`, `--raw`,
`--ignore-content-length`, `--tr-encoding`, `--expect100-timeout`,
`--request-target`, `-G` (done) edge cases, `--url-query`.
- HTTP/2/3 depth: `--http2-prior-knowledge`, server-push handling, HTTP/3
dynamic-table *encoder* (we decode; we still send literals).
**Delivers:** the long tail of `-H`-adjacent HTTP behavior. **Effort: L.**
## Milestone 5 — Transfer control & UX (needs M1)
- **Progress**: `-#/--progress-bar` and the default progress meter (live).
- **Rate/speed**: `--limit-rate`, `-Y/--speed-limit`, `-y/--speed-time`.
- **Retry family**: `--retry-delay`, `--retry-max-time`, `--retry-connrefused`,
`--retry-all-errors` (extend the existing `--retry`).
- **Resume**: `-C -` automatic resume (range/REST from existing file size).
- **Output**: full `-w/--write-out` variable set **with real phase timers**
(`time_namelookup/connect/appconnect/pretransfer/starttransfer/total`,
`speed_download`, `num_connects`, `ssl_verify_result`, `%{json}`, `%{header{}}`,
`%{certs}`), `--output-dir`, `--remove-on-error`, `--create-dirs` (done),
`-R` (done), `--fail-with-body`, `--fail-early`.
- **Tracing**: `--trace`, `--trace-ascii`, `--trace-time`, `--trace-ids`,
`--stderr`, `--styled-output`.
**Delivers:** scripting/observability parity. **Effort: L (after M1).**
## Milestone 6 — Connection & DNS control
- Binding/locality: `--interface`, `--local-port`, `--unix-socket`,
`--abstract-unix-socket`.
- TCP tuning: `--tcp-nodelay` (default on in curl), `--tcp-fastopen`,
`--no-keepalive`, `--keepalive-time`.
- Address selection: Happy Eyeballs (`-4`/`-6` done; add dual-stack racing +
`--happy-eyeballs-timeout-ms`), `--connect-to`, `--resolve` (done).
- DNS: custom servers/interface (`--dns-servers`, `--dns-interface`) — *caveat:
needs a pure-Rust resolver; today we use the system resolver. Likely a
pure-Rust DNS dependency or a built-in stub resolver.*
**Delivers:** placement/binding parity. **Effort: M–L.**
## Milestone 7 — URL handling & globbing
- **URL globbing**: `{a,b,c}` and `[1-100]`/`[a-z]`/`[001-100]` with `:step`,
plus `--globoff (-g)`. Expands one URL into many transfers.
- `\#N` output-name references for globs (`-o "#1.html"`).
- `--path-as-is`, IDN edge cases, `--url` (done).
**Delivers:** curl's batch-URL ergonomics. **Effort: M.**
## Milestone 8 — Parallelism in the CLI
The library already has `send_multiplexed` (true HTTP/2 concurrency).
- `-Z/--parallel`, `--parallel-max`, `--parallel-immediate` driving multiple
URLs/operations concurrently (across `--next` ops and globs).
- A shared progress meter for parallel transfers.
**Delivers:** `curl -Z` throughput. **Effort: M (after M1+M5).**
## Milestone 9 — New protocols
In rough value order:
- **SMTP / SMTPS** (`--mail-from`, `--mail-rcpt`, `-T` body, STARTTLS, SASL).
High value; reuses the IMAP/POP3 SASL + STARTTLS machinery.
- **TELNET** (`-t/--telnet-option`). Small.
- **SMB / SMBS**. Large; pure-Rust SMB is a significant dependency/build.
- **RTMP family**. Large; niche. Likely lowest priority / possible *out of
scope* depending on a pure-Rust impl.
**Effort: SMTP M, TELNET S, SMB XL, RTMP XL.**
## Milestone 10 — Protocol depth & polish
- FTP: active mode (`-P/--ftp-port`), `--ftp-method`, `--ftp-create-dirs`,
`--ftp-pret`, `--ftp-ssl-ccc`, MLSD listings, `--disable-epsv/eprt`.
- IMAP/POP3/SMTP: more URL verbs, `--login-options`, UID ranges, APOP.
- RTSP: RTP media reception (`--interleaved`), `--rtsp-request` aliases.
- WebSocket: `--ws` ping interval, close-code surfacing.
- LDAP: paged results, modify/add (read-only today).
- Combined short flags (`-sS`, `-fL`) — small, high compatibility value;
**good early win, can land before M1.**
**Effort: mixed S–L.**
## Milestone 11 — Compatibility & ecosystem polish
- Exit-code parity sweep (curl's full table), error-message wording.
- `--help category`, generated man page, `--manual`, `-q/--disable` (.curlrc),
shell completions.
- `libcurl`-shaped C API surface in `ffi.rs` for drop-in linking (stretch).
**Effort: M.**
---
## Suggested ordering (dependency-aware)
1. **Quick compatibility wins first** (no deps): combined short flags, `-z`,
`-e ;auto`, `--retry-*`, `--proto*`, `--output-dir`, `--fail-with-body`,
URL globbing (M7), `--connect-to`. These each close visible `-h` gaps cheaply.
2. **M1 Streaming I/O** — the keystone; schedule early because M5/M8 and large
files depend on it.
3. **M2 TLS** and **M3 Auth** in parallel (independent subsystems).
4. **M5 UX/transfer-control** once M1 lands; **M8 parallel CLI** after M5.
5. **M4 HTTP breadth** and **M6 connection control** as steady fill-in.
6. **M9 protocols** (SMTP first), **M10 depth**, **M11 polish** ongoing.
## Definition of "parity" (tiers)
- **Parity-90**: every commonly-used curl flag and all mainstream protocols
(HTTP(S)/FTP(S)/SFTP/SMTP/IMAP/POP3) behave identically — M1–M9 (minus SMB/
RTMP) + the M10 quick wins. This is the realistic target.
- **Parity-99**: the full option table and SMB/RTMP/TELNET, man page, libcurl-
shaped API — M10/M11 complete.
- **Never-parity (by design)**: features requiring a C toolchain or non-Rust
crypto/DNS — OpenSSL `--engine`, GSSAPI/Kerberos `--negotiate` (unless a
pure-Rust SPNEGO lands), c-ares-specific DNS knobs. Documented as
intentional divergences.
## Out of scope / caveats
- **`--negotiate` (Kerberos/GSSAPI)**: no pure-Rust GSSAPI today; NTLM and
Digest are feasible, Negotiate likely deferred or stubbed.
- **`--dns-servers`/`--dns-interface`**: require a pure-Rust resolver to honor
without C (c-ares). Tracked under M6 with that dependency.
- **`--engine`, PKCS#11**: OpenSSL-specific; not applicable to a pure-Rust TLS
stack.