storj-uplink
Native Rust Uplink client for Storj.
1.1.1 extends the public storj::* API: access grants, buckets, objects
(multi-segment upload/download), listing, copy/move, multipart, revoke, and
Object Lock. Spec: docs/design-native-uplink.md.
This is not an S3 SDK, not an FFI wrapper around
uplink-c, and not a drop-in for
crates.io uplink 0.11.0 (blocking,
!Send). Go is never required to build or use the crate.
Edge credential registration and linksharing are outside this crate's scope.
Install
[]
= "1.1"
= { = "1", = ["macros", "rt-multi-thread", "io-util"] }
Callers need their own Tokio runtime; tokio is not re-exported.
Git: storj = { git = "https://github.com/BradKollmyer/storj-uplink", tag = "v1.1.1" }.
From this workspace: storj = { path = "crates/storj" }.
The public API is storj::* only. Implementation crates (storj-access,
storj-ec, storj-encryption, storj-proto, storj-rpc, storj-uplink)
are on crates.io so Cargo can resolve them; do not depend on them directly.
storj-test is unpublished.
Quick start
Parse an access grant, open a project, upload and download. Full CLI:
crates/storj/examples/walkthrough.rs.
use ;
use AsyncWriteExt;
async
commit() is the only path that publishes an upload. Dropping Upload without
commit aborts. poll_shutdown does not commit.
Custom metadata is validated before upload setters, multipart commit, and metadata
updates: keys must be nonempty, and keys/values cannot contain NUL bytes. Empty
values and Unicode are accepted. Use verify_custom_metadata(&metadata) for
preflight validation; invalid input returns ErrorKind::MetadataInvalid.
Network transports and telemetry
Configure both satellite and storage-node connections with Config::transport:
Noise (the default), Tcp (TLS only), Quic (QUIC only), or Auto. Auto gives QUIC a 250 ms
head start, then races TCP/TLS; a failed QUIC attempt starts TCP immediately.
DNS, TLS authentication, and fallback share the configured dial deadline.
Both transports pin the peer's Storj NodeID and present the client identity.
QUIC carries DRPC directly with the storj ALPN, using Quinn and TLS 1.3.
TransportMode::Noise selects TCP with Noise IK for storage-node uploads and
downloads when the authenticated satellite advertises a Noise key. Metadata
calls and nodes without an advertised key use TCP/TLS. Invalid keys, unsupported
protocols, or failed Noise handshakes fail the dial without a TLS downgrade.
Both advertised ciphers (ChaCha20-Poly1305 and AES-GCM, with X25519/BLAKE2b) are
supported. The DRPC INVOKE frame is sent inside the IK handshake. DRPC flushes
that frame before sending the first piece request, so the order limit and upload
chunk travel after the handshake; they are not currently coalesced into early
data as in Go. Authentication completes during I/O and shares the dial deadline.
Upload response identities require a leaf signed by the supplied CA, whose
NodeID must match the order limit, before checking the signed piece hash.
Additional response certificates must parse but need not form a complete chain,
matching Go. TLS/QUIC handshakes still require full-chain signature validation.
Connection telemetry reports TransportKind::Noise for these connections.
Config::network controls noise_early_data, tcp_fast_open, and
background_qos (all enabled by default). Fast Open races ordinary TCP after
250 ms only when the satellite advertises Fast Open and a debounce limit of at
least two. Each resolved address uses a fresh Noise handshake, so a stalled
address cannot prevent another from authenticating. The Fast Open and ordinary
TCP legs for one address send at most two identical copies; without advertised
suppression, that address sends one copy over ordinary TCP.
Unavailable Fast Open support allows ordinary TCP to proceed. The optional
TFO dependency is compiled only on the platforms tokio-tfo implements
(Windows, Linux, Android, FreeBSD, and Apple platforms); other targets,
including the other BSDs, build without it and use ordinary TCP. TCP address candidates also race
to avoid a stalled IPv6 route blocking IPv4. On Linux, background QoS requests
Lower Effort DSCP; congestion_control can name a kernel TCP controller. These
socket hints are best-effort. Disable noise_early_data for an eager, empty-payload
handshake, or select Tcp to force TLS. Early data is restricted to piece Upload/Download.
use ;
let config = Config ;
// Project::open_with_config(&access, config).await?;
Telemetry is disabled by default. The callback receives connection-attempt events (transport, elapsed time, outcome) and one terminal event per upload, download, or multipart part. Transfers report plaintext bytes accepted from the writer or delivered to the reader, elapsed time from operation start, time to the first such byte, outcome, and configured transport mode. Upload success means commit succeeded; download success is reported on close/drop after the requested range was consumed. The copy helpers also report source read and destination write/flush failures. Explicit aborts and unfinished drops are cancellations. Initialization and transfer failures report errors. No bucket names, object keys, credentials, payloads, or error strings are included.
Transfer events also contain diagnostics: working time, canonical satellite
NodeID/address, OS/architecture/CPU count, expiration status, and a sanitized
error kind plus retryability. Downloads include the requested range, normalized
range, and full object size. Object size becomes available after initialization
for downloads and successful commit for uploads/parts. Unknown values remain
None, including size/range when download initialization fails.
Working time excludes idle gaps between completed API operations and includes
initialization, commit/abort, and pending read/write waits. An I/O operation stays
active from its first poll until it returns Ready or the transfer terminates;
the underlying AsyncRead/AsyncWrite traits cannot observe cancellation of an
individual caller-owned I/O future. It measures wall time, not CPU usage.
To reuse a TLS identity, parse a leaf-first certificate chain and its matching unencrypted P-256 private key (PKCS#8 or SEC1 PEM) before opening the project:
use ;
let config = Config ;
The constructor validates chain signatures and the leaf/key match, returning
ErrorKind::InvalidTlsIdentity for invalid input. All satellite and storage-node
TLS/QUIC connections reuse that identity, including new pooled connections.
None retains ephemeral identity generation. Noise still generates its own
X25519 initiator key. Debug output excludes PEM material, and stored private-key
bytes are zeroized when dropped.
Callbacks run synchronously and may run concurrently; keep them fast and nonblocking. With panic unwinding, callback panics are caught. There is no automatic network exporter or background delivery queue.
Remote-segment downloads launch extra pieces every download_hedge_delay
(default 1s) while more shares are needed; piece completions do not postpone
that deadline. Some(Duration::ZERO) disables hedging. Speculative launches
are capped at about 20% of the required shares. concurrent_segments
(default 10) caps how many remote segments transfer at once.
Existing Config struct literals from 1.0 must add the new fields
(tls_identity, transport, network, telemetry, download_hedge_delay,
and concurrent_segments) or use ..Default::default(). To retain the 1.0
transport policy, set transport: TransportMode::Tcp; the new default
selects advertised Noise. Public transport and telemetry types are defined
by storj; internal RPC types are not part of the facade API.
Comparison with uplink 0.11.0 (FFI)
uplink 0.11.0 |
storj 1.1.1 |
|---|---|
crate name uplink |
crate name storj |
uplink::access::Grant |
storj::Access (Access::parse) |
blocking std::io |
Tokio AsyncRead / AsyncWrite |
Project, Grant, … are !Send + !Sync |
public handle types are Send + Sync |
Project::open is infallible |
Project::open returns Result |
| Go required at build time | Go never required to build or use |
MSRV
Rust 1.88 (edition 2024). This matches the existing let-chain syntax and the
patched time dependency used by the certificate stack.
Build notes
On aarch64 (Apple Silicon, Graviton) the pinned aes/polyval crates only use
the ARMv8 AES and PMULL instructions when built with
--cfg aes_armv8 --cfg polyval_armv8; without them AES-GCM runs in software
(~10x slower). This workspace sets them in .cargo/config.toml; downstream
builds must set the same rustflags (x86_64 autodetects AES-NI).
Tests
STORJ_INTEROP=1
Object-matrix interop and storj-sim need a live grant (STORJ_INTEROP_ACCESS /
STORJ_SIM_ACCESS). Production-satellite smoke tests are #[ignore] and also
need STORJ_LIVE=1; they load STORJ_ACCESS from the environment or a .env
file:
STORJ_LIVE=1
See crates/storj/tests/README.md.
Contributing
See CONTRIBUTING.md. Changelog: CHANGELOG.md. Code of Conduct: CODE_OF_CONDUCT.md.
Security
Report vulnerabilities privately — see SECURITY.md.
License
Dual-licensed MIT OR Apache-2.0. See LICENSE-MIT and LICENSE-APACHE.