# Holger guards your artifacts at rest
**Immutable artifact repository in pure Rust.**
- **Airgap**: transfer from internet, Nexus, Artifactory → immutable znippy archive
- **Versioned**: atomic transactions, content-addressed (Blake3)
- **Extreme performance**: Apache Arrow zero-copy, multi-core parallel I/O
- **Zero dependencies**: no OS packages, no shell, no Python — 100% Rust (even decompressors rewritten from C)
- **Embeddable**: runs on bare-metal/OS-free devices, 2 MB RSS
---
### Artifact retrieval throughput — Holger vs Nexus 3
Same machine (32 cores, AMD), same artifact types, same dataset size per type.
Nexus 3 (v3.72): container, 32 CPUs, 32 GB RAM, 16 GB heap, 400 Jetty threads, G1GC.
Holger (znippy 0.8.0): in-process, **2 MB RSS**.
| **Rust crates** | 417,602 ops/s | **10,084,953 ops/s** | 105 ops/s | 1,265 ops/s | 519 ops/s |
| **Java JARs** | 283,015 ops/s | **7,307,041 ops/s** | 87 ops/s | 567 ops/s | 418 ops/s |
| **Python wheels** | 313,469 ops/s | **7,740,871 ops/s** | 91 ops/s | 955 ops/s | 726 ops/s |
> **Holger is 7,972× faster** on Rust crates at Nexus's sweet spot (16 clients).
> **12,887× faster** on Java JARs, **8,106× faster** on Python wheels.
> Single-threaded Holger still outperforms 32-client Nexus by **805×** (Rust), **677×** (Java), **432×** (Python).
> Nexus *degrades* past 16 clients on all types despite 400 Jetty threads and 16 GB heap.
> Holger uses **2 MB RSS** vs Nexus's **16 GB heap** — 8,000× less memory.
#### How it works
Each `get_file` is one `pread` from a shared fd plus either a zero-copy skip
(pre-compressed `.crate`/`.jar`) or an openzl decompress into a reused buffer —
fully parallel across cores, no shared reader thread. Backed by znippy 0.8.0's
N-worker positioned-I/O read path.
---
## Import / Export (holger-agent)
The agent CLI transfers artifacts between any combination of sources and destinations:
```
┌──────────────────────────────────────────────────────────────────────────────┐
│ holger-agent transfer paths │
├──────────────────────────────────────────────────────────────────────────────┤
│ │
│ Sources (--from) Destinations (--to) │
│ ───────────────── ────────────────── │
│ world (crates.io/PyPI/Maven Central) ──┐ │
│ nexus (Sonatype Nexus, basic auth) ──┤──▶ znippy (.znippy archive)│
│ holger (Holger server, OIDC/mTLS) ──┤──▶ tar-zstd (.tar.zst) │
│ znippy (.znippy archive) ──┤──▶ directory (flat files) │
│ tar-zstd (.tar.zst archive) ──┤──▶ nexus (push to Nexus) │
│ directory (flat .crate/.jar/.whl files) ──┘──▶ holger (push to Holger) │
│ │
└──────────────────────────────────────────────────────────────────────────────┘
```
### Examples
```bash
# ── Download from internet → znippy archive ─────────────────────────────────
# Rust crates (from Cargo.lock)
holger-agent --from world --lockfile Cargo.lock --to znippy -o rust.znippy
# Python packages (from requirements.txt)
holger-agent --from world --lockfile-kind requirements-txt \
--lockfile requirements.txt --to znippy -o pip.znippy
# Python packages (from uv.lock — URLs already embedded)
holger-agent --from world --lockfile-kind uv-lock \
--lockfile uv.lock --to znippy -o pip.znippy
# ── Download from internet → tar.zst archive ────────────────────────────────
# Rust crates (from Cargo.lock) — standard tar+zstd, no snappy dependency
holger-agent --from world --lockfile Cargo.lock --to tar-zstd -o rust-crates.tar.zst
# Python packages (from requirements.txt) → tar.zst
holger-agent --from world --lockfile-kind requirements-txt \
--lockfile requirements.txt --to tar-zstd -o pip-packages.tar.zst
# Python packages (from uv.lock) → tar.zst
holger-agent --from world --lockfile-kind uv-lock \
--lockfile uv.lock --to tar-zstd -o pip-packages.tar.zst
# ── Download from internet → directory ───────────────────────────────────────
# Rust crates to flat directory
holger-agent --from world --lockfile Cargo.lock --to directory -o ./crates/
# ── Nexus → znippy (mirror/export from Nexus) ──────────────────────────────
holger-agent --from nexus --nexus-url http://nexus.internal:8081 \
--repository maven-releases --username admin --password secret \
--to znippy -o maven-export.znippy
# ── Nexus → tar.zst (export from Nexus to standard archive) ─────────────────
holger-agent --from nexus --nexus-url http://nexus.internal:8081 \
--repository maven-releases --username admin --password secret \
--to tar-zstd -o maven-export.tar.zst
# ── Holger → tar.zst (export from Holger server) ────────────────────────────
holger-agent --from holger --holger-url grpc://holger.internal:50051 \
--auth oidc --token-url https://keycloak/token --client-id ci-agent \
--repository rust-prod --to tar-zstd -o rust-prod-backup.tar.zst
# ── znippy → Nexus (import into Nexus) ──────────────────────────────────────
holger-agent --from znippy --input maven-export.znippy \
--to nexus --nexus-url http://nexus.internal:8081 \
--repository maven-staging --username admin --password secret
# ── tar.zst → Nexus (import standard archive into Nexus) ────────────────────
holger-agent --from tar-zstd --input maven-export.tar.zst \
--to nexus --nexus-url http://nexus.internal:8081 \
--repository maven-staging --username admin --password secret
# ── znippy → Holger server (push to production) ─────────────────────────────
holger-agent --from znippy --input rust.znippy \
--to holger --holger-url grpc://holger.internal:50051 \
--auth oidc --token-url https://keycloak/token --client-id ci-agent \
--repository rust-prod
# ── tar.zst → Holger server (push standard archive to production) ────────────
holger-agent --from tar-zstd --input rust-crates.tar.zst \
--to holger --holger-url grpc://holger.internal:50051 \
--auth oidc --token-url https://keycloak/token --client-id ci-agent \
--repository rust-prod
# ── directory → Nexus (push loose files to Nexus) ────────────────────────────
holger-agent --from directory --input ./my-crates/ \
--to nexus --nexus-url http://nexus.internal:8081 \
--repository crates-raw --username admin --password secret
# ── directory → Holger (push loose files to Holger) ──────────────────────────
holger-agent --from directory --input ./my-crates/ \
--to holger --holger-url grpc://holger.internal:50051 \
--auth oidc --token-url https://keycloak/token --client-id ci-agent \
--repository rust-staging
# ── Nexus → directory (flat file export) ─────────────────────────────────────
holger-agent --from nexus --nexus-url http://nexus.internal:8081 \
--repository npm-hosted --username admin --password secret \
--to directory -o ./exported-packages/
# ── Holger → directory (export from Holger to flat files) ────────────────────
holger-agent --from holger --holger-url grpc://holger.internal:50051 \
--auth oidc --token-url https://keycloak/token --client-id ci-agent \
--repository rust-prod --to directory -o ./exported-crates/
```
### Airgap workflow
```
Internet side │ Airgapped side
│
crates.io ──┐ │
Maven ──────┤ │
PyPI ───────┤──▶ holger-agent │ holger-agent ──▶ holger-server
Nexus ──────┘ --to znippy │ --from znippy (gRPC / Rust fn)
--to tar-zstd │ --from tar-zstd
│ │ ▲
▼ │ │
.znippy file ─┼───────┘ (znippy = max perf, Arrow IPC)
.tar.zst file ┼───────┘ (tar.zst = universal, no snappy)
(USB/media) │
```
**When to use which format:**
| `.znippy` | Fastest (Arrow IPC), queryable by pyarrow/polars, sub-index per repo/pkg_type | Requires znippy/snappy support |
| `.tar.zst` | Universal (standard tar+zstd), works everywhere, `tar` can extract | No sub-indexing, slightly larger |
| `directory` | Human-inspectable, git-friendly | Many small files, slow on HDD |
### Lockfile formats
| `Cargo.lock` | `--lockfile-kind cargo` (default) | Crate names + exact versions → downloads from crates.io |
| `requirements.txt` | `--lockfile-kind requirements-txt` | `name==version` lines → downloads from PyPI |
| `uv.lock` | `--lockfile-kind uv-lock` | Full URLs embedded → downloads directly |
## Architecture
```
┌─────────────────────────────────────────────────────────────────────────┐
│ Holger Ecosystem │
├─────────────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────┐ ┌──────────────────┐ ┌────────────────┐ │
│ │ holger-agent │────▶│ holger-server │────▶│ Storage │ │
│ │ (CLI + daemon) │ │ (gRPC / Rust fn) │ │ (.znippy) │ │
│ │ │ │ │ │ │ │
│ │ --auth oidc │ │ OIDC validation │ │ Immutable │ │
│ └─────────────────┘ └──────────────────┘ │ Blake3 CAS │ │
│ │ │ └────────────────┘ │
│ │ │ │
│ ▼ ▼ │
│ ┌─────────────────┐ ┌──────────────────┐ │
│ │ Nexus/Registry │ │ cargo/mvn │ │
│ │ (connector) │ │ (clients) │ │
│ └─────────────────┘ └──────────────────┘ │
│ │
└─────────────────────────────────────────────────────────────────────────┘
```
## Access Modes
Holger exposes **two access modes** — no HTTP/REST:
### 1. Rust Function Bindings (embedded mode)
Use holger-server-lib directly in your Rust application:
```rust
use holger_server_lib::{read_ron_config, wire_holger};
use holger_traits::ArtifactId;
let mut holger = read_ron_config("holger.ron").unwrap();
holger.instantiate_backends().unwrap();
wire_holger(&mut holger).unwrap();
// Fetch artifact
let id = ArtifactId { namespace: None, name: "serde".into(), version: "1.0.219".into() };
let data = holger.fetch("rust-prod", &id).unwrap();
// List repositories
let repos = holger.list_repositories(); // ["rust-prod", "maven-staging"]
// Put artifact (write-enabled repos)
holger.put("rust-staging", &id, &bytes).unwrap();
```
### 2. gRPC (network mode)
Configure `exposed_endpoints` in RON → holger starts a tonic gRPC server automatically.
**Services** (defined in `proto/holger.proto`):
| `RepositoryService` | `FetchArtifact`, `ListArtifacts`, `PutArtifact`, `StreamArtifact` (server-streaming) |
| `ArchiveService` | `ListArchiveFiles`, `ArchiveInfo` |
| `AdminService` | `Health`, `ListRepositories` |
### 3. Apache Arrow Flight (batch mode) — planned
Arrow Flight as a **universal artifact service**: batch-fetch multiple artifacts (or files within artifacts) in a single call with zero-copy columnar transport.
| **Batch artifact fetch** | Request N artifacts in one Flight `DoGet` — server reads from znippy archive (already Arrow IPC), streams back as RecordBatches |
| **Files-within-artifact** | Fetch specific files from inside a .whl or .jar without downloading the whole archive (uses lzip/ljar filter) |
| **Index queries** | `GetFlightInfo` returns archive metadata (file list, sizes, checksums) as Arrow schema — zero parsing on client side |
| **Any-language client** | Python (pyarrow), Go, Java, Rust, C++ — all get native Flight clients for free |
Why Flight over raw gRPC for batch:
- znippy archives are already Arrow IPC → Flight serves them with near-zero serialization cost
- Columnar format means fetching 1000 small crates is one contiguous memory transfer, not 1000 RPCs
- `uv`/`pip`/`cargo` resolve+fetch could use a single Flight batch call instead of N HTTP GETs
**Connect with grpcurl:**
```bash
# Health check
grpcurl -plaintext localhost:50051 holger.AdminService/Health
# List repositories
grpcurl -plaintext localhost:50051 holger.AdminService/ListRepositories
# Fetch artifact
grpcurl -plaintext -d '{"repository":"rust-prod","name":"serde","version":"1.0.219"}' \
localhost:50051 holger.RepositoryService/FetchArtifact
# Stream large artifact (chunked 64KB)
grpcurl -plaintext -d '{"repository":"rust-prod","name":"tokio","version":"1.47.1"}' \
localhost:50051 holger.RepositoryService/StreamArtifact
```
## Crates
| `holger-server-lib` | Core server: RON config, wiring engine, gRPC services, auth module |
| `holger-server-cli` | Server binary (`holger-server start --config`) |
| `holger-agent-lib` | Agent logic: connectors, transfer operations |
| `holger-agent-cli` | Agent binary: airgap, push/pull between registries |
| `holger-traits` | Shared traits (`ConnectorTrait`, `RepositoryBackendTrait`) |
| `holger-nexus-connector` | Nexus/Artifactory connector (basic auth, OIDC) |
| `holger-world-connector` | crates.io + PyPI connector (reads Cargo.lock, requirements.txt, uv.lock) |
| `holger-rust-file-repository` | Rust crate repository backend (filesystem) |
| `holger-rust-znippy-repository` | Rust/Maven/Pip repository backend (znippy archive) |
| `holger-mannequin` | Dev daemon: mock PKI, OIDC, Nexus, egui GUI |
| `holger-distr` | Container/systemd distribution (Dockerfile, .service) |
## Configuration (RON)
Holger uses [RON](https://github.com/ron-rs/ron) for configuration — it can express the graph topology that TOML/YAML cannot.
### Rust crate repository (file-backed)
```ron
(
repositories: [
(
ron_name: "rust-prod",
ron_repo_type: "rust",
ron_upstreams: [],
ron_in: None,
ron_out: Some((
ron_storage_endpoint: "local-storage",
ron_exposed_endpoint: "grpc",
)),
),
],
exposed_endpoints: [
(
ron_name: "grpc",
ron_url: "0.0.0.0:50051",
),
],
storage_endpoints: [
(
ron_name: "local-storage",
ron_storage_type: "rocksdb",
ron_path: "/var/lib/holger/data",
),
],
)
```
### Rust crate repository (znippy archive)
```ron
(
repositories: [
(
ron_name: "rust-airgap",
ron_repo_type: "rust",
ron_archive_path: Some("/var/lib/holger/rust-airgap.znippy"),
ron_upstreams: [],
ron_in: None,
ron_out: Some((
ron_storage_endpoint: "znippy-storage",
ron_exposed_endpoint: "grpc",
)),
),
],
exposed_endpoints: [
(
ron_name: "grpc",
ron_url: "0.0.0.0:50051",
),
],
storage_endpoints: [
(
ron_name: "znippy-storage",
ron_storage_type: "znippy",
ron_path: "/var/lib/holger/rust-airgap",
),
],
)
```
### Python pip repository (znippy archive, PEP 503 simple index)
```ron
(
repositories: [
(
ron_name: "pip-internal",
ron_repo_type: "pip",
ron_archive_path: Some("/var/lib/holger/pip.znippy"),
ron_upstreams: [],
ron_in: None,
ron_out: Some((
ron_storage_endpoint: "pip-storage",
ron_exposed_endpoint: "grpc",
)),
),
],
exposed_endpoints: [
(
ron_name: "grpc",
ron_url: "0.0.0.0:50051",
),
],
storage_endpoints: [
(
ron_name: "pip-storage",
ron_storage_type: "znippy",
ron_path: "/var/lib/holger/pip",
),
],
)
```
The server exposes a PEP 503 simple index at `/{repo}/simple/` — point pip or uv directly at it:
```bash
pip install --index-url https://holger.internal/pip-internal/simple/ requests
uv pip install --index-url https://holger.internal/pip-internal/simple/ requests
```
### Maven repository (znippy archive)
```ron
(
repositories: [
(
ron_name: "maven-internal",
ron_repo_type: "maven3",
ron_archive_path: Some("/var/lib/holger/maven.znippy"),
ron_upstreams: [],
ron_in: None,
ron_out: Some((
ron_storage_endpoint: "maven-storage",
ron_exposed_endpoint: "grpc",
)),
),
],
exposed_endpoints: [
(
ron_name: "grpc",
ron_url: "0.0.0.0:50051",
),
],
storage_endpoints: [
(
ron_name: "maven-storage",
ron_storage_type: "znippy",
ron_path: "/var/lib/holger/maven",
),
],
)
```
## Authentication
| **OIDC** | Bearer token from token endpoint (Keycloak, AD, etc.) | Human users, CI/CD pipelines |
The agent CLI uses `--auth oidc` when talking to Holger server. Legacy Nexus connections still use `--username`/`--password` (basic auth via connector).
## Performance
Holger uses **znippy archives** (.znippy) as storage. A `.znippy` file is a valid **Arrow IPC File** — queryable directly by pyarrow/polars:
```python
# pyarrow
import pyarrow.ipc
reader = pyarrow.ipc.open_file("archive.znippy")
table = reader.read_all()
print(table.schema)
# relative_path: string, compressed: bool, uncompressed_size: uint64, chunks: list<...>
# polars
import polars as pl
df = pl.read_ipc("archive.znippy")
df.select("relative_path", "uncompressed_size", "compressed").head(10)
```
### Znippy archive benchmarks (v0.7.4 — io_uring + Gatling pipeline)
Packing source files into `.znippy` and extracting back. Files already compressed (.gz, .jar, .zip, etc.) are stored as-is (skipped). `.crate` files are gzipped tarballs — znippy compresses the *source contents* inside, not the .crate blob itself:
| text 500MB | 500 MB | 0.06 MB | 9,014× | 1,639 MB/s | 2,874 MB/s |
| single file 2GB | 2,048 MB | 0.22 MB | 9,509× | 5,802 MB/s | 3,066 MB/s |
| binary pattern 500MB | 500 MB | 0.07 MB | 7,338× | 2,242 MB/s | 2,976 MB/s |
| 100k small files (10KB) | 977 MB | 17.4 MB | 56× | 4,811 MB/s | 1,957 MB/s |
| random (incompressible) | 500 MB | 500 MB | 1.0× | 105 MB/s | 2,890 MB/s |
| mixed repo 530MB | 530 MB | 530 MB | 1.0× | 3,118 MB/s | 6,310 MB/s |
| **Real jars (4730 files, RAID)** | **5,124 MB** | — | — | **2,527 MB/s** | **9,950 MB/s** |
> **Note:** "Pack" = archiving + compression into `.znippy`. "Unpack" = decompression + extraction. Stream packer (in-memory/HTTP) numbers above; slot packer (io_uring disk reads) hits 678 MB/s on 10k×10KB synthetic. Already-compressed data (.jar, .crate) passes through at full I/O speed without re-compression.
### Host decompressors
The `host-decompressors` feature replaces miniz_oxide with custom parallel decompression:
| **linflate** | ~700 MB/s single-core | SIMD DEFLATE decoder (AVX2 match-copy) |
| **lgz** | 6,100+ MB/s multi-core | Parallel gzip decompression (full-flush segments) |
| **lgz** (speculative) | 527 MB/s | Two-pass pugz-style for arbitrary gzip streams |
| **ljar** | Multi-core | Parallel JAR/ZIP extraction (per-entry parallelism) |
| miniz_oxide (replaced) | ~190 MB/s | What most Rust projects use by default |
Single-core path is 3.7× faster than miniz_oxide. Multi-core scales linearly.
## GUI Builders
Holger has **no built-in web UI**. Instead, build your own GUI using either access mode:
### Option A: gRPC client (any language)
Use the proto definition (`proto/holger.proto`) to generate a client in any language.
**Getting OIDC tokens and TLS certs from mannequin (for GUI dev):**
Start mannequin first — it provides the complete auth infrastructure your GUI needs:
```bash
# 1. Start mannequin (mock PKI + OIDC + Nexus)
cargo run -p holger-mannequin -- serve --gui --port 9999
# 2. Get an OIDC token (accepts any credentials — it's a dev dummy)
TOKEN=$(curl -s -X POST http://127.0.0.1:9999/token \
-d 'grant_type=password&username=admin&password=anything&client_id=my-gui' \
| jq -r .access_token)
echo "Bearer token: $TOKEN"
# 3. Get a TLS client certificate (for mTLS to holger-server)
curl -s -X POST http://127.0.0.1:9999/pki/issue/client \
-H "Content-Type: application/json" \
-d '{"common_name": "my-gui-client"}' \
| jq -r '.cert_pem' > client.pem
curl -s -X POST http://127.0.0.1:9999/pki/issue/client \
-H "Content-Type: application/json" \
-d '{"common_name": "my-gui-client"}' \
| jq -r '.key_pem' > client-key.pem
# 4. Get the CA cert (to verify holger-server's TLS)
curl -s http://127.0.0.1:9999/pki/ca.pem > ca.pem
# 5. Connect to holger-server with OIDC auth
grpcurl -H "Authorization: Bearer $TOKEN" \
-cacert ca.pem -cert client.pem -key client-key.pem \
localhost:50051 holger.AdminService/ListRepositories
```
**In your GUI code (Rust + tonic):**
```rust
use tonic::transport::{Channel, ClientTlsConfig, Certificate, Identity};
use tonic::metadata::MetadataValue;
// Get token from mannequin OIDC endpoint
let token_resp: serde_json::Value = reqwest::Client::new()
.post("http://127.0.0.1:9999/token")
.form(&[("grant_type", "password"), ("username", "admin"),
("password", "dev"), ("client_id", "my-gui")])
.send().await?.json().await?;
let token = token_resp["access_token"].as_str().unwrap();
// Get CA + client cert from mannequin PKI
let ca_pem = reqwest::get("http://127.0.0.1:9999/pki/ca.pem").await?.bytes().await?;
let client_cert = reqwest::Client::new()
.post("http://127.0.0.1:9999/pki/issue/client")
.json(&serde_json::json!({"common_name": "my-gui"}))
.send().await?.json::<serde_json::Value>().await?;
// Build TLS channel
let tls = ClientTlsConfig::new()
.ca_certificate(Certificate::from_pem(&ca_pem))
.identity(Identity::from_pem(
client_cert["cert_pem"].as_str().unwrap(),
client_cert["key_pem"].as_str().unwrap(),
));
let channel = Channel::from_static("https://localhost:50051")
.tls_config(tls)?
.connect().await?;
// Attach OIDC bearer token to every request
let token_str: MetadataValue<_> = format!("Bearer {}", token).parse()?;
let mut admin = holger::admin_service_client::AdminServiceClient::with_interceptor(
channel.clone(),
move |mut req: tonic::Request<()>| {
req.metadata_mut().insert("authorization", token_str.clone());
Ok(req)
},
);
// Now use it
let repos = admin.list_repositories(holger::Empty {}).await?.into_inner();
```
**Plain examples (no auth, local dev):**
```rust
// Rust (tonic)
use tonic::transport::Channel;
let channel = Channel::from_static("http://[::1]:50051").connect().await?;
let mut admin = holger::admin_service_client::AdminServiceClient::new(channel.clone());
let mut repo = holger::repository_service_client::RepositoryServiceClient::new(channel);
// List repositories
let repos = admin.list_repositories(holger::Empty {}).await?.into_inner();
for name in &repos.names {
println!("{}", name);
}
// Fetch artifact
let artifact = repo.fetch_artifact(holger::FetchRequest {
repository: "rust-prod".into(),
name: "serde".into(),
version: "1.0.219".into(),
namespace: String::new(),
}).await?.into_inner();
// Stream large artifact (server sends 64KB chunks)
let mut stream = repo.stream_artifact(holger::FetchRequest {
repository: "rust-prod".into(),
name: "tokio".into(),
version: "1.47.1".into(),
namespace: String::new(),
}).await?.into_inner();
while let Some(chunk) = stream.message().await? {
// chunk.data: Vec<u8>, chunk.offset: u64
}
```
```python
# Python (grpcio)
import grpc
import holger_pb2, holger_pb2_grpc
channel = grpc.insecure_channel('localhost:50051')
admin = holger_pb2_grpc.AdminServiceStub(channel)
repo = holger_pb2_grpc.RepositoryServiceStub(channel)
# List repos
response = admin.ListRepositories(holger_pb2.Empty())
print(response.names)
# Fetch artifact
artifact = repo.FetchArtifact(holger_pb2.FetchRequest(
repository="rust-prod", name="serde", version="1.0.219"))
```
```go
// Go (google.golang.org/grpc)
conn, _ := grpc.Dial("localhost:50051", grpc.WithInsecure())
admin := pb.NewAdminServiceClient(conn)
repo := pb.NewRepositoryServiceClient(conn)
repos, _ := admin.ListRepositories(ctx, &pb.Empty{})
artifact, _ := repo.FetchArtifact(ctx, &pb.FetchRequest{
Repository: "rust-prod", Name: "serde", Version: "1.0.219"})
```
### Option B: Rust function bindings (same process, zero overhead)
For Rust-native GUIs (egui, iced, slint, dioxus), embed holger directly:
```rust
use holger_server_lib::{read_ron_config, wire_holger, Holger};
use holger_traits::ArtifactId;
// Initialize once at app startup
let mut holger = read_ron_config("holger.ron").unwrap();
holger.instantiate_backends().unwrap();
wire_holger(&mut holger).unwrap();
// Use from any thread — no serialization, no network
let repos = holger.list_repositories(); // Vec<&str>
let id = ArtifactId { namespace: None, name: "serde".into(), version: "1.0.219".into() };
let data: Option<Vec<u8>> = holger.fetch("rust-prod", &id).unwrap();
// Upload
holger.put("rust-staging", &id, &bytes).unwrap();
```
### gRPC services reference
| `AdminService` | `Health` | Server health + uptime |
| `AdminService` | `ListRepositories` | All configured repo names |
| `RepositoryService` | `FetchArtifact` | Download artifact (single response) |
| `RepositoryService` | `StreamArtifact` | Download artifact (64KB streaming chunks) |
| `RepositoryService` | `ListArtifacts` | List artifacts in a repository |
| `RepositoryService` | `PutArtifact` | Upload artifact |
| `ArchiveService` | `ListArchiveFiles` | List files in znippy archive |
| `ArchiveService` | `ArchiveInfo` | Archive metadata (size, entry count) |
## Development (holger-mannequin)
The `holger-mannequin` crate provides a full mock environment for development and integration testing:
```
┌─────────────────────────────────────────────┐
│ holger-mannequin │
├──────────┬──────────┬───────────┬───────────┤
│ PKI │ OIDC │ Nexus │ Auth │
│ │ │ │ │
│ CA mgmt │ /token │ proxy-only│ Users │
│ Issue │ /userinfo│ prod (4) │ Roles │
│ Regen │ Discovery│ upload │ Validate │
└──────────┴──────────┴───────────┴───────────┘
│
▼
egui GUI (--gui flag)
Live system map + request logs
```
**Subsystems:**
| **PKI** | Certificate Authority | Issue/revoke certs, CA chain |
| **OIDC** | Keycloak/Azure AD | `/token`, `/userinfo`, `/.well-known/openid-configuration` |
| **Nexus** | Sonatype Nexus | 3 repos: `proxy-only`, `prod` (4 seeded crates), `upload` |
| **Auth** | Identity provider | 4 seeded users: admin, reader, writer, ci-agent |
**Usage modes:**
```bash
# As a server (dev environment)
cargo run -p holger-mannequin -- serve --gui --port 9999
# In-process (integration tests — no network needed)
let mannequin = holger_mannequin::Mannequin::new();
let token = mannequin.oidc.issue_token("admin");
let cert = mannequin.pki.issue_client_cert("ci-agent");
# CLI (cert generation)
cargo run -p holger-mannequin -- pki generate --output ./certs
```
**egui GUI features:**
- Live system topology map (repositories ↔ storage ↔ endpoints)
- Request traffic visualization (gRPC calls in real-time)
- OIDC token issuer for quick dev testing
- Mock Nexus browser with seeded artifacts
## Quick Start
```bash
# Build everything
PROTOC=/path/to/protoc cargo build --workspace
# Run tests
PROTOC=/path/to/protoc cargo test --workspace
# Start mannequin (dev dummy with mock PKI, OIDC, Nexus + GUI)
cargo run -p holger-mannequin -- serve --gui
# Start holger server (starts gRPC on configured port)
cargo run -p holger-server-cli -- start --config holger.ron
# Agent: see "Import / Export" section above for all transfer examples
```
## Immutable Storage Model
All `.artifact` files are immutable. Holger can optionally be configured to allow **live ingest** of artifacts not found in the current archive (DEV mode only).
| V1 | Initial .artifact import | Immutable | Bootstrap, base snapshot |
| V2 | .artifact + live ingest | Appendable | DEV: allow dynamic additions |
| V3 | Promoted .artifact only | Immutable | PROD: strict airgapped enforcement |
## Status
- ✅ RON config + graph wiring engine
- ✅ gRPC server (tonic, 3 services)
- ✅ Rust function bindings API (embedded mode)
- ✅ Rust crate serving (file + znippy backends)
- ✅ Maven artifact serving (znippy backend)
- ✅ Python pip serving (znippy backend, PEP 503 simple index)
- ✅ Python pip airgap agent (requirements.txt + uv.lock → znippy)
- ✅ Airgap agent (world → znippy → nexus/holger)
- ✅ OIDC authentication
- ✅ Nexus connector (list/download/upload)
- ✅ Znippy host-decompressors (lgz, ljar, linflate)
- ✅ Container distribution (Dockerfile + systemd)
- ✅ Dev mannequin with egui system map GUI
- 🚧 Blake3 content-addressable verification
- 🚧 Authorization layer (role-based access on repos)
- 🚧 Go/npm format support
- 🚧 Live ingest proxy mode (DEV)
- 🚧 Promote workflow (DEV → PROD)
- 🚧 egui admin client for holger-server
- 🚧 Audit log (who accessed what, when)
## Testing
### Test Levels
Holger has three test levels:
1. **Unit / light integration** — no network, no containers, runs in seconds
2. **Synthetic benchmarks** — auto-generated fake data, no network, measures throughput regression
3. **Heavy / container tests** — spins up a real **Sonatype Nexus 3 (v3.72.0)** container via `testcontainers` crate + Podman, uploads real artifact archives, benchmarks Holger vs Nexus
### Unit / Light Integration Tests
These use `holger-mannequin` as an in-process mock (Nexus API + OIDC + mTLS + PKI). No external services needed.
```bash
# All quick tests (no network, no containers):
cargo test --workspace
# Mannequin (mock Nexus/OIDC/mTLS) tests:
cargo test -p holger-agent-cli --test integration_holger -- --nocapture
cargo test -p holger-agent-cli --test integration_pip -- --nocapture
# tar.zst round-trip tests (mannequin mock, no network):
cargo test -p holger-agent-cli --test integration_tar_zstd -- --nocapture
```
### Heavy Tests (Real Nexus Container)
These tests start a **real Sonatype Nexus 3** container using the [`testcontainers`](https://crates.io/crates/testcontainers) Rust crate. The container lifecycle is fully managed by the test — it starts Nexus, waits for readiness, creates repositories, runs the benchmark, and tears down automatically.
**How it works:**
1. `testcontainers` pulls `sonatype/nexus3:3.72.0` via Podman (or Docker)
2. Container starts with port 8081 exposed on a random host port
3. Test waits up to 180s for Nexus REST API (`/service/rest/v1/status`) to respond
4. Reads initial admin password from container (`/nexus-data/admin.password`)
5. Changes password to `admin123`, creates raw hosted repositories
6. Uploads cached znippy archives → benchmarks upload throughput (MB/s, files/s)
7. Downloads all artifacts back → benchmarks download throughput
8. Runs retrieval bombardment (multi-threaded random GETs) on Nexus, then same dataset on Holger znippy for comparison
9. Container is stopped and removed automatically
**The tail-latency test** additionally constrains Nexus to a 512 MB JVM heap (`-Xms512m -Xmx512m -XX:+UseG1GC`) to provoke GC pauses, measuring p99/p999 latency vs Holger's zero-GC constant-time reads.
**Running heavy tests:**
```bash
# Prerequisite: enable Podman socket (rootless)
systemctl --user start podman.socket
# Set DOCKER_HOST so testcontainers finds the Podman socket
export DOCKER_HOST=unix:///run/user/1000/podman/podman.sock
# Nexus load test (uploads ~14 GB, benchmarks Holger vs Nexus)
cargo test --release -p holger-agent-cli \
--test integration_nexus_load -- --ignored --nocapture
# Tail-latency / GC pressure test (60s bombardment, p99/p999 comparison)
cargo test --release -p holger-agent-cli \
--test integration_latency_tail -- --ignored --nocapture
```
**Or run everything via xtask:**
```bash
cargo xtask --heavy
```
### Synthetic Benchmarks (No Network)
The `bench_znippy_throughput` test generates 500K fake crates/JARs/wheels in `/tmp/holger_bench_throughput/` — auto-created on first run, cached for subsequent runs. Measures raw `get_file()` and HTTP handler ops/s for all 3 artifact types (single-thread and multi-thread).
```bash
cargo test --release -p holger-rust-znippy-repository \
--test bench_znippy_throughput -- --ignored --nocapture
```
### Integration Test Summary
| `integration_holger` | ✗ | ✗ | Mannequin mock: OIDC auth, mTLS auth, upload/download, cert issuance |
| `integration_pip` | ✗ | ✗ | Mannequin mock: Python package resolution and PEP 503 index |
| `integration_tar_zstd` | ✗ | ✗ | tar.zst create → verify → push round-trip (mannequin mock + OIDC) |
| `bench_znippy_throughput` | ✗ | ✗ | 500K synthetic artifacts, throughput regression gate |
| `integration_znippy_serve` | ✗ | **first run** | Downloads real crates/JARs/wheels from internet, packs znippy, verifies |
| `integration_nexus_load` | **Nexus 3** | ✗¹ | Real Nexus container, upload/download benchmark, bombardment vs Holger |
| `integration_latency_tail` | **Nexus 3** | ✗¹ | Nexus with 512MB heap, p99/p999 latency under GC pressure vs Holger |
¹ Container image must be available locally or pullable. Test data uses pre-cached archives.
### Test Cache
Heavy tests cache downloaded artifacts at `/home/rickard/work/holger_tests/`:
```
crates/ — 9,162 .crate files (4.2 GB) from crates.io
jars/ — 4,730 .jar files (5.0 GB) from Maven Central
wheels/ — 541 .whl files (5.0 GB) from PyPI
crates_archive.znippy — packed with znippy 0.7.4
jars_archive.znippy — packed with znippy 0.7.4
pip_archive.znippy — packed with znippy 0.7.4
```
**If files don't exist, tests create them automatically** — they download from crates.io / Maven Central / PyPI on first run. Subsequent runs reuse the cached archives. If znippy version changes, delete `*.znippy` and re-run to rebuild.
### Requirements
| **Podman** (or Docker) | Nexus container tests | `systemctl --user start podman.socket` |
| **Network access** | First run of `integration_znippy_serve` | Downloads ~14 GB from crates.io, Maven, PyPI |
| **~15 GB disk** | Test cache | `/home/rickard/work/holger_tests/` |
| **`sonatype/nexus3:3.72.0` image** | Heavy tests | Auto-pulled by testcontainers if not present |
## Release
### Pre-release gate: `cargo xtask --heavy`
**You MUST run `cargo xtask --heavy` before publishing any crate.** This is the full release gate — it runs every test level including a real Nexus 3 container benchmark. If it passes, the code is safe to publish. If it fails, do not release.
```bash
# Prerequisites
systemctl --user start podman.socket
export DOCKER_HOST=unix:///run/user/1000/podman/podman.sock
# THE release command — runs everything:
cargo xtask --heavy
```
**What `cargo xtask --heavy` does (in order):**
| 1 | `cargo test --workspace` | Any unit/integration test fails |
| 2 | `integration_tar_zstd` | tar.zst round-trip broken |
| 3 | `bench_znippy_throughput` (500K synthetic) | >20% throughput regression vs `bench_history.json` |
| 4 | Records benchmark to `bench_history.json` | — |
| 5 | **Starts Sonatype Nexus 3 container** | Container fails to start |
| 6 | Uploads all cached archives, bombardment-tests Nexus vs Holger | Upload/download failures |
| 7 | Tail-latency / GC pressure test (p99/p999) | Latency regression |
`cargo xtask` (without `--heavy`) skips steps 5–7 — use it for quick local iteration, but **never for release**.
### Bump versions and publish
```bash
# 1. Bump version in all Cargo.toml files (workspace members share versions)
# holger-traits, holger-*-repository, holger-server-*, holger-agent-*
# 2. Run the FULL release gate (mandatory — do not skip)
cargo xtask --heavy
# 3. Publish crates in dependency order (traits first, then backends, then server/agent)
cargo publish -p holger-traits
cargo publish -p holger-rust-file-repository
cargo publish -p holger-rust-znippy-repository
cargo publish -p holger-maven-znippy-repository
cargo publish -p holger-python-znippy-repository
cargo publish -p holger-nexus-connector
cargo publish -p holger-world-connector
cargo publish -p holger-server-lib
cargo publish -p holger-server-cli
cargo publish -p holger-agent-lib
cargo publish -p holger-agent-cli
cargo publish -p holger-mannequin
# 4. Tag and push
git tag v0.X.Y
git push && git push --tags
```
### Publish order matters
Crates.io requires dependencies to be published first:
```
holger-traits (no internal deps)
├── holger-rust-file-repository
├── holger-rust-znippy-repository
├── holger-maven-znippy-repository
├── holger-python-znippy-repository
├── holger-nexus-connector
├── holger-world-connector
├── holger-server-lib (depends on traits + all repos)
│ └── holger-server-cli
├── holger-agent-lib (depends on traits + connectors)
│ └── holger-agent-cli
└── holger-mannequin
```
## Fan art
<img width="1024" height="1536" alt="holger" src="https://github.com/user-attachments/assets/cbc60639-0025-4437-a088-c41f8deded2e" />
*"Allfadern ser varje bit"*