Holger guards your artifacts at rest

Holger, immutable artifact repository in pure Rust!

Capabilities:
Airgap: transfer from internet ,nexus, artifacty to immutable znippy archive.
Versioned: Atomic transactions
Extreme perfomance due to Apache Arrow and its zero copy, multicore
Zero depencies to OS , shell,python. 100% rust. Compiled depencies like C based decompressors rewritten in RUST!

Embeddable on os free devices

Artifact retrieval throughput

Random-access retrieval from artifact archives — Holger in-process (znippy 0.7.4, 2 MB RSS) vs Sonatype Nexus 3 (v3.72, container, 32 cores, 16 GB heap, 400 Jetty threads).
Same machine, same 32 cores. Nexus container stopped before Holger phase for clean isolation.

Holger is 13,724× faster than Nexus at peak (16 clients = Nexus's sweet spot).
Single-threaded Holger (442K ops/s) outperforms 32-client Nexus (590 ops/s) by 750×.
Nexus degrades past 16 clients despite 400 Jetty threads and 16 GB heap.
Holger uses 2 MB RSS vs Nexus's 16 GB heap — 8,000× less memory.

Raw archive retrieval — in-process get_file (znippy 0.7.4, 32-core)

Direct random-access extraction from a 50,000-artifact znippy archive — no HTTP, no network. This is the path cargo xtask benchmarks and regression-gates.

Backed by znippy 0.7.4's N-worker positioned-I/O read path: each get_file is one pread from a shared fd plus either a zero-copy skip (pre-compressed .crate/.jar) or an openzl decompress into a reused buffer — fully parallel across cores, no shared reader thread.

Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                          Holger Ecosystem                                │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│  ┌─────────────────┐     ┌──────────────────┐     ┌────────────────┐  │
│  │  holger-agent    │────▶│  holger-server    │────▶│  Storage       │  │
│  │  (CLI + daemon)  │     │  (gRPC / Rust fn) │     │  (.znippy)     │  │
│  │                  │     │                   │     │                │  │
│  │  --auth oidc     │     │  OIDC validation  │     │  Immutable     │  │
│  └─────────────────┘     └──────────────────┘     │  Blake3 CAS    │  │
│           │                        │               └────────────────┘  │
│           │                        │                                    │
│           ▼                        ▼                                    │
│  ┌─────────────────┐     ┌──────────────────┐                         │
│  │  Nexus/Registry  │     │  cargo/mvn        │                         │
│  │  (connector)     │     │  (clients)        │                         │
│  └─────────────────┘     └──────────────────┘                         │
│                                                                         │
└─────────────────────────────────────────────────────────────────────────┘

Access Modes

Holger exposes two access modes — no HTTP/REST:

1. Rust Function Bindings (embedded mode)

Use holger-server-lib directly in your Rust application:

use holger_server_lib::{read_ron_config, wire_holger};
use holger_traits::ArtifactId;

let mut holger = read_ron_config("holger.ron").unwrap();
holger.instantiate_backends().unwrap();
wire_holger(&mut holger).unwrap();

// Fetch artifact
let id = ArtifactId { namespace: None, name: "serde".into(), version: "1.0.219".into() };
let data = holger.fetch("rust-prod", &id).unwrap();

// List repositories
let repos = holger.list_repositories(); // ["rust-prod", "maven-staging"]

// Put artifact (write-enabled repos)
holger.put("rust-staging", &id, &bytes).unwrap();

2. gRPC (network mode)

Configure exposed_endpoints in RON → holger starts a tonic gRPC server automatically.

Services (defined in proto/holger.proto):

3. Apache Arrow Flight (batch mode) — planned

Arrow Flight as a universal artifact service: batch-fetch multiple artifacts (or files within artifacts) in a single call with zero-copy columnar transport.

Why Flight over raw gRPC for batch:

• znippy archives are already Arrow IPC → Flight serves them with near-zero serialization cost
• Columnar format means fetching 1000 small crates is one contiguous memory transfer, not 1000 RPCs
• uv/pip/cargo resolve+fetch could use a single Flight batch call instead of N HTTP GETs

Connect with grpcurl:

# Health check
grpcurl -plaintext localhost:50051 holger.AdminService/Health

# List repositories
grpcurl -plaintext localhost:50051 holger.AdminService/ListRepositories

# Fetch artifact
grpcurl -plaintext -d '{"repository":"rust-prod","name":"serde","version":"1.0.219"}' \
  localhost:50051 holger.RepositoryService/FetchArtifact

# Stream large artifact (chunked 64KB)
grpcurl -plaintext -d '{"repository":"rust-prod","name":"tokio","version":"1.47.1"}' \
  localhost:50051 holger.RepositoryService/StreamArtifact

Crates

Configuration (RON)

Holger uses RON for configuration — it can express the graph topology that TOML/YAML cannot.

Rust crate repository (file-backed)

(
    repositories: [
        (
            ron_name: "rust-prod",
            ron_repo_type: "rust",
            ron_upstreams: [],
            ron_in: None,
            ron_out: Some((
                ron_storage_endpoint: "local-storage",
                ron_exposed_endpoint: "grpc",
            )),
        ),
    ],
    exposed_endpoints: [
        (
            ron_name: "grpc",
            ron_url: "0.0.0.0:50051",
        ),
    ],
    storage_endpoints: [
        (
            ron_name: "local-storage",
            ron_storage_type: "rocksdb",
            ron_path: "/var/lib/holger/data",
        ),
    ],
)

Rust crate repository (znippy archive)

(
    repositories: [
        (
            ron_name: "rust-airgap",
            ron_repo_type: "rust",
            ron_archive_path: Some("/var/lib/holger/rust-airgap.znippy"),
            ron_upstreams: [],
            ron_in: None,
            ron_out: Some((
                ron_storage_endpoint: "znippy-storage",
                ron_exposed_endpoint: "grpc",
            )),
        ),
    ],
    exposed_endpoints: [
        (
            ron_name: "grpc",
            ron_url: "0.0.0.0:50051",
        ),
    ],
    storage_endpoints: [
        (
            ron_name: "znippy-storage",
            ron_storage_type: "znippy",
            ron_path: "/var/lib/holger/rust-airgap",
        ),
    ],
)

Python pip repository (znippy archive, PEP 503 simple index)

(
    repositories: [
        (
            ron_name: "pip-internal",
            ron_repo_type: "pip",
            ron_archive_path: Some("/var/lib/holger/pip.znippy"),
            ron_upstreams: [],
            ron_in: None,
            ron_out: Some((
                ron_storage_endpoint: "pip-storage",
                ron_exposed_endpoint: "grpc",
            )),
        ),
    ],
    exposed_endpoints: [
        (
            ron_name: "grpc",
            ron_url: "0.0.0.0:50051",
        ),
    ],
    storage_endpoints: [
        (
            ron_name: "pip-storage",
            ron_storage_type: "znippy",
            ron_path: "/var/lib/holger/pip",
        ),
    ],
)

The server exposes a PEP 503 simple index at /{repo}/simple/ — point pip or uv directly at it:

pip install --index-url https://holger.internal/pip-internal/simple/ requests
uv pip install --index-url https://holger.internal/pip-internal/simple/ requests

Maven repository (znippy archive)

(
    repositories: [
        (
            ron_name: "maven-internal",
            ron_repo_type: "maven3",
            ron_archive_path: Some("/var/lib/holger/maven.znippy"),
            ron_upstreams: [],
            ron_in: None,
            ron_out: Some((
                ron_storage_endpoint: "maven-storage",
                ron_exposed_endpoint: "grpc",
            )),
        ),
    ],
    exposed_endpoints: [
        (
            ron_name: "grpc",
            ron_url: "0.0.0.0:50051",
        ),
    ],
    storage_endpoints: [
        (
            ron_name: "maven-storage",
            ron_storage_type: "znippy",
            ron_path: "/var/lib/holger/maven",
        ),
    ],
)

Authentication

The agent CLI uses --auth oidc when talking to Holger server. Legacy Nexus connections still use --username/--password (basic auth via connector).

Airgap Workflow

Internet side                    │  Airgapped side
                                 │
  crates.io ──┐                  │
  Maven ──────┤                  │
  PyPI ───────┤──▶ holger-agent  │  holger-agent ──▶ holger-server
  npm ────────┘    --to znippy   │  --from znippy      (gRPC / Rust fn)
                       │         │       ▲
                       ▼         │       │
                   .znippy file ─┼───────┘
                   (USB/media)   │

Performance

Holger uses znippy archives (.znippy) as storage. A .znippy file is a valid Arrow IPC File — queryable directly by pyarrow/polars:

# pyarrow
import pyarrow.ipc
reader = pyarrow.ipc.open_file("archive.znippy")
table = reader.read_all()
print(table.schema)
# relative_path: string, compressed: bool, uncompressed_size: uint64, chunks: list<...>

# polars
import polars as pl
df = pl.read_ipc("archive.znippy")
df.select("relative_path", "uncompressed_size", "compressed").head(10)

Znippy archive benchmarks (v0.7.4 — io_uring + Gatling pipeline)

Packing source files into .znippy and extracting back. Files already compressed (.gz, .jar, .zip, etc.) are stored as-is (skipped). .crate files are gzipped tarballs — znippy compresses the source contents inside, not the .crate blob itself:

Note: "Pack" = archiving + compression into .znippy. "Unpack" = decompression + extraction. Stream packer (in-memory/HTTP) numbers above; slot packer (io_uring disk reads) hits 678 MB/s on 10k×10KB synthetic. Already-compressed data (.jar, .crate) passes through at full I/O speed without re-compression.

Host decompressors

The host-decompressors feature replaces miniz_oxide with custom parallel decompression:

Single-core path is 3.7× faster than miniz_oxide. Multi-core scales linearly.

GUI Builders

Holger has no built-in web UI. Instead, build your own GUI using either access mode:

Option A: gRPC client (any language)

Use the proto definition (proto/holger.proto) to generate a client in any language.

Getting OIDC tokens and TLS certs from mannequin (for GUI dev):

Start mannequin first — it provides the complete auth infrastructure your GUI needs:

# 1. Start mannequin (mock PKI + OIDC + Nexus)
cargo run -p holger-mannequin -- serve --gui --port 9999

# 2. Get an OIDC token (accepts any credentials — it's a dev dummy)
TOKEN=$(curl -s -X POST http://127.0.0.1:9999/token \
  -d 'grant_type=password&username=admin&password=anything&client_id=my-gui' \
  | jq -r .access_token)

echo "Bearer token: $TOKEN"

# 3. Get a TLS client certificate (for mTLS to holger-server)
curl -s -X POST http://127.0.0.1:9999/pki/issue/client \
  -H "Content-Type: application/json" \
  -d '{"common_name": "my-gui-client"}' \
  | jq -r '.cert_pem' > client.pem

curl -s -X POST http://127.0.0.1:9999/pki/issue/client \
  -H "Content-Type: application/json" \
  -d '{"common_name": "my-gui-client"}' \
  | jq -r '.key_pem' > client-key.pem

# 4. Get the CA cert (to verify holger-server's TLS)
curl -s http://127.0.0.1:9999/pki/ca.pem > ca.pem

# 5. Connect to holger-server with OIDC auth
grpcurl -H "Authorization: Bearer $TOKEN" \
  -cacert ca.pem -cert client.pem -key client-key.pem \
  localhost:50051 holger.AdminService/ListRepositories

In your GUI code (Rust + tonic):

use tonic::transport::{Channel, ClientTlsConfig, Certificate, Identity};
use tonic::metadata::MetadataValue;

// Get token from mannequin OIDC endpoint
let token_resp: serde_json::Value = reqwest::Client::new()
    .post("http://127.0.0.1:9999/token")
    .form(&[("grant_type", "password"), ("username", "admin"),
            ("password", "dev"), ("client_id", "my-gui")])
    .send().await?.json().await?;
let token = token_resp["access_token"].as_str().unwrap();

// Get CA + client cert from mannequin PKI
let ca_pem = reqwest::get("http://127.0.0.1:9999/pki/ca.pem").await?.bytes().await?;
let client_cert = reqwest::Client::new()
    .post("http://127.0.0.1:9999/pki/issue/client")
    .json(&serde_json::json!({"common_name": "my-gui"}))
    .send().await?.json::<serde_json::Value>().await?;

// Build TLS channel
let tls = ClientTlsConfig::new()
    .ca_certificate(Certificate::from_pem(&ca_pem))
    .identity(Identity::from_pem(
        client_cert["cert_pem"].as_str().unwrap(),
        client_cert["key_pem"].as_str().unwrap(),
    ));

let channel = Channel::from_static("https://localhost:50051")
    .tls_config(tls)?
    .connect().await?;

// Attach OIDC bearer token to every request
let token_str: MetadataValue<_> = format!("Bearer {}", token).parse()?;
let mut admin = holger::admin_service_client::AdminServiceClient::with_interceptor(
    channel.clone(),
    move |mut req: tonic::Request<()>| {
        req.metadata_mut().insert("authorization", token_str.clone());
        Ok(req)
    },
);

// Now use it
let repos = admin.list_repositories(holger::Empty {}).await?.into_inner();

Plain examples (no auth, local dev):

// Rust (tonic)
use tonic::transport::Channel;

let channel = Channel::from_static("http://[::1]:50051").connect().await?;
let mut admin = holger::admin_service_client::AdminServiceClient::new(channel.clone());
let mut repo = holger::repository_service_client::RepositoryServiceClient::new(channel);

// List repositories
let repos = admin.list_repositories(holger::Empty {}).await?.into_inner();
for name in &repos.names {
    println!("{}", name);
}

// Fetch artifact
let artifact = repo.fetch_artifact(holger::FetchRequest {
    repository: "rust-prod".into(),
    name: "serde".into(),
    version: "1.0.219".into(),
    namespace: String::new(),
}).await?.into_inner();

// Stream large artifact (server sends 64KB chunks)
let mut stream = repo.stream_artifact(holger::FetchRequest {
    repository: "rust-prod".into(),
    name: "tokio".into(),
    version: "1.47.1".into(),
    namespace: String::new(),
}).await?.into_inner();

while let Some(chunk) = stream.message().await? {
    // chunk.data: Vec<u8>, chunk.offset: u64
}

# Python (grpcio)
import grpc
import holger_pb2, holger_pb2_grpc

channel = grpc.insecure_channel('localhost:50051')
admin = holger_pb2_grpc.AdminServiceStub(channel)
repo = holger_pb2_grpc.RepositoryServiceStub(channel)

# List repos
response = admin.ListRepositories(holger_pb2.Empty())
print(response.names)

# Fetch artifact
artifact = repo.FetchArtifact(holger_pb2.FetchRequest(
    repository="rust-prod", name="serde", version="1.0.219"))

// Go (google.golang.org/grpc)
conn, _ := grpc.Dial("localhost:50051", grpc.WithInsecure())
admin := pb.NewAdminServiceClient(conn)
repo := pb.NewRepositoryServiceClient(conn)

repos, _ := admin.ListRepositories(ctx, &pb.Empty{})
artifact, _ := repo.FetchArtifact(ctx, &pb.FetchRequest{
    Repository: "rust-prod", Name: "serde", Version: "1.0.219"})

Option B: Rust function bindings (same process, zero overhead)

For Rust-native GUIs (egui, iced, slint, dioxus), embed holger directly:

use holger_server_lib::{read_ron_config, wire_holger, Holger};
use holger_traits::ArtifactId;

// Initialize once at app startup
let mut holger = read_ron_config("holger.ron").unwrap();
holger.instantiate_backends().unwrap();
wire_holger(&mut holger).unwrap();

// Use from any thread — no serialization, no network
let repos = holger.list_repositories();  // Vec<&str>

let id = ArtifactId { namespace: None, name: "serde".into(), version: "1.0.219".into() };
let data: Option<Vec<u8>> = holger.fetch("rust-prod", &id).unwrap();

// Upload
holger.put("rust-staging", &id, &bytes).unwrap();

gRPC services reference

Development (holger-mannequin)

The holger-mannequin crate provides a full mock environment for development and integration testing:

┌─────────────────────────────────────────────┐
│              holger-mannequin                 │
├──────────┬──────────┬───────────┬───────────┤
│   PKI    │   OIDC   │   Nexus   │   Auth    │
│          │          │           │           │
│ CA mgmt  │ /token   │ proxy-only│ Users     │
│ Issue    │ /userinfo│ prod (4)  │ Roles     │
│ Regen    │ Discovery│ upload    │ Validate  │
└──────────┴──────────┴───────────┴───────────┘
         │
         ▼
    egui GUI (--gui flag)
    Live system map + request logs

Subsystems:

Usage modes:

# As a server (dev environment)
cargo run -p holger-mannequin -- serve --gui --port 9999

# In-process (integration tests — no network needed)
let mannequin = holger_mannequin::Mannequin::new();
let token = mannequin.oidc.issue_token("admin");
let cert = mannequin.pki.issue_client_cert("ci-agent");

# CLI (cert generation)
cargo run -p holger-mannequin -- pki generate --output ./certs

egui GUI features:

• Live system topology map (repositories ↔ storage ↔ endpoints)
• Request traffic visualization (gRPC calls in real-time)
• OIDC token issuer for quick dev testing
• Mock Nexus browser with seeded artifacts

Quick Start

# Build everything
PROTOC=/path/to/protoc cargo build --workspace

# Run tests
PROTOC=/path/to/protoc cargo test --workspace

# Start mannequin (dev dummy with mock PKI, OIDC, Nexus + GUI)
cargo run -p holger-mannequin -- serve --gui

# Agent: download from crates.io → znippy archive
cargo run -p holger-agent -- --from world --to znippy --lockfile Cargo.lock --output ./airgap.znippy

# Agent: download Python packages (requirements.txt) → znippy archive
cargo run -p holger-agent -- --from world --lockfile-kind requirements-txt \
  --lockfile requirements.txt --to znippy --output ./pip-airgap.znippy

# Agent: download Python packages (uv.lock, URLs already embedded) → znippy archive
cargo run -p holger-agent -- --from world --lockfile-kind uv-lock \
  --lockfile uv.lock --to znippy --output ./pip-airgap.znippy

# Agent: push to Holger with OIDC auth
cargo run -p holger-agent -- \
  --from znippy --to holger \
  --input ./airgap.znippy \
  --holger-url grpc://holger.internal:50051 \
  --auth oidc --token-url https://keycloak/token --client-id ci-agent \
  --repository rust-prod

# Start holger server (starts gRPC on configured port)
cargo run -p holger-server-cli -- start --config holger.ron

Immutable Storage Model

All .artifact files are immutable. Holger can optionally be configured to allow live ingest of artifacts not found in the current archive (DEV mode only).

Status

• ✅ RON config + graph wiring engine
• ✅ gRPC server (tonic, 3 services)
• ✅ Rust function bindings API (embedded mode)
• ✅ Rust crate serving (file + znippy backends)
• ✅ Maven artifact serving (znippy backend)
• ✅ Python pip serving (znippy backend, PEP 503 simple index)
• ✅ Python pip airgap agent (requirements.txt + uv.lock → znippy)
• ✅ Airgap agent (world → znippy → nexus/holger)
• ✅ OIDC authentication
• ✅ Nexus connector (list/download/upload)
• ✅ Znippy host-decompressors (lgz, ljar, linflate)
• ✅ Container distribution (Dockerfile + systemd)
• ✅ Dev mannequin with egui system map GUI
• 🚧 Blake3 content-addressable verification
• 🚧 Authorization layer (role-based access on repos)
• 🚧 Go/npm format support
• 🚧 Live ingest proxy mode (DEV)
• 🚧 Promote workflow (DEV → PROD)
• 🚧 egui admin client for holger-server
• 🚧 Audit log (who accessed what, when)

Testing

Integration & Benchmark Tests

All heavy tests are #[ignore] — run explicitly with -- --ignored --nocapture.

Test Cache

Heavy tests cache downloaded artifacts at /home/rickard/work/holger_tests/:

crates/          — 9,162 .crate files (4.2 GB) from crates.io
jars/            — 4,730 .jar files (5.0 GB) from Maven Central
wheels/          — 541 .whl files (5.0 GB) from PyPI
crates_archive.znippy  — packed with znippy 0.7.2
jars_archive.znippy    — packed with znippy 0.7.2
pip_archive.znippy     — packed with znippy 0.7.2

If znippy version changes, delete *.znippy and re-run integration_znippy_serve to rebuild.

Unit / Light Integration Tests

# All quick tests (no network, no containers):
cargo test --workspace

# Mannequin (mock Nexus) tests:
cargo test -p holger-agent-cli --test integration_holger -- --nocapture
cargo test -p holger-agent-cli --test integration_pip -- --nocapture

Requirements

• Podman (or Docker) for Nexus container tests — enable socket: systemctl --user start podman.socket
• Network access for downloading real artifacts from crates.io, Maven Central, PyPI
• ~15 GB disk for test cache

Fan art

"Allfadern ser varje bit"