createrepo_rs 0.1.6

๐Ÿฆ€ Pure Rust implementation of createrepo_c โ€” generates RPM repository metadata (repodata). Drop-in replacement with identical output, zero FFI.
Documentation

createrepo_rs ๐Ÿฆ€

Rust License CI

100% pure Rust implementation of createrepo_c โ€” generates RPM repository metadata (repodata).

Drop-in replacement for the C version with identical output, zero FFI, and 3.5MB static binary.

๐ŸŽฏ Why createrepo_rs?

createrepo_c (C) createrepo_rs (Rust)
Output compatibility โœ… โœ… 100% byte-compatible
CLI arguments 55 52/55 (3 hard: split, zchunk)
Dependencies librpm, libxml2, glib2, zchunk... zero FFI โ€” pure Rust crates
Binary size ~200KB + shared libs 3.5MB static (musl)
Memory safety โŒ manual malloc/free โœ… borrow checker
Cross-compile painful cargo zigbuild
dnf compatible โœ… โœ… verified

๐Ÿš€ Quick Start

# Install
cargo install --git https://github.com/jamesarch/createrepo_rs createrepo_rs

# Or clone and build
git clone https://github.com/jamesarch/createrepo_rs
cd createrepo_rs
cargo build --release

# Generate metadata for a directory of RPMs
createrepo_rs /path/to/rpms/

# Or with options
createrepo_rs /path/to/rpms/ \
  --compress-type=zstd \
  --no-database \
  --workers=8 \
  --simple-md-filenames

๐Ÿ“ฆ Features

Core

  • โœ… primary.xml, filelists.xml, other.xml generation
  • โœ… repomd.xml with correct checksums
  • โœ… SQLite database generation (--no-database to disable)
  • โœ… Multi-threaded RPM parsing (configurable --workers)
  • โœ… Changlog extraction
  • โœ… Provides/Requires/Conflicts/Obsoletes/Suggests/Recommends
  • โœ… Weak dependencies (Supplements/Enhances)
  • โœ… File type detection (dir/symlink/regular)
  • โœ… Graceful Ctrl+C handling
  • โœ… --update incremental mode

Compression

  • โœ… gzip (default)
  • โœ… bzip2
  • โœ… zstd
  • โœ… xz
  • โœ… Configurable level

CLI (52/55 params โ€” 100% of commonly used)

createrepo_rs --help   # Full list

Notable:

Flag Description
--workers=N Parallel RPM parsing threads
--compress-type=zstd Compression algorithm
--checksum=sha512 Hash algorithm for metadata
--revision=12345 Custom repository revision
--baseurl=https://... Base URL for repository
--simple-md-filenames Clean filenames (no hash prefix)
--unique-md-filenames Hash-prefixed filenames (default)
--location-prefix=subdir/ Prefix before location href
--cut-dirs=2 Strip N directory components
--repomd-checksum=sha512 Checksum type for repomd.xml
--general-compress-type=xz Separate compression for XML files
--duplicated-nevra=error Error on duplicate packages
--retain-old-md-by-age=30d Auto-cleanup old metadata
--compatibility Max compatibility mode (gzip + simple filenames)
-q / -v Quiet / Verbose output
--no-pretty Compact XML without indentation

๐Ÿ“Š Performance

Tested on macOS (M1 Pro, 16GB RAM), createrepo_c run in Docker (Fedora 40). All tests use --compress-type=zstd --no-database unless noted.

Full Generation Time (lower is better)

RPMs createrepo_rs (4w) createrepo_c (4w) Speedup
10 0.01s 0.10s 10x
100 0.01s 0.11s 11x
500 0.03s 0.12s 4x
1000 0.05s 0.20sยน 4xยน

ยน Estimated: createrepo_c ~O(n) for RPM parsing

Worker Scaling (500 RPMs)

Workers createrepo_rs createrepo_c
1 0.05s 0.25s
4 0.03s 0.12s
8 0.03s โ€”

Compression Comparison (500 RPMs, 4 workers)

Algorithm Time Output Size Best For
zstd 0.03s 20KB Speed + size (default)
gzip 0.03s 40KB Max compatibility
xz 0.11s 20KB Smallest output
bzip2 0.14s 24KB Legacy support

Incremental Update (--update --skip-stat)

RPMs createrepo_rs Notes
100 0.01s Cache hit, skips re-parsing
500 0.03s Only processes changed packages
1000 0.05s Near-constant time for unchanged repos

Optimizations Applied (v0.1.4)

Optimization Impact
LTO + opt-level=3 + panic=abort ~5-10% runtime, ~7% binary size
Multi-worker deadlock fix Enables 2+ workers (was broken)
Arc<Package> update cache Avoids full clone on --update hits
SQLite batch transactions 10-50x filelists insert speed
Redundant stat() eliminated -2 syscalls per XML file
XML Vec::with_capacity Avoids reallocation during dump
SHA buffer 8KBโ†’64KB Fewer read() syscalls
From<DependencyInfo> impl -114 lines duplicated code

Primary XML generation is byte-identical to the createrepo_c C version.

๐Ÿณ Docker Test

cd docker-test
docker compose up -d
docker logs -f createrepo_client

Output:

โœ…โœ…โœ… Success! createrepo_rs generated metadata recognized and downloaded by dnf!

๐Ÿ”จ Building

# Debug build
cargo build

# Release build
cargo build --release

# Linux static binary (musl) โ€” requires Zig
# Install: https://ziglang.org/download/
# Then: cargo install cargo-zigbuild
cargo zigbuild --release --target x86_64-unknown-linux-musl

# Cross-compile for ARM
cargo zigbuild --release --target aarch64-unknown-linux-musl

๐Ÿ—๏ธ Architecture

createrepo_rs/
โ”œโ”€โ”€ lib.rs           # Library root + prelude re-exports
โ”œโ”€โ”€ src/main.rs      # Binary entry point, CLI handling, orchestration
โ”œโ”€โ”€ cli/mod.rs       # Clap argument parser (52/55 params)
โ”œโ”€โ”€ pool/mod.rs      # Parallel worker pool
โ”œโ”€โ”€ rpm/mod.rs       # RPM header parsing via `rpm` crate
โ”œโ”€โ”€ types/mod.rs     # Core types: Package, Dependency, RepomdRecord
โ”œโ”€โ”€ compression/     # gzip, bzip2, zstd, xz
โ”œโ”€โ”€ db/mod.rs        # SQLite database generation
โ”œโ”€โ”€ xml/
โ”‚   โ”œโ”€โ”€ error.rs     # XML error types
โ”‚   โ”œโ”€โ”€ mod.rs       # XML helpers
โ”‚   โ”œโ”€โ”€ parse.rs     # XML parsing
โ”‚   โ””โ”€โ”€ repomd.rs    # repomd.xml generation
โ””โ”€โ”€ walk/mod.rs      # Directory traversal

๐Ÿ“š Library Usage

createrepo_rs can also be used as a library:

[dependencies]
createrepo_rs = { git = "https://github.com/jamesarch/createrepo_rs" }
use std::path::Path;
use createrepo_rs::prelude::*;

let mut reader = RpmReader::open(Path::new("my-package.rpm")).unwrap();
let pkg = reader.read_package().unwrap();
println!("{} {}-{}", pkg.name, pkg.version, pkg.release);

The [prelude] module re-exports commonly used types and functions:

  • Compression: gzip_compress, gzip_decompress, zstd_compress, zstd_decompress, etc.
  • Types: Package, Dependency, ChecksumType, CompressionType, ChangelogEntry
  • RPM: RpmReader, parse_dep_version
  • DB: RepomdDb, DbError
  • Pool: WorkerPool, Job, ProcessingResult
  • XML: XmlError
  • Walk: DirectoryWalker, WalkError

๐Ÿ“ License

GPL-2.0-or-later โ€” same as createrepo_c.

๐Ÿ™ Credits

Original C implementation: rpm-software-management/createrepo_c

Built with: