Hardware-isolated code execution and judge engine written in Rust, powered by Firecracker microVMs.
"True hardware microVM isolation instead of shared-kernel container sandboxes. Self-hosted, one Rust binary."
Each execution runs inside an ephemeral Linux KVM microVM with a read-only rootfs, communicates strictly over vsock, and is destroyed immediately after execution.
Table of Contents
- Why Cratera
- Why Not Containers?
- Architecture
- Quick Setup Guide
- Interactive Command Center
- API Usage & Examples
- Verdict Codes
- Multi-Language Configuration
- Configuration Reference
- Systemd Service & Deployment
- Recommended Ingress: Cloudflare Zero Trust & Private Networks
- Development & Verification
- Troubleshooting & FAQ
- Community & Discussion
- Governance & Security
- License
Why Cratera
- Untrusted code, macro expansions, and system calls are isolated from the host kernel by a hardware virtualization boundary (Intel VT-x / AMD-V), not shared namespaces or cgroups. Guest syscalls terminate entirely inside the guest kernel, never reaching the host.
- Boots clean microVMs in milliseconds or restores from snapshots.
- MicroVMs have zero network devices attached. Host coordinator enforces systemd eBPF sandboxing (
IPAddressDeny=any) to block all non-localhost inbound and outbound traffic. - Measures in-guest user execution in microseconds ($\mu s$) and tracks anonymous RSS (
RssAnon), filtering out shared library noise. - Integrates with Firecracker Jailer for dropped UID/GID (
20001), chroot, cgroups v2, and isolated PID namespaces.
Why Not Containers?
The dominant open-source code execution judges — Judge0, Piston like self-hosted runners — rely on Linux containers (Docker + isolate, Docker --privileged, or managed cgroups). The shared-kernel model has a documented, public track record of full-host compromise from within sandboxed code:
| Judge / Engine | Isolation Model | Notable CVEs / Issues |
|---|---|---|
| Judge0 | Docker + isolate binary (shared kernel) |
CVE-2024-28189 (CVSS 10.0) — symlink attack → host file overwrite → RCE outside sandbox; CVE-2024-28185, CVE-2024-29021 — privileged container escape and SSRF chaining to full host root. Disclosed April 2024. |
| Piston | Docker containers (shared kernel) | Relies on Docker isolation; inherits shared-kernel namespace escape risk. No dedicated security model document. |
| Cratera | Firecracker KVM microVMs (hardware boundary) | Guest code interacts only with the guest Linux kernel. No shared namespaces. No privileged containers. See docs/threat_model.md for full analysis. |
The root cause in every container-based escape is the same: the attacker's code and the host OS share a single Linux kernel. A single namespace misconfiguration, privileged flag, or kernel LPE turns a "sandboxed" job into full host access.
Firecracker microVMs eliminate the shared-kernel surface entirely. Guest syscalls are trapped by KVM's hardware boundary, not by namespace filtering. A guest kernel panic or root-level exploit inside the VM does not propagate to the host. This structural difference is why Cratera uses microVMs rather than containers, and why it is self-hosted by design — you control the hypervisor, the host kernel patch level, and the entire stack.
Architecture
┌─────────────────────────────────────────────────────────────┐
│ HTTP Client / Web Gateway │
└──────────────────────────────┬──────────────────────────────┘
│ POST /harness (Bearer Token)
▼
┌─────────────────────────────────────────────────────────────┐
│ Cratera Host Coordinator │
│ • Bearer Token Auth • Fast Snapshot Restore (~5ms)│
│ • Template Splicing • Firecracker Jailer (20001) │
└──────────────────────────────┬──────────────────────────────┘
│
vsock:52 (IPC) │ Zero-NIC KVM Hardware Boundary
▼
┌─────────────────────────────────────────────────────────────┐
│ Firecracker MicroVM (2 vCPU, 2 GiB RAM) │
│ │
│ ┌─────────────────────────────────────────────────────┐ │
│ │ cratera-agent (PID 1) │ │
│ │ ├── Vsock Server (Port 52) │ │
│ │ ├── Compile / Interpret (tmpfs, 12s budget) │ │
│ │ └── Execute & Measure (Microsecond / RssAnon) │ │
│ └─────────────────────────────────────────────────────┘ │
│ │
│ Storage: Read-Only SquashFS / ext4 │ RAM: 256MB tmpfs │
└──────────────────────────────┬──────────────────────────────┘
│
│ JSON Verdict over Vsock
▼
┌─────────────────────────────────────────────────────────────┐
│ Reap & Destroy MicroVM │
│ • Wipe Jail Directory • Release cgroups v2 │
└─────────────────────────────────────────────────────────────┘
Quick Setup Guide
Prerequisites
- Linux with KVM (Fedora, Ubuntu, Debian, Arch, RHEL) with
/dev/kvmhardware virtualization. - Container Engine (Docker or rootless Podman) to assemble the guest rootfs.
- Rust Toolchain (
cargoandrustc1.80+).
Installation & Quick Start
# 1. Interactive setup (downloads kernel, tools, builds rootfs, installs cratera binary)
# 2. Or install the 'cratera' binary directly to ~/.cargo/bin:
Interactive Command Center
Launch the terminal control center from anywhere by typing cratera (or ./target/release/cratera locally):
╭─────────────────────────────────────────────────────────────╮
│ CRATERA INTERACTIVE COMMAND CENTER │
│ Hardware MicroVM Isolation & Multi-Language Sandbox │
│ Systemd: ● Service Active & Supervised │
╰─────────────────────────────────────────────────────────────╯
[1] System Diagnostics & Health Check (/dev/kvm, Jailer, storage, kernel)
[2] Multi-Language Toolchains Manager (Toggle 30 languages, apply presets)
[3] Resource Budgets & Limits Editor (vCPU, RAM, cgroups, timeouts)
[4] In-Guest MicroVM Smoke Tester (Measure microsecond execution)
[5] Build / Rebuild Guest Rootfs Image (SquashFS / ext4)
[6] Start / Stop Local Dev Server [Active on 127.0.0.1:3100]
[7] Systemd Service Manager [Active & Running]
[0] Exit Command Center
Key Subsystems:
1. System Doctor (cratera doctor)
Non-destructive 5-step diagnostic suite for /dev/kvm permissions, SMT hyperthreading status, Jailer UID 20001, Linux cgroups v2, guest kernel, and SquashFS rootfs validation.
2. Multi-Language Manager (cratera lang)
- Interactive Cursor Checklist: Launch
cratera langto navigate the 30-language table with↑/↓(orj/k) and toggle compilers on/off instantly withEnterorSpace. - Dynamic Viewport: Supports scrollable viewports on compact terminal windows.
- Curated Presets: Quick switch between
all,top10,systems,web,functional,scientific, andminimal(Rust only). - CLI & Numeric Indexing: Toggle by table number (
cratera lang disable 27 28) or language key (cratera lang enable go zig).
3. Resource Budgets & Limits Editor (cratera settings)
Interactive editor for per-VM hardware allocation (vCPUs, RAM MiB), execution timeouts, and Jailer cgroup limits with .env persistence.
4. MicroVM Smoke Tester (cratera test [lang])
Boots real isolated microVMs over KVM and reports boot latency ($ms$), compiler time ($ms$), and execution time ($\mu s$) with anonymous RSS profiling.
5. Persistent Background Coordinator (cratera serve)
- Starts the Axum HTTP coordinator on
127.0.0.1:3100as a detached daemon process with PID tracking. - Closing the Command Center leaves the server running so external clients and test scripts can continue submitting workloads.
- Select
[6]anytime from the Command Center to gracefully stop the background daemon.
API Usage & Examples
1. Request Schema
Execute untrusted code inside an ephemeral microVM by sending POST /harness:
| Parameter | Type | Required | Default | Description |
|---|---|---|---|---|
language |
string | Optional | rust |
Target language key defined in languages.toml (e.g. python, node, rust, cpp, go, zig). |
code |
string | Yes | — | Source code to execute in the guest. |
mode |
string | Optional | "run" |
Execution mode: "run" (2-second budget) or "submit" (5-second budget). |
harness |
string | Optional | "" |
Optional harness template for competitive judging or test assertions. |
2. Ready-to-Run Example Script
Cratera includes examples/submit.sh to quickly submit code in any language:
# Execute Python 3
# Execute JavaScript / Node
# Execute Rust
# Execute C++20
3. Direct cURL Examples
# Execute Rust (2024 Edition)
# Execute Python 3
JSON Response
Verdict Codes
| Verdict | Status | Description |
|---|---|---|
AC |
Passed | Solution passed all test assertions (exit code 0). |
WA |
Test Failed | Assertion failed (assert! panic). |
CE |
Compilation Error | Compilation failed. |
TLE |
Time Limit Exceeded | Execution exceeded runtime timeout. |
MLE |
Memory Limit Exceeded | Process exceeded memory budget. |
RE |
Runtime Error | Process crashed or exited with non-zero status. |
IE |
Internal Error | Infrastructure or sandbox initialization failure. |
Multi-Language Configuration
All 30 runtimes, compilers, and packages are defined declaratively in languages.toml.
Out-of-the-Box Supported Languages (Top 30):
- Systems & Low-Level: Rust (2024), C (C17/GCC 14), C++ (C++20), Go (1.24), Zig (0.14), Nim, D (DMD), Fortran (GFortran).
- General & Scripting: Python 3.12, JavaScript (Node.js 24), TypeScript (esbuild + Node), Ruby (3.3), PHP (8.3), Lua, Perl.
- Enterprise & JVM: Java (OpenJDK 21), C# (Mono), F# (.NET/Mono), Scala 3 (3.6), Kotlin (2.1), Clojure.
- Functional & Scientific: Julia (1.11), Haskell (GHC 9.6), OCaml (5.1), Elixir, Erlang.
- Mobile & Modern: Swift (6.0), Dart, R, Bash.
Declarative Recipe Engine
Each language in languages.toml uses one of four explicit install strategies:
install = "curl_tar": Direct download and extraction of official standalone releases (e.g. Zig, Scala 3, Julia).install = "docker_image": Extracts compiler binaries directly from official Docker/OCI images (e.g. Rust, Swift, Dart).install = "apt_core": Installs optimized packages from Ubuntu 24.04 repositories.install = "docker_image_base": Sets the base container image.
To add, toggle, or update any language:
- Edit
languages.tomlor runcratera langto toggle interactively. - Run
./scripts/build-rootfs.sh(orcratera build) to update the guest rootfs and verify in-guest execution.
See docs/languages.md for full recipe syntax and examples.
Configuration Reference
Set in .env or manage directly via cratera settings:
| Variable | Default | Description |
|---|---|---|
CRATERA_BIND |
127.0.0.1:3100 |
Host API bind address. |
CRATERA_INTERNAL_KEY |
(auto-generated) | Shared secret for Bearer authentication. |
CRATERA_RUN_MS |
2000 |
Execution time limit for test runs (ms). |
CRATERA_SUBMIT_MS |
5000 |
Execution time limit for formal submissions (ms). |
CRATERA_MAX_TIME_MS |
10000 |
Hard upper ceiling for execution timeouts (ms). |
CRATERA_COMPILE_TIMEOUT_SECS |
12 |
Guest compiler compilation budget (seconds). |
CRATERA_VCPU |
2 |
Virtual CPU cores allocated per MicroVM. |
CRATERA_MEM_MIB |
2048 |
Guest RAM memory allocated per MicroVM (MiB). |
CRATERA_JAIL_MEM_MAX |
3221225472 |
Host cgroup memory.max per Firecracker process (bytes). |
CRATERA_JAIL_PIDS_MAX |
64 |
Host cgroup pids.max process limit per MicroVM. |
CRATERA_FIRECRACKER |
./images/firecracker |
Path to Firecracker binary. |
CRATERA_JAILER |
./images/jailer |
Path to Jailer binary. |
CRATERA_KERNEL |
./images/vmlinux.bin |
Path to guest kernel. |
CRATERA_ROOTFS |
./images/rootfs.squashfs |
Path to guest SquashFS / ext4 rootfs disk image. |
CRATERA_WORK_DIR |
/var/tmp/cratera |
Directory for ephemeral VM roots. |
CRATERA_USE_JAILER |
0 |
Set 1 to enable Firecracker Jailer isolation. |
CRATERA_JAIL_UID |
20001 |
UID for unprivileged jailer process. |
CRATERA_JAIL_GID |
20001 |
GID for unprivileged jailer process. |
CRATERA_USE_SNAPSHOT |
0 |
Set 1 to enable fast snapshot restore. |
CRATERA_SNAPSHOT_DIR |
./images/snapshot |
Directory for golden snapshot files. |
Systemd Service & Deployment
Cratera includes a systemd unit at deploy/cratera.service. The unit provides verified default settings for unattended background execution across modern Linux distributions (Ubuntu, Debian, Fedora, Arch, RHEL).
- Host-level eBPF network sandboxing (
IPAddressDeny=any), cgroups v2 resource delegation (Delegate=yes), and unprivileged Jailer isolation (UID/GID20001) are enabled out of the box.- All paths are self-contained in
/opt/cratera. Custom ports, memory ceilings, and authentication keys can be set in/opt/cratera/.envwithout modifying the unit file.
[Unit]
Description=Cratera Firecracker harness judge service
After=network.target
[Service]
Type=simple
User=root
WorkingDirectory=/opt/cratera
Environment=NODE_ENV=production
Environment=CRATERA_BIND=127.0.0.1:3100
Environment=CRATERA_FIRECRACKER=/usr/local/bin/firecracker
Environment=CRATERA_JAILER=/usr/local/bin/jailer
Environment=CRATERA_KERNEL=/opt/cratera/images/vmlinux.bin
Environment=CRATERA_ROOTFS=/opt/cratera/images/rootfs.ext4
Environment=CRATERA_WORK_DIR=/var/lib/cratera
Environment=CRATERA_USE_JAILER=1
Environment=CRATERA_JAIL_UID=20001
Environment=CRATERA_JAIL_GID=20001
Environment=CRATERA_USE_SNAPSHOT=1
Environment=CRATERA_SNAPSHOT_DIR=/opt/cratera/images/snapshot
EnvironmentFile=-/opt/cratera/.env
ExecStart=/opt/cratera/cratera
Restart=on-failure
RestartSec=3
LimitNOFILE=65536
Delegate=yes
KillMode=mixed
IPAddressDeny=any
IPAddressAllow=localhost
[Install]
WantedBy=multi-user.target
Systemd Directives & Environment Breakdown
| Section | Directive / Variable | Configured Value | Architectural Purpose & Security Function |
|---|---|---|---|
[Unit] |
Description |
Cratera Firecracker harness judge service |
Identifies the judge daemon in system logs (journalctl -u cratera) and process supervisors. |
[Unit] |
After |
network.target |
Delays daemon execution until basic host network stack and loopback interface are initialized. |
[Service] |
Type |
simple |
Treats the service as active immediately upon launching the ExecStart process. |
[Service] |
User |
root |
Required on the host to open /dev/kvm ioctls, manage rootfs loop mounts, and spawn Firecracker Jailer (which drops unprivileged child permissions to UID/GID 20001). |
[Service] |
WorkingDirectory |
/opt/cratera |
Sets root execution context for resolving relative configuration files (languages.toml, images/). |
[Service] |
ExecStart |
/opt/cratera/cratera |
Absolute binary path to the Cratera CLI and HTTP Coordinator daemon (cratera serve). |
[Service] |
EnvironmentFile |
-/opt/cratera/.env |
Loads optional operator environment overrides. The leading - prevents service failure if .env is absent. |
[Service] |
Restart |
on-failure |
Automatically resurrects the service if the coordinator process terminates unexpectedly or crashes. |
[Service] |
RestartSec |
3 |
Imposes a 3-second delay before restarting to prevent rapid restart loops during hardware faults. |
[Service] |
LimitNOFILE |
65536 |
Raises file descriptor limits to accommodate high-concurrency microVM execution (epoll pipes, vsock descriptors, disk handles). |
[Service] |
Delegate |
yes |
Critical for cgroups v2: Grants Cratera authority over its own cgroup sub-hierarchy (/sys/fs/cgroup/system.slice/cratera.service/...) to enforce per-microVM CPU and memory budgets. |
[Service] |
KillMode |
mixed |
Sends SIGTERM to the main coordinator process on stop/restart, then sends SIGKILL to any lingering microVM child processes. |
[Service] |
IPAddressDeny |
any |
Host-level eBPF Sandboxing: Employs kernel eBPF cgroup network filters to drop all inbound and outbound IPv4/IPv6 packets. |
[Service] |
IPAddressAllow |
localhost |
Whitelists loopback traffic (127.0.0.1, ::1), allowing local API clients and reverse proxies (e.g. Nginx, Caddy) to submit evaluation jobs while preventing external internet egress. |
[Service] |
NODE_ENV |
production |
Sets standard production environment flag for Node runtime wrappers. |
[Service] |
CRATERA_BIND |
127.0.0.1:3100 |
Host HTTP API bind socket. Restricting to 127.0.0.1 ensures only authenticated local applications can access the judge. |
[Service] |
CRATERA_FIRECRACKER |
/usr/local/bin/firecracker |
Path to the installed AWS Firecracker VMM binary. |
[Service] |
CRATERA_JAILER |
/usr/local/bin/jailer |
Path to the unprivileged Firecracker Jailer wrapper binary. |
[Service] |
CRATERA_KERNEL |
/opt/cratera/images/vmlinux.bin |
Path to the uncompressed minimal Linux guest kernel image. |
[Service] |
CRATERA_ROOTFS |
/opt/cratera/images/rootfs.ext4 |
Path to the guest root filesystem disk image containing all 30 language compilers and the in-guest agent. |
[Service] |
CRATERA_WORK_DIR |
/var/lib/cratera |
Base scratch directory where ephemeral microVM jail root directories and vsock sockets are created. |
[Service] |
CRATERA_USE_JAILER |
1 |
Enables production chroot, UID/GID dropping, and cgroups v2 sandbox isolation (1 = enabled, 0 = disabled). |
[Service] |
CRATERA_JAIL_UID |
20001 |
Dedicated unprivileged UID for the jailed Firecracker process. |
[Service] |
CRATERA_JAIL_GID |
20001 |
Dedicated unprivileged GID for the jailed Firecracker process. |
[Service] |
CRATERA_USE_SNAPSHOT |
1 |
Enables ~5ms sub-millisecond VM restoration from memory snapshots instead of full cold boot. |
[Service] |
CRATERA_SNAPSHOT_DIR |
/opt/cratera/images/snapshot |
Directory holding the golden microVM memory and guest CPU state. |
[Install] |
WantedBy |
multi-user.target |
Directs systemd to start Cratera automatically on system boot when enabled. |
One-Command Activation
Run this single copyable command to install the service, reload systemd, and enable on boot:
&& &&
Automatic Setup Option: You can also let the installer handle systemd setup automatically by passing
--service:
Managing the Service via Cratera CLI & Command Center
You can control, supervise, and inspect the daemon directly using the cratera service CLI or via the Interactive Command Center:
# 1. Start or stop the background service
# 2. Restart the service (e.g. after updating .env or languages)
# 3. View live status, PID, and memory footprint
# 4. Stream real-time journald logs
# 5. Or launch the Interactive Command Center and select [7]
# => Select [7] Systemd Service Manager
Direct Systemctl Commands
# View live service status and PID
# Follow real-time coordinator journal logs
# Restart daemon
Recommended Ingress: Cloudflare Zero Trust & Private Networks
Cratera is designed as an isolated internal execution engine. By default, it listens exclusively on 127.0.0.1:3100 and drops direct external internet traffic via systemd eBPF rules (IPAddressDeny=any).
Do not expose port 3100 directly to the public internet. Submissions should be routed through a zero-trust network overlay:
1. Cloudflare Zero Trust Tunnels with Service Tokens
Deploying cloudflared on the judge host provides defense-in-depth isolation:
- Zero Open Inbound Ports: The host opens no listening ports on the public firewall. All traffic is tunneled through an encrypted outbound tunnel to Cloudflare's edge.
- Service Token Authentication: Worker jobs and backend queues must present Cloudflare Access Service Token headers (
CF-Access-Client-IdandCF-Access-Client-Secret) at Cloudflare's edge before traffic reaches your server. - Dual-Layer Authorization: Requests passing the Zero Trust boundary must also supply the
Authorization: Bearer <CRATERA_INTERNAL_KEY>header to interact with the judge coordinator.
# Example cloudflared configuration (/etc/cloudflared/config.yml)
tunnel: <TUNNEL_UUID>
credentials-file: /etc/cloudflared/<TUNNEL_UUID>.json
ingress:
- hostname: judge.yourdomain.org
service: http://127.0.0.1:3100
- service: http_status:404
# Submitting a job via Cloudflare Zero Trust with Service Tokens:
2. Alternative Private Overlays & Reverse Proxies
- Tailscale / WireGuard: Deploy the judge node onto an encrypted Tailscale private mesh or WireGuard VPN. Applications access
http://<tailscale-ip>:3100without exposing the engine publicly. - Local Reverse Proxy (Nginx / Caddy / Traefik): Terminate TLS on loopback with client-certificate mutual TLS (mTLS) or local UNIX domain sockets.
Development & Verification
# 1. Quick local pre-commit check (<2s: formatting, clippy, unit tests)
# 2. Full CI pipeline verification (fmt, clippy, workspace tests, release builds)
# 3. In-guest microVM smoke test (requires /dev/kvm)
Troubleshooting & FAQ
1. Permission denied: /dev/kvm
Ensure your user belongs to the kvm group:
# Log out and log back in, or run:
2. Language not found in manifest
Ensure the language key you pass in JSON is present and set to enabled = true in languages.toml. Then rebuild the rootfs:
3. Zero Network Isolation in MicroVMs
MicroVMs have no network interfaces attached by design. Package downloads (e.g. pip install, npm install, cargo install) will deliberately fail at execution time inside the VM. All dependencies, compilers, and packages must be declared in languages.toml at build time.
Community & Discussion
- Zulip Chat: Join real-time discussion and operator channels on Cratera Zulip.
- GitHub Discussions: Open architectural discussions and feature proposals via GitHub Discussions.
- Direct Contact: Inquiries and vulnerability disclosures can be sent to
contact@cratera.org.
Governance & Security
- See GOVERNANCE.md for the open-source commitment, BDFL/RFC process, and perpetual Apache-2.0 license guarantee.
- See SECURITY.md and docs/threat_model.md for private vulnerability reporting, threat modeling, Firecracker limitations, and sandbox architecture.
- See CONTRIBUTING.md for development workflows and CI standards.
License
Licensed under the Apache License, Version 2.0 (LICENSE or http://www.apache.org/licenses/LICENSE-2.0).