densol: on-chain LZ4 compression for Solana programs

Transparent LZ4 compression for Anchor account fields. Store less data, pay less rent.

I couldn't find measured data on whether on-chain compression is worth the extra CU, so I measured it. If you spot a mistake, open an issue

Use densol

densol = "0.1"

use densol::Lz4 as Strategy;
use densol::Compress;

#[account]
#[derive(Compress)]
pub struct MyAccount {
    #[compress]
    pub data: Vec<u8>,
}

// my_account.set_data(&raw_bytes)?;  // compress + store
// let raw = my_account.get_data()?;  // decompress + return

When to compress

Why LZ4 fits the SBF heap

The SBF VM has a 32 KB bump-allocated heap. Most compressors (zstd, gzip) fail here. LZ4 works because lz4_flex places its compression hash table ([u32; 4096] = 16 KB) on the stack, not the heap. Heap allocations during compression reduce to one output buffer (~N bytes), giving peak heap ≈ 2× input size.

This yields a practical in-program compression ceiling around 10–12 KB. The exact limit is data-dependent: the heap must hold both the compressed and original forms simultaneously, so well-compressed data allows larger inputs than random data at the same raw size. Above the ceiling, compress off-chain and upload pre-compressed bytes.

A note on CU cost and break-even

The Solana base fee (5,000 lamports/signature) is fixed regardless of CU consumed. Extra CU only costs lamports via the priority fee, which is optional. At zero priority fee — the default for most transactions — compression adds no extra lamport cost whatsoever. The rent saving is pure profit.

The break-even column below assumes a conservative priority fee of 1,000 µL/CU. At that rate the extra cost per read is 1–57 lamports; the one-time rent saving is hundreds of thousands of lamports. The numbers in the millions of reads are not a warning — they confirm that CU overhead is effectively never a binding constraint for compressible data.

Run benchmarks

# Pure-Rust: sizes and rent savings (no Solana CLI needed)
cargo test -p densol --test scenarios --features chunked_lz4 -- --nocapture

# CU measurements (requires Anchor + local validator)
anchor test

Results

Lz4 — write (tx-limited sizes)

Sizes are limited to 800 B by the transaction payload cap. Compression ratios grow with size because LZ4 builds up more back-reference context.

The orderbook type (8 B price + 8 B quantity + 1 B side + 63 B zero padding per entry) compresses better than even repetitive text because the dense zero runs are trivial for LZ4 to encode.

Lz4 — read (account-limited sizes)

Accounts are set up via chunked store_raw calls followed by an in-place compress_stored. OOM rows indicate that compress_stored exceeded the 32 KB heap — the raw data plus its compressed form would not both fit simultaneously. Well-compressed data (reptitive, orderbook) reaches 10 KB cleanly; random data cannot compress past ~4 KB because the "compressed" output is nearly the same size as the input.

¹ Break-even = rent_saving / (overhead_CU × 0.000001 L) at 1,000 µL/CU priority fee. At zero priority fee the break-even is infinite — compression is always net positive.

² OOM on compress_stored: compressed and raw forms of ~8 KB of random data both fit on the 32 KB heap simultaneously, but Anchor account deserialization leaves insufficient headroom. Well-compressed data (repetitive, orderbook) doesn't hit this because the compressed form is tiny.

ChunkedLz4 — large accounts

Plain Lz4 OOMs on decompression above roughly 10 KB because it allocates the entire output at once. ChunkedLz4<N> splits input into independent N-byte chunks and exposes decompress_chunk(data, i) — an O(chunk_size) heap call. On-chain code reads only the chunks it needs; the total account size is irrelevant to heap pressure.

The wire format is N-agnostic: data compressed with ChunkedLz4<512> can be decompressed by ChunkedLz4<4096> and vice versa.

Rent savings at large account sizes (ChunkedLz4<4096>):

The 90 KB orderbook row mirrors a realistic OpenBook BookSide account (90 KB of raw state → 1,391 B compressed, 66.25× ratio, ~0.63 SOL saved). The on-chain compression benchmark (largeAccountDemo) confirms this: 90,952 B → 1,386 B, compressCu=764,131 (~55% of the 1.4 M budget). Compression ratios plateau as chunk count grows because LZ4 already reaches maximum context within the first few chunks.

Chunk size vs compression ratio — because LZ4 only back-references within the current chunk, smaller chunks mean less context. Fixed input: 90 KB orderbook:

ChunkedLz4<4096> is the recommended default: near-maximum compression (the 80-byte pattern fills ~51× within a 4 KB chunk) while each decompress_chunk call uses only ~4 KB of heap.

ChunkedLz4 — on-chain compression CU (single-tx and multi-tx)

Single-transaction ceiling: compressStoredChunkedLarge processes up to ~163,840 bytes (40 × 4,096-byte chunks) in one transaction — ~34,572 CU/chunk, 1,382,886 CU total — within the 1,400,000 CU budget. 172,032 bytes (42 chunks) exceeds the limit and fails. Data pattern: orderbook (worst case — maximum chunks filled, so maximum CU).

Multi-transaction compression (compressLargeInit / compressLargeBatch / compressLargeFinalize) removes the per-tx ceiling. The init instruction pre-reads any "dangerous" raw chunks that would be overwritten by the header, then writes the ChunkedLz4 header. Each batch call compresses up to 38 chunks (~1.33M CU/tx). Finalize truncates the account to the compressed length.

On-chain measured results — orderbook-pattern data, localnet, 2026-03-22:

CU breakdown for 1 MB: 1 init + 7 batches of 38 chunks + 1 finalize = 9 transactions, ~34,996 CU/chunk in batch mode (~1.25% higher than single-tx due to per-chunk Vec allocation overhead).

ChunkedLz4 — per-chunk read CU

What this benchmark measures: The test accounts are 1–4 KB, which fit in exactly one 4 KB chunk (ceil(1024/4096) = ceil(4096/4096) = 1). The table shows the cost of decompressing one chunk. For multi-chunk accounts, this cost is paid once per chunk you access, independent of total account size. A 90 KB account has 23 chunks; reading one of them costs the same CU as shown here for 4 KB.

For structured data the per-chunk overhead is ~5,800–6,100 CU per 1 KB and ~22,800–23,200 CU per 4 KB chunk. The 1,400,000 CU budget can cover ~60 chunks of 1 KB or ~60 chunks of 4 KB in a single transaction — enough to scan a full 90 KB orderbook (23 × 4 KB chunks) with room to spare.

Real-world accounts (mainnet)

Measured on live mainnet accounts fetched 2026-03-21 via getAccountInfo.

Drift User accounts (4.4 KB, Borsh) are a direct densol integration target — well within the 32 KB heap limit. Even the worst case (active user) compresses 2.6× and saves rent permanently.

OpenBook BookSide accounts (90 KB, zero-copy bytemuck::Pod) show the rent-saving potential: ~54× compression, ~0.62 SOL per account. Plain Lz4 cannot process 90 KB on-chain (heap limit). ChunkedLz4<4096> with AccountInfo-based bypass deserialization keeps peak heap at ~3 KB, making large-account on-chain compression feasible. BookSide stores data as a bytemuck::Pod struct with an exact binary layout — storing compressed bytes in the same account would break the layout invariant, so densol integration requires off-chain compression or program-side adoption.

ChunkedLz4 — write and full-read CU

Note: For accounts ≤ 10 KB, prefer plain Lz4 — it gives better compression and lower write CU than ChunkedLz4 at small sizes. The overhead difference ranges from ~990 CU (repetitive 256 B) to ~4,250 CU (repetitive 800 B). The tables below are included for completeness and for cases where ChunkedLz4 format is required at all sizes.

Why ChunkedLz4 write CU is higher: The custom on-chain compressor allocates the LZ4 hash table (8 KB) once per compress() call and reuses it across all chunks, clearing between chunks. This replaces the old approach of delegating to lz4_flex per chunk, which stranded one hash table per chunk on the SBF bump allocator — 23 chunks × 8 KB = 188 KB for a 90 KB account. The new approach keeps peak heap at ~3 KB regardless of account size, enabling MB-scale on-chain compression. Write CU increased ~10–40%; the rent-saving break-even is unchanged.

Write (256–800 B):

Full read (256 B – 10 KB; random data OOMs at 10 KB, all other types succeed through 10 KB):

³ OOM on read_chunked_full for random 10 KB: the full decompressed output (~10 KB) plus the working buffers exceed the 32 KB heap. Use decompress_chunk to read individual chunks without this limit.

Missing pieces

• Mainnet priority fee sensitivity analysis (current benchmark assumes 1,000 µL/CU)
• Alternative algorithms: heatshrink and lzss use ~256–512 B heap vs LZ4's 16 KB stack (see ROADMAP.md)

License

Licensed under Apache License 2.0.