Qem

High-performance Rust text engine for massive files, with mmap-backed reads, incremental line indexing, and responsive editing.

Qem is built for editor-style workloads where opening huge files must stay responsive, scrolling should avoid full materialization, and saves should stream back to disk instead of rebuilding the entire document in memory.

Qem is intentionally a backend-first text engine, not a GUI widget toolkit. Applications are expected to own their cursor, scrollbar, selection, and rendering model while Qem owns the hard part: huge-file access, viewport reads, text mutation, save pipelines, and persistent edit sessions.

Small files take a fast inline path: Qem fully indexes them during open() so line counts, viewport reads, and first edits are exact immediately without paying background-thread overhead.

For large file-backed documents, Qem builds a sparse on-disk B-tree line index with an LRU page cache. This keeps global line navigation bounded without holding billions of line offsets in RAM.

When editing is enabled for large files, Qem can promote the document into a piece tree and persist its state to a sibling .qem.editlog sidecar using append-only copy-on-write commits and a bounded in-memory page cache.

The same sidecar also stores persistent edit sessions: the current piece-tree state, add buffer, and undo/redo roots can be recovered automatically on the next Document::open, as long as the source file metadata still matches.

Highlights

• Fast open and viewport reads for very large files
• Exact inline indexing for small files
• Background line indexing with a sparse on-disk line index
• Detected line-ending style exposed and preserved on save
• Lazy promotion to rope or piece-table editing buffers
• Persistent edit sessions with undo/redo recovery
• Async document-session open/save flows with progress polling
• Streaming atomic saves for large edited documents

Installation

[dependencies]
qem = "0.3"

To disable the editor/session wrapper and use only the document/storage layer:

[dependencies]
qem = { version = "0.3", default-features = false }

Cargo features

• editor (default): enables DocumentSession, EditorTab, CursorPosition, and async open/save helpers.
• tmp-auto (default): dynamic scratch-temp policy. On Windows Qem prefers the executable directory, then the edited file directory, then the system temp directory. On Unix-like systems Qem prefers the edited file directory, then the system temp directory, then the executable directory.
• tmp-source-dir: keeps snapshot/scratch temp files next to the edited file.
• tmp-system-dir: uses the OS temp directory for snapshot/scratch temp files.
• tmp-exe-dir: uses the executable directory for snapshot/scratch temp files.

Only scratch files such as .qem.snap.* follow this policy. The temp file used for atomic save replacement still stays next to the destination file so save_to() remains atomic.

Example:

[dependencies]
qem = { version = "0.3", default-features = false, features = ["editor", "tmp-exe-dir"] }

Runtime override is also available:

• QEM_TMP_POLICY=auto|source-dir|system-dir|exe-dir
• QEM_TMP_DIR=/absolute/path/to/custom/tmp/root

Core components

• FileStorage: mmap-backed file access.
• Document: background line indexing, fast line metrics, viewport reads, rope or piece-table editing, and persistent edit-session recovery.
• DocumentSession: backend-first document session wrapper with async open/save helpers, progress polling, status snapshots, forwarded viewport/edit helpers, and no GUI cursor assumptions.
• EditorTab: lightweight document session wrapper with cursor, async open/save helpers, and progress polling.

Positioning

• Qem is a text engine / editor backend for huge files.
• Qem is not a ready-made text widget.
• GUI code should keep ownership of visual cursor movement, custom scrollbars, line-number gutters, and selection painting.
• Qem should own viewport reads, typed range/selection reads, text positions/ranges, char-index conversions, mutation commands, save orchestration, and large-file storage behavior.
• Qem should also tell the frontend when an edit is safe, when it would require promotion, and when a huge-file safety limit makes it unavailable.

Choosing an integration layer

• Use Document when your application already owns tabs, session state, background jobs, and save orchestration.
• Use DocumentSession when you want a backend-first session wrapper with generation tracking, async open/save helpers, progress polling, and no built-in cursor state.
• Use EditorTab when you want the same session helpers plus convenience cursor state.
• A GUI typically renders visible rows through Document::read_viewport(...) or DocumentSession::read_viewport(...).
• Older compatibility helpers that silently swallow edit errors or expose raw tuple progress are still present for now, but they are deprecated in favor of the typed/session-first API.

Frontend integration pattern

Qem stays UI-agnostic on purpose, but the intended editor/frontend flow is:

1. Open a file with Document::open(...) for a synchronous viewer or DocumentSession::open_file_async(...) for a responsive frontend.
2. Poll poll_background_job() and cache status() or the focused loading_state(), save_state(), and indexing_state() values from your event loop.
3. Size scrollbars with display_line_count() while the line count is still estimated.
4. Render only the visible rows with read_viewport(ViewportRequest::new(...).with_columns(...)).
5. Use edit_capability_at(...) when the UI wants to disable or annotate edits before the user hits a huge-file safety limit.
6. Keep GUI selection state as TextSelection { anchor, head }, read it through read_selection(...) for copy flows, convert it into a TextRange with text_range_for_selection(...) when needed, or use try_replace_selection(...) / try_delete_selection(...) / try_cut_selection(...) directly. For key handling, try_backspace_selection(...) and try_delete_forward_selection(...) handle the usual "delete selection or fall back to caret command" path for you. Then save with save_to(...), DocumentSession::save_async(...), or DocumentSession::save_as_async(...).

The new frontend_session example demonstrates that engine-facing lifecycle without pulling any GUI toolkit into the crate:

cargo run --example frontend_session --features editor -- input.txt output.txt

Stability Notes

• The Rust API is intended to be stable within the 0.3.x release line except for clearly documented fixes to incorrect behavior.
• Sidecar artifacts such as .qem.editlog and .qem.lineidx are internal cache/session formats. Qem may regenerate them across releases, and callers should not treat them as long-term interchange formats.
• The minimum supported Rust version for this release line is 1.85.

Examples

Inspect a viewport from a huge file:

cargo run --example viewport -- "C:\path\to\huge.log" 1000000 20

Open a file through the editor wrapper and save a modified copy:

cargo run --example editor_session --features editor -- input.txt output.txt

The editor_session example now uses EditorTab::open_file_async(), save_as_async(), and poll_background_job() to show the non-blocking tab workflow end to end.

Model the data flow that a GUI frontend would use:

cargo run --example frontend_session --features editor -- input.txt output.txt

The frontend_session example shows how a frontend can poll load/save/index progress, compute a viewport, and render visible rows through DocumentSession::read_viewport(...) while keeping UI concerns out of the engine.

API Example

use qem::{Document, TextPosition, TextRange, ViewportRequest};
use std::path::Path;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let path = Path::new("huge.log");
    let mut doc = Document::open(path)?;

    let viewport = doc.read_viewport(ViewportRequest::new(10, 20).with_columns(0, 160));
    println!("scroll rows: {}", viewport.total_lines().display_rows());
    for row in viewport.rows() {
        println!(
            "{:>8}: [{}] {}",
            row.line_number(),
            if row.is_exact() { "=" } else { "~" },
            row.text()
        );
    }

    let _ = doc.try_insert(TextPosition::new(0, 0), "[Qem]\n")?;
    let _ = doc.try_replace(TextRange::new(TextPosition::new(1, 0), 4), "HEAD")?;
    doc.save_to(path)?;

    Ok(())
}

Benchmarks

Qem ships with Criterion benchmarks for:

• large-file open and indexing
• viewport/scroll reads through read_viewport
• viewport reads after large-file edits through the piece table
• streaming saves of edited large files

The built-in fixture sizes currently include:

• 1_000_000 lines for large-file open/indexing
• 400_000 lines for viewport read benchmarks
• 250_000 lines for save benchmarks

Run them with:

cargo bench --bench document_perf

Project Links

• CI lives in .github/workflows/ci.yml.
• Repository: https://github.com/MrHanty1488/Qem