Qem

High-performance Rust text engine for massive files, with mmap-backed reads, incremental line indexing, and responsive editing.

Qem is built for editor-style workloads where opening huge files must stay responsive, scrolling should avoid full materialization, and saves should stream back to disk instead of rebuilding the entire document in memory.

Qem is intentionally a backend-first text engine, not a GUI widget toolkit. Applications are expected to own their cursor, scrollbar, selection, and rendering model while Qem owns the hard part: huge-file access, viewport reads, text mutation, save pipelines, and persistent edit sessions.

Small files take a fast inline path: Qem fully indexes them during open() so line counts, viewport reads, and first edits are exact immediately without paying background-thread overhead.

For large file-backed documents, Qem builds a sparse on-disk B-tree line index with an LRU page cache. This keeps global line navigation bounded without holding billions of line offsets in RAM.

When editing is enabled for large files, Qem can promote the document into a piece tree and persist its state to a sibling .qem.editlog sidecar using append-only copy-on-write commits and a bounded in-memory page cache.

The same sidecar also stores persistent edit sessions: the current piece-tree state, add buffer, and undo/redo roots can be recovered automatically on the next Document::open, as long as the source file identity still matches.

Highlights

Fast open and viewport reads for very large files
Exact inline indexing for small files
Background line indexing with a sparse on-disk line index
Detected line-ending style exposed and preserved on save
Lazy promotion to rope or piece-table editing buffers
Persistent edit sessions with undo/redo recovery
Async document-session open/save flows with progress polling
Streaming atomic saves for large edited documents

Installation

[dependencies]

qem = "0.3"

To disable the editor/session wrapper and use only the document/storage layer:

[dependencies]

qem = { version = "0.3", default-features = false }

Cargo features

editor (default): enables DocumentSession, EditorTab, CursorPosition, and async open/save helpers.
tmp-auto (default): dynamic scratch-temp policy. On Windows Qem prefers the executable directory, then the edited file directory, then the system temp directory. On Unix-like systems Qem prefers the edited file directory, then the system temp directory, then the executable directory.
tmp-source-dir: keeps snapshot/scratch temp files next to the edited file.
tmp-system-dir: uses the OS temp directory for snapshot/scratch temp files.
tmp-exe-dir: uses the executable directory for snapshot/scratch temp files.

Only scratch files such as .qem.snap.* follow this policy. The temp file used for atomic save replacement still stays next to the destination file so save_to() remains atomic.

Example:

[dependencies]

qem = { version = "0.3", default-features = false, features = ["editor", "tmp-exe-dir"] }

Runtime override is also available:

QEM_TMP_POLICY=auto|source-dir|system-dir|exe-dir
QEM_TMP_DIR=/absolute/path/to/custom/tmp/root

QEM_TMP_DIR is only honored when it points to an absolute, writable directory. Invalid or unusable custom roots are ignored and Qem falls back to the selected temp policy.

Core components

FileStorage: mmap-backed file access.
Document: background line indexing, fast line metrics, viewport reads, rope or piece-table editing, and persistent edit-session recovery.
DocumentSession: backend-first document session wrapper with async open/save helpers, progress polling, status snapshots, forwarded viewport/edit helpers, and no GUI cursor assumptions.
EditorTab: lightweight document session wrapper with cursor, async open/save helpers, and progress polling.

Positioning

Qem is a text engine / editor backend for huge files.
Qem is not a ready-made text widget.
GUI code should keep ownership of visual cursor movement, custom scrollbars, line-number gutters, and selection painting.
Qem should own viewport reads, typed range/selection reads, text positions/ranges, char-index conversions, mutation commands, save orchestration, and large-file storage behavior.
Qem should also tell the frontend when an edit is safe, when it would require promotion, and when a huge-file safety limit makes it unavailable.

Choosing an integration layer

Use Document when your application already owns tabs, session state, background jobs, and save orchestration.
Use DocumentSession when you want a backend-first session wrapper with generation tracking, async open/save helpers, progress polling, and no built-in cursor state.
Use EditorTab when you want the same session helpers plus convenience cursor state.
A GUI typically renders visible rows through Document::read_viewport(...) or DocumentSession::read_viewport(...).
Older compatibility helpers that silently swallow edit errors or expose raw tuple progress are still present for migration only, but they are deprecated and hidden from the main rustdoc surface in favor of the typed/session-first API.

0.4.x Support Matrix

UTF-8 / ASCII text is the primary 0.4.x path: open, viewport reads, edits, undo/redo, and saves are supported.
Non-UTF8 or invalid UTF-8 bytes can still be opened and inspected, but text-facing APIs expose lossy UTF-8 views. Encoding-preserving edit/save behavior for arbitrary legacy encodings is not yet a stable 0.4.x contract.
Huge files are supported for mmap-backed reads, viewport rendering, line-count estimation, and background indexing without full materialization. Editing may be rejected when it would require unsafe rope promotion beyond the built-in size limits.
.qem.lineidx and .qem.editlog are internal cache/session sidecars. Qem invalidates them when file length, modification time, or sampled content fingerprint no longer match, and it may rebuild them across releases.
Typed progress/state APIs such as indexing_state(), loading_state(), loading_phase(), save_state(), background_issue(), take_background_issue(), and close_pending() are the supported session-facing surface for 0.4.x.
Literal search is available through find_next(...) and find_prev(...) on Document, DocumentSession, and EditorTab. For repeated searches with the same needle, LiteralSearchQuery lets you reuse a compiled literal query instead of rebuilding search state every time. Bounded variants search only within a typed TextRange and reject matches that would cross the range boundary. In 0.4.x this is a typed, case-sensitive literal search surface rather than a full regex/search subsystem.
DocumentSession and EditorTab typed edit helpers are idle-only in 0.4.x: while a background open/save is active they return EditUnsupported instead of mutating a document that may soon be replaced or saved from an older snapshot. Raw document_mut() and set_path() remain escape hatches, but using either while busy invalidates the in-flight worker result so the next poll returns an error instead of applying stale state. If a close was deferred at the time, that new state change also cancels the deferred close.
close_file() is also truthful in 0.4.x: if a background open/save is still running, the close is deferred until that job completes instead of silently dropping the worker result. Deferred closes after background saves are only applied on success; a failed save keeps the dirty document open.
Repeated async open/save requests use a first-job-wins policy in 0.4.x: while a load/save is active, later requests are rejected and the original worker remains authoritative until poll_background_job() consumes its result.
Typed positions, ranges, and viewport columns use document text units. For UTF-8 text, line-local columns count Unicode scalar values, not grapheme clusters and not display cells. Stored CRLF still counts as one text unit between lines.

Column Semantics

TextPosition::col0, TextRange::len_chars(), line_len_chars(), text_units_between(), and ViewportRequest::with_columns(...) all use document text units.
For UTF-8 text, a column is one Unicode scalar value. Combining marks therefore occupy their own columns, and wide CJK characters still count as one column even if they render as two terminal cells.
Qem intentionally does not try to own grapheme clustering, display-cell width, tab expansion, or visual cursor movement. Those are frontend concerns.
CRLF is treated as one text unit for typed range/edit/navigation semantics even though it is stored as two bytes in mmap-backed files.

Frontend integration pattern

Qem stays UI-agnostic on purpose, but the intended editor/frontend flow is:

Open a file with Document::open(...) for a synchronous viewer or DocumentSession::open_file_async(...) for a responsive frontend.
Poll poll_background_job() and cache status() or the focused loading_state(), loading_phase(), save_state(), background_issue(), close_pending(), and indexing_state() values from your event loop. loading_state() covers the asynchronous open path itself; once the document is ready, continued line indexing is reported through indexing_state(). If a background job fails or is intentionally discarded as stale, background_issue() keeps the last typed problem available even after background_activity() returns to Idle. If close_file() was requested while the engine was busy, close_pending() tells the frontend that the document is now scheduled to disappear once the active worker finishes. Once you have surfaced that async problem to the user, call take_background_issue() to acknowledge and clear it explicitly.
Size scrollbars with display_line_count() while the line count is still estimated.
Render only the visible rows with read_viewport(ViewportRequest::new(...).with_columns(...)).
Use edit_capability_at(...) when the UI wants to disable or annotate edits before the user hits a huge-file safety limit.
Avoid using text_lossy() / DocumentSession::text() / EditorTab::text() in hot paths for large files. They materialize the entire current document into a new String. Prefer read_viewport(...) or read_text(...) for bounded reads.
Treat session/tab edit helpers as idle-only. If is_busy() is true, keep polling poll_background_job() instead of mutating through try_insert(...), try_replace(...), or the selection helpers. If you intentionally use raw document_mut() or set_path() while busy, expect the active load/save result to be discarded on the next poll.
If the user closes a session/tab while is_busy() is true, keep polling until the current job completes; close_file() defers the actual close so the engine can finish and account for the in-flight worker result. If a background save fails, surface the error and keep the document open for retry or explicit discard.
While is_busy() is true, treat the current loading_state() or save_state() path as authoritative. New async open/save requests will be rejected until you poll and finish that active job. The file write runs on a worker thread, but save_async() and save_as_async() still snapshot the current document before that worker starts, so very large edited buffers can make the call itself noticeable.
Keep GUI selection state as TextSelection { anchor, head }, read it through read_selection(...) for copy flows, convert it into a TextRange with text_range_for_selection(...) when needed, or use try_replace_selection(...) / try_delete_selection(...) / try_cut_selection(...) directly. For key handling, try_backspace_selection(...) and try_delete_forward_selection(...) handle the usual "delete selection or fall back to caret command" path for you.
For literal search, use find_next(...) and find_prev(...) and keep the returned SearchMatch as a typed TextRange/selection source for highlight or replace flows. If you are repeating the same search many times from UI state, build one LiteralSearchQuery and call find_next_query(...) / find_prev_query(...) instead. If the search must stay inside a selection or other local region, use find_next_in_range(...) / find_prev_in_range(...) or the query-based bounded variants. find_prev(...) returns the last match whose end is at or before the boundary position you pass, and bounded search only returns matches fully contained within the requested TextRange. On clean mmap and piece-table backings this search follows stored bytes, including stored CRLF; rope-backed documents search the current in-memory \n representation.
Then save with save_to(...), DocumentSession::save_async(...), or DocumentSession::save_as_async(...).

The new frontend_session example demonstrates that engine-facing lifecycle without pulling any GUI toolkit into the crate:

cargo run --example frontend_session --features editor -- input.txt output.txt

Stability Notes

The Rust API is intended to be stable within the 0.4.x release line except for clearly documented fixes to incorrect behavior.
Sidecar artifacts such as .qem.editlog and .qem.lineidx are internal cache/session formats. Qem may regenerate them across releases, and callers should not treat them as long-term interchange formats.
Sidecar recovery trusts the source file identity, currently defined as file length, modification time, and a sampled content fingerprint.
The minimum supported Rust version for this release line is 1.85.

Examples

Inspect a viewport from a huge file:

cargo run --example viewport -- "C:\path\to\huge.log" 1000000 20

Open a file through the editor wrapper and save a modified copy:

cargo run --example editor_session --features editor -- input.txt output.txt

The editor_session example now uses EditorTab::open_file_async(), loading_phase(), save_as_async(), and poll_background_job() to show the non-blocking tab workflow end to end.

Model the data flow that a GUI frontend would use:

cargo run --example frontend_session --features editor -- input.txt output.txt

The frontend_session example shows how a frontend can poll load/save/index state, distinguish open phases from continued indexing, compute a viewport, and render visible rows through DocumentSession::read_viewport(...) while keeping UI concerns out of the engine.

Exercise the typed Document edit/read API directly:

cargo run --example typed_editing -- input.txt output.txt

The typed_editing example shows a minimal non-session flow built around read_selection(...), try_replace_selection(...), try_cut_selection(...), read_text(...), LiteralSearchQuery, find_next_query(...), find_prev_query(...), and save_to(...).

API Example

use qem::{Document, TextPosition, TextRange, ViewportRequest};
use std::path::Path;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let path = Path::new("huge.log");
    let mut doc = Document::open(path)?;

    let viewport = doc.read_viewport(ViewportRequest::new(10, 20).with_columns(0, 160));
    println!("scroll rows: {}", viewport.total_lines().display_rows());
    for row in viewport.rows() {
        println!(
            "{:>8}: [{}] {}",
            row.line_number(),
            if row.is_exact() { "=" } else { "~" },
            row.text()
        );
    }

    let _ = doc.try_insert(TextPosition::new(0, 0), "[Qem]\n")?;
    let _ = doc.try_replace(TextRange::new(TextPosition::new(1, 0), 4), "HEAD")?;
    doc.save_to(path)?;

    Ok(())
}

Benchmarks

Qem ships with Criterion benchmarks for:

large-file open and indexing
viewport/scroll reads through read_viewport
session-layer viewport reads through DocumentSession and EditorTab
session-layer text() and status() overhead relative to raw Document
viewport reads after large-file edits through the piece table
typed text reads through read_text and read_selection
typed literal search through find_next and find_prev
full-text materialization through text_lossy
typed edit commands such as try_insert, try_replace_selection, and try_delete_selection
streaming saves of edited large files

The built-in fixture sizes currently include:

1_000_000 lines for large-file open/indexing
400_000 lines for viewport read benchmarks
64_000 lines for steady-state piece_table edit benchmarks
4_096 lines for typed edit command benchmarks
250_000 lines for save benchmarks

Run them with:

cargo bench --bench document_perf

Run only the session-layer overhead benches with:

cargo bench --bench document_perf -- session_layer

Project Links

CI lives in .github/workflows/ci.yml.
Repository: https://github.com/MrHanty1488/Qem

qem 0.4.0

Qem