repl-lang 1.1.0

The model

Two independent halves, usable together or apart:

• Session state. A Session accumulates lines into an entry and calls your pipeline — a closure that lexes, parses, and evaluates — with an Input over the whole entry after each line. The pipeline answers Status::Complete(value) or Status::Incomplete; the session answers with a Feed, and on completion commits the entry and hands back an Entry with the value.
• Line editing. An Editor holds the line, the cursor, a kill buffer, and a bounded history. Your key handler maps keys to Edit commands; the editor applies them and reports whether to redraw.

What each part owns, and what it leaves to you:

Installation

[dependencies]
repl-lang = "1"
diag-lang = "1"     # rendering diagnostics against Session::sources

Or from the terminal:

cargo add repl-lang diag-lang

MSRV: Rust 1.85 (Rust 2024 edition).

Quick start

A session over a fixed script with a pipeline that keeps reading until parentheses balance. In a real REPL the lines come from a terminal and the pipeline is your parser.

use repl_lang::{Feed, Session, Status};

let mut session = Session::new();
let mut transcript = Vec::new();

for line in ["(define (square x)", "  (* x x))", "", "(square 12)"] {
    let prompt = if session.is_pending() { "..." } else { ">>>" };
    let feed = session.feed(line, |input| {
        let depth: i32 = input.text().chars().map(|c| match c {
            '(' => 1,
            ')' => -1,
            _ => 0,
        }).sum();
        if depth > 0 { Status::Incomplete } else { Status::Complete(input.text().lines().count()) }
    })?;
    match feed {
        Feed::Complete { entry, value } => {
            transcript.push(format!("{prompt} {line}   [entry {} spans {value} lines]", entry.number()));
        }
        Feed::Incomplete => transcript.push(format!("{prompt} {line}")),
        Feed::Empty => transcript.push(prompt.to_owned()),
    }
}

assert_eq!(transcript, [
    ">>> (define (square x)",
    "...   (* x x))   [entry 1 spans 2 lines]",
    ">>>",
    ">>> (square 12)   [entry 2 spans 1 lines]",
]);
assert_eq!(session.entry(1).map(|e| e.name()), Some("<repl:1>"));
# Ok::<(), repl_lang::SessionError>(())

Using parser errors to detect continuation

Input::is_incomplete turns any parser that reports a missing token at the end of input into a continuation detector. Errors in the middle of the input mean the entry is wrong, not unfinished, so it completes and the errors are reported. Use it when the parser's positions are global (lexed with Input::cursor). A parser run on Input::text alone, such as a lang-forge language, reports positions from the start of the entry; use Input::is_incomplete_relative for it. The same rule applies to both; see docs/API.md.

use diag_lang::{Diagnostic, Label, Severity};
use repl_lang::{Feed, Session, Span, Status};

// Stand-in for a parser: "expected expression" at the end when the text ends
// with an operator, "unexpected `)`" where a stray parenthesis appears.
fn parse(text: &str, base: u32) -> Vec<Diagnostic> {
    let trimmed = text.trim_end();
    let error = |at: usize, msg: &str| {
        let at = base + at as u32;
        Diagnostic::new(Severity::Error, msg.to_owned(), Label::new(Span::new(at, at), "here"))
    };
    if let Some(at) = trimmed.find(')') {
        vec![error(at, "unexpected `)`")]
    } else if trimmed.ends_with(['+', '*', '=']) {
        vec![error(trimmed.len(), "expected expression")]
    } else {
        Vec::new()
    }
}

let mut session = Session::new();
let mut run = |line: &str| {
    session.feed(line, |input| {
        let errors = parse(input.text(), input.base().to_u32());
        if input.is_incomplete(&errors) { Status::Incomplete } else { Status::Complete(errors.len()) }
    })
};

assert_eq!(run("let x = 1 +")?, Feed::Incomplete);         // error at the end: keep reading
assert!(matches!(run("    2")?, Feed::Complete { value: 0, .. }));
assert!(matches!(run("1 + ) 2")?, Feed::Complete { value: 1, .. })); // error mid-line: report it
# Ok::<(), repl_lang::SessionError>(())

Rendering diagnostics across entries

Every committed entry lives in Session::sources, a SourceMap that diag_lang renders against. Spans the pipeline records are global from the start, so a diagnostic can point into any entry, including earlier ones.

use diag_lang::{Diagnostic, Label, Renderer, Severity};
use repl_lang::{Feed, Session, Span, Status};

let mut session = Session::new();
let mut first_definition = None;

for line in ["let limit = 10", "let limit = 20"] {
    let feed = session.feed(line, |input| {
        let name = Span::new(input.base().to_u32() + 4, input.base().to_u32() + 9);
        match first_definition {
            None => {
                first_definition = Some(name);
                Status::Complete(None)
            }
            Some(first) => Status::Complete(Some(
                Diagnostic::new(Severity::Error, "`limit` is already defined", Label::new(name, "redefined"))
                    .with_secondary(Label::new(first, "first defined here")),
            )),
        }
    })?;
    if let Feed::Complete { value: Some(diagnostic), .. } = feed {
        let text = Renderer::new().render(&diagnostic, session.sources());
        assert!(text.contains("<repl:2>:1:5"));
        assert!(text.contains("<repl:1>:1:5"));
        assert!(text.contains("first defined here"));
    }
}
# Ok::<(), Box<dyn std::error::Error>>(())

Line editing

The editor never touches a terminal. Map key events to Edit commands, redraw from line() and column(), and call submit() on Enter.

use repl_lang::{Edit, Editor};

let mut editor = Editor::new();
editor.insert("let total = price");

assert!(editor.apply(Edit::KillWordLeft));   // Ctrl-W
editor.insert("cost * 2");
assert_eq!(editor.line(), "let total = cost * 2");

assert!(editor.apply(Edit::Home));           // Ctrl-A
assert!(editor.apply(Edit::WordRight));      // Alt-F: after "let"
assert_eq!(editor.cursor(), 3);

assert_eq!(editor.submit(), "let total = cost * 2");
assert!(editor.apply(Edit::HistoryPrev));    // Up: recall it
assert_eq!(editor.line(), "let total = cost * 2");

// Columns, not bytes, for placing the terminal cursor.
let mut wide = Editor::new();
wide.insert("名前 = 1");
assert_eq!(wide.cursor(), 10);
assert_eq!(wide.column(), 8);

Examples

Three runnable examples ship in examples/. Two share a small calculator language in examples/common: a lexer built on Input::cursor, a parser-lang Pratt grammar, Input::is_incomplete for continuation lines, and bindings that remember where they were defined. That module is the template for wiring repl-lang into a real language.

• Calculator REPL — an interactive loop over standard input with let bindings, multi-line entries, rendered diagnostics, and a :history command. It also runs non-interactively.

  cargo run --example calc
  echo "let x = (1 +
  2) * 7" | cargo run --example calc
  

• Notebook — a scripted session printed as In [n] / Out[n] cells: multi-line entries, blank lines that never consume a number, an interrupt, a redefinition error that underlines the original definition in an earlier cell, and the entry-size limit.

  cargo run --example notebook
  

• Line editing — replays a keystroke script through the editor and draws each state with a caret under the cursor column: word motion, kill and yank, a wide-character line, a combining accent kept with its letter, and history browsing that restores the half-typed line.

  cargo run --example line_editing
  

Performance

Feeding a line appends to one reused buffer and calls the pipeline; committing an entry makes one exact-size copy of its text and its name into the source map. The editor keeps its line, kill buffer, and every history slot between uses — once the history is full, submitting a line recycles the oldest slot instead of allocating. Display widths and word classes take an ASCII fast path before consulting the Unicode tables.

Measured with the benchmarks in benches/ (x86_64, Rust stable, release profile):

Per entry that is roughly 40–65 ns to feed, check, and commit a line, and 3–5 ns per keystroke.

Run them yourself:

cargo bench --bench bench

Criterion writes per-benchmark reports to target/criterion/. Numbers vary by CPU; use the trend across runs, not a single absolute.

Design notes

• The parser is the oracle. Completeness is a property of the language, so the session asks the language. There is no built-in bracket counter to disagree with your grammar about strings, comments, or significant indentation; a pipeline that already parses gets continuation lines from is_incomplete in one call.
• Check and work are the same call. Status::Complete carries the pipeline's result, so a finished entry is parsed once, not once to test and again to run.
• Spans are final when they are made. The pipeline learns its entry's global base before the entry is committed, and the session guarantees the entry lands exactly there. Nothing is rebased afterwards.
• Errors never echo input. SessionError reports sizes, not text, so a refused line cannot leak into a log through the error value.
• The terminal stays outside. Raw mode, key decoding, and drawing differ on every platform; editing logic does not. Keeping them apart makes the editor identical everywhere and testable without a terminal.
• Display units, not code points. The cursor moves over a visible character together with any combining marks attached to it, so it never separates a letter from its accent, and column() counts wide characters as two terminal columns.
• History keeps an entry's shape. A saved multi-line entry comes back with its newlines and tabs; every other control character, escape included, is removed before it can reach the line.

Testing

The suite runs on Windows, Linux (WSL2 Ubuntu), and macOS through the CI matrix, on stable and the 1.85 MSRV:

cargo test                       # unit + integration + property + doctests
cargo clippy --all-targets --all-features -- -D warnings
cargo bench --bench bench

The property tests in tests/proptests.rs hold both halves to reference models. Arbitrary sequences of edits, pastes, submits, and history operations — over text mixing ASCII, accents, combining marks, joiners, wide characters, emoji, and control characters — must leave the editor in exactly the state of a model that works on a Vec<char>. Arbitrary lines with arbitrary terminators and size limits must leave the session agreeing with a plain string accumulator on every pending buffer, entry number, base offset, span, and refusal. Every rust example in this README and in docs/API.md is compiled and run as a doctest, so the published examples cannot drift from the API.

Cross-platform support

• Linux (x86_64, aarch64)
• macOS (x86_64, Apple Silicon)
• Windows (x86_64)

The crate uses no operating-system facilities and no platform-specific code; behaviour is identical on every target. Line terminators are normalised on input, so "\r\n" from a Windows console and "\n" from a Unix pipe produce the same entry.

Contributing

See REPS.md for the engineering standards every change is held to, and dev/ROADMAP.md for the plan to 1.0. Before a PR: cargo fmt --all, cargo clippy --all-targets --all-features -- -D warnings, and cargo test --all-features must be clean.