scriptmark 0.2.0

Automated grading CLI for student programming assignments
Documentation

ScriptMark

CI Crates.io PyPI License

Automated grading CLI for student programming assignments. Rust core, TOML test specifications, Python bindings via PyO3.

Installation

Rust CLI

cargo install scriptmark

Python

pip install scriptmark

import scriptmark

results = scriptmark.grade(["submissions/"], "tests/")
for r in results:
    print(f"{r.student_id}: {r.grade:.1f} ({r.passed}/{r.total})")

From source

git clone https://github.com/Acture/scriptmark.git
cd scriptmark
cargo install --path crates/scriptmark

Requires python3 on PATH for running student Python code.

Features

  • Custom test engine -- runs student code in subprocess, no pytest/unittest required
  • TOML-based specs -- declarative test cases with expected values, checkers, and parametrization
  • 8 built-in checkers -- exact, approx, sorted, set_eq, contains, regex, text, plus Rhai expressions and external Python checkers
  • Parallel execution -- tokio-based orchestrator grades all students concurrently
  • Sandboxed execution -- env isolation, import allowlist, resource limits (setrlimit), timeout with process kill
  • Parametrize + oracle -- auto-generate test cases with random inputs and teacher reference implementations
  • Canvas LMS integration -- pull rosters, push grades
  • Similarity detection -- style + structural code similarity scoring
  • Interactive TUI -- browse students, sessions, and similarity in the terminal
  • HTML reports -- standalone dashboard with per-student breakdowns
  • SQLite storage -- persist grading history across sessions
  • Python API -- scriptmark.grade(), scriptmark.run(), scriptmark.discover(), scriptmark.load_spec()

Quick Start

  1. Write a TOML test spec:
[meta]
name = "find_max"
file = "lab5.py"
function = "find_max"
language = "python"

[[cases]]
name = "basic"
args = [[3, 1, 5, 2]]
expect = 5

[[cases]]
name = "negative"
args = [[-3, -1, -5]]
expect = -1

[[cases]]
name = "random inputs"
[cases.parametrize]
count = 20
seed = 42
[cases.parametrize.args]
nums = "list(int(-100, 100), 5, 20)"
[cases.parametrize.oracle]
rhai = "nums.sort(); nums[nums.len() - 1]"
  1. Grade submissions:
scriptmark grade submissions/ -t tests/

TOML Spec Format

Each .toml file defines tests for one function/file:

Section Purpose
[meta] Target file, function, language, teacher imports, allowed_imports
[vars] Constants injected as Python globals
[[setup]] Call functions, store results as $ref for later cases
[[cases]] Scored test cases with expect, check, or expect_error
[cases.parametrize] Auto-generate cases with random args + oracle
[lint] Optional style scoring via pylint/ruff

Checkers

Checker Usage
exact (default) expect = 42
approx expect = 3.14 with tolerance = 0.01
text Normalized multiline comparison (strips trailing whitespace + blank lines)
sorted Validates array is sorted
set_eq Unordered array comparison
contains Substring match
regex check = { regex = "^\\d+$" }
Rhai expression check = { rhai = "result > 0 && result < 100" }
Python script check = { python = "verifiers/check.py" }

CLI Commands

scriptmark grade        Run tests + summarize + display grades
scriptmark run          Run tests only, output JSON
scriptmark summarize    Re-analyze existing results
scriptmark similarity   Detect code similarity between submissions
scriptmark report       Generate HTML report
scriptmark db           Database management (init, import-roster, sessions, history)
scriptmark tui          Interactive terminal UI
scriptmark roster-pull  Pull roster from Canvas LMS
scriptmark grades-push  Push grades to Canvas LMS

Architecture

Single Rust crate (scriptmark) with modular structure:

models/       Data models, TOML spec parsing, grading policies
discovery     Student file discovery + ZIP extraction
runner/       PythonExecutor, orchestrator, sandbox, parametrize, oracle
checker/      Checker trait + 8 built-in implementations + Rhai + Python
db/           SQLite persistence (rusqlite, bundled)
canvas/       Canvas LMS API client (reqwest + rustls)
tui/          Interactive terminal UI (ratatui)

scriptmark-py is a separate PyO3 cdylib crate providing Python bindings, distributed via PyPI.

License

GPL-3.0-or-later