litnik 0.2.0

An embedded Lit and FileCheck runner for compiler test suites.
Documentation

litnik

Litnik embeds Lit integrated shell tests and FileCheck in a Rust test target. The test suite does not need the LLVM Lit or FileCheck executables.

Version 0.1.0 is yanked. It did not connect registered tools to command execution, did not contain FileCheck, and accepted failed discovery as an empty success. Version 0.2.0 is the replacement.

Set up a suite

Add Litnik as one development dependency:

[dev-dependencies]
litnik = "0.2"

[[test]]
name = "lit"
path = "tests/lit.rs"
harness = false

Put the suite configuration in tests/lit.rs:

fn main() {
    litnik::Suite::new(concat!(env!("CARGO_MANIFEST_DIR"), "/tests/lit"))
        .suffix("mlir")
        .tool("butcher-opt", env!("CARGO_BIN_EXE_butcher-opt"))
        .feature("butcher-registered-target")
        .run_from_args();
}

Write normal Lit syntax in the compiler test:

// RUN: butcher-opt %s | FileCheck %s
// REQUIRES: butcher-registered-target

// CHECK-LABEL: func.func @fold_me
// CHECK: arith.constant 42

Run it through Cargo:

cargo test --test lit
cargo test --test lit -- --list

FileCheck is an embedded command. Do not register a path for it. A tool that is registered with Suite::tool takes precedence over PATH and can be used by its exact name in each RUN: line.

Lit model

Litnik keeps test files in Lit's text format. Rust configures the suite around that format; it does not translate directives into Rust test cases.

The suite builder configures these Lit concepts:

  • source and execution roots;
  • test suffixes and excluded names;
  • tool paths and environment variables;
  • substitutions and available features;
  • test time limits, retries, and pipeline failure behavior.

Discovery is recursive. An unreadable root, an unreadable directory entry, or zero discovered tests fail the harness.

Compatibility boundary

Litnik reads RUN:, DEFINE:, REDEFINE:, REQUIRES:, UNSUPPORTED:, XFAIL:, ALLOW_RETRIES:, and END. from the test file. It supports Lit path, line-number, recursive, and conditional substitutions. The shell runs commands, pipelines, ordered redirections, &&, and || without a host shell. It implements cd, pushd, popd, export, echo, mkdir, rm, umask, and : in process. The cat and diff commands resolve through PATH. Litnik rejects ulimit because it cannot apply that limit correctly yet.

The embedded FileCheck supports the main LLVM directives, string and numeric variables, expressions, check prefixes, comment prefixes, implicit negative checks, whitespace modes, full-line matching, case control, and variable scope.

Litnik does not execute Python lit.cfg files or delegate commands to a host shell. Put the corresponding suite values in tests/lit.rs.

Embedded FileCheck is not byte-for-byte compatible with LLVM FileCheck yet. It requires UTF-8 check files and input, uses Rust regular-expression behavior, and does not expand response files. The pinned engine does not dump passing input for --dump-input=always; it also does not implement --dump-input-context. The -v option does not report successful matches, --color has no effect, and diagnostics do not use LLVM's exact layout. These differences are part of the current 0.2.0 compatibility boundary.

Platform and toolchain

Litnik requires Rust 1.92 or later. The runner supports Unix and Windows process execution.

The crate uses synchronous filesystem and process operations. Cargo's test harness can run independent test files in parallel. Time limits stop child processes. They do not preempt an embedded command such as FileCheck while that command is running.

The standard harness reports XFAIL and FLAKYPASS as successful tests because libtest-mimic has no custom success statuses. The low-level TestResult API preserves the exact Lit result code.