Skip to main content

Crate egui_kittest

Crate egui_kittest 

Source
Expand description

§egui_kittest

Latest version Documentation MIT Apache

Ui testing library for egui, based on kittest (an AccessKit based testing library).

§Example usage

use egui::accesskit::Toggled;
use egui_kittest::{Harness, kittest::{Queryable, NodeT}};

let mut checked = false;
let app = |ui: &mut egui::Ui| {
    ui.checkbox(&mut checked, "Check me!");
};

let mut harness = Harness::new_ui(app);

let checkbox = harness.get_by_label("Check me!");
assert_eq!(checkbox.accesskit_node().toggled(), Some(Toggled::False));
checkbox.click();

harness.run();

let checkbox = harness.get_by_label("Check me!");
assert_eq!(checkbox.accesskit_node().toggled(), Some(Toggled::True));

// Shrink the window size to the smallest size possible
harness.fit_contents();

// You can even render the ui and do image snapshot tests
#[cfg(all(feature = "wgpu", feature = "snapshot"))]
harness.snapshot("readme_example");

§Configuration

You can configure test settings via a kittest.toml file in your workspace root. All possible settings and their defaults:

# path to the snapshot directory
output_path = "tests/snapshots"

# maximum weighted squared YIQ color distance between two corresponding pixels
# (a per-pixel color tolerance, applied to each pixel pair on its own)
threshold = 0.6

# how many pixels may exceed the `threshold` before the test fails
# (an absolute pixel count, not a fraction of the image)
max_failed_pixels = 0

# how many steps past `max_steps` `Harness::run` keeps stepping to report how many steps the ui
# would have needed to settle
diagnostic_max_steps = 100

[windows]
threshold = 0.6
max_failed_pixels = 0

[macos]
threshold = 0.6
max_failed_pixels = 0

[linux]
threshold = 0.6
max_failed_pixels = 0

Raise max_failed_pixels only very carefully: a high value (more than ~10) is enough to hide a real change, such as a moved separator, a shifted one-pixel border, or a small icon rendering incorrectly. Prefer the smallest value that makes the test pass, and re-check it whenever you update the snapshot.

§Accessibility check

Whenever the ui has settled (after Harness::run and friends) and before every snapshot, the harness checks that every input widget (button, checkbox, slider, text field, …) has an accessible name, and panics if one does not. A widget without a name cannot be found by get_by_label, nor by a screen reader. Give icon-only buttons an alt text, tie text fields to their label with Response::labelled_by, or add an on_hover_text.

Turn the check off with Harness::builder().with_accessibility_check(false), or run it by hand with Harness::check_accessibility.

§Recording

When enabling the recording feature, you can record tests to mp4s via these env vars:

  • KITTEST_RECORD=1 cargo test writes numbered MP4s to tests/snapshots/recordings
  • KITTEST_RECORD=open cargo test writes each recording to a temporary file and opens it

Recording needs ffmpeg on the PATH. MP4 has no alpha channel, so transparent pixels turn black.

The recorder is an egui::Plugin (RecordingPlugin), so you can also register it on any egui::Context yourself to record any egui app.

§Snapshot testing

There is a snapshot testing feature. To create snapshot tests, enable the snapshot and wgpu features. Once enabled, you can call Harness::snapshot to render the ui and save the image to the tests/snapshots directory.

To update the snapshots, run your tests with UPDATE_SNAPSHOTS=true, so e.g. UPDATE_SNAPSHOTS=true cargo test. Running with UPDATE_SNAPSHOTS=true will cause the tests to succeed. This is so that you can set UPDATE_SNAPSHOTS=true and update all tests, without cargo test failing on the first failing crate.

UPDATE_SNAPSHOTS=true will only update the images of failing tests. If you want to update all snapshot images, even those that are within error margins, run with UPDATE_SNAPSHOTS=force.

If you want to have multiple snapshots in the same test, it makes sense to collect the results in a SnapshotResults (look here for an example). This way they can all be updated at the same time.

You should add the following to your .gitignore:

**/tests/snapshots/**/*.diff.png
**/tests/snapshots/**/*.new.png

§Guidelines for writing snapshot tests

  • Whenever possible prefer regular Rust tests or insta snapshot tests over image comparison tests because…
    • …compared to regular Rust tests, they can be relatively slow to run
    • …they are brittle since unrelated side effects (like a change in color) can cause the test to fail
    • …images take up repo space
  • images should…
    • …be checked in or otherwise be available (egui uses git LFS files for this purpose)
    • …depict exactly what’s tested and nothing else
    • …have a low resolution to avoid growth in repo size
    • …have a low comparison threshold to avoid the test passing despite unwanted differences (the default threshold should be fine for most usecases!)

§What to do when CI / another computer produces a different image?

The default tolerance settings should be fine for almost all gui comparison tests. However, especially when you’re using custom rendering, you may observe image differences with different setups leading to unexpected test failures.

First check whether the difference is due to a change in enabled rendering features, potentially due to differences in hardware (/software renderer) capabilities. Generally you should carefully enforce the same set of features for all test runs, but this may happen nonetheless.

Once you validated that the differences are miniscule and hard to avoid, you can try to carefully adjust the comparison tolerances (SnapshotOptions::threshold and, as a last resort, SnapshotOptions::max_failed_pixels) for the specific test. See also TODO(#5683).

⚠️ WARNING ⚠️ Picking too high tolerances may mean that you are missing actual test failures. It is recommended to manually verify that the tests still break under the right circumstances as expected after adjusting the tolerances.

In order to avoid image differences, it can be useful to form an understanding of how they occur in the first place.

Discrepancies can be caused by a variety of implementation details that depend on the concrete GPU, OS, rendering backend (Metal/Vulkan/DX12 etc.) or graphics driver (even between different versions of the same driver).

Common issues include:

  • multi-sample anti-aliasing
    • sample placement and sample resolve steps are implementation defined
    • alpha-to-coverage algorithm/pattern can vary wildly between implementations
  • texture filtering
    • different implementations may apply different optimizations even for simple linear texture filtering
  • out of bounds texture access (via textureLoad)
    • implementations are free to return indeterminate values instead of clamping
  • floating point evaluation, for details see WGSL spec § 15.7. Floating Point Evaluation. Notably:
    • rounding mode may be inconsistent
    • floating point math “optimizations” may occur
      • depending on output shading language, different arithmetic optimizations may be performed upon floating point operations even if they change the result
    • floating point denormal flush
      • even on modern implementations, denormal float values may be flushed to zero
    • NaN/Inf handling
      • whenever the result of a function should yield NaN/Inf, implementations are free to yield an indeterminate value instead
    • builtin-function precision & error handling (trigonometric functions and others)
  • partial derivatives (dpdx/dpdx)
    • implementations are free to use either dpdxFine or dpdxCoarse
  • […]

From this follow a few simple recommendations (these may or may not apply as they may impose unwanted restrictions on your rendering setup):

  • avoid enabling mult-sample anti-aliasing whenever it’s not explicitly tested or needed
  • do not rely on NaN, Inf and denormal float values
  • consider dedicated test paths for texture sampling
  • prefer explicit partial derivative functions

§Feature flags

  • wgpu — Adds a wgpu-based test renderer.
  • snapshot — Adds a dify-based image snapshot utility.
  • video — Record a test session as an MP4 video.
  • eframe — Allows testing eframe::App

Re-exports§

pub use kittest;

Modules§

wgpu

Structs§

ExceededMaxStepsError
Harness
The test Harness. This contains everything needed to run the test.
HarnessBuilder
Builder for Harness.
Node
OsThreshold
Helper struct to define a per-OS comparison tolerance.
RecordingOptions
How to record. Pass this to crate::Harness::start_recording or RecordingPlugin::new.
RecordingPlugin
Records an egui::Context using the integration’s screenshot support.
SnapshotOptions
SnapshotResults
Utility to collect snapshot errors and display them at the end of the test.

Enums§

LazyRenderer
A lazy renderer that initializes the renderer on the first render call.
RecordingError
What went wrong when saving a recording.
SnapshotError

Constants§

RECORD_ENV_VAR
Name of the environment variable that records every crate::Harness in the process.
THRESHOLD_SWEEP
The per-pixel thresholds that a failing snapshot is measured against, to help you pick a SnapshotOptions::threshold.

Traits§

TestRenderer

Functions§

image_snapshot
Image snapshot test.
image_snapshot_options
Image snapshot test with custom options.
try_image_snapshot
Image snapshot test.
try_image_snapshot_options
Image snapshot test with custom options.

Type Aliases§

EventQueue
SnapshotResult