Lucas Canade Optical Flow and Shi-Tomasi feature detection on Rust

High-performance Rust implementation of Lucas-Kanade optical flow and Shi-Tomasi feature detection, optimized for real-time applications and WebAssembly (Wasm) compatibility.

Features

• 🎯 Pyramidal Lucas-Kanade optical flow with per-point status and photometric error
• 🔁 Forward-backward consistency check to reject occlusions and outliers
• 🧭 Optional motion prediction (initial guess) for large inter-frame displacements
• 🔍 Shi-Tomasi feature detection, plus grid-based detection for uniform coverage
• ♻️ Zero-allocation steady-state path (TrackerContext, DetectorContext) for real-time per-frame tracking, with up-front capacity reservation
• 🌡️ f32 pipeline for images in physical units, e.g. thermal frames in °C, without 8-bit quantization
• ⚡ Fixed-point SIMD tracking kernels (SSE2/AVX2 on x86, simd128 in wasm) with results bit-identical across targets; SIMD gradients and pyramid: AVX2 (x86), NEON (aarch64), simd128 (wasm)
• 🌐 Built on the image crate; WebAssembly-ready

Usage

Add to your Cargo.toml:

[dependencies]

optical-flow-lk = "0.6"

Basic example — detect corners in one frame and track them into the next:

use image::{open, GrayImage};
use optical_flow_lk::{
    build_pyramid, calc_optical_flow_ex, good_features_to_track,
    Corner, TrackParams, TrackStatus, DEFAULT_MIN_EIGEN_THRESHOLD,
};

let prev: GrayImage = open("examples/input1.png").unwrap().into_luma8();
let next: GrayImage = open("examples/input2.png").unwrap().into_luma8();

let prev_pyr = build_pyramid(&prev, 4);
let next_pyr = build_pyramid(&next, 4);

// Keep corners scoring at least 1% of the strongest one, at least 5px apart.
let mut corners = good_features_to_track(&prev, 0.01, 5);
corners.truncate(100);
let points: Vec<(f32, f32)> = corners.iter().map(Corner::position).collect();

// 21px window, 30 iterations per level; `None` = no initial guess.
let params = TrackParams {
    window_size: 21,
    max_iterations: 30,
    min_eigen_threshold: DEFAULT_MIN_EIGEN_THRESHOLD,
};
let results = calc_optical_flow_ex(&prev_pyr, &next_pyr, &points, None, params);

for (start, r) in points.iter().zip(&results) {
    if r.status == TrackStatus::Tracked {
        println!("{start:?} -> {:?} (error {:.1})", r.pos, r.error);
    }
}

Real-time tracking

For per-frame tracking (e.g. a VIO front-end or the web demo), reuse a TrackerContext and pass it each new frame with advance. The pyramid of the last frame becomes the previous one, so only the new frame's pyramid is built and you keep no copy of the last frame. After the first frame it performs no heap allocation, and track_fb adds the forward-backward consistency check:

use optical_flow_lk::{TrackerContext, DEFAULT_FB_THRESHOLD};

let mut ctx = TrackerContext::new();

// Per frame (`frame` is a `&GrayImage`), with `params` as above:
ctx.advance(frame, 4);
let results = ctx.track_fb(&points, None, params, DEFAULT_FB_THRESHOLD);
// Points flagged `TrackStatus::FbInconsistent` failed the round-trip.

The context computes each frame's Scharr gradients once and keeps them with its pyramid, so the backward pass of one frame and the forward pass of the next share them. TrackOptions tunes a call further: fb_levels runs the backward pass over the finest levels only, which on a phone makes track_fb with one level about 30% cheaper, and start_level skips coarse levels when a prediction is already close:

use optical_flow_lk::{TrackOptions, DEFAULT_FB_THRESHOLD};

const OPTIONS: TrackOptions = TrackOptions::new().with_fb_levels(1);
let results = ctx.track_fb_with(&points, None, params, DEFAULT_FB_THRESHOLD, OPTIONS);

A single-level check is slightly more permissive: on the bundled example pair it accepts 14-16 of 354 points that the full check rejects, and rejects 0-2 that it accepts.

advance(frame, 4) gives the same results, bit for bit, as prepare(&last_frame, frame, 4), which builds both pyramids from an explicit frame pair. On the first call there is no last frame yet, so both pyramids show frame and tracking measures no motion. FloatTrackerContext has the same advance.

Detection has the same shape. A DetectorContext owns the gradient, tensor and score planes plus the spacing grid, so detect / detect_grid allocate nothing after the first frame:

use optical_flow_lk::{DetectorContext, Grid};

let mut detector = DetectorContext::new();
// Top up lost tracks every few frames: 6x4 grid, 3 per cell, quality 0.05,
// at least 12 px from each other and from the points already tracked.
let grid = Grid { cols: 6, rows: 4, max_per_cell: 3 };
let fresh = detector.detect_grid(&prev, grid, 0.05, 12, &points);

To keep even the first frame free of allocation, create the contexts with with_capacity. A DetectorCapacity reserves for the largest possible candidate count, so no image content can grow a buffer later:

use optical_flow_lk::{DetectorCapacity, DetectorContext, TrackerCapacity, TrackerContext};

let mut detector = DetectorContext::with_capacity(DetectorCapacity {
    width: 640, height: 480, min_distance: 12, grid_cells: 24, existing_points: 200,
});
let mut tracker = TrackerContext::with_capacity(TrackerCapacity {
    width: 640, height: 480, levels: 4, window_size: 21, points: 200,
});

f32 images in physical units

For sensors whose values carry a unit, such as thermal cameras in °C, FloatDetectorContext and FloatTrackerContext run the same algorithms on f32 frames, without range normalization or 8-bit quantization. Scores and the tracker's eigenvalue threshold are in (unit / pixel)^2, residuals in the unit itself. A FloatImageView borrows a frame after checking that every value is finite and within range:

use optical_flow_lk::{
    Corner, FloatDetectorContext, FloatImageView, FloatTrackerContext, ScoreThreshold,
    TrackParams, DEFAULT_FB_THRESHOLD,
};

// `prev_celsius` and `next_celsius` are 640x480 row-major `&[f32]` frames.
let prev = FloatImageView::new(prev_celsius, 640, 480)?;
let next = FloatImageView::new(next_celsius, 640, 480)?;

// Corners scoring at least 0.002 (°C / px)^2, at least 8 px apart.
let mut detector = FloatDetectorContext::new();
let corners = detector.detect(prev, ScoreThreshold::absolute(0.002), 8);
let points: Vec<(f32, f32)> = corners.iter().map(Corner::position).collect();

// The eigenvalue threshold is in (°C / px)^2 as well and has no default.
let params = TrackParams { window_size: 15, max_iterations: 15, min_eigen_threshold: 1e-4 };
let mut tracker = FloatTrackerContext::new();
tracker.prepare(prev, next, 3);
let results = tracker.track_fb(&points, None, params, DEFAULT_FB_THRESHOLD);

On frames holding 8-bit values the f32 detector returns the 8-bit detector's corners bit for bit. On a 640x480 frame on an Apple M2, a whole frame step (pyramids, forward-backward tracking, grid top-up) costs about 1.1x the 8-bit pipeline on the same data, natively and under WASI; tracking alone is faster in f32. See the measurements. Against the published 0.3.1, the f32 frame step is 3.3-3.6x faster natively and 2.9-3.1x faster under WASI (comparison).

Upgrading from 0.4: detection returns Corners, grid detection takes a Grid, and tracking takes TrackParams. The deprecated calc_optical_flow is gone; use calc_optical_flow_ex or TrackerContext. See the changelog.

Performance

Per-frame tracking

The step of a real-time front-end: TrackerContext::advance with the new frame, then track_fb of the strongest corners, on synthetic textured frames 3 px apart, 4 levels, 30 iterations. Measured for 0.6.0 on 2026-10-20, in alternation with 0.5.1.

On a phone, the deployment target: Xiaomi 11T (MediaTek Dimensity 1200, arm64), Android 14, Chrome 155, the web demo's ?bench=1 mode at 360×640. Medians per step over 60 samples of 5 steps, after the CPU clock ramped up.

Each tracked point costs 2.5-2.7x less than in 0.5.1. On a desktop, AMD Ryzen 9 9900X, benches/tracking.rs (regime) at 360×640 with 150 points and a 21 px window:

All cases, 640×480 included, are in the changelog.

Against OpenCV

Measured against OpenCV 4.11 on the bundled 720×405 frame pair; every row re-measured for 0.6.0 on 2026-10-20. The harness, parameters and caveats are in benches/opencv-compare/. Tracking is both pyramids plus Lucas-Kanade for 200 points, 21 px window, no forward-backward check.

On the phone above. Medians per call over 100 samples of 5 calls, steady-state runs; 0.5.1 measured 2.5 and 13.0 ms in the same session.

On the desktop above, wasm under Node 24. Medians of 100 calls.

The desktop wasm detection figure was 0.8 ms at 0.4.1. The grouped non-maximum suppression of 0.5.0 is faster natively, on the phone and on the synthetic benchmark inputs, but slower on this frame under V8 on x86-64. Details and the per-input figures are in the comparison README.

Corners agree with OpenCV's to the pixel: every corner this crate finds has an OpenCV Scharr corner within 1.5 px, and 99% of OpenCV's have one of this crate's. Tracked endpoints agree to a median of 0.004 px on points both libraries track.

Since 0.6.0 the 8-bit tracker computes its windows in fixed point on x86 and on wasm with simd128, bit for bit the same on every target; positions differ from the f32 arithmetic of 0.5 by a median of 0.0003 px on the bundled pair. TrackKernel::Reference selects the f32 arithmetic, see TrackOptions.

Live demo

A browser demo runs the tracker entirely client-side in WebAssembly: point your phone's camera at a scene and tap to drop points (or hit Auto to detect Shi-Tomasi corners) and watch them ride the optical flow.

▶ lk-demo.jt3.ru

Source and build instructions are in web-demo/.

WebAssembly

The hot image kernels — the fixed-point tracking windows, the Scharr gradients and the pyramid downsample — have hand-written simd128 paths that are selected automatically on wasm32 — but only when the target is built with SIMD enabled, since WASM has no runtime feature detection. The bundled .cargo/config.toml sets this for you:

[target.wasm32-unknown-unknown]

rustflags = ["-C", "target-feature=+simd128"]

If you build from a different working directory (so that config is not picked up), pass it yourself:

RUSTFLAGS="-C target-feature=+simd128" cargo build --release --target wasm32-unknown-unknown

Without +simd128 the crate still works, falling back to scalar loops, and the 8-bit tracker uses its f32 arithmetic (TrackKernel::FixedPoint selects the scalar fixed-point kernel, with the results of the simd128 build). For production WASM, also run wasm-opt -O3 on the output (wasm-pack does this automatically). Tracking 200 points on the bundled pair, the simd128 build is 3× faster than the scalar build in V8 (Node): 1.8 against 5.5 ms.