English | 한국어

cumulus3d-dense

Undistortion and multi-view densification: estimates, filters, and fuses depth maps from a registered sparse model and its images to produce a dense point cloud (PLY).

Pipeline stage: the last stage, after SfM (cumulus3d-sfm) and GPS alignment (cumulus3d-align). Sparse model → undistortion (PINHOLE) → densification scene → neighbor views and depth ranges → multi-scale PatchMatch depth maps → filtering and post-processing → fusion → dense point cloud.

PatchMatch execution itself (red–black propagation at one scale, view selection, refinement, readout) is handled by a backend implementing PatchMatchBackend; this crate handles everything around it (scene, neighbor selection, depth ranges, multi-scale progression, upsampling, detail-restoration decisions, filter thresholds, post-processing, fusion, statistics).

Main entry points

Public items

"Root" marks items re-exported at the crate root as cumulus3d_dense::name.

undistort — undistortion

image — image buffers

scene — densification scene

params — settings

kernel — backend boundary

densify — progression

fusion, fusion_score — fusion

neighbors — neighbor views and depth ranges

upsample — pyramid and upsampling

postproc — post-processing

stats — quality statistics

cache — depth map cache

math — numerical tools

synthetic — synthetic scenes for testing

Examples

Fuse the ground-truth depth maps of a synthetic scene on the CPU (no backend needed; same as the doc-test in src/lib.rs):

use cumulus3d_dense::neighbors::PairStats;
use cumulus3d_dense::synthetic::{make_scene, SynthConfig};
use cumulus3d_dense::{fuse, FusionInput, FusionParams};

let cfg = SynthConfig { width: 96, height: 72, focal: 75.0, num_points: 500, ..SynthConfig::default() };
let s = make_scene(&cfg);
let stats = PairStats::new(&s.scene);
let overlap: Vec<Vec<usize>> = (0..s.scene.views.len()).map(|v| stats.select(v, 50, 0.0)).collect();
let inputs: Vec<Option<FusionInput>> =
    s.depth.iter().zip(&s.normal).map(|(d, n)| Some(FusionInput::plain(d, n))).collect();
let out = fuse(&s.scene, &inputs, &overlap, &FusionParams::default(), 1);
assert!(out.cloud.len() > 0);

Real densification (requires a GPU backend and real data; the doc-test is no_run):

use cumulus3d_dense::{densify, undistort_from_dir, DenseScene, DensifyOptions, PatchMatchBackend, SceneOptions, UndistortCache, UndistortOptions};

fn run(backend: &dyn PatchMatchBackend) -> cumulus3d_core::Result<()> {
    let rec = cumulus3d_core::interop::read_model("sparse/0")?;
    let und = undistort_from_dir(&rec, "images", &UndistortOptions::pipeline(), &UndistortCache::new())?;
    let scene = DenseScene::from_reconstruction(&und.reconstruction, &und.images, &SceneOptions::default())?;
    let out = densify(&scene, &DensifyOptions::default(), backend, None)?;
    out.write_ply("dense/fused.ply")?;
    Ok(())
}

Pass cumulus3d_cuda::CudaPatchMatch as backend (see the cumulus3d-cuda crate).

Feature flags and hardware

• No feature flags. Pure Rust; builds on any machine.
• Densification (densify, compute_depth_maps) requires a PatchMatchBackend implementation. This crate has no CPU backend; the only current implementation is CudaPatchMatch in cumulus3d-cuda (NVIDIA GPU, CUDA 12.x driver).
• What works on the CPU without a backend: undistortion, scene construction, neighbor selection and depth ranges, upsampling and median filtering, post-processing, fusion (fuse, fuse_depth_maps, fuse_output), statistics, synthetic scenes. Parallelism via rayon.

Processing flow

1. Scene: undistorted model (pinhole) + images → views (K, pose, gray/color). Pixel coordinates inside the scene are integer = pixel center (principal point −0.5).
2. Neighbor views: from sparse point tracks, compute the shared point count and the 75th-percentile triangulation angle for each view pair, and pick up to 10 views with an angle of at least 1°, ordered by shared count. Optionally, per-bin attenuation over baseline azimuth spreads the chosen directions evenly (NeighborParams::diversity_decay < 1).
3. Depth range: the 1%/99% values of observed sparse point depths times 0.75/1.25. Used only for random initialization.
4. Multi-scale (downscale factor 0.5, up to 3 levels): the coarsest scale runs a photometric pass from random initialization; higher scales do joint bilateral upsampling → detail restoration (pixels where upsampled-hypothesis cost − photometric-pass cost > 0.1 are replaced with the photometric result) → 2 geometric-consistency runs (double-buffered: every reference view reads the previous round's depth snapshot, so the result is independent of execution order).
5. Readout and filtering: after a 5×5 median plane filter, keep only pixels for which at least 2 source views (or the number of source views, if fewer) satisfy all of: triangulation angle ≥ 3°, incidence cos > 0, emission visibility probability ≥ E (0.9), forward–backward reprojection ≤ 1 px.
6. Post-processing: remove small, high-cost components among relative-depth-connected regions; optionally fill holes, stopping at intensity edges.
7. Fusion: the default is consistency fusion — project the reference pixel's point into overlapping views; if at least 5 views agree in depth (1% relative), reprojection (2 px), and normal (10°), output the inverse-variance-weighted average of the reference and the consistent points (σ_d = d²·σ_px/(f·b), σ_px = 0.25 + cost). Traversal median fusion (FusionMode::Traversal) and score fusion (ScoreFusionOptions) are also available.
8. Output: PLY (x y z nx ny nz red green blue, 27 bytes/point).

Profiles

Determinism

Per-pixel random numbers come from a counter-based sequence determined by (seed, image name hash, scale, run, iteration, half-step, pixel), so they are independent of thread scheduling. Geometric runs use a double-buffered snapshot, so they are independent of view processing order. Consistency fusion and score fusion are also deterministic (traversal fusion is not when run in parallel).

Tests

• cargo test -p cumulus3d-dense: numerical reference values, neighbor selection, relative geometry, upsampling and median filtering, fusion of synthetic-scene ground-truth depth maps, second-pass fusion, post-processing, undistortion.
• GPU accuracy: tests/patchmatch_gpu.rs in cumulus3d-cuda (compared against synthetic-scene ground truth).

References

• M. Bleyer, C. Rhemann, C. Rother. PatchMatch Stereo – Stereo Matching with Slanted Support Windows. BMVC 2011.
• S. Galliani, K. Lasinger, K. Schindler. Massively Parallel Multiview Stereopsis by Surface Normal Diffusion. ICCV 2015.
• J. L. Schönberger, E. Zheng, J.-M. Frahm, M. Pollefeys. Pixelwise View Selection for Unstructured Multi-View Stereo. ECCV 2016.
• Q. Xu, W. Tao. Multi-Scale Geometric Consistency Guided Multi-View Stereo. CVPR 2019.
• J. Kopf, M. F. Cohen, D. Lischinski, M. Uyttendaele. Joint Bilateral Upsampling. SIGGRAPH 2007.

License

Apache-2.0 (LICENSE, NOTICE).