# pixelmap_multiview
Turn a set of ordinary photos of a scene into a 3D model.
[`pixelmap_model_3d`](https://github.com/d4per/pixelmap/tree/main/rust/model_3d) lifts a single mapping between two photos into a surface. This crate
goes further: it maps every pair of N ≥ 3 photos with [`pixelmap`](https://crates.io/crates/pixelmap), recovers
where each photo was taken from, and fuses all of the views into one mesh in a single
frame.
## From photos to a 3D model
These 23 photos were taken with a phone while walking once around a statue. Nothing else
went in: no calibration, no markers, no measurements.

`pixelmap_multiview` worked out where each photo was taken from and fused all 23 views into
this mesh, shown here untextured in MeshLab from three sides:

## Using it
Hand over the photos, say how much effort to spend, follow along, and save the result.
Decoding and PNG encoding are the caller's job; this example uses the `image` crate:
```rust,ignore
use std::{fs::File, io::BufWriter, sync::Arc};
use pixelmap::{Photo, Quality};
use pixelmap_multiview::{export, Flow, Options};
fn main() -> Result<(), Box<dyn std::error::Error>> {
// The photos must all be the same size.
let photos = ["a.jpg", "b.jpg", "c.jpg"]
.iter()
.map(|path| {
let rgba = image::open(path)?.into_rgba8();
let (width, height) = rgba.dimensions();
Ok(Arc::new(Photo::from_rgba(width as usize, height as usize, rgba.into_raw())?))
})
.collect::<Result<Vec<_>, Box<dyn std::error::Error>>>()?;
// No `.focal(...)`: the focal length is estimated from the photos and refined.
let options = Options::new().quality(Quality::Medium);
let model = pixelmap_multiview::run(&photos, &options, &mut |event| {
println!("{}: {}", event.stage(), event.message());
Flow::Continue(())
})?;
println!("{} triangles", model.mesh.triangles.len());
// The GLB and the web page both embed the texture atlas as PNG.
let atlas = &model.texture.atlas;
let mut atlas_png = std::io::Cursor::new(Vec::new());
image::RgbaImage::from_raw(atlas.width() as u32, atlas.height() as u32, atlas.as_rgba().to_vec())
.expect("the atlas buffer matches its dimensions")
.write_to(&mut atlas_png, image::ImageFormat::Png)?;
let atlas_png = atlas_png.into_inner();
// A single binary glTF file with the texture embedded. It imports straight into
// Blender (File → Import → glTF 2.0), as well as three.js and most game engines.
let mut glb = BufWriter::new(File::create("mesh.glb")?);
export::write_textured_glb(&mut glb, &model.mesh, &model.texture, &atlas_png)?;
// A web page that shows the model in 3D in any browser.
let mut html = BufWriter::new(File::create("mesh.html")?);
export::write_textured_html(&mut html, &model.mesh, &model.texture, &atlas_png, "My model")?;
Ok(())
}
```
`Options` is the whole of what a caller chooses: the quality, what is known about the
camera's focal length, the seed, whether to refine the focal length, and how large a
texture atlas to produce. The thresholds each stage judges by — about fifty of them — are
derived from those and are deliberately not part of the API: they are meaningful only
together, and publishing them would turn tuning decisions into this crate's contract.
`run` gives back a `Model`: the mesh, its texture, the camera intrinsics, where each photo
was taken from, and which photos made it. Everything is in one frame: the first camera of
the best pair sits at the origin, and the distance between that pair's cameras is the unit
of length, so nothing is metric. Everything computed along the way — tracks, both sparse
models, the depth maps and their per-sample statistics — is behind `Model::diagnostics()`,
for working out why a reconstruction came out as it did. Most callers never look, and
`Model::take_diagnostics()` frees that memory.
To reconstruct from correspondences that did not come from pixelmap, implement
`PairLookup` for them, build a `PairGraph` with `PairGraph::from_fn`, and call
`reconstruct`.
Positions and poses are `nalgebra` types, and photos are `pixelmap` types. Both crates
are re-exported as `pixelmap_multiview::nalgebra` and `pixelmap_multiview::pixelmap`, and a
new major version of either is a breaking change for this crate.
Each way a capture can fail — a camera that only turned, a flat scene, photos with too
little in common — comes back as an `Error` that names the stage and says what to do
differently.
## The stages
1. **Pairwise correspondence** — pixelmap over every pair, and a check that the
well-mapped pairs link enough views together.
2. **Two-view geometry** — the relative pose of each pair from an 8-point essential
matrix in RANSAC, plus triage that flags cameras which only rotated or moved too
little to measure depth.
3. **Tracks** — points followed across every view that sees them, kept only where each
observation agrees with every other view's mapping.
4. **Registration** — the best pair defines the frame, the other views are added by
PnP, and every track is triangulated over all the views that see it.
5. **Bundle adjustment** — every camera and point refined together by Levenberg–Marquardt
with a Schur complement and a Huber loss, optionally including the focal length.
6. **Dense depth** — a depth map per view from the full correspondences, solved along each
pixel's ray and kept only where the other views' depth maps agree. Each depth carries its
expected error, so a depth only two photos support is judged by what two photos can
deliver rather than by the standard of three.
7. **Fusion** — the depth maps merged in a sparse truncated signed distance grid, and the
surface extracted with surface nets into a mesh.
8. **Texturing** — a colour per vertex, and a texture atlas in which each part of the surface
comes from the photo that sees it best.
## Feedback while it runs
A run is minutes of work, so the caller hears about it throughout. `Event` carries data
rather than prose: which pair is being mapped and how far through it, why a pair was
rejected, which photo was left out and why. Showing progress never means parsing a
sentence, and the wording of a message is never part of the contract.
| `Started` | how many views and pairs the run is |
| `Stage` | the stage, how far through it, and a line for a log |
| `PairProgress` | the pair, its index of the total, and the solver step within it |
| `PairMapped` | the pair and its finished `PairMap` |
| `PairRejected` | the pair and a typed `Degeneracy` |
| `ViewDropped` | the photo left out, the stage, and why |
| `Log` | a level and a message |
`Event::progress()` weighs the stages against each other and gives one number for the whole
run. It is an `Option`: a log line or a dropped view can happen anywhere within a stage, and
reporting the stage's start for those would drive a progress bar backwards. `Event::message()`
renders any event as one line, for callers that want a log.
`PairMapped` carries that pair's `PairMap`: the dense correspondence itself, as the raw
grid, in the coordinates of the photos handed in rather than the lower resolution the solver
works at. It is plain data — no borrows, no generics — so a frontend can draw the match for
every pair as it is found. How to draw it is the caller's decision; the crate renders
nothing.
## Stopping a run
Returning `Flow::Break(())` from the callback stops the run, which comes back as
`Error::Cancelled` naming the stage it stopped in. A stop takes effect at the next
checkpoint. Within pairwise correspondence that is one pixelmap schedule step, which is the
longest a stop ever waits; every other stage reports often enough to be prompt.
Measured on three photos of stone steps at 1200 px, asking a run to stop at various points:
| medium | 2.3 s | 0.04 s | 0.04 s |
| high | 5.8 s | 0.12–0.18 s | 0.53 s |
So a run stops well inside a second even at the slowest setting, without interrupting a
pixelmap solve partway. `pixelmap-multiview --cancel-after SECONDS` is what measures this.
With the `threads` feature (on by default), `Job` runs a reconstruction on a worker thread
and hands back a channel of events, a `Status` to poll, and `Job::cancel()`. Turn the
feature off for `wasm32-unknown-unknown`, which has no threads to hand out; `run` works
there and is what `Job` drives underneath, so nothing is lost but the convenience.
## The command-line tool
Like the library it builds on, this crate does no I/O and has no platform dependencies, so
it builds for `wasm32-unknown-unknown` with `--no-default-features`. Decoding, EXIF and
resizing are the caller's job; the `pixelmap-multiview` binary in
[`rust/multiview_cli`](https://github.com/d4per/pixelmap/tree/main/rust/multiview_cli) of
the repository does them for photos on disc:
cargo run --release -p pixelmap_multiview_cli -- a.jpg b.jpg c.jpg d.jpg --dump-dir out
The textured surface is written to the current directory: as `mesh.obj`, with `mesh.mtl`
and `mesh_texture.png`; as X3D to `mesh.x3d`; as a single binary glTF file with the texture
embedded to `mesh.glb`; inside a web page to `mesh.html`, which shows it in 3D in any
browser (it fetches the `<model-viewer>` library from a CDN, so it needs a network
connection); and with a colour per vertex to `mesh_colours.ply`. They open in MeshLab,
Blender or CloudCompare; `mesh.glb` also opens in the Windows 3D Viewer, three.js and
`<model-viewer>`, and most game engines.
With `--dump-dir out`, the meshes go to `out` instead, along with the photos as used
(`out/view_N.png`), the sparse model (`out/sparse.ply`, with coloured points and a red
pyramid per camera), each depth map (`out/depth_N.png`) and a coverage map per photo
(`out/coverage_N.png`).
The CLI also prints, per photo, how much of it was matched to no other photo and where the
remaining depth samples were dropped.
Known limitation: pixelmap matches little within about 25–50 px (at 1200 px) of a photo's
edge, so a strip along each edge that only two photos share usually stays empty. Photos that
overlap generously avoid it; so does anything seen by a third photo away from its edge.
The command-line tool is not published to crates.io; build it from the repository.
## Time complexity
Pairwise correspondence dominates: N photos need N(N − 1)/2 pixelmap runs. One run on
a 4:3 photo pair, native build, measured with `pixelmap-multiview --time-pair`:
| low | 0.60 s | 0.56 s | 0.50 s |
| medium | 2.25 s | 2.30 s | 2.38 s |
| high | | 5.81 s | |
The input size barely matters, because pixelmap scales every photo to a fixed working
width (400 px for low, 800 px for medium, 1600 px for high) before it starts. What the input
resolution buys is the precision of the geometry that follows, not matching time.
End to end, those three photos of stone steps at 1200 px and low quality reconstruct in 1.9 s into a
mesh of 23,314 vertices and 45,708 triangles, textured from 1,528 charts in a 2048 × 2048
atlas.