Mosaicking

Illustration of a 3-camera array with overlapping fields of view. It is often desirable to combine the output of multi-camera systems into a single image, a mosaic. Such image aligns the different cameras into a common coordinate system and provides the "big picture" of the captured scene. Accurate and efficient estimate of this mosaic is necessary, as this is often the first step in a computer vision or image processing pipeline.

Our Solution

Our mosaicking algorithm estimates mosaics with unparalleled accuracy and efficiency, with only a minimal overlap between cameras needed. It achieves this by performing joint optimization of the mosaic parameters, handling lens or other camera distortions, and judiciously choosing the constraints in the objective function for maximum efficiency. Furthermore, it minimizes photometric differences between cameras to ensure a seamless render of the mosaic.

Joint Optimization

Line chart showing a rapidly decreasing function. Optimizing all the parameters at once allows for quick convergence. Moreover, it enables reaching the global optimum of the objective, provided a good parameter initialization.

Camera Distortion Model

Heatmap of an image showing a large amount of distortion in the corners. Our camera distortion model, based on the piecewise affine deformation of an image, is able to handle arbitrary non-linear distortions in images. Moreover, the change in the deformation with respect to its parameters can be expressed analytically, allowing an easy integration of the deformation model in the joint optimization and quick convergence.

Optimization Constraints

Aerial image with a sparse set of green dots indicating pixels used in the optimization. The joint optimization of the parameters can use feature correspondences or intensity variance of the overlap region (brightness constancy) as constraints. Intensity variance constraints are sampled sparsely, yet ensuring maximum effect on the optimization. An efficient context-sensitive validation of feature correspondences removes nearly all false matches.

Seamless Render

Brightness differences across cameras are compensated by modeling these differences with a simple, yet effective, parametric model and solving for a jointly optimal brightness correction that minimizes them. Any remaining photometric differences virtually disappear with a blending of the overlap regions.

Standard rendering of a mosaic.
Seamless rendering of a mosaic.

Results

Our mosaicking solution is able to efficiently generate mosaics for the entire CLIF2006 and WPAFB2009 datasets. The estimated parameters for these datasets are available for download and can be used for mosaic rendering with the provided demonstration program.

Mosaic Parameters

  • WPAFB2009 (819 KiB)

    The parameters are different for every frame, as there are slight shifts in the camera array.

  • CLIF2006 (2.7 MiB)

    The parameters are different for every frame, as there are slight shifts in the camera array.

Demonstration Program

Download (1.2 MiB)

Linux x86-64 (Red Hat 7 or later, Ubuntu 14.04 or later)

Video Demonstration

Licensing

Our mosaicking solution is available for commercial licensing. Please contact us at info@jgradient.com for a customized offer or any additional information.