ipol.im » pub » algo »  
Image burst denoising
edit This workshop may eventually be submitted and published.
  1. Overview
  2. References
  3. Datasets
  4. Algorithm
  5. Hybrid Denoising Scheme
  6. Details
  7. Experiments
  8. Conclusion

Overview

Taking photographs under low light conditions with a hand-held camera is problematic. A long exposure time can cause motion blur due to the camera shaking and a short exposure time gives a noisy image. We consider the new technical possibility offered by cameras that take image bursts. Each image of the burst is sharp but noisy. In this preliminary investigation, we explore a strategy to efficiently denoise multi-images or video. The proposed algorithm is a complex image processing chain involving accurate registration, video equalization, noise estimation and the use of state-of-the-art denoising methods. Yet, we show that this complex chain may become risk free thanks to a key feature: the noise model can be estimated accurately from the image burst. Preliminary tests will be presented. On the technical side, the method can already be used to estimate a non parametric camera noise model from any image burst.

References

  1. Toni Buades, Yifei Lou, J.M. Morel and Zhongwei Tang, "A Note on multi-image denoising",
    In the proceeding of the International Workshop on Local and Non-Local Approximation (LNLA) in Image Processing, 2009.
    pdf, talk slides

Datasets

painting MuseOrsay largeFrameLabCam.zip

3D scene with motion booksDiffLightsMouse.zip

Algorithm

Registration of Image Sequence

  1. Input is the image sequence V = {V0, · · · ,VN}.
  2. Compute SIFT points on V0 and the rest Vj.
  3. Keep SIFT matching points among all the frames, denoted as CommonSIFTpoints.
  4. Apply multiple ORSA on the set of CommonSIFTpoints to get the most meaningful 4 points.
  5. For each j > 0, compute the homography between Vj and V0 using 4 points and register Vj to V0.

Noise Estimation

We use the sequence of registered images to estimate the signal-dependent noise curve.

In a burst of images, if one pixel is well-registered, its values along time give samples to estimate the noise model.

  1. The histogram of the mean of each pixel along time is constructed with n uniform bins.
  2. Inside each bin, the median value of the standard deviations is computed, thus yielding a curve of mean versus standard deviation.

How to deal with the wrongly registered pixels?

  1. Exclude the edge points on which interpolation error occurs during registration. This can be done by Canny edge detector.
  2. Exclude the pixels whose standard deviation is large than a threshold.
  3. Exclude the bins that contain very few items.

Video equalization beforehand by Midway

Hybrid Denoising Scheme

Suppose we have two denoising results

  1. uA = averaging after registration
  2. uBM3D = any state-of-art denoising method (BM3D is used here)

Thanks to the noise estimation, it is possible to make (soft) decision on which method to denoise at certain pixel.

uH = W x uBM3D + (1 − W ) x uA where W is a weight function, depending on the expected standard deviation and the empirical standard deviation, such as

W(x) = 1-\exp\left\{-c\max\left(\frac {\sigma_r-\sigma_e}{\sigma_e}-1, 0\right)\right\}
with

Details

SIFT

Modified ORSA

Given a set S of the common points and the matching points in the j-th image, denoted as Sj. Set ε = +∞.

Repeat until the number of trials does not exceed N :

  1. Pick up 4 random points from S.
  2. For each image Vj,
    • compute the homography using these 4 points and the corresponding ones in Sj.
    • find the most meaningful subset of S with respect to S under this homography, save the meaningfulness parameter as εj.
  3. Compute the joint meaningfulness ε0 = Σ εj.
  4. If ε0 < ε, then ε = ε0, and save the meaningful subset for each pair of images as Tj and the 4 points, P4.

Return ε0 Tj and P4.

Experiments

Examples list of processed images.

Synthetic Data

one of the simulated data with translation and Poisson noise
The zoom-in part in one particular frame
Video NLM after registration
Video BM3D after registration
Averaging after registration
Hybrid scheme
The zoom-in part in the original image to synthesize the noisy data

Comments: block-matching schemes remove the textures that look like noise, which are well-preserved by averaging.

Noise Curves are expected to be square root function

Paintings

One particular frame Noise curve
one particular frame after color-balancing
Video NLM after registration
Video BM3D after registration
Average after registration
Hybrid scheme

3D Scene

One particular frame Noise curve
one particular frame
Video BM3D after registration
Average after registration
Hybrid scheme

3D Scene with Moving Object

Three frames from the raw data
After SIFT and MidWay
Noise curve before MidWay Noise curve after MidWay
one particular frame after color-balancing
Video NLM after registration
Video BM3D after registration
Average after registration
Hybrid scheme

Noise Curves

one of input images average after SIFT Noise curve
one of input images average after SIFT Noise curve

Conclusion

Pros

  1. Directly obtaining a noise model from a burst.
  2. Camera shaking is well approximated by homography.
  3. Denoising performance is predictable as square root of N, where N is the number of the images in a burst
  4. Images are averaged along time, thus noises are always independent.
  5. It works with JPEG, and can be extended to raw data

Cons

  1. no ground-breaking improvement over BM3D
  2. Complexity: N times of SIFT + 1 BM3D
  3. memory

Future Work

  1. denoise each frame in video
  2. involve 3D depth estimation
  3. replace SIFT by fast local registration
  4. beat ground-truth averaging
π