Neural Networks Learn To Remove Noise Without Seeing A Good Image
Written by Mike James
Sunday, 15 July 2018

Don't you just hate film grain? Except, of course, when you don't and set it as a special effect on your digital camera. NVIDIA now has a tool that will remove grain, noise and all manner of defects, and the amazing part is that it learned how to without seeing a clean image. How?

From the point of view of machine learning this is almost magical. How can you teach a neural network to remove noise from an image when it is never given an example of what a clean image is. In other words, it isn't given the target it is aiming for. From the point of view of computational photography, or more generally signal processing, this is more than magic:

We observe that under suitable, common circumstances, we can learn to reconstruct signals from only corrupted examples, without ever observing clean signals,and often do this just as well as if we were using clean examples.

The idea is almost lunatic in its conception, but it has its roots going back to the use of the simple mean, or average, to estimate a value. The point is that, while the noise varies between images, the image remains constant. Thus, by training a network using nothing but noisy images, the network slowly learns to produce the average value of the images.

To illustrate the idea the team took images from the well known ImageNet and added random noise to them. They then trained a neural network using just the corrupted images. That is, the network was attempting to reproduce a corrupted image using a different corrupt image as its input. Over time it learned to produce the average, or probably more accurately the invariant part, of the image. Then, when shown a corrupt image it hasn't been trained on, it produces a cleaned up version.

This works much better than you could reasonably expect. See the video:

From simple Gaussian noise, the team went on to remove more complex types of corruption from the images. In particular, they had success removing a particularly difficult form of noise - Monte Carlo noise - that other methods have a tough time with.

"We have shown that simple statistical arguments lead to surprising new capabilities in learned signal recovery; it is possible to recover signals under complex corruptions without observing clean signals, at performance levels equal or close to using clean target data. There are several real-world situations where obtaining clean training data is difficult: low-light photography (e.g., astronomical imaging),physically-based image synthesis, and magnetic resonance imaging."

This is a powerful idea - you can learn the noise structure just from degraded samples. As long as you can get multiple samples of the noise, you don't need the original.

Is there something deeper going on here?

Noise2Noise: Learning Image Restoration without Clean Data

Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala and Timo Aila, NVIDIA, Aalto University, and MIT.

#### Related Articles

Using Super Seeing To See Photosynthesis

Selfie Righteous - Corrects Selfie Perspective

Computational Photography Shows Hi-Res Mars

Super Resolution

Seeing Buildings Shake With Software

See Invisible Motion, Hear Silent Sounds Cool? Creepy?

Extracting Audio By Watching A Potato Chip Packet

 5 Low-Cost Ways To Improve the Security of Your eCommerce Website16/10/2020Running a website is not an easy task, as you have to make sure that it is secure. Here are some tips for those moving their retail outlets online.  + Full Story Learn JavaScript and Node.js With Microsoft15/10/2020Microsoft loves Open Source and loves Python. Now it seems, it loves JavaScript too? Who would have thought that someday Microsoft would promote and teach languages and frameworks not based on .N [ ... ] + Full Story More News