Neural Networks Learn To Remove Noise Without Seeing A Good Image
Written by Mike James
Sunday, 15 July 2018

Don't you just hate film grain? Except, of course, when you don't and set it as a special effect on your digital camera. NVIDIA now has a tool that will remove grain, noise and all manner of defects, and the amazing part is that it learned how to without seeing a clean image. How?

From the point of view of machine learning this is almost magical. How can you teach a neural network to remove noise from an image when it is never given an example of what a clean image is. In other words, it isn't given the target it is aiming for. From the point of view of computational photography, or more generally signal processing, this is more than magic:

We observe that under suitable, common circumstances, we can learn to reconstruct signals from only corrupted examples, without ever observing clean signals,and often do this just as well as if we were using clean examples.

The idea is almost lunatic in its conception, but it has its roots going back to the use of the simple mean, or average, to estimate a value. The point is that, while the noise varies between images, the image remains constant. Thus, by training a network using nothing but noisy images, the network slowly learns to produce the average value of the images.

To illustrate the idea the team took images from the well known ImageNet and added random noise to them. They then trained a neural network using just the corrupted images. That is, the network was attempting to reproduce a corrupted image using a different corrupt image as its input. Over time it learned to produce the average, or probably more accurately the invariant part, of the image. Then, when shown a corrupt image it hasn't been trained on, it produces a cleaned up version.

This works much better than you could reasonably expect. See the video:

From simple Gaussian noise, the team went on to remove more complex types of corruption from the images. In particular, they had success removing a particularly difficult form of noise - Monte Carlo noise - that other methods have a tough time with.

"We have shown that simple statistical arguments lead to surprising new capabilities in learned signal recovery; it is possible to recover signals under complex corruptions without observing clean signals, at performance levels equal or close to using clean target data. There are several real-world situations where obtaining clean training data is difficult: low-light photography (e.g., astronomical imaging),physically-based image synthesis, and magnetic resonance imaging."

This is a powerful idea - you can learn the noise structure just from degraded samples. As long as you can get multiple samples of the noise, you don't need the original.

Is there something deeper going on here?

Noise2Noise: Learning Image Restoration without Clean Data

Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala and Timo Aila, NVIDIA, Aalto University, and MIT.

#### Related Articles

Using Super Seeing To See Photosynthesis

Selfie Righteous - Corrects Selfie Perspective

Computational Photography Shows Hi-Res Mars

Super Resolution

Seeing Buildings Shake With Software

See Invisible Motion, Hear Silent Sounds Cool? Creepy?

Extracting Audio By Watching A Potato Chip Packet

 Udacity's New Discovering Ethical AI Course12/04/2024Udacity has just launched an hour-long course on Ethical AI. Intended for a wide audience across many industries, it introduces to basic concepts and terms needed to step into the world of Ethica [ ... ] + Full Story Avi Wigderson Gains Turing Award16/04/2024Israeli mathematician and computer scientist, Avi Wigderson, is the recipient of the 2023 ACM A.M Turing Award which carries a \$1 million prize with financial support from Google. + Full Story More News