This Neural Network That Colorizes And Gets Rid Of Flickers In Old Film Feels Like Magic
Peter Jackson's "They Shall Not Grow Old" brought colorization of old film into the mainstream, but how feasible is the process if you don't have a Hollywood studio backing you?
According to this paper from researchers Satoshi Iizuka and Edgar Simo-Serra, using a temporal convolutional neural networks along with reference images for color make the process relatively attainable. The technique has two parts — restoring the black and white footage, and then figuring out how to apply the colors from the reference images:
The model is input a sequence of black and white images which are restored using a pre-processing network and used as the luminance channel of the final output video. Afterwards, a source-reference network uses an arbitrary number of reference color images in conjunction with the output of the pre-processing network to produce the final chrominance channels of the video.
In each of the following clips produced by the researchers, the top left video is the original input and the bottom right is the eventual output after noise removal and colorization. The results are very impressive:
If you want a fuller understanding of how the process works, Two Minute Papers on YouTube broke it down: