In India, we celebrated the festival of color “Holi” last week. We celebrate the end of the winter with a splash of color because that’s what the spring will bring us in a few days. When I was young, the celebrations were sparse. It was the decade of frugal parenting. We waited for festivals so eagerly because it meant parent approved outing and fun. We barely slept in the night before and buzzed with excitement early morning. Tubs and buckets were sacrificed for the good cause of color mixing. Then started the actual riot. We went to people’s places, people came to our place… And nobody was left with the color that nature gave us. All with the reds, yellows, and greens!

The festival celebrates colors and how they add beauty to life. I began to think, how dull and boring would be life without colors. Looking back at the photos from that time, what strikes me is that they don’t quite carry the colors they were meant to. As much as we all love looking at those old black & white photos and feel the nostalgia, there can be more to those pictures.  Typically, coloring black and white photos is a manual and time-consuming task and is done using adobe photoshop by experts. Can we train neural networks to add colors to black and white photos? Yes, we can! That’s what I would demonstrate in today’s post.

I recently came across this facebook group of my college, IIT Kanpur. In this group, one of our seniors shared the black and white photos of his time(more than 40+ years old). These photos take you on a journey to the past as if you were there. I wondered if I could color these photos.

Image colorization demo in OpenCV

IIT Kanpur football field back in time. On the left: original black and white image. Right: Image colored using AI.

Students marching. On the left: original black and white image. Right: Image colored by a neural network

Image colorization in OpenCV

NCC training on campus. On the left: original black and white image. Right: Image colored by a neural network

Pretty exciting!! Isn’t it? In today’s post, we shall learn to add colors to black and white photos using Deep Learning in OpenCV’s DNN module. Let’s first look at how the color information is encoded in digital images. Most of the popular image formats use RGB color space.

RGB Color space:

In RGB color space, each pixel has three color values (Red, Green, and Blue). So, in an 8-bit image, each channel (R,G,B) can have a value between 0 and 255. The brightness of the image depends on all three channels. 

While in a grayscale(black & white) image, each pixel just has just the intensity value.

I know what you are thinking, we want to teach a neural network to convert this grayscale image to the colored one. i.e It needs to learn to map this single value to a three channel image. But experts do something different. To understand that we need to first know about the Lab color space.

Lab color space

The Lab is another color space like RGB. In this space:

L channel: Lightness

a channel: encodes green-red

b channel: encodes blue-yellow.

Here the grayscale images are encoded in only L channel. Hence, this color space seems more convenient for our problem. As we only need to learn how to map L channel to a and b channels.

Problem Formulation:

Given an input L(grayscale image), we need to learn to predict a and b channels.

There have been many efforts to colorize an image automatically. However, Colorful Image Colorization by zHang et. al. is one of the most successful automatic colorization approaches. This paper uses convolutional neural networks for this learning task.

Colorful Image Colorization:

It uses a simple convolutional neural network architecture. As explained above, we take the L channel image and learn to predict a and b channels. Combining the prediction and input would give us the colorized image which can be converted back to the RGB color space. In order to train the network, the authors created the grayscale version of Imagenet dataset.

deep learning based image colorization

 

The results of the training which was done in Caffe are very impressive. These results have been integrated into the DNN module of OpenCV. In a few moments, let’s write a code to use this model to colorize our images.

Colorization in OpenCV:

We would write a single script that would take images, videos or webcam feed as input and generate a colored output. Let’s download the pre-trained models’ weights and other dependencies. You can also run get_models.sh file to download the pretrained model.

This would download following files:

colorization_release_v2.caffemodel: The path to the model weights trained in Caffe.

colorization_deploy_v2.prototxt: Caffe specific file which defines the network.

kernel: Path to cluster center points stored in numpy format.

Now, let’s write the code. The first step is to handle the imports and define a way to take inputs to the script.

The script needs the following inputs:

input: Path to the input grayscale image or video.

caffemodel: Path to the model weights trained in Caffe.

prototxt: Caffe specific file which defines the network.

kernel: Path to cluster center points stored in numpy format.

Let’s create the network graph.

Take the input image and apply some pre-processing:

Now, we are ready to do the inference. Let’s run the model by calling the forward method of OpenCV’s DNN module.

Now, take the output out which is a and b channels predicted by the network. If we add the L channel(the input), we would have the complete image in Lab color space. Then we convert it back to the BGR color space.

Now, let’s look at the output of the network:

Deep Learning based Image colorization

Iconic Black and white image converted to color

We can also use this to convert black and white videos to the colored. The code is very similar, we just read the video or webcam depending on the input. Here is how the results look like.

As usual, the complete code can be found here.

 

References:

  1. Photos: Unsplash.com and this bit of that iitk.

2. Colorful image colorization by Richard Zhang.