2 of 9 for Introduction to Image Processing
Enhancing Images
How do we improve the quality of images using basic image processing techniques?
What is image enhancement?
Nowadays, image enhancement may simply involve pressing a button in your phone or computer. However, there’s more than meets the eye which we will try to talk in more detail in the succeeding parts of this topic.
Image enhancement is simply a process of improving the visual quality of an image by performing adjustments on the image brightness, contrast, color, and many more. There are several techniques which can be employed in image enhancement but we will talk about three popular methods in this post: Fourier Transform, White Balancing, and Histogram Manipulation.
Fourier Transform
Fourier Transform is a mathematical method used to analyze signals and images. It breaks down a complex signal or image into its individual frequency components, making it easier to analyze and manipulate. In the context of image processing, Fourier Transform is often used to remove periodic noise from an image or to sharpen blurry images.
Fourier Transform is simple a mathematical method which breaks down a complex signal or image in our case from its spatial components into its frequency components. This transformation makes the image easier to manipulate considering periodic noise in the image is more visible in the frequency domain than in the original spatial domain.
Fourier transform allows for representation of an image whose intensities can be thought of as a signal consisting of multiple frequencies. These images may contain noise caused by various factors such as camera shake, imperfect lenses, or other environmental factors such as wind and vibrations. These unwanted fluctuations can be isolated and removed once the image is viewed in terms of the frequencies. The noise in the image can often be captured as distinct frequencies hence noise removal is as simple as isolating these noise frequencies and masking them out.
Once the noise frequencies have been masked out in the frequency domain, the Fourier transformed-image can be reverted back to the spatial domain using inverse Fourier transform — in order to see the image in a human-understandable form.
Take for example one of the Moon exploration photos from the final Apollo mission, Apollo 17, which has a checkered artifact pattern over it (just for illustration purposes).
moon = imread('moon_image.png')
The image can be converted into the frequency domain using the following code. There are two major functions involved in the Fourier transformation of the image. The first function affecting our image moon is np.fft.fft2() which computes for the two-dimensional discrete Fourier transform of the image. The output of this function is then fed to another function np.fft.fftshift() which shifts the zero-frequency component of the Fourier transform towards the center of the spectrum. In essence, it takes the zero-frequency component (simply the average value) of the image, which is normally located at the corners of the transformed image, into the center. Moving this to the center will prove helpful during the image manipulation process since removing this zero-frequency will lead to removal of almost all information from the original image. Hence it is important to isolate that critical information in only one point during the manipulation process.
import numpy as np
import matplotlib.pyplot as plt
import skimage.io as skio
from skimage import img_as_ubyte, img_as_float
moon_fft = np.fft.fftshift(np.fft.fft2(moon))
skio.imshow(np.log(abs(moon_fft)), cmap='gray');
The bright spot in the middle of the transformed image is the zero-frequency component which holds almost all the non-noise information of the image. The horizontal and vertical white lines correspond to the repeating noise patterns present in the original image. Masking these frequencies involves setting their values in the frequency domain low enough such that they are not expressed anymore upon conversion back to the real domain later on.
moon_fft2 = moon_fft.copy()
moon_fft2[:236, moon_fft2.shape[1]//2] = 1
moon_fft2[-236:, moon_fft2.shape[1]//2] = 1
moon_fft2[:, 236] = 1
moon_fft2[:, 393] = 1
moon_fft2[177, :] = 1
moon_fft2[297, :] = 1
moon_fft2[358, :] = 1
moon_fft2[117, :] = 1
moon_fft2[:, 158] = 1
moon_fft2[:, 473] = 1
imshow(np.log(abs(moon_fft2)), cmap='gray');
Transforming the manipulated transformed image back into the real domain requires the use of np.fft.ifft2() which is the inverse Fourier transform. This will give us the final image with less noise compared to the original image.
imshow(abs(np.fft.ifft2(moon_fft2)), cmap='gray');
Using Fourier transform in filtering out noise from images can be both intuitive and not intuitive at the same time. For images with more regular noise patterns, transforming the image into the frequency domain will typically show one or two frequencies which can be easily masked to remove noise from the original image. However, once the noise in the image becomes more complex, using Fourier transform to convert the image from the spatial domain to the frequency domain might cause confusion due to more frequencies appearing. The more frequencies there are, the trickier it is to isolate the noise frequency. Furthermore, Fourier transform assumes that the noise pattern is repetitive. Regardless of the application, Fourier transform remains a powerful and essential tool in image processing.
White Balancing
Have you ever taken a photograph that turned out too reddish or too bluish for your taste? This is a common problem in photography and can be attributed to color temperature. Color temperature refers to the color of the light source for the object or scene being photographed. Outdoor lighting is typically bluish but indoor lighting tends to be more on the yellowish side.
This issue is typically corrected by a process called white balancing. White balancing adjusts the colors in an image in order to make it appear more natural. Any neutral white areas in the image are manipulated to appear truly white. We will discuss the three most common white balancing algorithms: white patch algorithm, gray-world algorithm, and the ground-truth algorithm.
As example, let us call up Frankie whose photo has somewhat of a bluish tint.
frankie = skio.imread('frankie.jpg')
skio.imshow(frankie);
We will go through all three white balance algorithms but let us first start with the white patch algorithm:
White Patch Algorithm
The white patch algorithm assumes that the whitest patch in the image is white and scales all the pixels across the three channels (RGB) in the image such that the brightest patch will have values of white (255, 255, 255). However, this will fail if there is already a real white patch in the image. As a workaround, the color channels are instead scaled using a certain top percentile for each color channel.
First, a quick look at Frankie’s smiling photo reveals an open window at the back of the room. The bright light spilling through this window may have registered as white (255, 255, 255) in the camera even with the bluish tint. Hence, we should be ready to use percentiles instead of the max color channel values in scaling the colors of this photo.
Let us use the following function in fixing Frankie’s photo using the white patch algorithm:
def white_patch(image, percentile=100):
'''
White balance image using White patch algorithm
Parameters
----------
image : numpy array
Image to white balance
percentile : integer, optional
Percentile value to consider as channel maximum
Returns
-------
image_wb : numpy array
White-balanced image
'''
image_wp = img_as_ubyte((image*1.0 / np.percentile(image, percentile, axis=(0, 1)))
.clip(0, 1))
skio.imshow(image_wp);Performing the white patch algorithm at the max percentile (whitest patch in the image) does not seem to change the image any bit.
white_patch(frankie, percentile=100)
However, taking a lower percentile results in significant improvement! Using the 85th percentile as the threshold means that pixels with color channel values above the 85th percentile for each color channel are already set to 255 (the maximum!). Choice of percentile will depend on domain expertise and patience. Trust me, it’s an iterative process of finding out.
This comes at the expense of over-saturation of parts which are already maxed out to begin with, such as the window area. Much of the details surrounding the window have been washed out but Frankie’s colors are now warmer and her strikingly red bowtie stands out more.
Gray-world algorithm
The gray-world algorithm works on the assumption that the average color of any given image should be gray. Basically, what this algorithm tries to achieve is to make the average value of each color channel (R, G, and B) across the entire image close, if not equal, to the global average color value for all channels for the entire image. Since the final average value of each color channel after application of the algorithm will tend to be equal, the average color for the entire image becomes a shade of gray.
Let us use the following function in fixing Frankie’s photo using the gray-world algorithm:
def gray_world(image):
'''
White balance image using Gray-world algorithm
Parameters
----------
image : numpy array
Image to white balance
Returns
-------
image_wb : numpy array
White-balanced image
'''
image_gw = ((image * (image.mean() / image.mean(axis=(0, 1))))
.clip(0, 255).astype(int))
skio.imshow(image_gw[:, :, :3]);For reference, the global average color value for all channels for Frankie’s original photo is 110.99.
gray_world(frankie)Performing the gray-world algorithm on Frankie’s photo gives us a less balanced image compared to the white patch algorithm. The average values of each color channel on the adjusted image are (110.49 , 108.88, 110.46) — completely close to the global average of the original image.
Ground-truth Algorithm
This algorithm is more straightforward that the previously discussed algorithms. In a nutshell, this algorithm takes in a patch from the image that is known to be actually white and uses it to scale all other pixels.
Let us use the following function in fixing Frankie’s photo using the ground-truth algorithm. Notice that it also takes in a patch argument, which will be the area of the image we know to be white. Additionally, the function also allows the user to specify whether to use the mean or the max (brightest) of the patch as the ground-truth color.
def ground_truth(image, patch, mode='mean'):
'''
White balance image using Ground-truth algorithm
Parameters
----------
image : numpy array
Image to white balance
patch : numpy array
Patch of "true" white
mode : mean or max, optional
Adjust mean or max of each channel to match patch
Returns
-------
image_wb : numpy array
White-balanced image
'''
if mode == 'mean':
image_gt = (image*1.0 / patch.mean(axis=(0, 1))).clip(0, 1)
if mode == 'max':
image_gt = (image*1.0 / patch.max(axis=(0, 1))).clip(0, 1)
skio.imshow(image_gt);In our case, we know that the inside surface of the striped paper bag behind Frankie in the middle right of the image is white.
Applying the algorithm to the image and using the mean value of the patch as ground-truth color gives us a balanced photo where the area previously enclosed in a red box is now white and Frankie has much warmer colors than the original photo.
frankie_patch = frankie[590:615, 1390:1415]
ground_truth(frankie, frankie_patch)
Overall, each of the white balance algorithms discussed has their unique approach to performing white balance. They all play an important role in achieving accurate and natural colors in photographs. The white patch algorithm is useful in correcting color casts caused by lighting conditions while the gray-world algorithm works the best in overall color casts that affect all colors equally. Lastly, the ground-truth algorithm is particularly useful in scientific and industrial applications where color accuracy is absolutely critical.
Histogram Manipulation
A histogram is simply a graphical representation of the tonal values present in the image. It provides a distribution of the brightness levels in the image. Ranging from left to right, it shows the fraction of pixels in the image with dark patches to bright highlights. There are different ways to effectively manipulate the histogram of an image such as histogram equalization, Gaussian equalization, and contrast stretching.
Consider the following images and their respective histograms. The x-axis of the histogram is the intensity value which is a range from 0 to 255 with 0 corresponding to the darkest regions while 255 corresponds to the brightest spots in the image. The y-axis represents the fraction of pixels which have that specific intensity value.
From the name itself, histogram manipulation is a type of image enhancement which involves the manipulation of the histogram itself to adjust the tonal values of the image in order to improve its visual quality. If the image appears too dim, the histogram can be moved to the right to increase the proportion of bright pixels in the image. On the other hand, images that appear too bright can have their histograms adjusted towards the left in order to decrease the brightness of the image.
Let us use this night photo which appears too dim for histogram manipulation:
rally = imread('rally.jpg')
imshow(rally);
Generating a histogram for the photo above confirms that much of the image contain extremely low intensity values, with only the stage area being visibly bright.
from skimage.color import rgb2gray
from skimage.exposure import histogram, cumulative_distribution
rally_intensity = img_as_ubyte(rgb2gray(rally))
freq, bins = histogram(rally_intensity)
plt.step(bins, freq*1.0/freq.sum())
plt.xlabel('intensity value')
plt.ylabel('fraction of pixels');
Histogram equalization
Histogram equalization, from the name itself, involves equalizing the number of pixels for each intensity value across all intensity values from 0 to 255.
A cumulative distribution function (CDF) can also be generated from the histogram above. The CDF is simply the cumulative sum of all the pixels going from the left to the right of the x-axis.
freq, bins = cumulative_distribution(rally_intensity)
target_bins = np.arange(255)
target_freq = np.linspace(0, 1, len(target_bins))
plt.step(bins, freq, c='b', label='actual cdf')
plt.plot(target_bins, target_freq, c='r', label='target cdf')
plt.plot([50, 50, target_bins[-16], target_bins[-16]],
[0, freq[50], freq[50], 0],
'k--',
label='example lookup')
plt.legend()
plt.xlim(0, 255)
plt.ylim(0, 1)
plt.xlabel('intensity values')
plt.ylabel('cumulative fraction of pixels');
For histogram equalization, a uniform CDF is employed. The slope of the target CDF is constant — thus all intensity values will have similar numbers of pixels. This is the part where the manipulation occurs. The intensity values from the actual CDF will be mapped into the target CDF. As for the case in the sample mapping above, the intensity value of 50 is at around the 95th percentile of the actual CDF. Mapping this to the target CDF, the 95th percentile of the target CDF has an intensity value of around 240. Thus, all intensity value of 50 in the original image will be replaced by 244 in the enhanced image.
This mapping will be performed for all other values until the target CDF has already been achieved.
new_vals = np.interp(freq, target_freq, target_bins)
rally_eq = img_as_ubyte(new_vals[rally_intensity].astype(int))
imshow(rally_eq);
While the target CDF has been achieved using histogram equalization, the image quality which has an equal distribution of all intensities does not look particularly pleasing to the eye.
Gaussian equalization
Since in reality, most of the images in a photo have middle intensity values instead of completely uniform as previously tested, other target distributions can be used such as the Gaussian distribution.
from scipy.stats import norm
gaussian = norm(32, 64)
freq, bins = cumulative_distribution(rally_intensity)
target_bins = np.arange(255)
target_freq = np.linspace(0, 1, len(target_bins))
plt.step(bins, freq, c='b', label='actual cdf')
plt.plot(gaussian.cdf(np.arange(0,256)), c='r', label='target cdf')
plt.plot([50, 50, target_bins[-125], target_bins[-125]],
[0, freq[50], freq[50], 0],
'k--',
label='example lookup')
plt.legend()
plt.xlim(0, 255)
plt.ylim(0, 1)
plt.xlabel('intensity values')
plt.ylabel('cumulative fraction of pixels');
Contrast stretching
A more straightforward and simpler approach involves simply rescaling a certain range of the histogram based on a lower and an upper percentile.
Applying contrast stretching on the previous image did not produce results as good as that of the Gaussian equalization. However, since the percentile bounds are technically hyperparameters, a lot of combinations can still be explored. For this example, we will use bounds at 5th and 90th percentile.
rally_contrast = rescale_intensity(rally_intensity,
in_range=tuple(np.percentile(rally_intensity, (5, 90))))
imshow(rally_contrast);
The most appropriate histogram manipulation method depends on the specific characteristics of the image being processed. The most suitable target CDF for equalization is mainly dependent on the colors and intensities present in the image. Contrast stretching works well in cases where the contrast must be enhanced without over-amplifying noise present in the image.
Conclusion
Ultimately, the key to effective image enhancement is to use these techniques judiciously and in combination with other methods. By understanding the strengths and weaknesses of each technique, you can create images that truly stand out and capture the essence of your subject.
Ultimately, the key to effective image enhancement is having the proper skills to use different techniques with proper understanding of their individual strengths and weaknesses. Image enhancement has a multitude of applications whether you are a professional photographer or just a casual hobbyist. Learning how to use basic enhancement methods such as Fourier transform, white balancing, and histogram manipulation can make great strides in improving your skills in imaging and photography.
References
NASA. (1972). Apollo 17 Extravehicular Activity. photograph. Retrieved from https://www.nasa.gov/multimedia/imagegallery/image_feature_2129.html.
Shutterstock. (n.d.). 3Ds rendered image of 10 hanging lamps which use different bulbs. photograph. Retrieved from https://www.shutterstock.com/image-illustration/3ds-rendered-image-10-hanging-lamps-384971908.
