6 of 9 for Introduction to Image Processing
Chromaticity and Image Differencing for Image Segmentation
How do we segment objects in the image based on their chromaticity and temporal movement?
Segmentation from the previous blog post involved explicit colors which are somewhat easily described. However, for colors which are complex combinations of the primary colors, such segmentations may not suffice.
In this post, we will discuss more image segmentation by chromaticity and image differencing.
Chromaticity Segmentation
Segmentation by chromaticity uses the Red-Green (RG) chromaticity space which is a visualization of the RGB color space sans the intensity. The red and green color channels are normalized with respect to their sum.
Only the red and green channels are presented in the chart since the values for the three color channels sum to 1 per the following equations.
Two approaches for chromaticity segmentation will be discussed: parametric and nonparametric approach.
Parametric Approach
To demonstrate the parametric approach of this segmentation method, let’s use a vibrant photo of the Garden of Morning Calm taken on a cold winter night.
When we take all the colors in this photo and project them into the RG chromaticity space, we can see that most of the colors are bluish but close to white.
morn_calm_R = morn_calm[:, : ,0] * 1.0 / morn_calm.sum(axis=2)
morn_calm_G = morn_calm[:, : ,1] * 1.0 / morn_calm.sum(axis=2)
plt.figure(figsize=(5, 5))
plt.hist2d(morn_calm_R.flatten(), morn_calm_G.flatten(), bins=100, cmap='binary')
plt.xlim(0,1)
plt.ylim(0,1)
plt.xlabel('red', color='r')
plt.ylabel('green', color='g')
plt.show()
Let’s assume that we are interested in segmenting trees lit up in blue similar to the blue tree in the righthand side of the image. Consider the following patch boxed in red.
patch = morn_calm[750:800, 3300:3350, :]
imshow(patch)
plt.show()
When we look at the projection of this patch onto the RG chromaticity space from earlier, it is well within the space inside the blue section of the chromaticity space as show in the histogram below.
patch_R = patch[:, :, 0] * 1.0 / patch.sum(axis=2)
patch_G = patch[:, :, 1] * 1.0 / patch.sum(axis=2)
plt.figure(figsize=(5, 5))
plt.hist2d(patch_R.flatten(), patch_G.flatten(), bins=100, cmap='binary')
plt.xlim(0, 1)
plt.ylim(0, 1)
plt.xlabel('red', color='r')
plt.ylabel('green', color='g')
plt.show()
Using this histogram, we are going to fit a Gaussian distribution over the distribution in this patch histogram.
std_patch_R = np.std(patch_R.flatten())
mean_patch_R = np.mean(patch_R.flatten())
std_patch_G = np.std(patch_G.flatten())
mean_patch_G = np.mean(patch_G.flatten())We can define the Gaussian functions as follows:
def gaussian(p, mean, std):
return np.exp(-(p-mean)**2/(2*std**2))*(1/(std*((2*np.pi)**0.5)))Plotting the two Gaussian functions of the two color channels on top of each other per the RG chromaticity space shows the enclosed area bounded by these two Gaussian functions. This area will be the color of interest to us.
x_R = np.linspace(0, 1)
y_R = gaussian(x, mean_patch_R, std_patch_R)
x_G = np.linspace(0, 1)
y_G = gaussian(x, mean_patch_G, std_patch_G)
fig, ax = plt.subplots()
plt.xlabel('red', color='r')
plt.ylabel('green', color='g')
ax.plot(x_R, y_R, 'r')
ax.plot(y_G, x_G, 'g')
plt.show()
We can apply this to the image of the Garden of Morning Calm by taking the probabilities of the color appearing in our image using the R and G channels, independently.
prob_R = gaussian(morn_calm_R, mean_patch_R, std_patch_R)
plt.imshow(prob_R)
plt.axis('off')
plt.show()
prob_G = gaussian(morn_calm_G, mean_patch_G, std_patch_G)
plt.imshow(prob_G)
plt.axis('off')
plt.show()
Performing element-wise multiplication of the values in the probabilities for both red and green channels will give us the area bounded by the two Gaussian functions. This will be the final probability matrices which captures the probabilities for the entire RG color band.
This is already a mask which we can apply to the original image to highlight where our color of interest possibly appears in the image.
masked_morn_calm = morn_calm.copy()
prob = prob_R * prob_G
prob = prob.astype(np.uint8)
masked_morn_calm[:, :, 0] *= prob.clip(0, 1)
masked_morn_calm[:, :, 1] *= prob.clip(0, 1)
masked_morn_calm[:, :, 2] *= prob.clip(0, 1)
plt.axis('off')
imshow(masked_morn_calm);
Nonparametric Approach
Sometimes, our region of interest cannot be approximated by a 2D Gaussian curve. Hence, we have to use a nonparametric approach to chromaticity segmentation.
To achieve this, we need to use a 2D histogram of the original image and perform histogram backprojection to mask the original image with the computed histogram of a reference patch.
The nonparametric approach is particularly useful for multi-color segmentation. For this example, we will use the nonparametric approach to segment skin colors of different shades from an image. Let us consider the following image.
Say we want to segment skin colors based on the following patch of different skin colors. This would definitely not be feasible using the parametric approach.
skin_patch = imread('skin_patch.png')
plt.axis('off')
imshow(skin_patch)
plt.show()
We can perform backprojection in order to mask the original image with the computed histogram of the reference skin patch above.
# Convert both images to the HSV color space
patch_hsv = colors.rgb_to_hsv(skin_patch[:, :, :3])
image_hsv = colors.rgb_to_hsv(skin_colors)
# Compute the 2D histogram of the reference patch in the H and S channels
patch_hist, x_edges, y_edges = np.histogram2d(
patch_hsv[:,:,0].flatten(), patch_hsv[:,:,1].flatten(), bins=16, range=[[0, 1], [0, 1]])
# Normalize the histogram to have a maximum value of 1
patch_hist = patch_hist / np.max(patch_hist)
# Compute the 2D histogram of the image in the H and S channels
image_hist, x_edges, y_edges = np.histogram2d(
image_hsv[:,:,0].flatten(), image_hsv[:,:,1].flatten(), bins=16, range=[[0, 1], [0, 1]])
# Normalize the image histogram to have a maximum value of 1
image_hist = image_hist / np.max(image_hist)
# Compute the backprojection of the image histogram using the reference patch histogram
backproj = np.zeros_like(image_hsv[:,:,0])
for i in range(image_hsv.shape[0]):
for j in range(image_hsv.shape[1]):
backproj[i,j] = patch_hist[np.searchsorted(x_edges, image_hsv[i,j,0])-1, np.searchsorted(y_edges, image_hsv[i,j,1])-1]
# Normalize the backprojection to have a maximum value of 1
backproj = backproj / np.max(backproj)
# Display the original image and the backprojection side by side
plt.figure(figsize=(10,5))
plt.subplot(1,2,1)
plt.imshow(skin_colors)
plt.title('Original Image')
plt.subplot(1,2,2)
plt.imshow(backproj)
plt.title('Backprojection')
plt.show()
Image Differencing
Instead of color, we can also segment images based on movement which assumes that the POV of the camera is stationary and only the objects in the images are moving.
Let’s take a photo of my desk for example. Consider the before and after photo and take note of which items have gone missing on the second photo.
fig, ax = plt.subplots(1, 2, figsize=(12, 10))
img_desk_before = imread('img_diff_before.jpg')
ax[0].imshow(img_desk_before)
ax[0].set_title('Before', fontsize=24)
ax[0].axis('off')
img_diff_after = imread('img_diff_after.jpg')
ax[1].imshow(img_diff_after)
ax[1].set_title('After', fontsize=24)
ax[1].axis('off')
fig.show()
Or you can simply run this code and know in less than a second.
img_desk_before_gray = rgb2gray(img_desk_before)
img_desk_after_gray = rgb2gray(img_desk_after)
desk_diff = img_desk_before_gray - img_desk_after_gray
masked_img_desk_before = img_desk_before_gray * desk_diff
plt.imshow(masked_img_desk_before, cmap='gray')
plt.axis('off')
plt.show()
It simply converts both photos to grayscale and subtracts the second photo from the first one. If an object on the first image did not move in the second image, it will simply be a case of the same values getting deduced from the same values, zeroing out those pixels. However, when there is movement or change, this will result to different pixel values for the before and after. These pixels will not be zero and hence will be visible when displayed.
Conclusion
In class, we learned about chromaticity-based image segmentation and image differencing for object detection. Chromaticity segmentation involves representing colors in the RG chromaticity space, which normalizes the red and green color channels. By analyzing the distribution of colors in this space and using techniques like Gaussian fitting, we can identify specific color regions of interest.
On the other hand, image differencing allows us to detect changes between two images by subtracting their grayscale representations. The resulting difference image highlights areas where movement or changes have occurred.
Through these techniques, we can effectively segment objects based on color or detect changes in a scene. By applying these concepts, we gain valuable insights into object segmentation and detection, enabling us to analyze images in various applications such as computer vision and surveillance systems.
References
Wikimedia Foundation. (2023, June 14). Human skin color. Wikipedia. https://en.wikipedia.org/wiki/Human_skin_color
째재와 육아 일기♡ : 네이버 블로그. (n.d.). https://blog.naver.com/hellomilja/220813661310
