ACTIVITY 19 Restoration of Blurred Image

In this activity, we demonstrate the degradation process and the restoration of the degraded image. A diagram is shown in Figure 1 to illustrate the process of degradation and then restoration of an image. The degradation part is equivalent to equation 1 where G, H, F and N are the Fourier transforms of the degraded image, degradation function, original image and noise, respectively. We used the model of motion blur as our degradation function (see equation 2). The assumption of this model is that the blurring is due to linear motion between the image and the sensor during image acquisition. In equation 2, T is the duration of the exposure, a and b are the total distances for which the image has been displaced in the x- and y-direction, respectively. In addition to degradation, noise was also introduced to the image in the form of Gaussian. The equation for this type of noise was already shown in the previous activity. The restoration process is through minimum mean square error (Weiner) filtering. Equations 3 and 4 give us the Fourier transform of the restored image. So, the restored image can be shown by getting the inverse Fourier of these transforms. Note that u and v are just the indices of the pixels in the image.

Figure 1.

Equation 1.

Equation 2.

Equation 3.

Equation 4.

Figure 1.

The following are the investigations I did in relation to the degradation and restoration of the image shown in Figure 1.

1.) The value of T in H (degradation function, equation 2) was varied.
Figure 2 shows the degraded and restored images for different values of T. As you can see, the only detectable difference in the images but not that obvious is that the image becomes whiter as the value of T is increased. This difference becomes more obvious when the images are restored as shown in Figure 2. This is because T is the duration of the exposure. So, the longer the time the image is exposed, the whiter the image will be.

2.) The values of a and b (same value) were varied.
Figure 3 shows the degraded and restored images for different values of a and b (same value). a and b are the total displacement in the x- and y-direction. So, the larger the values of these variables are,the more blurry the image will be. Moreover, the blurriness is equal in the two directions since a and b are equal. The more blurry the image is, the harder it is to restore. As you can see in Figure 3, the level of restoration decreases as the values of a and b increase.

3.) The values of a and b (different values) were varied.
Figure 4 shows the degraded and restored images when a and b (different values) were varied. Since a and b are not equal anymore, the blurriness is not the same in the two directions (x and y). As you can see, when a is larger(smaller), the degraded image appears to be stretched horizontally (vertically). Comparing the restored images, it can be observed that the lines (contour) of the texts are darker when a is larger than b.

4.) The standard deviation and mean in the Gaussian noise were varied.
Figure 5 shows the degraded and restored images for different values of the standard deviation and mean. The larger the value of these parameters are, the noisier the image is. Similar to the results of varying a and b but of equal values, it is harder to restore the image when it is noisier.

4.) The value of K in equation 4 was varied.
This time, we did not vary the degradation process but the restoration process. Figure 6 shows the restored images for different values of K. When the value of K is at extreme 0 or 1, the filter failed to restore the image. When it is near 0, the blurriness is removed but the noise remains. In fact, the noise already covers the entire image that it is not visible anymore. On the other hand, when it is near 1, the noise is removed but the image becomes more blurry. Thus, the value of K must be properly chosen to produce the best restored images. Compared to the restored image using equation 6, equation 7 produces images that are still noisy. This is because this equation assumes that the noise is spectral white. However, in our case, the noise we used is Gaussian. This is why equation 6 has better results because it is able to use the right noise.

Figure 2.

Figure 3.

Figure 4.

Figure 5.

Figure 6.

I would give myself a grade of 10 for this activity.

ACTIVITY 18 Noise Modeling and Basic Image Restoration

In this activity, we explored the different types of noise and used the common filters to clean the noisy images. I used Figure 1 as my original image. The noise we introduced on the image are the common types of noise, namely, Gaussian (normal), exponential, gamma, Rayleigh, uniform and impulse (salt& pepper). These noises are described in equations 1 to 6. The resulting noisy images are shown in Figure 2. On the right side of the images are the histogram (PDF) of the noisy images. Given a noisy image with unknown type of noise, one can then determine the type of noise from its PDF. However, it must be noted that the images I am talking here are grayscale images.

Figure 1.

Figure 2.

Equation 1. Gaussian (normal) noise.

Equation 2. Rayleigh noise.

Equation 3. Erlang (gamma) noise.

Equation 4. Exponential noise.

Equation 5. Uniform noise.

Equation 6. Impulse (salt & pepper) noise.

The filters I used are the following: arithmetic mean filter, geometric mean filter, harmonic mean filter and contraharmonic mean filter. Equations 7 to 10 illustrate how these filters work. Figure 3 shows the cleaned images after applying the different filters. As you can see, no filter was able to perfectly restore the noisy images to the original image. For different types of noise introduced, a filter cleans the noisy images differently. Consider the cleaned images when ageometric mean filter was used (3rd row of Figure 3). The image with impulse noise obviously has a different cleaned image as compared to the other noises. Arithmetic mean and geometric mean slightly clean the image but it can be observed that there is partial blurring. Harmonic mean filter has the best result in some images (gamma and exponential) but it also fails to restore some images (normal and uniform). I think this is due to the type of distribution these noises produce. Gamma and exponential have similar PDF, as well as normal and uniform. The cleaned images using contraharmonic filter is shown in Figure 4. Recall equation 10. When Q is +/-, salt(white)/pepper (black) noise is removed. This is why the images is dark/bright when Q is +/-.

Equation 7. Arithmetic mean filter.

Equation 8. Geometric mean filter.

Equation 9. Harmonic mean filter.

Equation 10. Contraharmonic mean filter.

Figure 3. The first row is the set of images with noise incorporated in the following sequence: salt & pepper, exponential, normal (Gaussian), gamma, uniform. In the succeeding rows are the resulting images after applying the arithmetic filter, geometric filter and harmonic filter, respectively.

Figure 4. The first row is the set of images with noise incorporated in the following sequence: salt & pepper, exponential, normal (Gaussian), gamma, uniform. In the succeeding rows are the resulting images after applying the contraharmonic mean filter with Q (-) and Q (+), respectively.

In equations 7 to 10, notice that the computation of the new pixel values is confined in a window of size m x n. I investigated how the cleaning varies for different window sizes. Figure 5 shows the cleaned images. As you can see, more noise is removed when the window size is bigger. However, the images become more blurry also. So, the window size must be properly chosen such that enough noise is removed and the restored image is not so blurry.

Figure 5. The resulting images after applying all the filters with different window sizes
(1st row - 3 x 3, 2nd - 5 x 5, 3rd - 9 x 9) in one of the noisy images.

I also tried applying the filters to the noisy images I downloaded from the internet (see Figure 6). From their PDFs,it can be inferred that the noise is Gaussian. The cleaned images are shown in Figures 7 to 10. As you can see, the restoration is still not perfect. The results are just the same as the cleaning of the images in Figure 2.

Figure 6. PDFs of the images.

Figure 7.

Figure 8.

Figure 9.

Figure 10.

I would give myself a grade of 10 for this activity.

ACTIVITY 17 Photometric Stereo

In this activity, we used the technique photometric stereo in constructing the 3D shape of an object. This technique works under the assumption that the source is at infinity and so, the radial waves reaching the surface appear as planar waves. Moreover, it is assumed that the intensity captured by a camera at point (x,y) is directly proportional to the brightness of the surface at that point.

By capturing multiple images with the sources at different locations, one can estimate the shape of the object. The sources can be represented by matrix V in equation 2 where Vn1, Vn2 and Vn3 are the component of the source n in the x-, y- and z- direction, respectively. If we know the intensity of the surface image captured by source n, then we can solve for g as illustrated in equation 3. The surface normals can then be calculated by normalizing g with its length (see equation 4). The surface elevation, z, is a function of x and y. The partial derivatives of this function with respect to x and y are shown in equation 5. Therefore, the surface elevation is given by equation 6.

In this activity, we used four sources and the surface images captured by these sources are shown in Figures 1 to 4. After implementing equations 2 to 6, we were able to generate a 3D shape of the object as shown in Figure 5.

Figure 1.

Figure 2.

Figure 3.

Figure 5.

ACTIVITY 16 Neural Networks

In this activity, we classify the patterns presented in Activity 14 using artificial neural networks. An advantage of this method over the algorithms reported in the previous activities is that one does not need heuristics and recognition rules to perform classification.

A neural network is a computational model of the neural structure of the brain. As implied by the name, it is a network of neurons that are either sets of input values (xi) and associated weights or functions that sum the input values according to their weights and map the results to an output (y). See Figure 1 for illustration. These neurons are organized into layers as shown in Figure 2.

Figure 1.

Figure 2.

The key features of neural networks are:
(1) the evaluation of the patterns one at a time and
(2) the manner of learning through comparison of their classification with the known actual classification of the pattern.
The learning process of a neural network is iterative. Moreover, the classification process is continuously modified based on the errors fed back from the previous classification stage. The modification is through the adjustment of weights associated with the input values in order to have a more accurate prediction of the class of the pattern.

We are lucky enough that SciLab has a toolbox for artificial neural neutworks. We need not worry anymore of generating a code for the learning process of a neural network. All we need is initialize the necessary input parameters. A code made by Jeric Tugaff was used as a guide in understanding what the needed parameters are. Below is the discussion of the code. We also used the blog of Cole Fabros as a reference.

SciLab Code:
rand('seed',0);
//this is to ensure that it has same starting point each time

N = [a,b,c];
//definition of network: a - number of neurons (features) in the input layer, b - number of
hidden layers, c - number of outputs/classes to be identified
// in this case, a = 3, b = 2 and c = 4

x = [];
x = x';
//feature vectors of the training set (must be an m x a matrix, a - total number of patterns)
//in this case, 20 x 3 since there are 4 classes of 5 samples each and 3 features are considered
(see Table 1)
//Note: If the features have very large values, these must be normalized. Otherwise, the results
would be wrong or the neural network won't work properly.

Table 1.

y = [];
//desired output/known actual classification of the patterns (must be a c x m matrix)
//in this case, 4 x 20 since members of the 4 classes are to be determined simultaneously
//value is either 1 or 0, 1 if right classification and 0 if wrong classification
//in this case, according to the order of patterns listed in Table 1, the desired output must
be similar to the matrix shown below (1st row - members of orange class, 2nd - nagaraya,
3rd - grapes, 4th - eggnog).


lp = [s,t];
//s - learning rate; t - threshold for the error tolerated by the network
//in this case, s is varied while t is always 0

W = ann_FF_init(N);
//initialization of weights

T = r;
//training cycles: r - number of iterations
//in this case, r is also varied

W = ann_FF_Std_online(x,y,N,W,lp,T);
//variables are already explained above
//adjusted weights after T iterations

ann_FF_run(x,N,W)
//testing the neural network, outputs its classification of the training patterns

test = [];
//feature vectors of the test set (must be an m1 x a matrix, m1 - total number of patterns)
//in this case, same as training set size: 20 x 3 matrix (see Table 2)
//Note: If the features have very large values, these must be normalized. Otherwise, the results
would be wrong or the neural network won't work properly.

Table 2.

class = ann_FF_run(test,N,W);
class = round(class);
//testing the neural network with a set of data it is not trained
//outputs its classification of the test patterns
// in this case, recall: 1st row - oranges, 2nd - nagaraya, 3rd - grapes, 4th - eggnog

oranges = find(class(1,:) == 1)
nagaraya = find(class(2,:) == 1)
grapes = find(class(3,:) == 1)
eggnog = find(class(4,:) == 1)
//outputs the data indices of the members of the said classes

First, I will disscuss the training results of the neural network. As mentioned above, the output must either be 1 or 0: 1 if the classification is right and 0 if not. But as you can notice in Table 3 and 4, the output values are not 1 or 0 but values that are near 0 and 1. This means the neural network had not learned enough to be precise in the classification of the patterns. However, the obtained values are already acceptable as classification of the patterns.

Table 3. Prediction after applying neural networks. Different learning rates were tried and compared.

Table 4. Prediction after applying neural networks. Different numbers of training cycles were tried.

Before proceeding with the discussion of the training results, I want to note that the output of the code is a 4 x 20 matrix and not just 4 x 4 (size of the tables). The reason why the tables are just 4 x 4 because data of the members of one class are averaged as a summary of the results. That is the output values of the 5 patterns belonging to the classes oranges, nagaraya, grapes and eggnog are averaged.

As an investigation, I varied the learning rate (s) and number of training cycles or iterations (r) to see if the output values get nearer to 1 and 0. Table 3 is the summary when the learning rate is varied while Table 4 is the summary when the number of training cycles is varied. Results show that the output values get nearer to 1 and 0 if both parameters are increased.

The next thing I did is test the neural network to a set of feature vectors it is not trained. The classification of the neural network is correct, i.e., the value close to 1, if it has the same row and column label. Looking at the values in Table 5, we can say that the neural network was able to classify correctly all the patterns (100%) even the data it was not trained.

Table 5.

As another investigation, I tried disarranging the test set as illustrated in Tables 6 and 7 to test whether the neural network is dependent on the order of the patterns. Table 6 is disarranged in such a way that the eggnog and oranges were interchanged in their positions. Table 7, on the other hand, is a random arrangement of the patterns. Tables 8 and 9 are the resulting classification of the patterns for the arrangement of data in Tables 6 and 7, respectively. Results show that using the training parameters mentioned above, the neural network was still able to classify correctly all the patterns (100%).

Table 6.

Table 7.

Table 8.

Table 9.

Since I was able to classify correctly all the patterns and then give a nice discussion
above (I think), I give myself a grade of 10 for this activity.

ACTIVITY 15 Linear Discriminant Analysis

For this activity, we did pattern recognition using linear discriminant analysis (LDA). The patterns I used are the images shown in the previous activity. I also used the same features and method of determining the feature values. The images were also divided into two sets: training set and test set.

LDA can be used to classify patterns if the classes can be assumed to be linearly separable, i.e., classes can be separated by a linear combination of the features. Since three features of the pattern were used, the separator must be a plane.

In LDA, two variables are needed:
(1) dependent variable (y) which is the class of the pattern and
(2) independent variable (x) which is the feature of the pattern.
The size of y must be n x 1, where n is the total number of patterns. Moreover, x must be of size n x 3; 3 since there are three features. Same row in x and y represent one pattern. The diagram below shows the sequence of the necessary parameters needed to be calculated in LDA.


In evaluating f, all the variables except those with subscript T were calculated using the training set. The patterns were classified by first calculating f for all classes. It was then assigned to the class with the maximum f.

Tables 1 and 3 show the feature vectors of the patterns. Table 2 and 4 are summaries of the calculated f for all classes. The column of the highlighted f (red color) is the classification of the pattern. As you can see, LDA was also able to correctly assign all the patterns to their right classes (100%).

Figure 1. Feature vectors (green, red, area) of the training images of oranges, nagaraya, grapes and eggnog.

I give myself a grade of 10 for this activity.