ACTIVITY 16 Neural Networks

In this activity, we classify the patterns presented in Activity 14 using artificial neural networks. An advantage of this method over the algorithms reported in the previous activities is that one does not need heuristics and recognition rules to perform classification.

A neural network is a computational model of the neural structure of the brain. As implied by the name, it is a network of neurons that are either sets of input values (xi) and associated weights or functions that sum the input values according to their weights and map the results to an output (y). See Figure 1 for illustration. These neurons are organized into layers as shown in Figure 2.

Figure 1.

Figure 2.

The key features of neural networks are:
(1) the evaluation of the patterns one at a time and
(2) the manner of learning through comparison of their classification with the known actual classification of the pattern.
The learning process of a neural network is iterative. Moreover, the classification process is continuously modified based on the errors fed back from the previous classification stage. The modification is through the adjustment of weights associated with the input values in order to have a more accurate prediction of the class of the pattern.

We are lucky enough that SciLab has a toolbox for artificial neural neutworks. We need not worry anymore of generating a code for the learning process of a neural network. All we need is initialize the necessary input parameters. A code made by Jeric Tugaff was used as a guide in understanding what the needed parameters are. Below is the discussion of the code. We also used the blog of Cole Fabros as a reference.

SciLab Code:
rand('seed',0);
//this is to ensure that it has same starting point each time

N = [a,b,c];
//definition of network: a - number of neurons (features) in the input layer, b - number of
hidden layers, c - number of outputs/classes to be identified
// in this case, a = 3, b = 2 and c = 4

x = [];
x = x';
//feature vectors of the training set (must be an m x a matrix, a - total number of patterns)
//in this case, 20 x 3 since there are 4 classes of 5 samples each and 3 features are considered
(see Table 1)
//Note: If the features have very large values, these must be normalized. Otherwise, the results
would be wrong or the neural network won't work properly.

Table 1.

y = [];
//desired output/known actual classification of the patterns (must be a c x m matrix)
//in this case, 4 x 20 since members of the 4 classes are to be determined simultaneously
//value is either 1 or 0, 1 if right classification and 0 if wrong classification
//in this case, according to the order of patterns listed in Table 1, the desired output must
be similar to the matrix shown below (1st row - members of orange class, 2nd - nagaraya,
3rd - grapes, 4th - eggnog).


lp = [s,t];
//s - learning rate; t - threshold for the error tolerated by the network
//in this case, s is varied while t is always 0

W = ann_FF_init(N);
//initialization of weights

T = r;
//training cycles: r - number of iterations
//in this case, r is also varied

W = ann_FF_Std_online(x,y,N,W,lp,T);
//variables are already explained above
//adjusted weights after T iterations

ann_FF_run(x,N,W)
//testing the neural network, outputs its classification of the training patterns

test = [];
//feature vectors of the test set (must be an m1 x a matrix, m1 - total number of patterns)
//in this case, same as training set size: 20 x 3 matrix (see Table 2)
//Note: If the features have very large values, these must be normalized. Otherwise, the results
would be wrong or the neural network won't work properly.

Table 2.

class = ann_FF_run(test,N,W);
class = round(class);
//testing the neural network with a set of data it is not trained
//outputs its classification of the test patterns
// in this case, recall: 1st row - oranges, 2nd - nagaraya, 3rd - grapes, 4th - eggnog

oranges = find(class(1,:) == 1)
nagaraya = find(class(2,:) == 1)
grapes = find(class(3,:) == 1)
eggnog = find(class(4,:) == 1)
//outputs the data indices of the members of the said classes

First, I will disscuss the training results of the neural network. As mentioned above, the output must either be 1 or 0: 1 if the classification is right and 0 if not. But as you can notice in Table 3 and 4, the output values are not 1 or 0 but values that are near 0 and 1. This means the neural network had not learned enough to be precise in the classification of the patterns. However, the obtained values are already acceptable as classification of the patterns.

Table 3. Prediction after applying neural networks. Different learning rates were tried and compared.

Table 4. Prediction after applying neural networks. Different numbers of training cycles were tried.

Before proceeding with the discussion of the training results, I want to note that the output of the code is a 4 x 20 matrix and not just 4 x 4 (size of the tables). The reason why the tables are just 4 x 4 because data of the members of one class are averaged as a summary of the results. That is the output values of the 5 patterns belonging to the classes oranges, nagaraya, grapes and eggnog are averaged.

As an investigation, I varied the learning rate (s) and number of training cycles or iterations (r) to see if the output values get nearer to 1 and 0. Table 3 is the summary when the learning rate is varied while Table 4 is the summary when the number of training cycles is varied. Results show that the output values get nearer to 1 and 0 if both parameters are increased.

The next thing I did is test the neural network to a set of feature vectors it is not trained. The classification of the neural network is correct, i.e., the value close to 1, if it has the same row and column label. Looking at the values in Table 5, we can say that the neural network was able to classify correctly all the patterns (100%) even the data it was not trained.

Table 5.

As another investigation, I tried disarranging the test set as illustrated in Tables 6 and 7 to test whether the neural network is dependent on the order of the patterns. Table 6 is disarranged in such a way that the eggnog and oranges were interchanged in their positions. Table 7, on the other hand, is a random arrangement of the patterns. Tables 8 and 9 are the resulting classification of the patterns for the arrangement of data in Tables 6 and 7, respectively. Results show that using the training parameters mentioned above, the neural network was still able to classify correctly all the patterns (100%).

Table 6.

Table 7.

Table 8.

Table 9.

Since I was able to classify correctly all the patterns and then give a nice discussion
above (I think), I give myself a grade of 10 for this activity.