3.1 A simple network consisting of a single neuron

We will begin with exploring a single neuron. The interactive figure below demonstrates a neural “network” with two inputs. The two dashed circles symbolise the points at which the data flows into the neural network, the so-called inputs. Incidentally, these are not independent neurons, even though they are often referred to as such.

As the neuron has two inputs and one output, the training data must also be structured in such a way that there are two numbers as input and one number as output. The data which this particular neural network is to be trained with is listed below the figure and is kept the same throughout the whole chapter. For reasons of clarity, for the first three training sets the input x always consists of exactly four pairs of training data, each with exactly two values (as the network has two inputs). In the following, a line in the input x field is called a pattern with the values x1 and x2. In the pre-selected setting, for example, (0.1 / 0.2) is the first pattern for which the network should output the value of the target y in the adjacent field, in this case 0. In the third field next to it, the actual value that the neuron is outputting in its current training state is displayed as output o. Without previous training, this value is approx. 0.5. Therefore, the network initially makes a large error, which can now be minimised by clicking on Train.

During training, the current error of the neural network is displayed in the graph. The training stops after 300 training epochs, i.e. after all training data (all patterns with their associated classes) have been sent through the neural network 300 times and the learning algorithm ideally has reduced the error 300 times. After training, it is worth comparing the desired target y with the actual output o. The network does not achieve the target 100 per cent of the time, but usually performs well enough.

Likewise, during training, colours indicate the current line learned by the neuron in order to separate the patterns of class 0 from the patterns of class 1, as intended in the training data. To show this, all four patterns in the figure are entered in a coordinate system in which x1 is the x-axis and x2 is the y-axis.

You can see how the data flows through the network by clicking on one of the four patterns to the left of the neural network. It is then highlighted in green. If the network is as small as in the examples in this chapter, the output can actually be calculated manually. The activation of the neuron is x1⋅w1 + x2⋅w2. The threshold value is subtracted from this value. The result is then entered as the x value in the activation function, which calculates the output as a number between 0 and 1. You can recalculate this last step at the bottom of the page by entering the result of the subtraction (activation minus threshold) as the x value and clicking Calculate.

In all training examples, there are always two patterns of class 0 and two patterns of class 1 – except for the last example. All data sets will be discussed in detail below.

Calculation of the activation function:

The training examples in detail

 Neural networks usually output numbers with values between 0 and 1 (when using other activation functions, ranges from -1 to 1 are also possible). As all calculations within the neural network take place in this interval between 0 and 1, the values for the input data should also be between 0 and 1. Some tools, such as the software used here, can cope with larger number ranges; nevertheless, the training examples on these pages are set up in such a way that they only contain numbers between 0 and 1. This is good practice and should also be followed when creating your own neural networks by carefully pre-processing the data.

The pre-selected training data example contains two patterns with a combination of low numbers and two patterns with a combination of high numbers. Therefore, the two classes to be separated are basically low numbers and high numbers.

Boolean functions are popular introductory examples for neural networks. They consist only of zeros and ones, standing for logically false and logically true. With two inputs and one output, there are exactly 16 different Boolean functions, 14 of which are available here and can be selected at random. As a single neuron cannot find a separation line for the two functions XOR and XNOR, they have been omitted here for reasons of comparability of the three example networks in this chapter.

The random number examples generate data that are not entirely random: There are always two patterns per class, but they are designed in such a way that the patterns of both classes are not opposite to each other at the same time. This is because a case like the previously mentioned XOR would then occur, and the single neuron used here would not be able to solve this situation. Additionally, it is avoided as far as possible that two (nearly) identical patterns belong to different classes. But despite this, interesting situations sometimes occur, as can be seen from the error curve which then starts to oscillate briefly. However, this is just as normal as the fact that some neural networks can get stuck, and errors get large. In that case, the only thing that helps is restarting with better randomly selected values.

The circle example clearly shows the limits of a single neuron. One neuron by itself is not able to separate the patterns of class 0 from those of class 1, as it can only generate a single separation line. Such a task requires more neurons and more layers, as we will see in the following units.