1.2 Neurons – the elements of neural networks

The human brain has around 100 billion nerve cells (also known as neurons), which can be regarded as components of information processing in the brain and form a very complex biological neuronal network. But what exactly is an artificial neuron?

An artificial neuron is a small piece of programming code which attempts to simulate – in highly simplified form – the functioning of a biological neuron. A graphic model as shown below can help us visualise this. An artificial neuron receives inputs (e.g. numbers), processes them using mathematical operations, and returns an output. The output of one neuron can serve as input for another neuron, which in turn generates its own output. In this way, the neurons work together to recognise complex patterns and correlations in data and make decisions.

Artificial neurons are modelled on real biological nerve cells in a very approximate and very simple mathematical way. As difficult as the subject of artificial intelligence may seem, the artificial neuron as the basic component of neural networks is very easy to understand. By the way: the same artificial neurons that we learn about on this page are also used in the most complex and largest artificial neural networks.

The first attempts of modelling artificial neurons are over 80 years old, and surprisingly, very little has changed in the artificial neuron model since then. Nowadays, however, the networks into which they are interconnected are much larger, the internet has made tons of training data available, and the computers on which such networks are run have become much faster.

A first approach

To begin with, an artificial neuron can be imagined as a light switch. When the neuron is activated by enough predecessor neurons, it fires, which in the example of the light switch would be switching on the light.

Over time, however, it has been found that artificial neural networks work much better with neurons not only mastering the two states ON/OFF, but always firing a little. The more they are activated, the more they should fire. You can think of it more like a dimmer light switch: The light gets brighter the more you turn the dial. In this context, firing means that the neuron calculates a number that it passes on to other neurons. The more predecessor neurons have activated the artificial neuron, the larger this number is (analogous to the dimmer light switch).

Model of an artificial neuron

• An artificial neuron has one to several inputs indicated by the letter x. All inputs are simply numbered from top to bottom: x1, x2, x3, and so on. These are decimal numbers. The neuron either receives these numerical values directly or from the neurons in the layer before it.
• Each input is connected to the neuron via a variable weight. This weight represents the strength of the connection, as it is assumed that such a connection becomes stronger in biological neurons the more often it is used. It is simply represented by a decimal number. The weights are labelled with the letter w in the same way as the input and numbered consecutively: w1, w2, w3, etc.
  Note: The weights are changed during training.
• The neuron itself is able to calculate its activation, a very simple formula: You add up all the products of input and weight in pairs, i.e. x1⋅w1 + x2⋅w2 + x3⋅w3 etc. In the example shown, this results in an activation of 2, because 1⋅2 + 0⋅1 = 2.
• In addition to the activation, the neuron has a variable threshold value labelled with the letter b (as in bias). In the classic “light switch model”, the neuron would only fire if the activation of the neuron was greater than this threshold value (this would be the case in the example shown). In the latter variant (the “dimmer switch” variant), the activation and the threshold value are incorporated into a mathematical formula that is used to calculate the neuron’s output. This formula is relatively easy to understand with a little upper secondary mathematics and has a very simple structure. The only important thing here is that the result is a decimal number between 0 and 1.
  Note: Like the weights, the threshold value is changed during training.
• Each neuron has exactly one output to transmit its signal (the “firing”) to other connected neurons or to return it as total output if it is part of the last layer. In these learning units, the output of the neuron is labelled with the letter o (as in output). (Often, the letter y can also be found in the literature. However, this has been avoided here to prevent confusion later on.)

Several neurons together form a neural network. Neurons are usually grouped into layers, and the layers are connected to each other. Now, neurons no longer act individually, but in a network. The output of one neuron may be combined with the output of other neurons to form the input of yet another neuron, whose output can in turn activate other neurons, etc.

However, a neural network’s weights and thresholds do not have to be set manually. They are all generated with random numbers, and then an algorithm helps to change them during the training process so that the network works as well as possible in the end. As there are quite a lot of parameters that have to be optimally coordinated, the training of a neural network is often very difficult to understand. However, the following pages should help you develop an understanding of the concept of learning or training and the concept of error.

Addendum

To show that there really is no magic in a neuron, and to get a better idea of how the graphical representation of the neuron is converted into programming code, the following image shows the source code of a neuron in the Java programming language. Even those who are not yet familiar with it may be able to guess that there is no higher intelligence at work here, just a few assignments and a mathematical calculation. All in all, this is surprisingly little – even when neglecting the fact that the weights and thresholds can change over time.