A while back, I read an amazing book by Michael Nielsen about neural networks. It is a short yet dense book that goes into depth about neural networks and deep learning, covering all the fundamentals of deep learning.
After reading it, I decided to write down my own understanding and explain the book again as if I were teaching it to myself.
If you want to read the book yourself, you can find it here: Neural Networks and Deep Learning.
What Is a Neural Network?
Neural networks are certainly complex. However, the key ideas behind their structure are fairly simple. I will try to explain them with an analogy.
Imagine building a Lego tower. This tower will consist of countless Lego bricks. We will arrange these bricks into layers. Then, these layers will be stacked on top of each other to build the tower.
Neural networks follow the same structure. They are built out of small units called neurons. Each neuron holds a single number. These neurons are organized into vertical layers. Then, all these layers are stacked to build our network, where each layer is connected to the one before it.
A neural network consists of an input layer, an output layer, and one or several in-between layers called hidden layers.
1. Deep Neural Networks
We said that neural networks have an input layer, an output layer, and in-between layers called hidden layers.
If a network has one hidden layer, then it is called a shallow neural network. If it has multiple hidden layers, then it is called a deep neural network. This is where the term "deep learning" comes from.
Deep neural networks have been behind many of the current advances in AI, such as ChatGPT. However, deep networks are even harder to train. The problem comes from the fact that these many layers of the network will start learning at different rates.
This is not an easy problem to solve, and it is still an area of open research.
2. What Is the Neural Network Doing Exactly?
Well, the answer is simple: it is calculating a mathematical function.
Let us say that you have data about lung cancer patients classified according to their age and whether they smoke or not. The neural network will graph these patients as (x, y) points, where x is the age and y is whether they smoke or not.
Then, it will use this training data to try to compute a mathematical function that fits these points. In other words, a function where, if you input the x value for the patient, it will output the y value.
This is useful because now, if you have a new lung cancer patient, the neural network can use this function to predict things about them.
Although the concept is simple, it can be extremely difficult to find this function, especially with complex data such as images. The job of neural network engineers is to design a network that can figure out this function efficiently.
3. Generalization and Overfitting
It is not well understood why neural networks can generalize the way they do. That is, they are able to recognize new data that they have not seen before.
Let us say we are trying to teach a child multiplication. Usually, a child in school will spend their time memorizing the famous multiplication table. However, when faced with a problem outside that table, such as 8 × 42, the child will not be able to figure out that we are just adding 42 eight times, as they did not understand what multiplication is. The result is that they will fail.
The same thing happens with neural networks.
Let us say we are making a network that can recognize handwritten digits, and we train it on 1,000 images. If trained blindly, the network will just memorize these 1,000 images. When we test it with new images, it will fail miserably.
This problem is called overfitting.
However, with good design and the use of certain mathematical functions, the network can generalize and recognize new images of handwritten digits with human-like accuracy.
It is not understood why these design choices and these functions let the network generalize in such a good way, and most good results are based on trial and error.
4. Neural Networks Are Not a New Idea
Interestingly, neural networks are not a new idea.
The most basic neuron, called the perceptron, was first proposed all the way back in 1943. By the 1990s, you could build complex neural networks.
However, as we mentioned earlier, neural networks are hard to train, and computers back then were weak. This is why AI engineers preferred other methods that used less computational power.
Interest in neural networks picked up again in 2005, and they became popular around 2012 for two main reasons.
First, certain neural networks achieved groundbreaking results in image recognition, which let people realize the potential of neural networks.
Second, computer graphics cards had become powerful by then. They were designed for the sake of playing video games. However, AI engineers realized that these cards could be used to train neural networks at a much faster rate than could have ever been done before.
5. How Does a Neural Network Work?
We have a basic idea about what a neural network does now, but how does it work?
I will try to go a little in depth here.
A neural network uses two mathematical functions, two algorithms, as well as a few parameters—which are just numbers—to do its job. I will go into detail about all of these, using an analogy to explain each of them.
Mathematical Functions
- Activation function
- Cost function
Algorithms
- Backpropagation
- Gradient descent
Parameters
- Learning rate
- Regularization parameter
- Number of epochs
- Mini-batch size
Let us go into detail about each of these.
Activation Function
Earlier, we said that a neuron is a basic unit that holds a number. Well, the number that this neuron holds is the output of this activation function.
This output is called the activation value of that neuron.
In this case, what are the inputs?
The inputs for this function are two numbers called a weight and bias.
A neuron is connected to each neuron in the layer before it. Each of these connections has a weight to define the relationship between the two neurons. A higher weight means these two neurons are strongly connected, and vice versa.
A bias is a number that is unique to each neuron. You can say that a bias defines how important this neuron is to the overall network. If a network has 1,000 neurons, then the neurons with higher biases are more important for the network to do its job than the neurons with lower biases.
Now we have a neuron that is connected to each neuron in the previous layer with a weight. We will multiply each of these weights by the activation value of the neuron it came from. We sum up all these values and add our bias.
This final number will be the input to our activation function, which will output the activation value that our neuron will hold.
We mentioned that the activation for a neuron will need the weights and activations of the neurons in the previous layers. This means that the process of calculating the activations will start layer by layer from the input layer until we reach the output layer.
The activations in the output layer are actually the output from our whole network.
For example, let us say that we make a network that takes an image as an input, and the output is the network recognizing whether the image shows a human or not.
In this case, our network will have one output neuron. Usually, a neuron's activation value is between 0 and 1. So, if our output neuron has an activation value of 0.95, then it means that our network is 95% confident that our image shows a human.
On the other hand, if the activation value was 0.01, then we can assume that the image does not show a human.
Examples of activation functions are the sigmoid function and the softmax function.
Cost Function
For a neural network to work well, it needs to be trained on large amounts of data.
If we make a network that recognizes handwritten digits, we will need to use thousands of images of handwritten digits by different people. We will use each of these images as the input for our network, and the output will be which digit it thinks this image is showing.
This decision could be correct, or it could be wrong and our network did not recognize the correct digit.
Our cost function tells the network how wrong this answer is. The network will use this cost to do a better job in the next run using the backpropagation algorithm explained later.
The output of our cost function is a number that shows how far off the answer is. A higher output will mean that our network is further away from the correct answer.
The input for our cost function will be the activation values of our output neurons.
In this example, our network will have 10 output neurons corresponding to each of the digits. If the activation value of the first neuron is 0.2, then this neuron is 20% confident that this image shows a zero. The same applies to the other neurons.
For our answer, we take the neuron that is most confident. If the third neuron has the highest activation value, then the network will answer that this image shows a two.
It is interesting to note that this cost function takes into account all the weights, biases, and activations of the whole network.
The reason is that, as we said, the activation function takes the weights, biases, and activations of the neurons in the previous layer. This means that the activation values of the output layer take into account the weights, biases, and activations from all the previous layers.
This, in turn, means that our output from the cost function is a good measure of the network's performance.
An example of a cost function is the cross-entropy function.
Something to note is that each cost function will work better with a certain activation function.
Tuning Weights and Biases
Now that we know whether our network is doing a good job or not using the cost function, we need a way to tune our weights and biases to make the network better.
We cannot tune our activation values directly, as they are dependent on the weights, biases, and activation values from the previous layer. These activation values are, in turn, dependent on the previous layer, and so on until we reach the input layer.
The activation values in the input layer are just the input data that we give the network. For example, a neuron in the input layer can represent the brightness of a pixel in an image.
This means that the only things we can tune in our network are the weights and biases.
We optimize these weights and biases using the backpropagation and gradient descent algorithms.
Backpropagation
The job of this algorithm is to calculate the derivative of our cost function.
Earlier, we said our cost function is a mathematical function that takes the activation values of the neurons in the output layer. Each of these values comes from the activation function of that neuron.
In a sense, our cost function is a giant combination of activation functions inside activation functions all the way down, because each activation function takes the output from other activation functions as input.
These functions, in turn, have only two variables: the weight and bias.
In a sense, this means that our cost function is a giant equation that includes all the weights and biases in the network as different variables.
This is such a large expression that it is difficult to wrap our heads around it. However, at its core, it is not that different from a function that can be represented by the equation:
f(x) = x + y + z
where x, y, and z are all different variables.
If we want to differentiate this function, we can differentiate it with respect to x, y, or z. For each variable, the derivative will look different. This is basic calculus.
If our cost function is an equation with the weights and biases as variables, it means that we can differentiate this function with respect to any of the weights and biases in the network.
This is useful because it will help us tune any of the weights and biases in our network by differentiating our cost function with respect to that weight or bias.
In essence, the backpropagation algorithm calculates the derivative of the cost function with respect to each of the weights and biases in the network.
The algorithm starts by calculating the derivative with respect to the neurons in the output layer, then works its way back to the neurons in the input layer.
To understand the reason it works backwards, it is useful to see the form of our input:
z = wa + b
where z is the input we will then give to our activation function.
If this is the last layer, then the output of this will be the input to our cost function, so they are closely related.
It is also worth noting that we said our input takes the sum of the weight times the activation of each of the neurons in the previous layer. We can think of wa as being that sum. Finally, b is the bias for that specific neuron.
If we want to differentiate this expression with respect to the weight in the output layer neurons, then it will be straightforward. We will treat a and b as constants and differentiate.
However, if we want to differentiate with respect to a weight in the layer before the last, then we will need to expand our expression, and it will look something like this:
w(wa + b) + b
Now, what is outside the parentheses are the weights and biases of the output layer. What is inside the parentheses are the weights and biases of the layer before the last, while a will be the activations of all the previous layers, which will also be treated as a constant.
Now we can see how complexity will ramp up as we go back through the layers and why it is better to start backwards from the output.
To understand it better, it is useful to use our Lego tower analogy from above.
If we want to change the orientation of one of the pieces in the top layer, then we will only need to be concerned with the pieces in that layer and how they fit together.
However, if we want to change the orientation of a piece in one of the middle layers, then we will need to disassemble our tower from the top and work our way down to the piece we want, and then fit everything together again.
This is how backpropagation works.
It will calculate the derivative with respect to each of the weights and biases in the network. It will then plug this derivative into gradient descent to calculate a new value for that weight and bias.
Gradient Descent
Now that we have calculated the derivative of our cost function with respect to each of the weights and biases in our network, let us see how this is useful in tuning these weights and biases to make the network better.
The goal is to tune them in a way that reduces our cost function because that is the measure of how good our network is.
Let us say we want to tune a certain weight w. We calculated the derivative with respect to w using backpropagation. Now, we will plug this derivative into gradient descent to find a new value for w.
The way gradient descent works is too math-involved to explain here, but at the core of what it is doing is observing the relationship between changes in w and changes in our cost function.
As we know, a derivative of a variable is the rate of change of that variable. This is why it was useful to calculate the derivative in the first place.
Now gradient descent will use a parameter called the learning rate to find a new value for w.
Imagine the process as turning the knobs on a radio to find the sweet spot where you can get a good signal. Gradient descent is the one turning this knob to find a good value for w, while the amount it turns the knob by is determined by the learning rate, which will be explained later.
Hyperparameters
Now that we have a good idea about how a neural network works, the functions, and the algorithms involved, let us take a look at the parameters that we need to set in order to train our network.
These are known as the hyperparameters.
Learning Rate
Earlier, we explained how gradient descent is used to tune our weights and biases. We compared it to a person turning the knobs on a radio to get a good signal.
We said that the learning rate is the amount by which you turn this knob.
Should you turn only a few degrees? Or should you turn it 180 degrees?
If we choose a learning rate that is too small, then it might take a long time to reach the sweet spot we are looking for. If we choose it to be too big, we might pass by the spot without noticing it and miss our target, having to go back.
Usually, at the beginning, we will need to take bigger turns because we are far away from getting a good signal.
In other words, it is unlikely that the random weights and biases that we started with are a good choice, and we need to adjust them by a great amount.
As we progress through training and get better performance, it is better to take smaller steps and adjust our weights by a smaller amount to reach optimal values.
This is why it is usually a good idea to start with a bigger learning rate and then make it lower as training progresses.
Regularization Parameter
Earlier, we explained a big problem when training a neural network called overfitting.
This is where our network will learn to memorize our training data instead of understanding it. Thus, it will achieve worse results in testing.
Regularization is a method to reduce overfitting.
Our network might have thousands of weights. Using gradient descent to tune these weights might make some of them significantly large.
Regularization forces our network to learn and reduce the cost function while, at the same time, focusing on not making the weights too large.
Our regularization parameter defines this process. A high value of this parameter will tell the network to focus more on using lower weights, while a low value will tell it to focus more on lowering the cost function.
It is not understood why this process helps to reduce overfitting, but it works pretty well in practice.
There are different methods of regularization, as well as other methods to reduce overfitting.
Another method is called the dropout method, where, in each epoch of training, you choose random neurons to exclude from the training. This makes the network more robust and less dependent on any single group of neurons.
Another effective method is training data expansion.
Gathering training data is expensive and difficult. However, we can artificially expand our training data by rotating our images by a few degrees or adding some noise. This will make these images be treated as new training images, different from the original ones.
Reducing overfitting is very important in training a network, and all these methods help with that.
Number of Epochs
If we have 1,000 images that we want to use to train a neural network, then, as we learned, we will run these images one by one through our network, calculate our cost function, and then tune the weights and biases of the network.
After we finish running all 1,000 images, it is said that we have completed one epoch of training.
This is usually not enough, so we run these images again for multiple epochs.
Does this mean that the more epochs we train, the better?
Well, the answer is no.
Training for a lot of epochs is bad for two main reasons.
The first is that training takes a lot of time and computation. Each training epoch might take hours or days.
Second, the more epochs you use, the more you are subjected to overfitting. In other words, the more your network sees the training data, the more likely it is to start memorizing this data instead of understanding it.
This is why it is important to choose a good number of epochs that will provide the best performance with the least computation.
It is a good idea to test the network after each epoch to monitor improvements in its accuracy.
However, if we use the test dataset to do this after each epoch, does this mean that the network will start to memorize the test data this time?
The answer is yes.
That is why we usually add a third part of the dataset called validation data.
We train our network using the training data, we test it after each epoch with validation data, and after we finish training, we do a final test using testing data.
Mini-Batch Size
As we know by now, if we want to train our network, we will run our data piece by piece through our network. After each one, we will adjust our weights and biases to better minimize the cost function.
However, it might be computationally expensive to do this process after each training example.
This is why it is a good idea to group training examples into mini-batches, where the weights and biases are updated after each batch.
The algorithm that does this is called stochastic gradient descent, which is the same as normal gradient descent, but it works on multiple training examples instead of one.
It is a good idea to choose a mini-batch size that is big enough to save computation but not too big, where you start losing performance.
Putting It All Together
All that has been explained until now is the fundamentals of how a neural network works.
As we have seen, the choice of all parameters is a delicate balance between good performance and computation.
The job of an AI engineer is to build a network that is best for the job at hand.
They start by choosing the number of layers and the number of neurons in each layer. They will then choose the cost function and activation function that they think are best. Finally, they will choose the parameters.
They will train the network, monitor its performance, and then adjust the parameters to get better results.
This can be a long and tedious process of trial and error. You will need to wait hours or days to finish training, adjust your parameters, and then start the process all over again.
It is usually a good idea to start with a small portion of your dataset.
If you had 50,000 images, you can start with only 1,000 images. This will make training much faster, and it will help you understand how to best choose a network that can be trained on this dataset.