Among all machine learning models, perhaps the most interesting and profound is the neural network model. I would like to offer my humble opinion and talk about neural networks. Of course, this article does not intend to introduce neural networks from scratch, but rather to discuss my personal understanding of them. For friends who wish to further understand neural networks and deep learning, please refer to the following tutorials:
http://blog.csdn.net/zouxy09/article/details/8775360
Machine Classification
Taking classification as an example, there are many classification problems in data mining or machine learning. For instance, classifying a sentence roughly as "positive" or "negative," or more finely as happy, angry, sad, etc. Another typical classification problem is handwritten digit recognition, which involves classifying images into 10 categories (0, 1, 2, 3, 4, 5, 6, 7, 8, 9). Consequently, many classification models have been developed.
The essence of a classification model is fitting— a model is actually a function (or a family of functions) with some undetermined parameters. Based on existing data, a loss function is determined (the most common loss function is the sum of squared errors; readers who are unclear can recall the process of the least squares method). Then, the loss function is optimized to be as small as possible to find the values of the undetermined parameters. Once the parameter values are found, this function can be used to make predictions. This is the basic idea of classification. As for preventing overfitting and similar issues, they are detailed matters and will not be discussed here for now.
The above idea seems simple, but there are two fundamental and difficult-to-solve problems: 1. What are the independent variables of the function? 2. What is this function? In other words, how do I know which things (features) are helpful for the classification task I want? Secondly, complex non-linear phenomena are everywhere; after finding these features, how do I know which function to use to fit them? In fact, there are still no perfect answers to these two questions. Before the emergence of deep learning, the selection of models and features was basically done manually. In other words, the field of machine learning has developed for decades, yet it hasn’t even solved these two fundamental problems!
The emergence of deep learning has brought significant hope for solving these two problems. The foundation of deep learning is the neural network.
Neural Networks
Neural networks solve the second problem: what is this function. Traditional models, such as linear regression and logistic regression, basically require us to manually specify the form of the function. However, there are so many non-linear functions that simply giving a few ready-made functions often results in limited fitting effects. Moreover, the fitting effect largely depends on finding good features—which is the first problem that hasn’t been solved yet: what are the independent variables of the function. (For example, if a function is y=x^2+x, which is a quadratic non-linear function, then if we use linear regression to fit it, the effect will never be good. However, if I define a new feature (independent variable) t=x^2, then y=t+x is a linear function with respect to t and x. At this point, a linear model can solve it. The problem is, without knowing the specific form of y, how do we find the feature t=x^2? This basically relies on experience and luck.)
To solve the problem of "what is this function," there can be various ideas. For example, we have already learned Taylor series and know that general non-linear functions can be approximated through Taylor expansion. So a natural thought is: why not use high-degree polynomials to approximate it? High-degree polynomials are indeed a good idea, but students who have studied computational methods probably know that the problem with polynomial fitting is that the fitting effect is very good on training data but poor on test data, which is the phenomenon of overfitting. So, is there any other way? Yes! The publisher of neural networks said—use composite functions to fit!
That’s right, neural networks fit through multiple composite functions! And it is the composition of the simplest functions—one is a linear function, and the other is the simplest non-linear function: the binary function \theta(x): \theta(x)=\left\{\begin{aligned}1,&\,x\geq 0\\-1,&\,x < 0\end{aligned}\right.
Taking the binary classification problem of the simplest three-layer neural network as an example:
First, a weighted linear combination of input features is performed (with weights w^{(1)}_{j,i}), then \theta(x) is applied to the combination result in the hidden layer. Then, the results are again linearly combined with weights (weights w^{(2)}_{k,j}), and finally, \theta(x) is applied again in the output layer for output. The entire process consists of two linear combinations and two applications of \theta(x), performed alternately. It can be seen that the entire process is a double composite function: \theta\left(\sum_{j} w^{(2)}_{k,j} \theta\left(\sum_{i} w^{(1)}_{j,i} x_i\right)\right)
The non-linear function \theta(x) is called the activation function. Next, we need to define a loss function, which is the function to be optimized to find the minimum value. However, the optimization process involves derivation, and since \theta(x) is discrete and cannot be derived, continuous activation functions are usually used to replace it, such as the Sigmoid function: S(x)=\frac{1}{1+e^{-kx}}
The question now is whether the model proposed above actually possesses the powerful capabilities we imagine. The following passage demonstrates the power of neural networks:
In 1943, psychologist W.S. McCulloch and mathematical logician W. Pitts established the aforementioned neural network and its corresponding mathematical model, known as the MP model. Through the MP model, they proposed a formalized mathematical description of neurons and network structure methods, proving that a single neuron can perform logical functions. Note that being able to complete logical functions means being able to do everything that today’s computers can do—today’s computers are just a combination of a series of logical instructions. (Of course, there is a matter of speed here; without considering speed, the MP model can indeed complete all the work that current computers can perform.)
Deep Learning
What was mentioned above is only a double composite function. Readers will easily think that since this path is feasible, would taking triple, quadruple, or even more composite functions yield better results? Exactly, this is the original idea of deep learning. After talking for so long, we have finally reached deep learning.
Neural networks, including multi-layer neural networks, are not models that were published recently. In fact, these models were published decades ago, and some solving algorithms were proposed. So, what exactly is the difference between deep learning and previous neural networks? Some common answers are that deep learning is generally much deeper (having five or six hidden layers is common) and introduces more effective solving algorithms that can better solve problems like gradient divergence and convergence to local minima. But I don’t think these are the fundamental differences. I believe the difference between deep learning and traditional neural networks lies in the fact that deep learning is dedicated to solving the first problem we mentioned earlier: what are the independent variables of the function?
In other words, deep learning algorithms are used to discover good features, a task that previously relied on manual selection, and the results were not necessarily "good." Now, machines can automatically (unsupervisedly) complete this task, and the results are no worse than those selected manually. Once good features are found, even linear models can perform well. That’s why it is said that deep learning has pushed the field of artificial intelligence a big step forward, and "deep learning" is called the algorithm currently closest to artificial intelligence.
Why dedicate effort to selecting good features? This is due to several reasons:
1. Accuracy: Finding good features means better excluding interference factors, thereby ensuring that the model’s inputs are beneficial and reducing redundancy;
2. Computational complexity: For example, to classify images of 1000 \times 1000 pixels, if all pixels are input directly, there are 10^6 features. If all these features are input into a deep learning model, the number of parameters in the model could be 10^{12} or more. Solving a model with so many parameters is almost impossible. Finding good features means only inputting useful information into the model, thereby reducing the number of parameters and making solving possible;
3. Storage capacity: Good feature algorithms can often also be used to achieve effective file compression, reducing storage capacity and the memory required during training.
Autoencoders
How does deep learning achieve the above functions? Two words: Information Loss!
First, we need to realize that no matter how intelligent our algorithms are, they are designed to complete a certain task in a certain field. Therefore, it is certain that the original data contains information we don’t need, and it’s even possible that there is more unnecessary information than necessary information. If there were a way to remove this unnecessary information and leave only the useful information, how great that would be! (In information loss, ideally, what is lost is the information we don’t need.) Perhaps everyone hasn’t realized it yet, but in fact, our human brains have been doing the same thing all along. We call that thinking process—abstraction!
Yes, it is abstraction. Abstraction is a process of information loss. For example, if there is a pile of balls—basketballs, footballs, volleyballs, table tennis balls, etc.—our first perception of them is that they are all (approximately) "spheres"—a set of points equidistant from a certain point! But don’t forget, this is a pile of diverse balls; their size, color, material, etc., are all different. Recognizing them as "balls" is equivalent to losing information such as size, color, and material, thereby leaving only the common feature of this pile of balls—their shape is roughly spherical. Such a process is exactly our abstract thinking, and it is also a process of information loss!
How does deep learning realize this process? The "Autoencoder" is one of its core components, which is also based on neural networks. Please look at the following three-layer neural network:
Where the input and output of the network are the same (both 100-dimensional), while the nodes in the middle hidden layer are 50-dimensional. The autoencoder hopes to train a simple function x=x through the above neural network, meaning it hopes the input and output are the same. However, in the training process, from the input layer to the hidden layer, the dimension is reduced from 100 to 50, which means information is lost. Yet from the hidden layer to the output layer, the dimension is restored to 100. Since information has already been lost, theoretically, this reconstruction process is impossible to achieve perfectly.
However, we insist on forcing it to train this way. What will be the result? The machine has no choice but to reluctantly reconstruct the original data from these 50-dimensional data points. To make the reconstruction effect as good as possible, the machine has to extract the common features of a large batch of input data as the result of the reconstruction. As a simple example, please look at the following four images:
These are four different images. How do readers see the differences between them? Obviously, the backgrounds of the images are the same; the difference lies in the letters on the images. If I asked you to remember the differences between these images in a short time, you could only remember the few letters on the images. But don’t forget these are four complete images; remembering only four letters means ignoring the background entirely—losing most of the information!
But if our task is only to "recognize the letters on the image," then ignoring the background is completely correct! After removing the background, we have good features for judgment; the existence of the background would instead be a distraction.
This is the process of self-encoding, the process of information loss, and the process of abstraction!
Summary
Of course, the content of deep learning is very rich, and there are various variations. It is worth mentioning that many models based on deep learning have achieved State Of The Art results (artistic level, i.e., the most advanced), which shows the power of deep learning.
This article is not an introductory tutorial on deep learning, but merely my shallow understanding of neural networks and deep learning. If readers want to systematically study the relevant theories, it is best to first read some books on data mining and machine learning to understand the basic concepts, and then read the two articles mentioned at the beginning.
http://ufldl.stanford.edu/wiki/UFLDL Tutorial
http://blog.csdn.net/zouxy09/article/details/8775360
When reprinting, please include the original address of this article: https://kexue.fm/archives/3331
For more detailed reprinting matters, please refer to: "Scientific Space FAQ"