SVD Decomposition (I): Autoencoders and Artificial Intelligence

Equivalence

Suppose we have a large matrix M_{m \times n} with m rows and n columns. This might pose problems for computation or even storage. Therefore, we consider a decomposition, hoping to find matrices A_{m \times k} and B_{k \times n} such that: M_{m \times n} = A_{m \times k} \times B_{k \times n} Here, the multiplication is standard matrix multiplication. As shown in the figure:

In this way, the original mn elements are transformed into (m+n)k elements on the right side. If k is small enough, then (m+n)k < mn, which reduces both storage and computation. Of course, if k is sufficiently small, the equality generally does not hold strictly; it holds under the condition of minimizing error. We will not make a strict distinction here. This is the core of the SVD decomposition described in this article. If the matrix M_{m \times n} is originally sparse (such as a document matrix represented by one-hot encoding or a user rating matrix), k can usually be several orders of magnitude smaller while maintaining basic accuracy. One of the uses of SVD is for topic modeling, recommendation systems, and so on. (Note: SVD might be defined differently in some contexts, but its fundamental purpose remains the same.)

So, what about Autoencoders? Here, let us first ignore the activation functions and look only at the linear structure. An Autoencoder aims to train an identity function f(x)=x, but with a compression at the hidden layer nodes.

Mathematically, it seeks to find matrices C_{n \times k} and D_{k \times n} such that: M_{m \times n} = M_{m \times n} \times C_{n \times k} \times D_{k \times n} Again, the equality is not strict but optimal in a certain sense. The illustration is as follows:

In this case, if we let: A_{m \times k} = M_{m \times n} \times C_{n \times k}, \quad B_{k \times n} = D_{k \times n} Then aren’t the two equivalent? As long as they are optimized under the same error function (loss function), the results obtained must be equivalent.

Thus, we have proven that a three-layer Autoencoder without activation functions is actually equivalent to traditional SVD decomposition. Although they are equivalent, the Autoencoder is still an innovation because it transforms matrix decomposition into a neural network compression and encoding problem, which is clearer and more intuitive. Furthermore, we can train in batches, unlike SVD which typically requires all data to be input at once (of course, one could devise a "batch SVD" algorithm by mimicking neural network training, but that would be redundant).

Compression and Intelligence

While Autoencoders are more intuitive in terms of compression and encoding, the physical meaning of SVD is more explicit. This shows that looking at the same thing from different perspectives is very meaningful. The physical meaning of SVD can be observed by considering document encoding. As shown in the figure below, each column represents a word and each row represents an article. The figure illustrates 6 articles using 5 words: A, B, C, D, and E.

As shown, we can view the physical meaning of SVD this way: The first decomposed matrix can be seen as clustering the rows, i.e., a simple clustering of documents; the second decomposed matrix can be seen as clustering the columns, i.e., clustering each word. Matrix multiplication then signifies the matching between document clusters and word clusters. Readers might ask why words and documents must be clustered into the same number of categories. In fact, a more reasonable formulation would be: M_{m \times n} = A_{m \times k} \times P_{k \times l} \times Q_{l \times n} This is equivalent to saying documents are clustered into k categories, words into l categories, and P_{k \times l} is the matching matrix between document categories and word categories. We then simply set B_{k \times n} = P_{k \times l} \times Q_{l \times n}.

Why construct such a seemingly forced explanation? My purpose is to explain the reason why "intelligence" is manifested in artificial models. We know that humans have memory, but after reaching a certain level of memory, we are often not satisfied with pure rote memorization; instead, we find the patterns in what we remember and memorize through patterns. For example, in language, we categorize words into verbs, nouns, etc., and then discover patterns like "verb + noun" usually forming a phrase. We might not explicitly state these patterns, but it is undeniable that our brains are performing such "pattern finding" and "classification" processes. From a data mining perspective, I believe this is a process of compression and clustering. That is, by reconstructing data after compression, we can mine the commonalities of the data—what we call patterns—and obtain more generalized results.

This might answer some questions, such as why deep learning-based word segmentation models (whether LSTM or CNN) have such good recognition capabilities for new words. It is because they contain structures like Autoencoders, which are equivalent to SVD. As explained earlier, SVD carries the physical meaning of clustering (pattern finding), making it more generalized and higher-performing.

Deriving New Words

Too abstract? Hard to follow? Don’t worry, here is an example. This example involves our understanding of words. Please look at the following table:

Suppose this is a statistical result from a science fiction novel. It indicates that words like "God-Beast," "God-World," "God-Dragon," "God-Spirit," "Magic-Beast," "Magic-World," "Magic-Dragon," "Magic-Spirit," and "Magic-Art" all appeared in the novel with a frequency of 1, but the word "God-Art" never appeared. Would we then infer: "Magic" and "God" seem to have similar usage, so is "God-Art" a valid word? How do we express this mathematically?

Surprisingly, one way to do this is SVD! That is to say, we should perform some clustering on the usage of the first character (perhaps as adjectives, verbs, or nouns?) and some clustering on the usage of the second character, and then consider the pairings between categories. Of course, we don’t actually need to tell the computer to cluster into nouns or verbs; the computer doesn’t need to understand language the way humans do. We only need to tell the computer to perform clustering, and since SVD inherently implies clustering, we just need to perform SVD. The specific experiment is as follows:

Select all two-character words with a frequency of at least 100 from the Jieba segmentation dictionary. Create a matrix where the first character represents the row, the second character represents the column, and the frequency is the value (default to 0 if the word does not exist). Perform SVD decomposition on this matrix, obtain two matrices, and then multiply them back together (reconstruction). Treat the reconstructed matrix as new frequencies and compare it with the original matrix to see which words with higher frequencies have emerged that were not in the original set.

As a result, I obtained some words that were not in the input dictionary, such as (the following is just a small sample):

Dragon-Brain 10.271244
Dragon-Foot 12.496673
Dragon-Wax 16.860170
Dragon-Leg 12.172765
Dragon-Lotus 11.362767
Dragon-Cane 67.800909
Dragon-Potato 30.580730
Seven-Lectures 11.439969
Seven-Reviews 12.362163
Seven-Lessons 11.587767
Seven-Son 12.789438
Seven-Hidden 10.479609
Seven-Pages 15.499356
Seven-Items 18.802959
Resentment-Anger 29.831461
Resentment-Evil 15.875075
Resentment-Bitterness 24.979326
Strange-Rabbit 10.246062
Strange-Odd 12.768122
Strange-Child 14.391967
Strange-Shape 14.856068

Among these words, many are unreliable; in fact, unreliable ones are far more numerous than reliable ones. But this is not important, because if we process natural language, unreliable ones are not sentences that appear in real environments. This yields a magical result: Through SVD decomposition, our original purpose might have been compression or dimensionality reduction, but after reconstruction, it derived richer results with a larger vocabulary. This seems to be the result of clustering and pattern finding. This is different from statistical new word discovery; these new words do not exist in the corpus but are derived through the construction rules of existing words. It is a true process of inference, i.e., "intelligence." To put it hyperbolically, SVD brings about a preliminary form of artificial intelligence (though it is certainly not the only method).

Activation Functions

What on earth is an activation function? In the experiments above, I also discovered a practical physical meaning for activation functions. When we perform SVD on a matrix and then reconstruct it, some elements in the reconstructed matrix will be negative. However, the meaning of our matrix is frequency, and frequency cannot be negative. Therefore, we discard (truncate) the negative numbers and record them as 0.

Wait? Isn’t this exactly the operation of the function \max(x, 0)? Isn’t this the most widely used ReLU activation function? It turns out that the physical meaning of the activation function here is simply the discarding (truncation) of irrelevant elements. This kind of operation has been used in statistics for a long time (discarding negative numbers, discarding small statistical results, etc.).

Of course, in neural networks, activation functions have deeper significance, but in shallow networks (matrix decomposition), the intuitive feeling they give us is just truncation.

Summary

This article centered on SVD, exploring its physical meaning (clustering), the connection between SVD decomposition and Autoencoders, the intricate relationship between SVD and intelligence, and a related experiment. Most of this content is conceptual and interpretive, aimed at deepening our understanding of models and AI, so that we have more "confidence" when using models rather than just blindly applying them.

When reposting, please include the original address: https://kexue.fm/archives/4208

For more detailed reposting matters, please refer to: Scientific Space FAQ