What Exactly Are Word Vectors and Embeddings?

Fifty Steps Laughing at a Hundred Steps

One-hot encoding is the most primitive way to represent characters or words. For simplicity, this article uses characters as an example, but the concept is similar for words. Suppose our vocabulary contains six characters: "Ke (Science), Xue (Study), Kong (Space), Jian (Between), Bu (Not), Cuo (Bad)." One-hot encoding assigns a 0-1 code to each of these six characters:

Then, to represent the word "Science" (composed of Ke and Xue), we can use the matrix: \begin{pmatrix} 1 & 0 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 & 0 \end{pmatrix}

You might sense the problem here: the number of dimensions in the vector must match the number of characters. If there are 10,000 characters, each character vector is 10,000-dimensional (common characters might not be many, around a few thousand, but for words, common ones could reach hundreds of thousands). Consequently, continuous vector representations emerged, such as using a 100-dimensional real-valued vector to represent a character. This significantly reduces dimensionality, lowers the risk of overfitting, and so on. This is what beginners say, and it is also what many experts say.

However, the truth is: Nonsense! Nonsense! Nonsense! Important things must be said three times.

Let me give you a problem to clarify: If I give you two arbitrary 100 \times 100 real matrices and ask you to calculate their product, few people could do it quickly. However, if I give you two 1000 \times 1000 matrices, but one of them is a one-hot matrix (where each row has only one element as 1 and the rest are 0), you can calculate the product very quickly. If you don’t believe me, try it.

Do you see the point? One-hot matrices are huge, but they are easy to calculate. Your "fancy" real-valued matrices might have smaller dimensions, but they are more troublesome to compute (though this amount of computation is negligible for a computer)! Of course, the deeper reason lies below.

Seemingly False but Actually True

Let’s actually perform a calculation: \begin{pmatrix} 1 & 0 & 0 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 & 0 & 0 \end{pmatrix} \begin{pmatrix} w_{11} & w_{12} & w_{13} \\ w_{21} & w_{22} & w_{23} \\ w_{31} & w_{32} & w_{33} \\ w_{41} & w_{42} & w_{43} \\ w_{51} & w_{52} & w_{53} \\ w_{61} & w_{62} & w_{63} \end{pmatrix} = \begin{pmatrix} w_{11} & w_{12} & w_{13} \\ w_{21} & w_{22} & w_{23} \end{pmatrix}

The form on the left shows that this is a fully connected neural network layer with a 2 \times 6 one-hot matrix as input and 3 nodes in the hidden layer. But look at the right side: isn’t this equivalent to selecting the 1st and 2nd rows from the matrix w_{ij}? Isn’t this exactly the same as the so-called "lookup table" for character vectors (finding the corresponding vector for a character from a table)? In fact, that is exactly what it is! This is the so-called Embedding layer. The Embedding layer is a fully connected layer that takes one-hot as input and has the character vector dimension as the number of hidden nodes! And the parameters of this fully connected layer are the "character vector table"! From this perspective, character vectors haven’t "done" anything new! They are just one-hot; stop mocking the issues of one-hot. Character vectors are simply the parameters of a fully connected layer with one-hot input!

So, is there really no innovation in character and word vectors? There is. From a computational standpoint, it was discovered that multiplying by a one-hot matrix is equivalent to a lookup. Therefore, the operation is implemented directly as a lookup rather than writing it as a matrix multiplication, which greatly reduces the computational load. To emphasize again, the reduction in computation is not because of the "invention" of word vectors, but because the one-hot matrix operation was simplified into a lookup operation. This is at the computational level. At the conceptual level, once the parameters of this fully connected layer are obtained, they are used directly as features—or rather, as representations of characters or words. This yields character/word vectors, which were later found to have interesting properties, such as the cosine of the angle between vectors representing the similarity between characters or words to some extent.

By the way, some people criticize Word2Vec (CBOW) for being only a three-layer model, not truly "deep" learning. In fact, if you count the one-hot fully connected layer, it has 4 layers, which qualifies as a small deep model.

Where Do They Come From?

Wait, if character vectors are treated as parameters of a fully connected layer (dear reader, I must correct you: they aren’t "treated as" such; they are such), then you haven’t told me how to get these parameters! The answer is: I don’t know either. Don’t the parameters of a neural network depend on your task? You should ask yourself what your task is; why ask me? You say Word2Vec is unsupervised? Let me clarify that.

Strictly speaking, neural networks are all supervised. Models like Word2Vec are more accurately described as "self-supervised." In fact, they train a language model, obtaining word vectors through the language model. A language model predicts the probability of the next character given the previous n characters; it is simply a multi-class classifier. We input one-hot, connect a fully connected layer, then connect several other layers, and finally a softmax classifier to get the language model. Then we train it on a large batch of text. Finally, the parameters of the first fully connected layer become the character/word vector table. Of course, Word2Vec also implemented many simplifications, but those were simplifications made to the language model itself. Its first layer remains a fully connected layer, and the parameters of that layer are the character/word vector table.

Looking at it this way, the problem is quite simple. I don’t necessarily have to use a language model to train vectors, right? Exactly. You can use other tasks, such as a supervised text sentiment classification task. As I said, it’s just a fully connected layer; what follows it is entirely up to you. However, since labeled data is usually scarce, this approach is prone to overfitting. Therefore, we generally use large-scale corpora for unsupervised training of character/word vectors first to reduce the risk of overfitting. Note: The reason for reducing the risk of overfitting is that unlabeled corpora can be used to pre-train word vectors (unlabeled corpora can be very large, and a sufficiently large corpus eliminates the risk of overfitting). It has nothing to do with the word vectors themselves. A word vector is just a layer of parameters to be trained; what power does it have to reduce overfitting on its own?

Finally, let me explain why these character and word vectors have certain properties, such as cosine similarity or Euclidean distance reflecting the similarity between words. This is because when we use a language model for unsupervised training, we use a window. We predict the probability of the next character based on the previous n characters, where n is the window size. Words within the same window will receive similar updates. These updates accumulate, and words with similar patterns will accumulate these similar updates to a significant degree. For example, the Chinese characters "Tan" and "Te" (which together form the word "Tan-Te" meaning nervous) are almost always used together. When "Tan" is updated, "Te" is almost always updated as well; thus, their updates are nearly identical, and their character vectors will inevitably be almost the same. "Similar patterns" means that in a specific linguistic task, they are interchangeable. For instance, in a general corpus, "like" in "I like you" and "like" in general contexts can be replaced by "hate" to still form a valid sentence. Therefore, "like" and "hate" will inevitably have similar word vectors. However, if the word vectors were trained specifically for a sentiment classification task, "like" and "hate" would have very different word vectors.

Not to be Continued

I feel like there is more to say, but it seems there isn’t much else that needs saying. I hope these few words help everyone understand the concepts of character and word vectors and clarify the essence of one-hot and Embedding. If you have questions or new insights, feel free to leave a comment.

When reposting, please include the original address: https://kexue.fm/archives/4122

For more details on reposting, please refer to: Scientific Space FAQ