In the previous article “The Principle of Minimum Entropy (VI): How to Choose the Dimension of Word Vectors?”, we derived a word vector dimension formula “n > 8.33 \log N” based on the idea of minimum entropy. Later, in “The Amazing Johnson-Lindenstrauss Lemma: Application Edition”, we further pointed out that this result is consistent with the \mathcal{O}(\log N) provided by the JL Lemma.
Since it looks perfect in theory, readers naturally asked: What about the experimental results? Is the coefficient 8.33 optimal? This article provides a brief summary of relevant content regarding these questions.
Word Vectors
First, we can calculate directly: when N is 100,000, 8.33 \log N \approx 96; when N is 5 million, 8.33 \log N \approx 128. This indicates that, at least in terms of order of magnitude, the results given by this formula are very consistent with the dimensions we use in practice. In the era of word embeddings, the dimensions of word vectors we train ourselves are usually around 100. Some readers might question that most open-source word vectors are 300-dimensional, and the Embedding layer of BERT even reaches 768 dimensions—doesn’t this clearly deviate from your results?
In fact, the fact that open-source word vectors like FastText are 300-dimensional does not negate the possibility that 128 dimensions could achieve similar effects. As for BERT, it is not a word vector model itself, so its choice of dimension has no direct relationship with the choice of word vector dimensions. Furthermore, ALBERT has already shown that performing low-rank decomposition on the Embedding layer (reducing it to 128 dimensions) hardly changes the model’s performance. Therefore, the 768-dimensional Embedding in BERT is more or less redundant.
Regarding the evaluation of word vectors, a comprehensive 2015 paper “How to Generate a Good Word Embedding?” can be referenced. The paper shows that the improvement of word vectors becomes quite weak after exceeding 50 dimensions, which can be considered supporting evidence for n > 8.33 \log N.
Attention
Another indirect experimental proof of the formula n > 8.33 \log N comes from the attention
mechanism. In “The Amazing
Johnson-Lindenstrauss Lemma: Application Edition”, we analyzed that
the calculation formula for the Attention matrix is mathematically
equivalent to the Skip-Gram model for word vectors. This means that the
formula n > 8.33 \log N can also be
applied to the problem of choosing the head_size in
attention mechanisms.
In the attention mechanism, N is the
sequence length to be processed. A common pre-training length is 512.
Substituting this gives 8.33 \log 512 \approx
52, which is very close to the current mainstream
head_size of 64. This indirectly proves the usability of
n > 8.33 \log N. Conversely, if we
accept this formula, it explains why the head_size of the
attention mechanism only needs to be 64, and also indirectly explains
why the attention mechanism uses multiple small heads instead of one
large head.
For more on the choice of head_size and expressive power
in attention mechanisms, one can also refer to “On the Expressive Power of
Self-Attention Matrices”.
Graph Networks
If we treat each word as a node and the co-occurrence between words as edges, Skip-Gram can also be viewed as a simple graph model. Therefore, the results regarding the choice of word vector dimensions can theoretically be applied to the selection of embedding dimensions in graph networks.
Results in this area can be found in the paper “Graph Entropy Guided Node Embedding Dimension Selection for Graph Neural Networks”. The paper considers both the feature entropy and structural entropy of the graph. The feature entropy part is similar to Skip-Gram and adopts the same approximation as in “The Principle of Minimum Entropy (VI): How to Choose the Dimension of Word Vectors?”, so this part is essentially the formula n > 8.33 \log N.
After combining feature entropy and structural entropy, the results calculated are used as the embedding dimensions for various graph tasks. Experimental results show that this method indeed yields superior dimension selection results:
Summary
This article analyzes the usability of the previously derived dimension selection formula n > 8.33 \log N. By synthesizing existing experimental results from word vectors, attention mechanisms, and graph networks, it is shown that the formula provides reasonable dimension estimates. It also suggests that using entropy to further determine the constant for \log N in the JL Lemma may be a feasible approach.
Original address: https://kexue.fm/archives/8711
For more details on reposting, please refer to: “Scientific Space FAQ”