Opening Remarks
In today’s world, where end-to-end solutions like deep learning have gradually swept through NLP, are you still willing to think about the fundamental principles behind natural language? We often talk about “text mining”; do you truly feel the essence of “mining”?
An Unexpected Encounter
Some time ago, I read an article on unsupervised syntactic analysis. From its references, I discovered the paper “Redundancy Reduction as a Strategy for Unsupervised Learning.” This paper describes how to recover English words from English articles where spaces have been removed. Corresponding to Chinese, isn’t this exactly vocabulary construction? So, I read it with great interest and found the logic clear, the theory complete, and the results beautiful—it was truly delightful.
Although the value of this paper might not seem significant today, and its results may have been learned by many, it is important to note: this is a paper from 1993! In an era before PCs were popular, such forward-looking research was conducted. While deep learning is popular now and NLP tasks are becoming increasingly complex—which is indeed a great step forward—our true understanding of NLP principles might not have surpassed that of our predecessors from decades ago by much.
This paper achieves unsupervised vocabulary construction through “Redundancy Reduction.” From the perspective of information theory, “redundancy reduction” is the minimization of information entropy. The article on unsupervised syntactic analysis also pointed out that “information entropy minimization is the only feasible solution for unsupervised NLP.” I further studied some related materials and reflected on them combined with my own understanding, finding this comment to be quite thought-provoking. I believe that not just for NLP, information entropy minimization is likely the fundamental principle of all unsupervised learning.
What is the Principle of Minimum Entropy?
Readers may have heard of the Maximum Entropy Principle and the Maximum Entropy Model. Now, what is this Minimum Entropy Principle? Does it not contradict Maximum Entropy?
We know that entropy is a measure of uncertainty. The Maximum Entropy Principle suggests that when making inferences about outcomes, we must acknowledge our ignorance; therefore, we should maximize uncertainty to obtain the most objective results. Regarding the Minimum Entropy Principle, we have two perspectives for understanding:
1. Intuitive Understanding: The evolution of civilization is always a process of exploration and discovery. Through our efforts, more and more things change from uncertain to certain, and entropy gradually tends toward minimization. Therefore, to discover hidden patterns from a pile of raw data (replaying the evolution of civilization), one must see if these patterns help reduce the overall information entropy, as this represents the direction of civilizational evolution. This is the “Principle of Minimum Entropy.”
2. Rigorous Understanding: “Knowledge” has an inherent information entropy, representing its essential information content. However, before we fully understand it, there are always unknown factors, which cause us to have redundancy when expressing it. Thus, the information entropy estimated according to our current understanding is actually an upper bound of the inherent information entropy. Minimizing information entropy means we must find ways to lower this upper bound, which implies reducing unknowns and approaching the inherent information entropy.
Consequently, following the path of the “Minimum Entropy Principle,” I have reorganized previous works and made some new extensions, which eventually resulted in these writings. Readers will gradually see that the Minimum Entropy Principle can derive rich results using a highly explanatory and heuristic method.
Information of Language
Let us begin by examining the information entropy of language and slowly enter the world of the Minimum Entropy Principle.
Information Entropy = Learning Cost
From previous discussions, we know that the information entropy of an object is proportional to the negative logarithm of its probability, namely: I(c) \sim -\log p_c \tag{1.1} If we consider the basic unit of Chinese to be the character, then Chinese is a combination of characters. p_c represents the probability of a corresponding character, and -\log p_c is the information content of that character. We can estimate the average information of each Chinese character through a large corpus: \mathcal{H}_c = -\sum_{c \in \text{Characters}} p_c \log p_c \tag{1.2} If the \log is base 2, then according to data circulating online, this value is approximately 9.65 bits (I have calculated some articles myself and obtained a value of about 9.5; the two are comparable). Similarly, the average information of each letter in English is about 4.03 bits.
What does this number mean? Generally, it can be assumed that the speed at which we receive or memorize information is fixed. Therefore, the size of this information content actually corresponds to the time we need to receive this information (or the effort spent, etc.). Thus, it can also be said that this number represents the difficulty of learning this thing (memory load). For example, assuming we can only receive 1 bit of information per second, then memorizing an 800-character article character-by-character would require 9.65 \times 800 seconds.
Chinese vs. English: Which is Superior?
Since the information entropy of a single Chinese character is 9.65, while that of an English letter is only 4.03, does this mean English is a more efficient form of expression?
Clearly, we cannot be so hasty. Is it necessarily easier to memorize an English essay than a Chinese one?
For instance, if an 800-character Chinese essay is translated into English, it might consist of 500 words. If each English word averages 4 letters, then the information content is 4.03 \times 500 \times 4 \approx 9.65 \times 800. It can be seen that they are equivalent. In other words, comparing the information content of different linguistic units is meaningless; what is meaningful is the total information—that is, who is more concise when describing the same meaning.
When the meaning of two sentences is the same, the inherent information content of this “meaning” is constant. However, when expressed in different languages, “redundancy” is inevitably introduced. Therefore, the information content of different languages varies when expressing the same thing. This information content is actually equivalent to the memory load; the more redundant the language, the greater the information content and the memory load. It is like teaching the same course: some teachers teach clearly and concisely, and students understand easily; other teachers are wordy and repetitive, and students learn painfully. For the same lesson, the essential amount of knowledge is the same, but a poor teacher introduces excessive irrelevant information during the teaching process—this is “redundancy.” Therefore, we must find ways to “reduce redundancy.”
The estimated results for Chinese and English mentioned above are similar, indicating that both Chinese and English have undergone long-term optimization. Both sides have roughly reached a relatively optimized state, and there is no clear claim that one is superior to the other.
The Path of Routines
Note that in the above estimation, we emphasized “memorizing by character.” Perhaps we are too familiar with Chinese to realize what this implies. In fact, it represents a very mechanical way of memorizing, which is not how we actually do it.
There is Knowledge even in Chanting
Recalling how we memorized ancient poems and classical Chinese as children, at first, we memorized them without understanding, swallowing them whole—recognizing every character but not knowing the meaning when they were strung together. This is the so-called “reading by character.” Obviously, the difficulty of such memory is very high. Later, we gradually began to figure out the patterns of classical Chinese and could understand the meaning of some poems or prose; the difficulty of memorization then decreased. By high school, we also learned grammatical rules in classical Chinese such as “object fronting” and “attributive postpositioning,” which were very helpful for our memory and understanding.
Here comes the key point!
From the example of classical Chinese, we can feel that reciting character-by-character like chanting is very difficult. It becomes easier after grouping characters into words for understanding, and even easier if some grammatical rules can be found. However, our speed of receiving (memorizing) information remains fixed. This means that steps like word segmentation and grammar reduce the information content of the language, thereby reducing our learning cost!
Thinking further, it is not just language; everything we learn is like this. If there are only a few things to learn, we can just memorize them by brute force. But when there are many things to learn, we try to find the “routines” (patterns) within them. For example, in Chinese chess, it is divided into the opening, middle game, and endgame. Each phase has many “fixed patterns” (joseki) used to reduce the learning difficulty for beginners and serve as a basis for adapting to complex situations. Similarly, we have The Art of War and The Thirty-Six Stratagems; these are all “compendiums of routines.” By mining “routines” to alleviate the burden of individual memorization, the emergence of a “routine” is a process of reducing information content.
Ultimately, if you chant enough, you can discover the routines of the scripture.
Adapting to Changes with the Invariant
In a nutshell, our speed of receiving information is fixed, so the only way to accelerate our learning progress is to reduce the redundant information content of the learning target. This is the so-called “extracting the essence,” which is the Minimum Entropy Principle in NLP—the “redundancy reduction” mentioned at the beginning. We can understand it as “saving unnecessary learning costs.”
In fact, an efficient learning process must reflect this idea. Similarly, teachers design their teaching plans based on this idea. When teaching, teachers prefer to teach “general solutions and methods” (even if they involve more steps) rather than choosing a unique, clever solution for every single problem. When preparing for the college entrance examination, we work hard to figure out various question patterns and problem-solving routines. These are all processes of reducing information entropy and thus reducing learning costs by mining “routines.” A “routine” is a method of “redundancy reduction.”
A “routine” is a “fixed pattern.” With enough routines, we can adapt to all changes with the invariant. As the saying goes, “ten thousand changes do not depart from the origin”; this “origin” is likely the routine. When there are too many “routines,” we further look for “meta-routines”—that is, routines of routines—to alleviate the burden of memorizing routines. This is a progressive process. It seems that elevating individual phenomena into routines is precisely the manifestation of human intelligence!
Well, enough empty talk. Next, we will formally embark on the journey of cultivating routines.
Reprinting: Please include the original address of this article: https://kexue.fm/archives/5448
Detailed Reprinting Matters: Please refer to “Scientific Space FAQ.”