The previously discussed scheme is essentially sufficient if the crawled pages contain only a single effective area, such as blog pages or news pages. However, for websites with distinct hierarchical divisions like forums, we need further subdivision. This is because, although the aforementioned steps allow us to extract the effective text, the result is that all the text is lumped together.
Depth-First Search
To further “block” the content, we need to utilize the position information of the DOM tree. As shown in the DOM tree diagram from the previous article, we need to number every node and leaf; that is, we need a way to traverse the DOM tree. Here, we adopt a “Depth-First Search” (DFS) approach.
Depth-First Search (DFS) is an algorithm for traversing or searching tree or graph data structures. The algorithm starts at the root node (selecting some arbitrary node as the root node in the case of a graph) and explores as far as possible along each branch before backtracking. This process continues until all nodes reachable from the source node have been discovered. If any undiscovered nodes remain, one of them is selected as a new source, and the process is repeated until all nodes have been visited.
After a depth-first search of the DOM tree from the previous article, the numbering of each node and leaf is shown in the figure below.
Differential Peak Clustering
After extracting the effective text through the previous steps and numbering it using the depth-first method described above, we obtain a sequence regarding text positions. We find that the difference between text numbers within the same module is relatively small, while the difference between text numbers of different modules is relatively large. For instance, the position numbers for the title, date, and content of the same article are very close, and the position numbers for the title, date, and reply content of the same post floor in a forum are also quite close, as shown below. This induces us to use these position numbers to further cluster the effective text.
To this end, we examine the differential graph of the position sequence and describe the actual floor boundaries with red dashed lines, resulting in the following figure:
The figure above reveals a very obvious pattern: The boundary of each floor is basically located at the local maximum of the differential sequence. Therefore, we can use this scheme to perform segmentation-based clustering, i.e., using local peaks as split points to segment the text. This clustering method has excellent properties, such as:
1. It can automatically determine the number of clusters, meaning we do not need to know in advance how many floors there are;
2. It adapts to special cases. For example, the second and third to last floors are wider than others because they contain comments within the floor (so-called “floor-within-a-floor”), leading to more content. Even so, this pattern still holds.
This indicates that using this pattern to block text and divide different floors is quite reliable. Furthermore, this clustering can be run repeatedly to adapt to different granularity requirements. In fact, for forums, running it approximately 1 \sim 2 times is usually enough to obtain the floor divisions.
Content Recognition
Finally, specifically for forums, we categorize the content of each text block. To avoid complicating the crawler and to ensure its efficiency, the recognition methods used here are relatively simple and are purely rule-based classifications.
Title
Title recognition is relatively simple because normal page source
codes contain a <title> field, which basically
contains complete title information. Therefore, we only need to extract
the title directly using regular expressions.
Date
In Chinese forums, common date formats include:
2017-1-9 15:42
2017-1-9 15:42 (Year-Month-Day in Chinese)
3 hours ago
Yesterday 20:48
The first two date formats are easy to recognize, while the latter two are more difficult. The latter two formats mainly appear in posts published within a short period on certain forums; generally, after a day or two, they automatically revert to the first two formats.
In this case, still proceeding from the perspective of simplicity and stability, we can adopt a compromise strategy: we do not recognize the dates for posts on these websites within the short term and save them mixed with other content. From a continuous monitoring perspective, after a day or two, we will be able to recognize the date. This neither affects the real-time nature of monitoring nor makes the program overly complex.
Main Body
After identifying the time, the single text area is split into upper and lower halves by the time marker. We assume that one of these two halves is the main content. To identify which half contains the main body, we perform global statistics on both halves and compare which half contains more Chinese characters. The part with more characters is considered the main body.
Of course, from a more rigorous perspective, one could follow the academic approach mentioned in the previous article, using language models to determine which part is closer to natural language to identify the main body. However, this scheme is less efficient, and since we are primarily concerned with crawling Chinese content, we excluded this scheme from our experiments.
When reprinting, please include the original address of this article: https://kexue.fm/archives/4422
For more detailed reprinting matters, please refer to: “Scientific Space FAQ”