Recently, I had a requirement to crawl some children’s story corpora to train word vectors. Therefore, I found some fairy tale websites and crawled the fairy tale articles from the entire sites. Below, I will share this process implemented in Python and combine it with my previous experience crawling Baidu Baike. This tutorial is suitable for the following needs: the need to traverse and crawl a specified website, where the specified website has no anti-crawling measures. Under this premise, the only challenges we face are the traversal algorithm and programming skills.
Assumptions
To reiterate our assumptions:
1. We need to traverse the entire website to crawl the information we need;
2. The website has no anti-crawling measures;
3. All pages of the website can eventually be reached from the homepage by clicking hyperlinks step-by-step.
What kind of websites fit these assumptions? The answer is quite a few, such as the story website we are about to crawl, as well as Baidu Baike, Hudong Baike, etc.
First, let’s look at how to crawl this story website:
http://wap.xigushi.com/
(For educational purposes only, no malicious intent!)
Breadth-First Search
The Breadth-First Search (BFS) algorithm is actually very simple: Every time a page is crawled, save all internal hyperlinks from that page, and then add the hyperlinks that have not yet been added to the queue into the queue.
Is that it? Yes! It’s that simple. Note that a queue follows the “First-In, First-Out” (FIFO) principle; therefore, what is described above is effectively the BFS algorithm. Writing it in Python is also very straightforward:
#! -*- coding:utf-8 -*-
import requests as rq
import re
import time
import codecs
from multiprocessing.dummy import Pool, Queue # dummy is a multi-threading library
import HTMLParser
unescape = HTMLParser.HTMLParser().unescape # Used to unescape HTML characters
tasks = Queue() # Link queue
tasks_pass = set() # Set of links already queued
results = {} # Variable for results
count = 0 # Total number of pages crawled
tasks.put('/index.html') # Add homepage to the link queue
tasks_pass.add('/index.html') # Add homepage to the processed set
def main(tasks):
global results, count, tasks_pass # Multi-threading can easily share variables
while True:
url = tasks.get() # Get a link
url = 'http://wap.xigushi.com' + url
web = rq.get(url).content.decode('gbk') # Encoding depends on actual situation
urls = re.findall('href="(/.*?)"', web) # Find all internal links
for u in urls:
if u not in tasks_pass: # Add links not yet queued to the queue
tasks.put(u)
tasks_pass.add(u)
text = re.findall('<article>([\s\S]*?)</article>', web)
# Crawl the information we need; requires regex knowledge based on source code
if text:
# Simple processing of the crawled results
text = ' '.join([re.sub(u'[ \n\r\t\u3000]+', ' ', re.sub(u'<.*?>|\xa0', ' ', unescape(t))).strip() for t in text])
results[url] = text # Add to results; using a dict allows deduplication by URL
count += 1
if count % 100 == 0:
print u'%s done.' % count
pool = Pool(4, main, (tasks,)) # Multi-threaded crawling, 4 is the number of threads
total = 0
while True: # This part ends the script if there is no activity for 20 seconds
time.sleep(20)
if len(tasks_pass) > total:
total = len(tasks_pass)
else:
break
pool.terminate()
with codecs.open('results.txt', 'w', encoding='utf-8') as f:
f.write('\n'.join(results.values()))With just a few lines of code, we have implemented a general scraping framework with multi-threaded concurrency. Much of this code consists of boilerplate patterns with high reusability. This is the elegance of Python—as the saying goes, “Life is short, use Python.”
Baidu Baike
The code above has completed a general scraping framework. However,
if we want to crawl Baidu Baike or Hudong Baike, new problems arise.
Because our previous code performed all data I/O in memory, it works
fine for small sites, but for sites like Baike with millions or even
tens of millions of pages, it will naturally be overwhelmed. Therefore,
two issues must be considered: 1. Breakpoint
resume (resumable crawling); 2. Memory efficiency. In fact, both
problems are solved by the same solution: a database. Previously, we stored the queue in a
Queue and the results in a dictionary, both of which reside
in memory. If we place them in a database, these two problems are
naturally resolved.
For databases, I personally prefer MongoDB. Aside from its other
advantages, it feels very “Pythonic” to me—operating it with
pymongo makes you feel like you aren’t even using a
database, but just pure Python (by contrast, using SQL through Python
usually still requires writing SQL statements). I won’t introduce the
installation of MongoDB here; assuming it and pymongo are
installed, here is the reference code for crawling Baidu Baike:
#! -*- coding:utf-8 -*-
import requests as rq
import re
import time
import datetime
from multiprocessing.dummy import Pool
import pymongo # Use database for storage
from urllib import unquote # Used to decode URLs
from urlparse import urlparse, urlunparse # Used to split long URLs
import HTMLParser
unescape = HTMLParser.HTMLParser().unescape # Used to unescape HTML characters
pymongo.MongoClient().drop_database('baidubaike')
tasks = pymongo.MongoClient().baidubaike.tasks # Store queue in database
items = pymongo.MongoClient().baidubaike.items # Store results
tasks.create_index([('url', 'hashed')]) # Create index to ensure query speed
items.create_index([('url', 'hashed')])
count = items.count() # Total number of pages already crawled
if tasks.count() == 0: # If queue is empty, use this page as the initial page
tasks.insert({'url': u'http://baike.baidu.com/item/Science'})
url_split_re = re.compile('&|\+')
def clean_url(url):
url = urlparse(url)
return url_split_re.split(urlunparse((url.scheme, url.netloc, url.path, '', '', '')))[0]
def main():
global count
while True:
# Get a URL and delete it from the queue
url_data = tasks.find_one_and_delete({})
if not url_data:
break
url = url_data['url']
sess = rq.get(url)
web = sess.content.decode('utf-8', 'ignore')
urls = re.findall(u'href="(/item/.*?)"', web) # Find all internal links
for u in urls:
try:
u = unquote(str(u)).decode('utf-8')
except:
pass
u = 'http://baike.baidu.com' + u
u = clean_url(u)
if not items.find_one({'url': u}): # Add links not yet crawled to queue
tasks.update({'url': u}, {'$set': {'url': u}}, upsert=True)
text = re.findall('<div class="content">([\s\S]*?)<div class="content">', web)
# Crawl required info; requires regex knowledge based on source code
if text:
text = ' '.join([re.sub(u'[ \n\r\t\u3000]+', ' ', re.sub(u'<.*?>|\xa0', ' ', unescape(t))).strip() for t in text])
title = re.findall(u'<title>(.*?)_Baidu Baike</title>', web)[0]
items.update({'url': url}, {'$set': {'url': url, 'title': title, 'text': text}}, upsert=True)
count += 1
print u'%s, Crawled "%s", URL: %s, Total: %s' % (datetime.datetime.now(), title, url, count)
pool = Pool(4, main) # Multi-threaded crawling, 4 threads
time.sleep(60)
while tasks.count() > 0:
time.sleep(60)
pool.terminate()Output effect:
-05-17 20:20:18.428393, Crawled “Physical Anthropology”, URL: http://baike.baidu.com/item/Physical_Anthropology, Total: 167
2017-05-17 20:20:18.502221, Crawled “Group Dynamics”, URL: http://baike.baidu.com/item/Group_Dynamics, Total: 168
2017-05-17 20:20:18.535227, Crawled “Biological Taxonomy”, URL: http://baike.baidu.com/item/Biological_Taxonomy, Total: 169
2017-05-17 20:20:18.545897, Crawled “Virology”, URL: http://baike.baidu.com/item/Virology, Total: 170
2017-05-17 20:20:18.898083, Crawled “Chromatography (Book)”, URL: http://baike.baidu.com/item/Chromatography, Total: 171
2017-05-17 20:20:18.929467, Crawled “Molecular Biology (Natural Science)”, URL: http://baike.baidu.com/item/Molecular_Biology, Total: 172
2017-05-17 20:20:18.974105, Crawled “Geochemistry (Subject)”, URL: http://baike.baidu.com/item/Geochemistry, Total: 173
2017-05-17 20:20:18.979666, Crawled “Nanotechnology (Physics Term)”, URL: http://baike.baidu.com/item/Nanotechnology, Total: 174
2017-05-17 20:20:19.077445, Crawled “Theoretical Chemistry”, URL: http://baike.baidu.com/item/Theoretical_Chemistry, Total: 175
2017-05-17 20:20:19.143304, Crawled “Thermochemistry”, URL: http://baike.baidu.com/item/Thermochemistry, Total: 176
2017-05-17 20:20:19.333775, Crawled “Acoustics”, URL: http://baike.baidu.com/item/Acoustics, Total: 177
2017-05-17 20:20:19.349983, Crawled “Mathematical Physics”, URL: http://baike.baidu.com/item/Mathematical_Physics, Total: 178
2017-05-17 20:20:19.662366, Crawled “High Energy Physics”, URL: http://baike.baidu.com/item/High_Energy_Physics, Total: 179
2017-05-17 20:20:19.797841, Crawled “Physics (Natural Science)”, URL: http://baike.baidu.com/item/Physics, Total: 180
2017-05-17 20:20:19.809453, Crawled “Condensed Matter Physics”, URL: http://baike.baidu.com/item/Condensed_Matter_Physics, Total: 181
2017-05-17 20:20:19.898944, Crawled “Atom (Physics Concept)”, URL: http://baike.baidu.com/item/Atom, Total: 182
The basic framework is implemented here. Of course, when readers use it, they need to adjust it according to their needs—for example, strengthening regular expressions to clean out useless information like “Collect View My Collection 0 Useful +1 Voted,” or extracting structured information to store separately (don’t be lazy and use it directly, as the results will contain a lot of noise; there is no such thing as a free lunch). All in all, everyone can build upon this foundation. After sufficient crawling with the above code, approximately 2.8 million entries can be collected, which basically covers commonly used terms.
Reflection & Summary
Some readers might ask: Doesn’t Baidu Baike have links like http://baike.baidu.com/view/52650.htm? Can’t we just iterate through the numbers to traverse the whole site?
If you are only considering Baidu Baike, that logic is correct. However, that approach is not universal; for instance, Hudong Baike does not have such links. Since we are interested in a universal crawling solution, we stick with the BFS traversal approach.
Once again: the websites and code demonstrated in this article are for educational purposes only and have no malicious intent.
When reposting, please include the original address of this article: https://kexue.fm/archives/4385
For more detailed information on reposting, please refer to: Scientific Space FAQ