Last year, Data Fountain hosted a competition titled "Power Professional Domain Vocabulary Mining". The interesting part of this competition was that it was an "unsupervised" competition, meaning it tested the ability to mine professional vocabulary from a large corpus without supervision.
This is clearly a valuable capability in the industry. Since I had previously conducted research on unsupervised new word discovery and was intrigued by the novelty of an "unsupervised competition," I participated without hesitation. However, my final ranking was not particularly high.
Regardless, I would like to share my approach. It is a truly unsupervised method and may provide some reference value for some readers.
Benchmark Comparison
First, for the new word discovery part, I used my own library nlp_zero. The basic idea is to perform new word discovery on the "competition corpus" and a "self-crawled encyclopedia corpus" separately, and then compare the two to find the characteristic words of the "competition corpus."
The reference source code is as follows:
from nlp_zero import *
import re
import pandas as pd
import pymongo
import logging
logging.basicConfig(level=logging.INFO, format='%(asctime)s - %(name)s - %(message)s')
class D: # Read the corpus provided by the competition
def __iter__(self):
with open('data.txt') as f:
for l in f:
l = l.strip().decode('utf-8')
# Keep only Chinese characters
l = re.sub(u'[^\u4e00-\u9fa5]+', ' ', l)
yield l
class DO: # Read own corpus (equivalent to a parallel corpus)
def __iter__(self):
db = pymongo.MongoClient().baike.items
for i in db.find().limit(300000):
l = i['content']
l = re.sub(u'[^\u4e00-\u9fa5]+', ' ', l)
yield l
# New word discovery in the competition corpus
f = Word_Finder(min_proba=1e-6, min_pmi=0.5)
f.train(D()) # Calculate mutual information
f.find(D()) # Build vocabulary
# Export word list
words = pd.Series(f.words).sort_values(ascending=False)
# New word discovery in own corpus
fo = Word_Finder(min_proba=1e-6, min_pmi=0.5)
fo.train(DO()) # Calculate mutual information
fo.find(DO()) # Build vocabulary
# Export word list
other_words = pd.Series(fo.words).sort_values(ascending=False)
# Normalize total word frequency for comparison
other_words = other_words / other_words.sum() * words.sum()
"""
Compare word frequencies between the two corpora to obtain characteristic words.
The comparison metric is (competition corpus frequency + alpha) / (own corpus frequency + beta);
The calculation of alpha and beta refers to http://www.matrix67.com/blog/archives/5044
"""
WORDS = words.copy()
OTHER_WORDS = other_words.copy()
total_zeros = (WORDS + OTHER_WORDS).fillna(0) * 0
words = WORDS + total_zeros
other_words = OTHER_WORDS + total_zeros
total = words + other_words
alpha = words.sum() / total.sum()
result = (words + total.mean() * alpha) / (total + total.mean())
result = result.sort_values(ascending=False)
idxs = [i for i in result.index if len(i) >= 2] # Exclude single-character words
# Export to CSV format
pd.Series(idxs[:20000]).to_csv('result_1.csv', encoding='utf-8', header=None, index=None)Semantic Filtering
Note that the word list exported according to the above method can at most be considered "corpus characteristic words," but not entirely "power professional domain vocabulary." If we focus on power-related vocabulary, we need to perform semantic filtering on the word list.
My approach was: use the exported word list to segment the competition corpus, then train a Word2Vec model, and cluster the words based on the word vectors obtained from Word2Vec.
First, training Word2Vec:
# nlp_zero provides a good wrapper to export a tokenizer directly.
# The vocabulary is the one obtained from new word discovery.
tokenizer = f.export_tokenizer()
class DW:
def __iter__(self):
for l in D():
yield tokenizer.tokenize(l, combine_Aa123=False)
from gensim.models import Word2Vec
word_size = 100
word2vec = Word2Vec(DW(), size=word_size, min_count=2, sg=1, negative=10)Then comes the clustering. However, this is not clustering in the strict sense, but rather finding a set of similar words based on several seed words we selected. The algorithm uses the transitivity of similarity (somewhat similar to connectivity-based clustering algorithms): if A and B are similar, and B and C are similar, then A, B, and C are clustered together (even if A and C are not similar according to the metrics). Of course, this transitivity could potentially traverse the entire word list, so the constraints on similarity must be gradually strengthened. For example, if A is a seed word and B and C are not, we might define A and B as similar if their similarity is 0.6, but B and C must have a similarity greater than 0.7 to be considered similar (one could consider calculating the similarity threshold via exponential decay). Otherwise, as the transitivity continues, subsequent words will drift further and further away from the semantics of the seed words.
The clustering algorithm is as follows:
import numpy as np
from multiprocessing.dummy import Queue
def most_similar(word, center_vec=None, neg_vec=None):
"""Find the most similar words based on a given word, center vector, and negative vector.
"""
vec = word2vec[word] + center_vec - neg_vec
return word2vec.similar_by_vector(vec, topn=200)
def find_words(start_words, center_words=None, neg_words=None, min_sim=0.6, max_sim=1., alpha=0.25):
if center_words == None and neg_words == None:
min_sim = max(min_sim, 0.6)
center_vec, neg_vec = np.zeros([word_size]), np.zeros([word_size])
if center_words: # Center vector is the average of all center seed word vectors
_ = 0
for w in center_words:
if w in word2vec.wv.vocab:
center_vec += word2vec[w]
_ += 1
if _ > 0:
center_vec /= _
if neg_words: # Negative vector is the average of all negative seed word vectors
_ = 0
for w in neg_words:
if w in word2vec.wv.vocab:
neg_vec += word2vec[w]
_ += 1
if _ > 0:
neg_vec /= _
queue_count = 1
task_count = 0
cluster = []
queue = Queue() # Establish a queue
for w in start_words:
queue.put((0, w))
if w not in cluster:
cluster.append(w)
while not queue.empty():
idx, word = queue.get()
queue_count -= 1
task_count += 1
sims = most_similar(word, center_vec, neg_vec)
min_sim_ = min_sim + (max_sim-min_sim) * (1-np.exp(-alpha*idx))
if task_count % 10 == 0:
log = '%s in cluster, %s in queue, %s tasks done, %s min_sim'%(len(cluster), queue_count, task_count, min_sim_)
print log
for i,j in sims:
if j >= min_sim_:
if i not in cluster and is_good(i): # is_good is a manual filtering rule
queue.put((idx+1, i))
if i not in cluster and is_good(i):
cluster.append(i)
queue_count += 1
return clusterRule Filtering
Generally speaking, unsupervised algorithms are always difficult to make perfect. In engineering, a common method is to observe the results manually and then write some rules to handle them. In this task, since the previous steps were purely unsupervised, even after semantic clustering, some non-power professional words (such as "Maxwell’s equations") or even "non-words" remained. Therefore, I wrote a set of rules for filtering (they are a bit messy...):
def is_good(w):
if re.findall(u'[\u4e00-\u9fa5]', w) \
and len(w) >= 2\
and not re.findall(u'[comparative/adjective characters]', w)\
and not u'of' in w\
and not u'already/past tense' in w\
and not u'this' in w\
and not u'that' in w\
and not u'to/arrive' in w\
and not w[-1] in u'[list of invalid suffixes: person, in, middle, etc.]'\
and not w[0] in u'[list of invalid prefixes: every, each, by, etc.]'\
and not w[-2:] in [u'problem', u'market', u'email', u'contract', u'hypothesis', u'number', u'budget', u'apply', u'strategy', u'status', u'work', u'assessment', u'evaluation', u'demand', u'communication', u'stage', u'account', u'awareness', u'value', u'accident', u'competition', u'transaction', u'trend', u'director', u'price', u'portal', u'region', u'cultivation', u'responsibility', u'society', u'socialism', u'method', u'cadre', u'committee', u'business', u'development', u'reason', u'situation', u'country', u'park', u'partner', u'opponent', u'target', u'member', u'personnel', u'as follows', u'under circumstances', u'see figure', u'national', u'innovation', u'sharing', u'information', u'team', u'rural', u'contribution', u'competitiveness', u'area', u'customer', u'field', u'query', u'application', u'can', u'operation', u'member', u'secretary', u'nearby', u'result', u'manager', u'degree', u'management', u'thought', u'supervision', u'ability', u'responsibility', u'opinion', u'spirit', u'speech', u'marketing', u'business', u'president', u'see table', u'power', u'editor', u'author', u'album', u'journal', u'create', u'support', u'funding', u'planning', u'plan', u'fund', u'representative', u'department', u'publisher', u'indicate', u'prove', u'expert', u'professor', u'teacher', u'foundation', u'as shown', u'located', u'engaged', u'company', u'enterprise', u'professional', u'idea', u'group', u'construction', u'management', u'level', u'leadership', u'system', u'government', u'unit', u'part', u'director', u'academician', u'economy', u'significance', u'internal', u'project', u'service', u'headquarters', u'discussion', u'improvement', u'literature']\
and not w[:2] in [u'consider', u'in figure', u'each', u'attend', u'one', u'with', u'will not', u'this time', u'produce', u'query', u'whether', u'author']\
and not (u'PhD' in w or u'Master' in w or u'Graduate' in w)\
and not (len(set(w)) == 1 and len(w) > 1)\
and not (w[0] in u'one to ten' and len(w) == 2)\
and re.findall(u'[^list of simple/invalid characters]', w)\
and not u'further' in w:
return True
else:
return FalseAt this point, we can execute the algorithm in full:
# Seed words, picked from the top of the word list obtained in the first step.
# They don't need to be perfectly accurate.
start_words = [u'power grid', u'voltage', u'DC', u'power system', u'transformer', u'current', u'load', u'generator', u'substation', u'unit', u'busbar', u'capacitance', u'discharge', u'equivalent', u'node', u'motor', u'fault', u'transmission line', u'waveform', u'inductance', u'conductor', u'relay', u'transmission', u'parameter', u'reactive power', u'line', u'simulation', u'power', u'short circuit', u'controller', u'harmonic', u'excitation', u'resistance', u'model', u'switch', u'winding', u'electric power', u'power plant', u'algorithm', u'power supply', u'impedance', u'dispatch', u'power generation', u'field strength', u'power source', u'load', u'disturbance', u'energy storage', u'arc', u'distribution', u'coefficient', u'lightning', u'output', u'parallel', u'loop', u'filter', u'cable', u'distributed', u'fault diagnosis', u'charging', u'insulation', u'grounding', u'induction', u'rated', u'high voltage', u'phase', u'reliability', u'mathematical model', u'wiring', u'steady state', u'error', u'electric field strength', u'capacitor', u'electric field', u'coil', u'nonlinear', u'access', u'modal', u'neural network', u'frequency', u'wind speed', u'wavelet', u'compensation', u'circuit', u'curve', u'peak', u'capacity', u'effectiveness', u'sampling', u'signal', u'electrode', u'measured', u'transformation', u'gap', u'module', u'test', u'filtering', u'measurement', u'component', u'optimal', u'loss', u'characteristic', u'resonance', u'live', u'instantaneous', u'damping', u'speed', u'optimization', u'low voltage', u'system', u'outage', u'selection', u'sensor', u'coupling', u'oscillation', u'linear', u'information system', u'matrix', u'controllable', u'pulse', u'control', u'bushing', u'monitoring', u'steam turbine', u'breakdown', u'delay', u'tie line', u'vector', u'rectification', u'transmission', u'maintenance', u'simulation', u'high frequency', u'measurement', u'sample', u'senior engineer', u'transformation', u'specimen', u'experimental research', u'average', u'vector', u'eigenvalue', u'conductor', u'corona', u'magnetic flux', u'kV', u'switching', u'response', u'efficiency']
cluster_words = find_words(start_words, min_sim=0.6, alpha=0.35)
result2 = result[cluster_words].sort_values(ascending=False)
idxs = [i for i in result2.index if is_good(i)]
pd.Series([i for i in idxs if len(i) > 2][:10000]).to_csv('result_1_2.csv', encoding='utf-8', header=None, index=None)Final results (partial):
Transformer
Generator
Substation
Overvoltage
Reliability
Controller
Circuit Breaker
Distributed
Transmission Line
Mathematical Model
Filter
Capacitor
Fault Diagnosis
Neural Network
DC Voltage
Plasma
Tie Line
Sensor
Steam Turbine
Thyristor
Electric Motor
Constraint Conditions
Database
Feasibility
Duration
Rectifier
Stability
Regulator
Electromagnetic Field
Postscript and Reflections
The algorithm in this article achieved a score of approximately 0.22 on the leaderboard, ranking around 100th when the board was frozen. The top score was already 0.49, so from a performance standpoint, there is not much to boast about. However, I heard at the time that many people were using existing professional dictionaries for character labeling. I stopped working on it then because if that was the case, I felt it lost its meaning...
In summary, this article provides an implementation template for the unsupervised extraction of professional terms. If readers find it useful, feel free to use it; if you find it worthless, please feel free to ignore it.
Original Address: https://kexue.fm/archives/6540