Last year, I was fortunate enough to receive a batch of Tencent CAPTCHA samples from a netizen. I conducted some research on them, and the process was recorded in "End-to-End Tencent CAPTCHA Recognition (46% Accuracy)".
Later, that article attracted interest from many readers, some asking for samples, some for models, and others engaging in discussions, which was quite unexpected. In fact, the original model was relatively crude; specifically, its accuracy was not high enough for practical use, and its reference value was limited. Over the past few days, I have revisited this and developed a model with higher accuracy, while also making the samples public for everyone.
The approach for this model is the same as in "End-to-End Tencent CAPTCHA Recognition (46% Accuracy)", except that the CNN part has been replaced with the off-the-shelf Xception architecture. Of course, readers can also experiment with VGG, ResNet50, etc. In fact, for CAPTCHA recognition, these models are all capable of the task. I chose Xception because it has fewer layers and smaller model weights, which I personally prefer.
Code
Github: https://github.com/bojone/n2n-ocr-for-qqcaptcha/
import glob
samples = glob.glob('sample/*.jpg')
import numpy as np
np.random.shuffle(samples) # Shuffle training samples
nb_train = 90000 # Total 100,000 samples, 90,000 for training, 10,000 for testing
train_samples = samples[:nb_train]
test_samples = samples[nb_train:]
from keras.applications.xception import Xception,preprocess_input
from keras.layers import Input,Dense,Dropout
from keras.models import Model
img_size = (50, 120) # All images are resized to this dimension
input_image = Input(shape=(img_size[0],img_size[1],3))
base_model = Xception(input_tensor=input_image, weights='imagenet', include_top=False, pooling='avg')
predicts = [Dense(26, activation='softmax')(Dropout(0.5)(base_model.output)) for i in range(4)]
model = Model(inputs=input_image, outputs=predicts)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.summary()
from scipy import misc
def data_generator(data, batch_size): # Sample generator to save memory
while True:
batch = np.random.choice(data, batch_size)
x,y = [],[]
for img in batch:
x.append(misc.imresize(misc.imread(img), img_size))
y.append([ord(i)-ord('a') for i in img[-8:-4]])
x = preprocess_input(np.array(x).astype(float))
y = np.array(y)
yield x,[y[:,i] for i in range(4)]
# The training process will display the accuracy per label
model.fit_generator(data_generator(train_samples, 100), steps_per_epoch=1000, epochs=10, validation_data=data_generator(test_samples, 100), validation_steps=100)
# Evaluate the model's overall accuracy (all four characters correct)
from tqdm import tqdm
total = 0.
right = 0.
step = 0
for x,y in tqdm(data_generator(test_samples, 100)):
_ = model.predict(x)
_ = np.array([i.argmax(axis=1) for i in _]).T
y = np.array(y).T
total += len(x)
right += ((_ == y).sum(axis=1) == 4).sum()
if step < 100:
step += 1
else:
break
print 'Model overall accuracy: %s'%(right/total)It is important to note: the pre-trained weights for Xception are from the ImageNet image classification task, which is clearly not suitable for CAPTCHA recognition. Therefore, all layers were unfrozen for training here, rather than fixing most of the weights as is common in general classification tasks.
Results
After training with the code above, the recognition rate on the test set (where all four characters must be correct to count as a success) can reach over 85%. With more refined hyperparameter tuning (considering adjustments to the learning rate, increasing or decreasing iterations, modifying the model structure, etc.), it can reach over 90%.
Additionally, you can refer to the masterpiece by Yang Peiwen, which uses CTC for the final classification: "Using Deep Learning to Crack CAPTCHAs".
Resources
The 100,000 CAPTCHA samples are publicly available at the following link:
Link: https://pan.baidu.com/s/1mhO1sG4
Password: j2rj
When reprinting, please include the original address of this article: https://kexue.fm/archives/4503
For more detailed information regarding reprinting, please refer to: "Scientific Space FAQ"