End-to-End Tencent CAPTCHA Recognition (46% Accuracy)

Samples

This batch of CAPTCHAs is relatively simple, consisting of 4-digit English letters. They include both uppercase and lowercase letters, but the input is not case-sensitive. The patterns have a certain amount of noise/interference, making traditional segmentation-based solutions likely difficult to implement. The end-to-end approach involves feeding the CAPTCHA directly into the model, passing it through several convolutional layers, and then connecting it to several classifiers (26-way classification) to directly output the four letter labels. In fact, there isn’t much else to say; if you have samples, you can do it. Moreover, this framework is universal and can be applied to case-sensitive scenarios (52-way classification) or mixed alphanumeric scenarios (by simply adding 10 more categories).

However, there is one aspect I find quite difficult to handle: the labels are case-insensitive. In this batch of samples, all labels are lowercase, but the CAPTCHAs in the images contain both uppercase and lowercase letters. Thus, if we only use 26-way classification, we are forced to categorize ’A’ and ’a’ into the same class. Since ’A’ and ’a’ look quite different visually, forcing them into the same category seems to be “asking too much of the model”... I suspect this is one of the reasons why my model’s accuracy hasn’t improved further. But I don’t have any better ideas for now.

Code

Without further ado, here is the code:
https://github.com/bojone/n2n-ocr-for-qqcaptcha

The model is very concise and conventional (it’s just a single file, how complex could it be?).

It uses 4 convolutional layers to extract image features, and then connects these image features to 4 separate softmax layers, each classifying into 26 categories. Note that you cannot use TimeDistributed for convenience here; TimeDistributed uses weight sharing, whereas here we need to output different labels for the same feature. If the weights were the same, wouldn’t the results be the same? Additionally, please note that my Keras uses Theano as the backend, not TensorFlow. The two handle image processing differently, so friends using TensorFlow will need to adjust accordingly.

_____________________________________________________________
Layer (type)                     Output Shape          Param #     Connected to
=============================================================
input_15 (InputLayer)            (None, 3, 129, 53)    0
_____________________________________________________________
convolution2d_40 (Convolution2D) (None, 32, 127, 51)   896         input_15[0][0]
_____________________________________________________________
maxpooling2d_48 (MaxPooling2D)   (None, 32, 63, 25)    0           convolution2d_40[0][0]
_____________________________________________________________
convolution2d_41 (Convolution2D) (None, 32, 61, 23)    9248        maxpooling2d_48[0][0]
_____________________________________________________________
maxpooling2d_49 (MaxPooling2D)   (None, 32, 30, 11)    0           convolution2d_41[0][0]
_____________________________________________________________
activation_37 (Activation)       (None, 32, 30, 11)    0           maxpooling2d_49[0][0]
_____________________________________________________________
convolution2d_42 (Convolution2D) (None, 32, 28, 9)     9248        activation_37[0][0]
_____________________________________________________________
maxpooling2d_50 (MaxPooling2D)   (None, 32, 14, 4)     0           convolution2d_42[0][0]
_____________________________________________________________
activation_38 (Activation)       (None, 32, 14, 4)     0           maxpooling2d_50[0][0]
_____________________________________________________________
convolution2d_43 (Convolution2D) (None, 32, 12, 2)     9248        activation_38[0][0]
_____________________________________________________________
maxpooling2d_51 (MaxPooling2D)   (None, 32, 6, 1)      0           convolution2d_43[0][0]
_____________________________________________________________
flatten_15 (Flatten)             (None, 192)           0           maxpooling2d_51[0][0]
_____________________________________________________________
activation_39 (Activation)       (None, 192)           0           flatten_15[0][0]
_____________________________________________________________
dense_63 (Dense)                 (None, 26)            5018        activation_39[0][0]
_____________________________________________________________
dense_64 (Dense)                 (None, 26)            5018        activation_39[0][0]
_____________________________________________________________
dense_65 (Dense)                 (None, 26)            5018        activation_39[0][0]
_____________________________________________________________
dense_66 (Dense)                 (None, 26)            5018        activation_39[0][0]
=============================================================
Total params: 48712
_____________________________________________________________

After dozens of training rounds, the model was obtained. The recognition accuracies for the 1st, 2nd, 3rd, and 4th characters are 0.89, 0.72, 0.73, and 0.87, respectively. Thus, the accuracy for getting all four correct should be: 0.89 \times 0.72 \times 0.73 \times 0.87 \approx 0.41 That is, there should be a 41% overall accuracy rate. Based on actual testing, the performance is even better, with an overall accuracy rate of 46%. This means nearly one out of every two images is recognized correctly, which should be practical in many situations. Of course, this accuracy is specific to this batch of samples; the actual accuracy might be lower, but I estimate it should be at least 10%? ^_^

It is not convenient to make the training samples public, nor is it easy to directly share the model weights. If you need them, please contact me privately.

Afterword

According to the friend who sent me the samples, he is currently using an interface provided by someone else that has an overall accuracy rate of over 95%. I was instantly filled with respect and really want to learn from that person. However, that program has already been commercialized, so it’s unlikely I’ll be able to observe it. I suspect they have been focusing on the specific demand for Tencent CAPTCHA recognition for a long time, unlike me, who is a “jack of all trades, master of none.”

Readers are welcome to provide better modeling ideas. Please feel free to give me your advice. The current model has fewer than 50,000 parameters and might be underfitting; I will try to adjust it further when I have time.