A Brief Attempt
Yesterday, I briefly experimented with a GAN model on Fashion MNIST and found that it worked reasonably well. Of course, that attempt didn’t require much technical skill; I simply modified the paths in an existing script and ran it. Today, I returned to the primary task of Fashion MNIST—10-class classification. I tested several models using Keras and eventually achieved an accuracy of around 94.5%. With data augmentation via random horizontal flipping, this reached 95%.
Initially, I manually constructed some model combinations, but the test results were not very good. It seems that for this dataset, designing a custom model architecture is quite difficult. Therefore, I decided to use existing model structures. When we think of ready-made CNN models, we usually think of VGG, ResNet, Inception, Xception, etc. However, these models were designed to solve the 1000-class ImageNet problem and seem overly massive for this entry-level dataset, making them prone to overfitting. Then I remembered that Keras includes a model called MobileNet. After checking the model weights, I found that the number of parameters is small, but the capacity should be sufficient. Thus, I chose MobileNet for my experiments.
In-depth Exploration
I won’t go into a detailed introduction of MobileNet, as there are many articles online explaining it. Simply put, its philosophy is similar to Xception; it replaces most standard convolutions with depthwise separable convolutions. This depthwise convolution is somewhat analogous to the SVD decomposition of a matrix, where a large convolution kernel matrix is decomposed into two smaller matrices. This results in fewer parameters and often better performance. There is also more recent work like ShuffleNet, but since there is no Keras version available yet, I skipped it.
The experiment is straightforward: load the MobileNet model, using the default ImageNet pre-trained weights. (It is not necessarily true that ImageNet weights directly help this specific dataset, but they do help speed up convergence and improve precision, as many visual features are generic.) Then, I attached a 10-class classifier and trained the model by unfreezing all weights. A few things to note:
The original design of MobileNet uses a 224 \times 224 input, while Fashion MNIST images are only 28 \times 28. Although direct input won’t trigger an error, I upscaled the images by a factor of two to 56 \times 56 to avoid losing detailed information. They could be scaled larger, but the effect on performance was not significant and would waste computation.
MobileNet requires a three-channel image input. To accommodate this, I simply replicated the grayscale image three times.
The complete code is as follows:
import numpy as np
import mnist_reader
from tqdm import tqdm
from scipy import misc
import tensorflow as tf
np.random.seed(2017)
tf.set_random_seed(2017)
X_train, y_train = mnist_reader.load_mnist('../data/fashion', kind='train')
X_test, y_test = mnist_reader.load_mnist('../data/fashion', kind='t10k')
height,width = 56,56
from keras.applications.mobilenet import MobileNet
from keras.layers import Input,Dense,Dropout,Lambda
from keras.models import Model
from keras import backend as K
input_image = Input(shape=(height,width))
input_image_ = Lambda(lambda x: K.repeat_elements(K.expand_dims(x,3),3,3))(input_image)
base_model = MobileNet(input_tensor=input_image_, include_top=False, pooling='avg')
output = Dropout(0.5)(base_model.output)
predict = Dense(10, activation='softmax')(output)
model = Model(inputs=input_image, outputs=predict)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.summary()
X_train = X_train.reshape((-1,28,28))
X_train = np.array([misc.imresize(x, (height,width)).astype(float) for x in tqdm(iter(X_train))])/255.
X_test = X_test.reshape((-1,28,28))
X_test = np.array([misc.imresize(x, (height,width)).astype(float) for x in tqdm(iter(X_test))])/255.
model.fit(X_train, y_train, batch_size=64, epochs=50, validation_data=(X_test, y_test))The code is very simple and clear, so I haven’t added extra comments.
After multiple tests, the model generally reaches over 94.5% accuracy within 20 epochs. (Even though we set a random seed, results may vary slightly across runs due to cuDNN). Later epochs become unstable, suggesting potential overfitting.
Fine-tuning
I feel that achieving over 94.5% accuracy without data augmentation is quite satisfactory. I then tested the model with data augmentation. For this dataset, there aren’t many suitable augmentation methods; the only one I could think of was random horizontal flipping. Let’s see the results:
import numpy as np
import mnist_reader
from tqdm import tqdm
from scipy import misc
import tensorflow as tf
np.random.seed(2017)
tf.set_random_seed(2017)
X_train, y_train = mnist_reader.load_mnist('../data/fashion', kind='train')
X_test, y_test = mnist_reader.load_mnist('../data/fashion', kind='t10k')
height,width = 56,56
from keras.applications.mobilenet import MobileNet
from keras.layers import Input,Dense,Dropout,Lambda
from keras.models import Model
from keras import backend as K
input_image = Input(shape=(height,width))
input_image_ = Lambda(lambda x: K.repeat_elements(K.expand_dims(x,3),3,3))(input_image)
base_model = MobileNet(input_tensor=input_image_, include_top=False, pooling='avg')
output = Dropout(0.5)(base_model.output)
predict = Dense(10, activation='softmax')(output)
model = Model(inputs=input_image, outputs=predict)
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
model.summary()
X_train = X_train.reshape((-1,28,28))
X_train = np.array([misc.imresize(x, (height,width)).astype(float) for x in tqdm(iter(X_train))])/255.
X_test = X_test.reshape((-1,28,28))
X_test = np.array([misc.imresize(x, (height,width)).astype(float) for x in tqdm(iter(X_test))])/255.
def random_reverse(x):
if np.random.random() > 0.5:
return x[:,::-1]
else:
return x
def data_generator(X,Y,batch_size=100):
while True:
idxs = np.random.permutation(len(X))
X = X[idxs]
Y = Y[idxs]
p,q = [],[]
for i in range(len(X)):
p.append(random_reverse(X[i]))
q.append(Y[i])
if len(p) == batch_size:
yield np.array(p),np.array(q)
p,q = [],[]
if p:
yield np.array(p),np.array(q)
p,q = [],[]
model.fit_generator(data_generator(X_train,y_train), steps_per_epoch=600, epochs=50, validation_data=data_generator(X_test,y_test), validation_steps=100)As expected, data augmentation helps. I ran it twice: once achieving 95.04% and once 94.91%. This means that within 50 epochs, it can reach approximately 95% accuracy. Note that not all data augmentation methods improve performance; I tried adding a random mask, but the performance actually dropped. Therefore, data augmentation must be adapted to the dataset, especially the test set. To put it bluntly, while data augmentation is performed on the training set, its essence is to introduce prior knowledge about the test set.
A Long Way to Go
It seems that Fashion MNIST is indeed quite challenging. Unlike the original MNIST, where a single Dense layer can easily reach over 90% accuracy, Fashion MNIST serves as a representative benchmark for CNN algorithms. While MNIST test accuracy generally reaches over 99%, the highest accuracy I have seen for Fashion MNIST is around 96% (and that was without public source code). There is still a long way to go to reach 99%.
I wonder which model will be the first to reach 99% accuracy on this dataset.
Reprinting Notice: Please include the original link when reprinting: https://kexue.fm/archives/4556
For more details on reprinting, please refer to: Scientific Space FAQ