Digit Recognizer (Kaggle) - over 99% accuracy
Apr 3, 2020

This is a very simple tutorial for tensorflow 2.0 beginners!

This tutorial is based on Digit Recognizer presented by Kaggle

You may be able to submit with the output generated by your model

Goal

  1. using tensorflow 2.0 frameworks
  2. using Sequential Models
  3. applying Image Augmentation by using ImageDataGenerator
  4. Make submission to Kaggle

Import Libraries

In [1]:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import warnings

warnings.filterwarnings('ignore')
%matplotlib inline

Load Data

In [2]:
train = pd.read_csv('data/train.csv')
test = pd.read_csv('data/test.csv')
In [3]:
x_train = train.drop('label', 1)
y_train = train['label']

Explore class distributions

In [4]:
y_train.value_counts()
Out[4]:
1    4684
7    4401
3    4351
9    4188
2    4177
6    4137
0    4132
4    4072
8    4063
5    3795
Name: label, dtype: int64
In [5]:
x_train.shape, y_train.shape
Out[5]:
((42000, 784), (42000,))

Reshaping Model to 28 * 28

change shape to (28, 28, 1)

28 X 28 picture size with graysclae channel of 1

In [6]:
x_train = x_train.values.reshape(-1, 28, 28, 1)
In [7]:
y_train = y_train.values.reshape(-1, 1)
In [8]:
y_train.shape
Out[8]:
(42000, 1)

x_test refers to test datasets that I need to predict

In [9]:
x_test = test.values.reshape(-1, 28, 28, 1)
In [10]:
x_test.shape
Out[10]:
(28000, 28, 28, 1)

Using ImageDataGenerator to augment Images

In [11]:
from tensorflow.keras.preprocessing.image import ImageDataGenerator
In [12]:
datagen = ImageDataGenerator(
        rescale=1.0/255,
        featurewise_center=False,  # set input mean to 0 over the dataset
        samplewise_center=False,  # set each sample mean to 0
        featurewise_std_normalization=False,  # divide inputs by std of the dataset
        samplewise_std_normalization=False,  # divide each input by its std
        zca_whitening=False,  # apply ZCA whitening
        rotation_range=10,  # randomly rotate images in the range (degrees, 0 to 180)
        zoom_range = 0.1, # Randomly zoom image 
        width_shift_range=0.1,  # randomly shift images horizontally (fraction of total width)
        height_shift_range=0.1,  # randomly shift images vertically (fraction of total height)
        horizontal_flip=False,  # randomly flip images
        vertical_flip=False)  # randomly flip images
In [13]:
from sklearn.model_selection import train_test_split
In [14]:
x_train, x_val, y_train, y_val = train_test_split(x_train, y_train, test_size=0.2)
In [15]:
x_train.shape, x_val.shape
Out[15]:
((33600, 28, 28, 1), (8400, 28, 28, 1))

Modeling

In [16]:
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Conv2D, MaxPool2D, Flatten, Dropout
from tensorflow.keras.callbacks import ModelCheckpoint
In [17]:
model = Sequential()
model.add(Conv2D(filters=32, kernel_size=(5, 5), padding='same', activation='relu', input_shape=(28, 28, 1)))
model.add(Conv2D(filters=32, kernel_size=(5, 5), padding='same', activation='relu', input_shape=(28, 28, 1)))
model.add(MaxPool2D(pool_size=(2, 2), strides=2))
model.add(Conv2D(filters=64, kernel_size=(3, 3), padding='same', activation='relu'))
model.add(Conv2D(filters=64, kernel_size=(3, 3), padding='same', activation='relu'))
model.add(MaxPool2D(pool_size=(2, 2), strides=2))
model.add(Dropout(rate=0.25))
model.add(Flatten())
model.add(Dense(1024, activation='relu'))
model.add(Dropout(rate=0.25))
model.add(Dense(10, activation='softmax'))
In [18]:
model.summary()
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d (Conv2D)              (None, 28, 28, 32)        832       
_________________________________________________________________
conv2d_1 (Conv2D)            (None, 28, 28, 32)        25632     
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 14, 14, 32)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 14, 14, 64)        18496     
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 14, 14, 64)        36928     
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 7, 7, 64)          0         
_________________________________________________________________
dropout (Dropout)            (None, 7, 7, 64)          0         
_________________________________________________________________
flatten (Flatten)            (None, 3136)              0         
_________________________________________________________________
dense (Dense)                (None, 1024)              3212288   
_________________________________________________________________
dropout_1 (Dropout)          (None, 1024)              0         
_________________________________________________________________
dense_1 (Dense)              (None, 10)                10250     
=================================================================
Total params: 3,304,426
Trainable params: 3,304,426
Non-trainable params: 0
_________________________________________________________________
In [19]:
batch_size = 128
epochs = 20
In [20]:
model_path = 'model/tmp-best-model.h5'
In [21]:
checkpoint = ModelCheckpoint(model_path, 
                             verbose=1, 
                             monitor='val_loss',
                             save_best_only=True,
                             save_weight_only=True, 
                             mode='auto')
In [22]:
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])
In [23]:
history = model.fit(datagen.flow(x_train, y_train, batch_size=batch_size),
                              epochs = epochs, validation_data = (x_val, y_val),
                              steps_per_epoch=x_train.shape[0] // batch_size,
                              callbacks=[checkpoint],
                             )
Train for 262 steps, validate on 8400 samples
Epoch 1/20
260/262 [============================>.] - ETA: 0s - loss: 0.3495 - accuracy: 0.8886
Epoch 00001: val_loss improved from inf to 13.87659, saving model to model/tmp-best-model.h5
262/262 [==============================] - 8s 29ms/step - loss: 0.3477 - accuracy: 0.8893 - val_loss: 13.8766 - val_accuracy: 0.9755
Epoch 2/20
261/262 [============================>.] - ETA: 0s - loss: 0.1016 - accuracy: 0.9692
Epoch 00002: val_loss improved from 13.87659 to 8.74911, saving model to model/tmp-best-model.h5
262/262 [==============================] - 7s 26ms/step - loss: 0.1014 - accuracy: 0.9692 - val_loss: 8.7491 - val_accuracy: 0.9845
Epoch 3/20
261/262 [============================>.] - ETA: 0s - loss: 0.0749 - accuracy: 0.9769
Epoch 00003: val_loss improved from 8.74911 to 5.84947, saving model to model/tmp-best-model.h5
262/262 [==============================] - 7s 26ms/step - loss: 0.0752 - accuracy: 0.9769 - val_loss: 5.8495 - val_accuracy: 0.9860
Epoch 4/20
261/262 [============================>.] - ETA: 0s - loss: 0.0627 - accuracy: 0.9807
Epoch 00004: val_loss improved from 5.84947 to 5.80423, saving model to model/tmp-best-model.h5
262/262 [==============================] - 7s 25ms/step - loss: 0.0627 - accuracy: 0.9807 - val_loss: 5.8042 - val_accuracy: 0.9896
Epoch 5/20
259/262 [============================>.] - ETA: 0s - loss: 0.0536 - accuracy: 0.9834
Epoch 00005: val_loss did not improve from 5.80423
262/262 [==============================] - 7s 25ms/step - loss: 0.0537 - accuracy: 0.9834 - val_loss: 6.2066 - val_accuracy: 0.9899
Epoch 6/20
261/262 [============================>.] - ETA: 0s - loss: 0.0435 - accuracy: 0.9858
Epoch 00006: val_loss improved from 5.80423 to 3.94841, saving model to model/tmp-best-model.h5
262/262 [==============================] - 7s 26ms/step - loss: 0.0435 - accuracy: 0.9858 - val_loss: 3.9484 - val_accuracy: 0.9914
Epoch 7/20
261/262 [============================>.] - ETA: 0s - loss: 0.0449 - accuracy: 0.9859
Epoch 00007: val_loss improved from 3.94841 to 3.75455, saving model to model/tmp-best-model.h5
262/262 [==============================] - 7s 26ms/step - loss: 0.0448 - accuracy: 0.9859 - val_loss: 3.7546 - val_accuracy: 0.9929
Epoch 8/20
260/262 [============================>.] - ETA: 0s - loss: 0.0374 - accuracy: 0.9889
Epoch 00008: val_loss did not improve from 3.75455
262/262 [==============================] - 7s 26ms/step - loss: 0.0374 - accuracy: 0.9889 - val_loss: 5.2989 - val_accuracy: 0.9899
Epoch 9/20
260/262 [============================>.] - ETA: 0s - loss: 0.0346 - accuracy: 0.9895
Epoch 00009: val_loss did not improve from 3.75455
262/262 [==============================] - 7s 26ms/step - loss: 0.0348 - accuracy: 0.9895 - val_loss: 4.9360 - val_accuracy: 0.9926
Epoch 10/20
259/262 [============================>.] - ETA: 0s - loss: 0.0330 - accuracy: 0.9900
Epoch 00010: val_loss did not improve from 3.75455
262/262 [==============================] - 7s 26ms/step - loss: 0.0328 - accuracy: 0.9900 - val_loss: 4.1454 - val_accuracy: 0.9929
Epoch 11/20
260/262 [============================>.] - ETA: 0s - loss: 0.0315 - accuracy: 0.9898
Epoch 00011: val_loss did not improve from 3.75455
262/262 [==============================] - 7s 25ms/step - loss: 0.0316 - accuracy: 0.9898 - val_loss: 5.1087 - val_accuracy: 0.9914
Epoch 12/20
261/262 [============================>.] - ETA: 0s - loss: 0.0303 - accuracy: 0.9908
Epoch 00012: val_loss did not improve from 3.75455
262/262 [==============================] - 7s 25ms/step - loss: 0.0307 - accuracy: 0.9908 - val_loss: 4.8329 - val_accuracy: 0.9907
Epoch 13/20
261/262 [============================>.] - ETA: 0s - loss: 0.0301 - accuracy: 0.9906
Epoch 00013: val_loss did not improve from 3.75455
262/262 [==============================] - 7s 25ms/step - loss: 0.0300 - accuracy: 0.9906 - val_loss: 4.1114 - val_accuracy: 0.9917
Epoch 14/20
260/262 [============================>.] - ETA: 0s - loss: 0.0272 - accuracy: 0.9918
Epoch 00014: val_loss improved from 3.75455 to 3.31105, saving model to model/tmp-best-model.h5
262/262 [==============================] - 7s 26ms/step - loss: 0.0271 - accuracy: 0.9918 - val_loss: 3.3111 - val_accuracy: 0.9938
Epoch 15/20
260/262 [============================>.] - ETA: 0s - loss: 0.0261 - accuracy: 0.9920
Epoch 00015: val_loss did not improve from 3.31105
262/262 [==============================] - 7s 25ms/step - loss: 0.0262 - accuracy: 0.9920 - val_loss: 4.6014 - val_accuracy: 0.9923
Epoch 16/20
261/262 [============================>.] - ETA: 0s - loss: 0.0252 - accuracy: 0.9921
Epoch 00016: val_loss did not improve from 3.31105
262/262 [==============================] - 7s 26ms/step - loss: 0.0253 - accuracy: 0.9921 - val_loss: 4.4429 - val_accuracy: 0.9927
Epoch 17/20
261/262 [============================>.] - ETA: 0s - loss: 0.0252 - accuracy: 0.9925
Epoch 00017: val_loss did not improve from 3.31105
262/262 [==============================] - 7s 25ms/step - loss: 0.0251 - accuracy: 0.9925 - val_loss: 4.2451 - val_accuracy: 0.9926
Epoch 18/20
261/262 [============================>.] - ETA: 0s - loss: 0.0267 - accuracy: 0.9920
Epoch 00018: val_loss did not improve from 3.31105
262/262 [==============================] - 7s 25ms/step - loss: 0.0266 - accuracy: 0.9920 - val_loss: 6.1762 - val_accuracy: 0.9904
Epoch 19/20
261/262 [============================>.] - ETA: 0s - loss: 0.0224 - accuracy: 0.9935
Epoch 00019: val_loss did not improve from 3.31105
262/262 [==============================] - 7s 26ms/step - loss: 0.0225 - accuracy: 0.9934 - val_loss: 4.8530 - val_accuracy: 0.9926
Epoch 20/20
260/262 [============================>.] - ETA: 0s - loss: 0.0214 - accuracy: 0.9934
Epoch 00020: val_loss did not improve from 3.31105
262/262 [==============================] - 7s 26ms/step - loss: 0.0214 - accuracy: 0.9933 - val_loss: 4.1306 - val_accuracy: 0.9944

Visualize

In [24]:
plt.figure(figsize=(12, 6))
plt.plot(np.arange(20)+1, history.history['loss'], label='Loss')
plt.plot(np.arange(20)+1, history.history['val_loss'], label='Validation Loss')
plt.title('losses over training', fontsize=20)

plt.xlabel('epochs', fontsize=15)
plt.ylabel('loss', fontsize=15)

plt.legend()
plt.show()
In [25]:
plt.figure(figsize=(12, 6))
plt.plot(np.arange(20)+1, history.history['accuracy'], label='Accuracy')
plt.plot(np.arange(20)+1, history.history['val_accuracy'], label='Validation Accuracy')
plt.title('Accuracy over training', fontsize=20)

plt.xlabel('epochs', fontsize=15)
plt.ylabel('Accuracy', fontsize=15)

plt.legend()
plt.show()

Load best model

In [26]:
model.load_weights(model_path)

Make Prediction

In [27]:
result = model.predict(x_test)
In [28]:
result = np.argmax(result, axis=1)
In [29]:
result = pd.Series(result, name='label')

Make Submission

In [30]:
submission = pd.read_csv('data/sample_submission.csv')
In [31]:
submission['Label'] = result
In [32]:
submission['Label'].value_counts()
Out[32]:
1    3196
7    2906
2    2813
3    2787
0    2778
8    2759
9    2757
6    2755
4    2751
5    2498
Name: Label, dtype: int64
In [33]:
submission.to_csv('submission/final_submission.csv', index=False)

Now You can submit this file Kaggle!

Here is the link for your submission

Submission

In [ ]:
 


관련 글 더보기

- tensorflow 2.0 ImageDataGenerator / Convolution Neural Network(CNN) 을 활용한 이미지 분류

- tensorflow 2.0 Dataset, batch, window, flat_map을 활용한 loader 만들기

- TensorFlow 2.0 - 단어 토큰화, Embedding, LSTM layer를 활용한 뉴스 데이터 sarcasm 판단

- 딥러닝(LSTM)을 활용하여 삼성전자 주가 예측을 해보았습니다

- [Keras] 콜백함수 (3) - 조기종료: EarlyStopping

데이터 분석, 머신러닝, 딥러닝의 대중화를 꿈 꿉니다.