Image Classification using ResNets

This project aimed to replicate and improve ResNet SOTA results on CIFAR10. I achieved a 6.90% error rate using a 20-layer ResNet with 0.27M parameters, improving upon the original ResNet20’s 8.75% error rate and matching ResNet56’s 6.97% (presented here). Replacing ResNet blocks with ResNeXt further reduced the error rate to 5.32%.

• Model Architecture updates

  1. All the updates mentioned in the Bag of Tricks paper - XResNet
  2. Mish activation instead of ReLU - MXResNet
  3. Squeeze-Excite (SE) blocks wherever possible - SE-MXResNet

• Updates to the Training procedure

  1. MixUp training.
  2. Cosine-decayed learning rate schedule.
  3. Adding Label Smoothing.
  4. Using reflection padding instead of zero padding for the input images.

Results for other models

Update:

I have updated the repository with ResNeXt based models to assess their influence in improving the performance. For this purpose, I have modified the original ResNeXt models presented here, such that they have roughly the same complexities as their ResNet counterparts.

1. Addition of bottlenecks: Since ResNeXt models make use of bottleneck residual blocks, I have increased the width of the ResNet models by 4x, which accounts for the reduction the feature maps undergo while entering a bottleneck block. The ResNeXt models therefore have [64, 64, 128, 256] filters as opposed to [16, 16, 32, 64] in ResNet.
2. Determining cardinality: To do so, I have referred to the process followed in the original paper, which is demonstrated in the following table. I finally settled on using a cardinality of 16, which translates to a bottleneck width of 2. In other words, the bottleneck conv layer is implemented as a grouped convolution consisting of 16 groups, each having 2 feature maps.

Following this process, I developed a model called XResNeXt29_16x2d. This model has 0.32M parameters, comparable to XResNet20. The extra 9 layers are a result of using the bottleneck blocks, which consist of 3 conv layers as opposed to the basic block’s 2. The performance of this model is shown below.
| MODEL | PARAMS | TEST ERR | TEST ACC |
| :—————————: | :——: | :———: | :———-: |
| XResNet29 | 0.31M | 7.62 | 92.38 |
| XResNeXt29_16x2d | 0.32M | 6.70 | 93.30 |
| SE-MXResNeXt29_16x2d | 0.36M | 5.32 | 94.68 |

After adding all the updates, this 29 layer model outperforms the 56 layer SE-MXResNet, while using less than half the number of parameters.

Replicating the results

• To replicate the results obtained above, first, clone this repository to your local machine and install all the necessary packages. Optionally, prior to running these commands, you can create a virtual environment by following the steps listed at this link

  $ git clone https://github.com/iamVarunAnand/image_classification.git
  $ cd image_classification
  $ pip install -r requirements.txt

• All training related configurations are specified in a separate config file, located in utils/config.py. All the available options are listed below:
  ```python

  dataset configs

  USE_MIXUP = False # determines whether to use mixup training
  USE_REFLECTION_PAD = False # determines if reflection pad is to be used for the input images, instead of zero pad

model configs

MODEL_NAME = “xresnet20” # model to be used for training

training configs

EPOCHS = 180 # number of training epochs
START_EPOCH = 0 # epoch to start training at (useful for stop-start training)
BS = 128 # batch size to be used while training
INIT_LR = 1e-1 # starting learning rate. (original ResNet paper recommends setting this to 1e-1)
USE_LBL_SMOOTH = False # determines if label smoothing is used while training
USE_COSINE = False # determines if the learning rate is to be scheduled using the cosine decay policy.

[**NOTE**] For the complete list of supported models, refer to the *dispatcher.py* file in the *utils* folder. This file consists a dictionary, mapping model names to the corresponding *tf.keras.Model* object.
- After setting all the necessary parameters in the configuration file, to train the model, run the following command ***from the base directory of the project***.

$ python train.py
```

Note on callbacks

During training, calls to the following callbacks are made either at the end of every batch or every epoch, dependent on the particular callback.

• LearningRateScheduler: Schedules the learning rate as per the policy specified in the config file.
• ModelCheckpoint: Serializes model weights to disk after every epoch. By default, the model weights are stored in the weights folder.
• TrainingMonitor: This callback is responsible for plotting the loss and accuracies at the end of every epoch and saving the plot to disk. By default, the plots (and optionally a json file containing the model metrics) are saved to the output directory.