最近發現了一份不錯的原始碼，作者使用 PyTorch 實現瞭如今主流的卷積神經網路 CNN 框架，包含了 12 中模型架構。所有程式碼使用的資料集是 CIFAR。
專案地址：
https://github.com/BIGBALLON/CIFAR-ZOO
CNN 經典論文
該專案實現的是主流的 CNN 模型，涉及的論文包括：
1. CNN 模型（12 篇）
(lenet) LeNet-5, convolutional neural networks
論文地址：http://yann.lecun.com/exdb/lenet/
(alexnet) ImageNet Classification with Deep Convolutional Neural Networks
論文地址：https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks
(vgg) Very Deep Convolutional Networks for Large-Scale Image Recognition
論文地址：https://arxiv.org/abs/1409.1556
(resnet) Deep Residual Learning for Image Recognition
論文地址：https://arxiv.org/abs/1512.03385
(preresnet) Identity Mappings in Deep Residual Networks
論文地址：https://arxiv.org/abs/1603.05027
(resnext) Aggregated Residual Transformations for Deep Neural Networks
論文地址：https://arxiv.org/abs/1611.05431
(densenet) Densely Connected Convolutional Networks
論文地址：https://arxiv.org/abs/1608.06993
(senet) Squeeze-and-Excitation Networks
論文地址：https://arxiv.org/abs/1709.01507
(bam) BAM: Bottleneck Attention Module
論文地址：https://arxiv.org/abs/1807.06514
(cbam) CBAM: Convolutional Block Attention Module
論文地址：https://arxiv.org/abs/1807.06521
(genet) Gather-Excite: Exploiting Feature Context in Convolutional Neural Networks
論文地址：https://arxiv.org/abs/1810.12348
(sknet) SKNet: Selective Kernel Networks
論文地址：https://arxiv.org/abs/1903.06586
2. 正則化（3 篇）
(shake-shake) Shake-Shake regularization
論文地址：https://arxiv.org/abs/1705.07485
(cutout) Improved Regularization of Convolutional Neural Networks with Cutout
論文地址：https://arxiv.org/abs/1708.04552
(mixup) mixup: Beyond Empirical Risk Minimization
論文地址：https://arxiv.org/abs/1710.09412
3. 學習速率排程器（2 篇）
(cos_lr) SGDR: Stochastic Gradient Descent with Warm Restarts
論文地址：https://arxiv.org/abs/1608.03983
(htd_lr) Stochastic Gradient Descent with Hyperbolic-Tangent Decay on Classification
論文地址：https://arxiv.org/abs/1806.01593
需求和使用
1. 需求
執行所有程式碼的開發環境需求為：
- Python >= 3.5
-
PyTorch >= 0.4
-
TensorFlow/Tensorboard
其它依賴項 (pyyaml, easydict, tensorboardX)
作者提供了一鍵安裝、配置開發環境的方法：
pip install -r requirements.txt
2. 模型程式碼
作者將所有的模型都存放在 model 資料夾下，我們來看一下 PyTorch 實現的 ResNet 網路結構：
# -*-coding:utf-8-*-
import math
import torch
import torch.nn as nn
import torch.nn.functional as F
__all__ = ['resnet20', 'resnet32', 'resnet44',
'resnet56', 'resnet110', 'resnet1202']
def conv3x3(in_planes, out_planes, stride=1):
"3x3 convolution with padding"
return nn.Conv2d(in_planes, out_planes, kernel_size=3, stride=stride,
padding=1, bias=False)
class BasicBlock(nn.Module):
expansion = 1
def __init__(self, inplanes, planes, stride=1, downsample=None):
super(BasicBlock, self).__init__()
self.conv_1 = conv3x3(inplanes, planes, stride)
self.bn_1 = nn.BatchNorm2d(planes)
self.relu = nn.ReLU(inplace=True)
self.conv_2 = conv3x3(planes, planes)
self.bn_2 = nn.BatchNorm2d(planes)
self.downsample = downsample
self.stride = stride
def forward(self, x):
residual = x
out = self.conv_1(x)
out = self.bn_1(out)
out = self.relu(out)
out = self.conv_2(out)
out = self.bn_2(out)
if self.downsample is not None:
residual = self.downsample(x)
out += residual
out = self.relu(out)
return out
class Bottleneck(nn.Module):
expansion = 4
def __init__(self, inplanes, planes, stride=1, downsample=None):
super(Bottleneck, self).__init__()
self.conv_1 = nn.Conv2d(inplanes, planes, kernel_size=1, bias=False)
self.bn_1 = nn.BatchNorm2d(planes)
self.conv_2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride,
padding=1, bias=False)
self.bn_2 = nn.BatchNorm2d(planes)
self.conv_3 = nn.Conv2d(planes, planes * 4, kernel_size=1, bias=False)
self.bn_3 = nn.BatchNorm2d(planes * 4)
self.relu = nn.ReLU(inplace=True)
self.downsample = downsample
self.stride = stride
def forward(self, x):
residual = x
out = self.conv_1(x)
out = self.bn_1(out)
out = self.relu(out)
out = self.conv_2(out)
out = self.bn_2(out)
out = self.relu(out)
out = self.conv_3(out)
out = self.bn_3(out)
if self.downsample is not None:
residual = self.downsample(x)
out += residual
out = self.relu(out)
return out
class ResNet(nn.Module):
def __init__(self, depth, num_classes, block_name='BasicBlock'):
super(ResNet, self).__init__()
# Model type specifies number of layers for CIFAR-10 model
if block_name == 'BasicBlock':
assert (
depth - 2) % 6 == 0, 'depth should be 6n+2, e.g. 20, 32, 44, 56, 110, 1202'
n = (depth - 2) // 6
block = BasicBlock
elif block_name == 'Bottleneck':
assert (
depth - 2) % 9 == 0, 'depth should be 9n+2, e.g. 20, 29, 47, 56, 110, 1199'
n = (depth - 2) // 9
block = Bottleneck
else:
raise ValueError('block_name shoule be Basicblock or Bottleneck')
self.inplanes = 16
self.conv_1 = nn.Conv2d(3, 16, kernel_size=3, padding=1,
bias=False)
self.bn_1 = nn.BatchNorm2d(16)
self.relu = nn.ReLU(inplace=True)
self.stage_1 = self._make_layer(block, 16, n)
self.stage_2 = self._make_layer(block, 32, n, stride=2)
self.stage_3 = self._make_layer(block, 64, n, stride=2)
self.avgpool = nn.AvgPool2d(8)
self.fc = nn.Linear(64 * block.expansion, num_classes)
for m in self.modules():
if isinstance(m, nn.Conv2d):
# nn.init.xavier_normal(m.weight.data)
nn.init.kaiming_normal_(m.weight.data)
elif isinstance(m, nn.BatchNorm2d):
m.weight.data.fill_(1)
m.bias.data.zero_()
def _make_layer(self, block, planes, blocks, stride=1):
downsample = None
if stride != 1 or self.inplanes != planes * block.expansion:
downsample = nn.Sequential(
nn.Conv2d(self.inplanes, planes * block.expansion,
kernel_size=1, stride=stride, bias=False),
nn.BatchNorm2d(planes * block.expansion),
)
layers = []
layers.append(block(self.inplanes, planes, stride, downsample))
self.inplanes = planes * block.expansion
for i in range(1, blocks):
layers.append(block(self.inplanes, planes))
return nn.Sequential(*layers)
def forward(self, x):
x = self.conv_1(x)
x = self.bn_1(x)
x = self.relu(x) # 32x32
x = self.stage_1(x) # 32x32
x = self.stage_2(x) # 16x16
x = self.stage_3(x) # 8x8
x = self.avgpool(x)
x = x.view(x.size(0), -1)
x = self.fc(x)
return x
def resnet20(num_classes):
return ResNet(depth=20, num_classes=num_classes)
def resnet32(num_classes):
return ResNet(depth=32, num_classes=num_classes)
def resnet44(num_classes):
return ResNet(depth=44, num_classes=num_classes)
def resnet56(num_classes):
return ResNet(depth=56, num_classes=num_classes)
def resnet110(num_classes):
return ResNet(depth=110, num_classes=num_classes)
def resnet1202(num_classes):
return ResNet(depth=1202, num_classes=num_classes)
其它模型也一併能找到。
3. 使用
簡單執行下面的命令就可以執行程式了：
## 1 GPU for lenet
CUDA_VISIBLE_DEVICES=0 python -u train.py --work-path ./experiments/cifar10/lenet
## resume from ckpt
CUDA_VISIBLE_DEVICES=0 python -u train.py --work-path ./experiments/cifar10/lenet --resume
## 2 GPUs for resnet1202
CUDA_VISIBLE_DEVICES=0,1 python -u train.py --work-path ./experiments/cifar10/preresnet1202
## 4 GPUs for densenet190bc
CUDA_VISIBLE_DEVICES=0,1,2,3 python -u train.py --work-path ./experiments/cifar10/densenet190bc
我們使用 yaml 檔案 config.yaml 儲存引數，檢視 ./experimets 中的任何檔案以瞭解更多詳細資訊。您可以透過 tensorboard 中 tensorboard –logdir path-to-event –port your-port 檢視訓練曲線。培訓日誌將透過日誌轉儲，請檢查您工作路徑中的 log.txt。
模型在 CIFAR 資料集上的結果
1. 12 種 CNN 模型：
2. 正則化
預設的資料擴充方法是 RandomCrop+RandomHorizontalLip+Normalize，而 √ 表示採用哪種附加方法。
PS：Shake_Resnet26_2X64d 透過剪下和混合達到 97.71% 的測試精度！很酷，對吧？
3. 不同的學習速率排程器
最後，再附上專案地址：
https://github.com/BIGBALLON/CIFAR-ZOO
