{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"colab_type": "text",
"execution": {},
"id": "view-in-github"
},
"source": [
"![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"/)
![\"Open](\"https://kaggle.com/static/images/open-in-kaggle.svg\"/)
"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"# Data Augmentation in image classification models\n",
"\n",
"**By Neuromatch Academy**\n",
"\n",
"__Content creators:__ [Jama Hussein Mohamud](https://engmubarak48.github.io/jmohamud/index.html), [Alex Hernandez-Garcia](https://alexhernandezgarcia.github.io/)\n",
"\n",
"__Production editors:__ Spiros Chavlis, Saeed Salehi\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"----\n",
"# Objective\n",
"\n",
"Data augmentation refers to synthetically increasing the amount of training data by transforming the existing training examples. Data augmentation has been shown to be a very useful technique, especially in computer vision applications. However, there are multiple ways of performing data augmentation and it is yet to be understood which transformations are more effective and why, and how data augmentation interacts with other techniques. In fact, it is common to see different augmentation schemes and setups in different papers. For example, there are perceptually possible image transformations (related to human visual perception), simple synthetic transformations such as cutout, more artificial transformations such as mixup that even transform the class labels, among many others. \n",
"\n",
"In this notebook, we will show how to train deep neural networks for image classification with data augmentation and analyse the results."
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"---\n",
"# Setup\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Install dependencies\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"execution": {},
"tags": [
"hide-input"
]
},
"outputs": [],
"source": [
"# @title Install dependencies\n",
"!pip install pandas --quiet"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {}
},
"outputs": [],
"source": [
"# imports\n",
"import os\n",
"import csv\n",
"import multiprocessing\n",
"\n",
"import numpy as np\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"\n",
"import torch\n",
"import torch.nn as nn\n",
"import torch.optim as optim\n",
"import torch.nn.functional as F\n",
"import torch.backends.cudnn as cudnn\n",
"from torch.autograd import Variable\n",
"\n",
"import torchvision\n",
"import torchvision.transforms as transforms"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Set random seed\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" Executing `set_seed(seed=seed)` you are setting the seed\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"execution": {},
"tags": [
"hide-input"
]
},
"outputs": [],
"source": [
"# @title Set random seed\n",
"\n",
"# @markdown Executing `set_seed(seed=seed)` you are setting the seed\n",
"\n",
"# for DL its critical to set the random seed so that students can have a\n",
"# baseline to compare their results to expected results.\n",
"# Read more here: https://pytorch.org/docs/stable/notes/randomness.html\n",
"\n",
"# Call `set_seed` function in the exercises to ensure reproducibility.\n",
"import random\n",
"import torch\n",
"\n",
"def set_seed(seed=None, seed_torch=True):\n",
" if seed is None:\n",
" seed = np.random.choice(2 ** 32)\n",
" random.seed(seed)\n",
" np.random.seed(seed)\n",
" if seed_torch:\n",
" torch.manual_seed(seed)\n",
" torch.cuda.manual_seed_all(seed)\n",
" torch.cuda.manual_seed(seed)\n",
" torch.backends.cudnn.benchmark = False\n",
" torch.backends.cudnn.deterministic = True\n",
"\n",
" print(f'Random seed {seed} has been set.')\n",
"\n",
"# In case that `DataLoader` is used\n",
"def seed_worker(worker_id):\n",
" worker_seed = torch.initial_seed() % 2**32\n",
" np.random.seed(worker_seed)\n",
" random.seed(worker_seed)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Set device (GPU or CPU)\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"execution": {},
"tags": [
"hide-input"
]
},
"outputs": [],
"source": [
"# @title Set device (GPU or CPU)\n",
"\n",
"# inform the user if the notebook uses GPU or CPU.\n",
"\n",
"def set_device():\n",
" device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n",
" if device != \"cuda\":\n",
" print(\"WARNING: For this notebook to perform best, \"\n",
" \"if possible, in the menu under `Runtime` -> \"\n",
" \"`Change runtime type.` select `GPU` \")\n",
" else:\n",
" print(\"GPU is enabled in this notebook.\")\n",
"\n",
" return device"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Random seed 2021 has been set.\n",
"GPU is enabled in this notebook.\n"
]
},
{
"data": {
"application/vnd.google.colaboratory.intrinsic+json": {
"type": "string"
},
"text/plain": [
"'cuda'"
]
},
"execution_count": 5,
"metadata": {
"tags": []
},
"output_type": "execute_result"
}
],
"source": [
"set_seed(seed=2021)\n",
"set_device()"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"### Training hyperparameters\n",
"\n",
"**Note:** We have reduced the number of epochs, `end_epochs`. The value was set to 200. Please, change it back and run the code."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {}
},
"outputs": [],
"source": [
"# hyper-parameters\n",
"use_cuda = torch.cuda.is_available()\n",
"alpha = 1 # alpha for mixup augmentation\n",
"best_acc = 0 # best test accuracy\n",
"start_epoch = 0 # start from epoch 0 or last checkpoint epoch\n",
"batch_size = 128\n",
"end_apochs = 15 # Please change this to 200\n",
"base_learning_rate = 0.1\n",
"cutout = True # True/False if you want to use cutout augmentation\n",
"mixup = False # True/False if you want to use mixup augmentation\n",
"n_holes = 1 # number of holes to cut out from image for cutout\n",
"length = 16 # length of the holes for cutout augmentation\n",
"torchvision_transforms = False # True/False if you want use torchvision augmentations"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"---\n",
"# Augmentation"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"## Cutout\n",
"Randomly mask out one or more patches from an image."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" `Cutout` Augmentation class\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"execution": {},
"tags": [
"hide-input"
]
},
"outputs": [],
"source": [
"# @markdown `Cutout` Augmentation class\n",
"\n",
"class Cutout(object):\n",
" \"\"\"\n",
" code from: https://github.com/uoguelph-mlrg/Cutout\n",
"\n",
" Randomly mask out one or more patches from an image.\n",
" Args:\n",
" n_holes (int): Number of patches to cut out of each image.\n",
" length (int): The length (in pixels) of each square patch.\n",
" \"\"\"\n",
" def __init__(self, n_holes, length):\n",
" self.n_holes = n_holes\n",
" self.length = length\n",
"\n",
" def __call__(self, img):\n",
" \"\"\"\n",
" Args:\n",
" img (Tensor): Tensor image of size (C, H, W).\n",
" Returns:\n",
" Tensor: Image with n_holes of dimension length x length cut out of it.\n",
" \"\"\"\n",
" h = img.size(1)\n",
" w = img.size(2)\n",
"\n",
" mask = np.ones((h, w), np.float32)\n",
"\n",
" for n in range(self.n_holes):\n",
" y = np.random.randint(h)\n",
" x = np.random.randint(w)\n",
"\n",
" y1 = np.clip(y - self.length // 2, 0, h)\n",
" y2 = np.clip(y + self.length // 2, 0, h)\n",
" x1 = np.clip(x - self.length // 2, 0, w)\n",
" x2 = np.clip(x + self.length // 2, 0, w)\n",
"\n",
" mask[y1: y2, x1: x2] = 0.\n",
"\n",
" mask = torch.from_numpy(mask)\n",
" mask = mask.expand_as(img)\n",
" img = img * mask\n",
"\n",
" return img"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"## Mixup\n",
"\n",
"Mixup is a data augmentation technique that combines pairs of examples via a convex combination of the images and the labels. Given images $x_i$ and $x_j$ with labels $y_i$ and $y_j$, respectively, and $\\lambda \\in [0, 1]$, mixup creates a new image $\\hat{x}$ with label $\\hat{y}$ the following way:\n",
"\n",
"\\begin{align}\n",
"\\hat{x} &= \\lambda x_i + (1 - \\lambda) x_j \\\\\n",
"\\hat{y} &= \\lambda y_i + (1 - \\lambda) y_j\n",
"\\end{align}\n",
"\n",
"You may check the [original paper](https://arxiv.org/abs/1710.09412) and [code repository](https://github.com/hongyi-zhang/mixup)."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" `mixup_data` Augmentation function\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"execution": {},
"tags": [
"hide-input"
]
},
"outputs": [],
"source": [
"# @markdown `mixup_data` Augmentation function\n",
"\n",
"def mixup_data(x, y, alpha=1.0, use_cuda=True):\n",
" '''Compute the mixup data. Return mixed inputs, pairs of targets, and lambda\n",
" - https://github.com/hongyi-zhang/mixup\n",
" '''\n",
" if alpha > 0.:\n",
" lam = np.random.beta(alpha, alpha)\n",
" else:\n",
" lam = 1.\n",
" batch_size = x.size()[0]\n",
" if use_cuda:\n",
" index = torch.randperm(batch_size).cuda()\n",
" else:\n",
" index = torch.randperm(batch_size)\n",
"\n",
" mixed_x = lam * x + (1 - lam) * x[index, :]\n",
" y_a, y_b = y, y[index]\n",
"\n",
" return mixed_x, y_a, y_b, lam"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"---\n",
"# Data"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"## Datasets\n",
"\n",
"We will start using CIFAR-10 data set from PyTorch, but with small tweaks we can get any other data we are interested in. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" Download and prepare Data\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"execution": {},
"tags": [
"hide-input"
]
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"==> Preparing data...\n",
"Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./CIFAR10/cifar-10-python.tar.gz\n"
]
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "3a3a8eff5d924100831e7e7cd0202361",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
"HBox(children=(FloatProgress(value=0.0, max=170498071.0), HTML(value='')))"
]
},
"metadata": {
"tags": []
},
"output_type": "display_data"
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Extracting ./CIFAR10/cifar-10-python.tar.gz to ./CIFAR10\n",
"Files already downloaded and verified\n"
]
}
],
"source": [
"# @markdown Download and prepare Data\n",
"print('==> Preparing data...')\n",
"def percentageSplit(full_dataset, percent=0.0):\n",
" set1_size = int(percent * len(full_dataset))\n",
" set2_size = len(full_dataset) - set1_size\n",
" final_dataset, _ = torch.utils.data.random_split(full_dataset,\n",
" [set1_size, set2_size])\n",
" return final_dataset\n",
"\n",
"# CIFAR100 normalizing\n",
"# mean = [0.5071, 0.4866, 0.4409]\n",
"# std = [0.2673, 0.2564, 0.2762]\n",
"\n",
"# CIFAR10 normalizing\n",
"mean = (0.4914, 0.4822, 0.4465)\n",
"std = (0.2023, 0.1994, 0.2010)\n",
"\n",
"# torchvision transforms\n",
"transform_train = transforms.Compose([])\n",
"if torchvision_transforms:\n",
" transform_train.transforms.append(transforms.RandomCrop(32, padding=4))\n",
" transform_train.transforms.append(transforms.RandomHorizontalFlip())\n",
"\n",
"transform_train.transforms.append(transforms.ToTensor())\n",
"transform_train.transforms.append(transforms.Normalize(mean, std))\n",
"if cutout:\n",
" transform_train.transforms.append(Cutout(n_holes=n_holes, length=length))\n",
"\n",
"transform_test = transforms.Compose([\n",
" transforms.ToTensor(),\n",
" transforms.Normalize(mean, std),\n",
"])\n",
"\n",
"trainset = torchvision.datasets.CIFAR10(\n",
" root='./CIFAR10', train=True, download=True,\n",
" transform=transform_train)\n",
"\n",
"testset = torchvision.datasets.CIFAR10(\n",
" root='./CIFAR10', train=False, download=True,\n",
" transform=transform_test)"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"#### CIFAR-10\n",
"\n",
"CIFAR-10 is a data set of 50,000 colour (RGB) training images and 10,000 test images, of size 32 x 32 pixels. Each image is labelled as 1 of 10 possible classes: \n",
"```\n",
"'plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck'\n",
"```\n",
"The data set is stored as a custom `torchvision.datasets.cifar.CIFAR` object. You can check some of its properties with the following code:"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Object type: \n",
"Training data shape: (50000, 32, 32, 3)\n",
"Test data shape: (10000, 32, 32, 3)\n",
"Number of classes: 10\n"
]
}
],
"source": [
"print(f\"Object type: {type(trainset)}\")\n",
"print(f\"Training data shape: {trainset.data.shape}\")\n",
"print(f\"Test data shape: {testset.data.shape}\")\n",
"print(f\"Number of classes: {np.unique(trainset.targets).shape[0]}\")"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"size of the new trainset: 50000\n"
]
}
],
"source": [
"# choose percentage from the trainset. set percent = 1.0 to use the whole train data\n",
"percent = 1.0\n",
"trainset = percentageSplit(trainset, percent = percent)\n",
"print(f\"size of the new trainset: {len(trainset)}\")"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"## Data loaders\n",
"\n",
"A dataloader is an optimized data iterator that provides functionality for efficient shuffling, transformation and batching of the data."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"----> number of workers: 4\n"
]
}
],
"source": [
"# Dataloader\n",
"num_workers = multiprocessing.cpu_count()\n",
"\n",
"print(f'----> number of workers: {num_workers}')\n",
"\n",
"trainloader = torch.utils.data.DataLoader(\n",
" trainset, batch_size=batch_size, shuffle=True, num_workers=num_workers)\n",
"testloader = torch.utils.data.DataLoader(\n",
" testset, batch_size=batch_size, shuffle=False, num_workers=num_workers)"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"## Visualization\n",
"\n",
"To visualize some of the augmentations, make sure you set to ```True``` their corresponding flags in the hyperparameters section"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {}
},
"outputs": [],
"source": [
"# get batch of data\n",
"batch_X, batch_Y = next(iter(trainloader))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {}
},
"outputs": [],
"source": [
"def plot_mixed_images(images):\n",
" inv_normalize = transforms.Normalize(\n",
" mean= [-m/s for m, s in zip(mean, std)],\n",
" std= [1/s for s in std]\n",
" )\n",
" inv_PIL = transforms.ToPILImage()\n",
" fig = plt.figure(figsize=(10, 8))\n",
" for i in range(1, len(images) + 1):\n",
" image = images[i-1]\n",
" ax = fig.add_subplot(1, 4, i)\n",
" inv_tensor = inv_normalize(image).cpu()\n",
" ax.imshow(inv_PIL(inv_tensor))\n",
" plt.show()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {}
},
"outputs": [],
"source": [
"# Mixup Visualization\n",
"if mixup:\n",
" alpha = 0.9\n",
" mixed_x, y_a, y_b, lam = mixup_data(batch_X, batch_Y,\n",
" alpha=alpha, use_cuda=use_cuda)\n",
" plot_mixed_images(mixed_x[:4])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {}
},
"outputs": [
{
"data": {
"image/png": "iVBORw0KGgoAAAANSUhEUgAAAlAAAACaCAYAAABmDna+AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO2daZAl2XXXz83Mt9V7r9burq7u6unumZ7d0kijlrWMFluyvAi0GBxgB9gjQlgEAYQI+ICACAITRCCIwHwwYCNshwawJWwQSAaBtXiQLcmWNNJsPUv3TM/03l1d+/L2zLx86FLd+/+/rvfqdb1auuv8IiYmT91cbt48eV92nn+eY6y1oiiKoiiKomycYKc7oCiKoiiKcruhD1CKoiiKoig9og9QiqIoiqIoPaIPUIqiKIqiKD2iD1CKoiiKoig9og9QiqIoiqIoPbKpByhjzE8bY04bY141xnyqX51S9hbqR8pmUR9S+oH6kdIL5lbzQBljQhE5IyIfEJFLIvI9EfkFa+2L620zMlKyhw6NuT9YQ51xy2lg120TETFJCnbSbIGdeucV5XO4cUjPjX1NhWU6msLjze2wbo/H6m3jLtgOFuPG88qVGVlYWO7UMaBXPxoYLtqhg6P+HnB/nXrJ/1wwvDbvy7fJH7ts2876171tX7Rq11vU31eXftiu5+FIU7zH2s7YBGS3OTu1r9vU9odzz78+Y63dv27n8Lg9z0XDIyP20MTEOp3rfDU7jdmdCv9OsB+xk3by2fZt29fotC/YvkPb9anrsri4uGVzkYhIeXDQ7tt/YL3udKbLjb2pGbzLmPbU2qUjmzqLNr/qsHmPt133Rxt7k6V2FuZmpbqyctOjR711CfhREXnVWvuaiIgx5vMi8hERWdfZDh0ak8997h+v2WEzhPbQO+NqNoG2LD1sZeaqYC9cuAz2Squxtrz/4fugLSgWwE7pYYwvVMA/FB1f3HV+gLKWfpTM+heRf8D4gVNsp8vXZZIKcHzbnlDJ+/wH0rZupO4B9a/+0q906NNN6cmPhg6Oysc+8/fW7ICeinw7onPI5dHfbIR2IBmwo8CNr7U4XpkI1217GGu73fBa+puHIe4rTXDjVkzXijDePwiikH0Cj5umdF/laH3/HqzWoCkwOF65KIvtAY1fZKndHTtI8RzTBPv1sWO/cF42Tu9z0cSE/Jff/V33B7p+ke9XAbWFOA7dHqZ9i9dtezBpe1ro/CtgvH5ani82AfcjpuvDfhS3mmAnyfo+m1A/+drzsXkebMWxO27K/XLrfvLvfHLdPqxDz360b/8B+Wef/tV1++PDD4425WuLdtKlHVq6PMCmXfzKb0+7PMC2Pzuvvz7vq+2BicaLfQPGiH6H+TeL+9E2fm3n4f4Qd/gHwn/81/9C1mMzIbzDInLRsy+t/g0wxnzCGPOUMeap+fmVTRxOuUPp6ke+D1UXKtvaOeW2oPe5aGFh2zqn3Db07EfLS0vb1jll97HlInJr7WestSettSdHRkpbfTjlDsT3oYHh4k53R7lNgbloeHinu6Pcpvh+VB4c3OnuKDvIZkJ4l0XkiGdPrv5tXazNSit1myQpTmL52L02M4tT0DZ/6RzYU0+fBvvaqVfA9qNyb/lb74S2wonjuG6C+ikOv4QUuujlubNdFdIpXNjlFS+FPQJLYSS7rtGGpRCeDei1etsG6wejg9gL5dh8x+PehN78yBqxqXPbtiinF27JZdG9WzFeZw5L5DO4fpRx42tCDFdxyIdfz/O+o4hutdBtH9N1rtQaYNfIPzMUpiuGbsw5HN2o16lfMdpN9CE/JLTSoPsiojBmivvi8FRRKERvXN84LF4yPfuNT+9zkYi0xJ3PIkkAfJ1kVC5DU5bGP5Oh0CW1h17ILwhwTCIKD5o2oR5BY5x44axGE6+HZZ1oyjbNAZ4f8r3SbGCIrlHH8G6UwX4NDg25/baFbTr3i8M87eHD9KbLbXbvIqKe/cgYEf/nIeoQom2TaAivi+1RB6luW0juJv0Cu0tkOPB0GTz3dw/hoe1PZ0GXa5CSv/P1xPHDtoB1iyy1oYO3hYY9k9QGkmzQdzbzBup7InKvMea4MSYrIj8vIl/axP6UvYn6kbJZ1IeUfqB+pPTELb+BstbGxpi/LSJ/KCKhiPy2tfaFvvVM2ROoHymbRX1I6QfqR0qvbCaEJ9baL4vIl/vUF2WPon6kbBb1IaUfqB8pvbCpB6heCayRgZaLW9oZ/Cpv/uyZteXrz38H2uZOo+Zp+uo02JY+Lc7t27e2vEKfY2cSXDclDUpIcdmEI538HX9PsAaqQ6KLtuAzxftD1CVAbLprEgz6pD9lV1g/XmwNai0kcl80pdL5k/vNEphASqFLQ5ElnRMMYYh9adKY8LZhgDqn2NeQkI8EpJcieZrEFJ9vtkgvBKkJSAtBmruBHGpscpRCoZR34xHQvso5FN3z5+WsG/B1MPkCfvQR5fGckybpvEL63J/SMZjQy81GWqDWlW3+utJg+ofmFdRcpp6mp1AcgLY6Xct6A3VmrO8KfZvTJdC8FZCdIe0c68wuXLiwtryyjPNci/rZaqGuiX3B1xr52iqRdm1KrY5pZA6Mj4H98BaKq/0xYB1M/xI5bAxr8VN81nf52kjWLfWaHgBSyXRJS9CWMoHn8zZ9raxPl9+SNgkb5OnqLU1HwP30OpaSMIlz3rXlwOsllQP3w7NZP4b9VRRFURRFUXpCH6AURVEURVF6ZFtDeGmtJvXnnSbv/Leeh/brr5xaW65cvobbtjB7uIyNgmmOHwI7mRxfW55uLENbOH0R7EIeP6EOyW7P7uyeO7uVdWgrW9DV7tBGh0rp+TeFLOdt6dTRTOkTddv55befJT6lc26JXypna5/JkziRhfnFNbtcRr/wL1WYob6QzeGSBn22n7bcmMQxhUspizl/c2zIZ9pex4vbX4YyehdyeE6Wwim1BqY5sMH6/pihcGBbBvC26+7aQyp5lDbpFTqFfSlDQtun70HB7TtD/jh1HkNo24F/bw7FeG7Xp6+vLWeP3gVtaYxjVi5iqDMIeU5w48hpDIImXksOnSV0PZcoceO5c+fWlht1TitB4VmeTjiFAhyL7hUOM7MSgT+b9/7QXr2K9t0Wauk8F7XPqTc/bq+lP24FIxji4fscRrRL1KytzJKsH95qyxbeZVuGZ2mYFnrNW9AWOusYD+xos09G8FvbLUDbeQzCtjFbP3XDRtE3UIqiKIqiKD2iD1CKoiiKoig9og9QiqIoiqIoPbKtGqjawryc+sIX1uz5l16H9qoXv05K+FlsOHEP2IXjR8GOD+0HOxl1n9HWSMsxPY16i0wWNSi5XA7sImkcCoXiuutyKYb26uv8zefG48XWsnYAj+2nX+AyDrZFxwlQn8Jp8tt75X9Cy/oeXzO2tcIDExrJlN3x6gY/xfZ1Bqw5iEnjZElLZFJK7eDrYiIuoYH7Yl0HuVzbJ7ahp5GqUj/qNdSyZFhvRfuuiFu/LU1Bk3UUuHFEJWr8T5INl/uhfRfzeF/kItISkp7BL/9RyKLvbm3yi5vja1fSQSqL9IrTQEVUsmaB7qVhzmFR4LQnboyDBI8z+91vgL04OwP2vR/6CNhBxHoTb5l1SKTFCmmFlKvZe2k/ohT7GfLcE7C/o4/6/m67fF4eGNIecj/pRoZUKZwqo4smtd/Uq8ty5pk/2dZjboTD9z0KdpturE2K1ClFApdX6aEjXdIWdNuZWWf5xqZdOtL12OkG113/OPoGSlEURVEUpUf0AUpRFEVRFKVH9AFKURRFURSlR7ZVAyXNWOSKi/HnSH/RPPrg2nLhxH3QFh7B0gCFYcwDlQSoqRDjYviFLObVaZHCJyHtS6OO0dYVKtUQhk4TUSjgvsvlMthZ0khlMnjOkNujLUyNf8hQvwsJlm5YnnP5kfIDWH4ipjxF1+exjE55BDVnYjo8W5N2woivp+olQN47QRjIwJAbw3qT8jN52oyYRVCCuo6A8haFWWyPvXIrAY1fSLqkMuVyiihXUJ1UPibjjsX6KUulgiLymUKG9CkZ15eQnKjRJP0CtRvSrjQ97VycoDal2aTcYQ0qHUIaqTzloKp745lBuZQE2zwVGcHcRplRKj3ilT2JFxehaY70EmHK9yWeXKHg9t2oYskaujxyfXoe7KMt0ulR7jJfc9kt5xznDmorSRW7fSekkWwJXvuG4JwYtwmwvNxXND5xhvwqwnOUOs6pQZP0adbT/LUJY0QRadfWdilrYjtpfmjdmOYBS7ozP78ezzc36SiZrLfqpEXauK7rprb0sO910DdQiqIoiqIoPaIPUIqiKIqiKD2iD1CKoiiKoig9sq3CAxOEEuWdRii+535oH3n0bW7dEdQ4CUp62nQjhQzlRBIXhyXlhoQhxdSJLGlhEtKzJJ62plJHTUNap3g+aaCiIg75SOSOlaN+ca6VocUFsJun/hTs5RdeXlvOPIBjW33gONjzs1WwB4eGwE5IpBJ48ePAUtYeP3eQbC1x0pS5+ctrNkeqCwNOP8HatoE86tMKRdRaNBPORuTGPyKxRdHitSpcQT1Q82Wst2hC3HfmxMTacn0Ux7o1gD4SkY4gs4LiraKvQVik60o5i7IB7jvIkv4v7+xKFtddoRqRJo/arCzV/0u4tqCngapV8L6p1+i+2WJSa6Xp6+dIw2a9+zaemcWNR1AvdXkBczc1zqG+MJNz+95fxOOM3Idaz5EC5tZqUA3GMIvXIJt3/eS6eRHdjXXKN8ZallbdHauakj+TpKaQxQk5yuC19+tlBkJiw2UcrySmWqW5I7i+pTxo3lyU0L1hfb3VHtZDmTb5T2etkfFrnZJmLZvF36FmA69npY5z22DR5UlMU2xrUq3HlPIVhjTn+EduO4d20TA2d9Fypd4PV/vw+P3SPFCKoiiKoih9Qx+gFEVRFEVRekQfoBRFURRFUXpkWzVQqYjUvLpRLy1j7Du65nQjB6mO02j+ANgjIyO4cw5iejH4IuVOSdJOuSZEUtLCRJTzx897wbXvbIv2RXqhZhVjwLFxeoliC3UHQ1exZl96FnU1lXNnwS4nXn21ly5D24VXzoM9fnAC7BGqPVg7jO0NbwzCthwi2+dGSZzI0pzTgiWUG2t50V2PLNVqSws49o0C6oViPi9P11GM8RwHLy2BfenpC926jse+4upAsm4sSzYrs+a72NvF5F99J9gLgnqdprCvOw1OLsLxbMbo+1uOFcyDRNqi8ePH1pZZT1Iewbqb10nnMU8XrFJ19/jyIp5nLcadDx7E+zBboIRZpMWL/HFs4PgvLFD+qrk53BXl+SoNOD3V1CzOzZV53PYTv/gXwB47Mgl2UHQ6MUu60NrFL4G9ePVJsPc99NfBTnMPYL9jf87Fwfb1O+15sfYOMdXqlLaarJTbydMiLVdwXhwdRB1vQDUumy28z1uezqlFGj4/t56ISK2O+cSyKR4rn+XZ0KObJqpdCNbZ7rDpeugbKEVRFEVRlB7RByhFURRFUZQe2eb6CUYk7z7/Xqzi576vPfu0M55/AdqOTOKnrffR579HjmD7vjH3KpzLrXAae/5sk4np00vxwpCWXmfyiAb0er9AIb+MF8bMnsFzzr/4Mti5FXwV3qjivpdHvTCnoU/hX8fwX34GX9HPLeG1yL//MbDDiYNrywmVgNjOMhxRGMnYiDvPBoUt/JBqgT4Jt/RettXE65oJ8dpkvdDk+HV8JT52FsMjl7p1/A4kfO062KUHMOy7nKHQTd2lA1imV/crMYaItpooDGVkcHjNtvQZ/qxXCmnhwjlo2z+A8oEDJ+4Ce4jCgdWmC4mUqHzV3FXcd1LFgKyJMDQmNFf5qQmuXrsGbeVCEew3PvJGsA9TmH70oLvHaxW8r07/2b8Huzr1BNiTd30S7PzQ3WvLrQG875aLh8CergyDfTB3EOzyIIZMm4mbq2KWY3hhnCja3p+33cTULPpRJsNlqnDuSzzZSovS9pRy9BtHv3krlJKkWvVCgCTFKQ3gvcESl4UFlEbkvHQiGZLSBFRuLKJ+tei3N6TfXj+cmJBsx3j7TjuUo9E3UIqiKIqiKD2iD1CKoiiKoig9og9QiqIoiqIoPbKtQeIgDCQ/5HQpdgXjnaWiK7WRUvr+axTfZ7tcQr3LseOudMk9d9+9bpuISLmMJT7a+k2fgPpfzrYoPtrMoJ2r4yehY5ev4L5ecLqn8gtnoGmQPkO241huZbmFpV1WjNMSjOVx3bvpc9DlCmqgFpdwX1n6bHnkoKdboDIm6ZYXcHEkSSyLC9Nr9uDgPmgvFNzn03GC18JQOotSHs8jQ/+cKF11n5zvv4ianQMlSqOxhNdqL/Dc//oG2HeP/jmwawXUjU1PudQaS9OYZqM5t72lXMQIzH4h3R9p3ukmUyrXtHzmFNiDk5hiJbH0KXbeOVa9gn5SzOIYZQZQD5QX1G/GVJiqVnV9Kw5geZUPf+SjYI/tRy0Rf47uV/1h2Us6hXqq1575M7Dr9V8CO2w5DVUc47U9cPTPg10YexceLEMlllq4fcYrtRV2+NycNTJ7ieUqjpkVnL94xs5l3NwYkx6I9UEBvXfhcizwe0B6WZtyCRnqCMmRVypuDk5Iuxxy6SJO1UA7M+QP/v4iQ+t659BJI713PUxRFEVRFOUW0QcoRVEURVGUHun6AGWM+W1jzHVjzCnvb6PGmK8aY15Z/f9Ip30oivqRslnUh5R+oH6k9IuNaKA+KyL/VkT+k/e3T4nI1621nzbGfGrV/gdd92SMiJfLoVpDfdCip1kJAtQk5EijMDoyCjbntXjuuefWlk+dQs3CxATmPzlxzz1g30s5piapTEEx6zQOsaCOppHHeOkwlUvIfPNPwU5f9nI/zaHWomZRC1AlvcRoHWPCBxru2BHl2blWR73ZxSLuW0ZRe5Gp4vblposJh3RtjF+yY31NwmelD35krJGg4dw2WcF8NUno+m1JA1WvU9mHBM+jxD53yuXOKkRYYuMalS3Zi7z2/Otgm++/BHZtEu+NuTmXN6q2jBq7hZkN5YH6rPRrLhIRXxIRRfhvySB0fT84fhjaVqZQx9i4+BrYw4+cBLvuiYtqNdJTzWNpFzOAesy0gD6cpKT78DQmE6RxGiyiJqpRxWNz3hyT9/w/QB1XnD6E/TDvBbtaw2td9DSshrQoNsRcQEEW9VXXpjCr2tVrqJd75A0/srac6VTqY30+K330o91ISjmQWD/E83QAf6AyL6wBYtuS7WmRLJUqqlbR3y1piFm3lHj6qoT6xemZSG4lORK1tqiMjF/qJ5el+99f7iDx7foGylr7xyLCs9tHROSHmdSeEJGPiqJ0QP1I2SzqQ0o/UD9S+sWtaqDGrbVXV5evicj4eisaYz5hjHnKGPPUckP/1a4AG/Ij34dqS9WbraLsXW5pLpqf26kSzMou5Zb8iN+oKHuLTYvI7Y2yzusGbqy1n7HWnrTWniznbul1q7IH6ORHvg8VBgdutoqi9DQXjYyqxEW5Ob340cBAYb3VlD3AreaBmjLGTFhrrxpjJkTketct5EY+hUrNPbEXi5i76fA+l09lZgbzEi1STqR6DTU6XOtndNRppPx6OiIis7OzYJ87dw7sb37rW2BPTKJm6p1veuva8kMPPwpt5THUZoUXsAZd9QoNlXH6oeUA39DZGC/P1AzqmIbrmOvjYMvlxnrlMuoI/iDB8QyGsH7XvfQoPbyMb3rSFWdHJdQw2MDNNR3SstyMnv1oOD8kP/vgz6zZDRqz15YvrC1fq05DW66APlLK4wN98hxeq+yM87mFEvrQ5UX0P9mD8+jiIuWaaeDVHxvDHF1B6DR8i3gpRCTPf9gotzQXGTESerleghDvNb8u3NTMDLSlBjvffBo1lsUJzAs1cNjLQ9fCB7cFg/qfyhTOTckw5rvLFHB+gVpjeTyHxQZpnujmZP1QUPf0JhnUW95/8sNgn7gfNVAN+rd4o+q2rzfxXpm7PgX28jLO7RcuXAD729/G+fi+EyfWlgco95Wfs8dw/r7O3JIf7VoS1AS3zcskGPJz5kVUE7TZwPt8uYpzboP0xwUvp1S1iX5E5VzbcizmQn6n4/plSMdlKJtVQqIozk8V0vr+ebblevQ1ZKwf87jVN1BfEpHHV5cfF5Ev3uJ+lL2N+pGyWdSHlH6gfqT0zEbSGHxORP5URO43xlwyxnxcRD4tIh8wxrwiIj+xaivKuqgfKZtFfUjpB+pHSr/oGsKz1v7COk3v73NflDsY9SNls6gPKf1A/UjpF9taC89aK9aLl1aWsB7b1bqLlw6WMdY/+SDmYlpZXgH78hXMzbLo7TuXRc3CsbtQ/3Ps6FGwr1y5CvaLL74I9pnnnH1g7OvQ9uDb3wD2T45h/qBDb0XNVHXZHWv+y7ivWoLagd+8hPlmlkOM2z4euHw1Dapf9MdV/Go3qGPceoL6OUFjFi46SUAwjNq1ONm+WnjZKCNH9rnzPD17DtqXPf+K81STLEfnVKMaTqdRm2Fjt/25a+gTSUiipwIeay+wRFqIOmkQDo6g3icqOc3NyEH8yCmKUAP15W2IoAS+JiLAvj/92itryy++gjUq30r3yugsahOXv/09sEd+xo1DbhDzPB17FOeDqWefBrv+Gmqghh7GYyeesCmgvDesZalVcM5cTlGfUo48PRbVBksDFK/EpB9cXsC56oqXy+ncuVehbWEedV6PvumNYI+MYB3PpUUc35Z3j3M9v5STAe1RkhivrQ3QF7IR1gX1abVw28VltJdrlHuPtEV5L6cayZDackqlXN9OsF8GV4Y2vtRNyvPUauKxsvR7WR5y/h7TtkK/n+uhpVwURVEURVF6RB+gFEVRFEVRekQfoBRFURRFUXpkWzVQYRhI2avPtK+GOpxKaXBteWkR4/XTc6hPGd+HdZ/uo/p1y17ds4VZzONy8SLm+7l0EfOODFOdvR95CHVN1RUXk5+fxRwmX/+DPwR7lrQEH//xH8d9ebqvqInx3zSHOpsXBWPRl5YwFj0UuTH70DjqCIoLuG4zxnjwOGXUvfsa5pEaKrm+LE5QEsJg+5ISLjYr8uXz312zlxL0k0rWaTFyGdRp5BMc32AWNXjRCsa9z604rcbFRVw3X0Ad2IGhY116fucxv4K6l+l51NkV5yinkZcTxpB2JcqQWGKrMZj7hXPKLDRc3rNXpzGfWGEIcw+dOIF6rvHrOCcs/anz15F3PQZtrQHMAzX+ANaca7bQJ6skXRnwtC25eZozv4/5qYYOoH7q9AzWmCsUnT6rRnPLvnE8x6lpTJP0R3/0NbAve3Nsneb5++9HDep73/sOsFPSuuzfj3m1/PppxlANM9NjJro7lBblZkppnDIB6+XcsiVdUkz17BLSMbVitFe89pTuq5S2tZShitJXSejngaI6eXFbLitsD9pq+KFZ8+ajOlVJKec29mikb6AURVEURVF6RB+gFEVRFEVRemR70xgEoTSL7pX1ykUMrQWDLuQyMTEMbY0GliKxc2hfuoCfmZtBt/3IIQz3RdlBsCtL+Oq7XsUyJmdfxs+Yc0Wvn4cPQdvIML4mzy3g6/9pi68KV865UNkIvb+MsviK8jh94jlLr0evNdx55AyG/x47gGMwduRhsN8+i+83DwsWW43n3RjUDx2DtsoxN578+rffNJKmvL5yzv2BXk1HqTcmTWwrZDDsVrhC4b86hkue8UpKTM3geAxRCYkD9x7r1O07kvllCmuW0eeGxvA+qy+7kF9g0JebDQxPbzfsth/+0EfWln/qJz8Ibb/9n58A+zf+6Btgf/zdWObkyLTznaWnnoG28tveCnYwTOVvKO5Qb1AJq8Cl5ggs+ntrHj//r9BJnn75ZbBnFtyceuaFs9D2S4//NbC/+WffBPv4gRLZD8pGefL//u+O7SeO4Zz6J/+v8/o/ZH5upvtKdyiWPsOnSkXSamJaioGy+12OArw365TWoC2Ex6Ey72fM8DuaLlkmmk38ffT9P5/Fk7CWUiIklObAUjyQunnVSz+SDfGcy1kN4SmKoiiKomwJ+gClKIqiKIrSI/oApSiKoiiK0iPbq4ESI4lxh8yVsKzB5anLsLbP0THURP3IIMbF5ysYO51ecrqD6Qp9Tl1E/cogfSY7PITHkiL2pZY6HcJ5SomQJPhMum8MP1NeKGI5kbTixaKzeJxCDmO4f3kM9RH35CtgDwVubI9TDPfAEJ7jgTc8APahC3ge4RJqLZan3Cfq9bP4+bMcv98tb3ElhVbSkiuzrrxFJovjWyo5O0OfoqYV1OyUz+Nn99PX0E8uTLlPta/NoJ5irIjH3YvUY9JRDON9NbQPU2nkS04nY0ivkG6zBsqIiPHSGFj6t2TilwshfVSGtIivvXoe7H899Qdg/4UPfGBt+Z3z6IPB976DO6dSUJkItUV5LjHh3fNxhOWEEtKZxcLaFTyxF045TdTMFKYpyGRw3VIBU4Qouwsj6Ce5COfC5RXU+RayzleCCH+jJMV9pSlpi+j+SH0dKq1rqZxKasimlAi+BioK6Lj0Y8ParJA0UoY6Wm9593+E+1r2Sp0l6fq6Xn0DpSiKoiiK0iP6AKUoiqIoitIj+gClKIqiKIrSI9uqgTJpKhmvZMjJMuqYgiEXwz+7iPmTglnUq0wOoqbnDVTa5Wrs9EGXKN47RTql6SXUEl1uYM6fUh61HQMl18+JiYPQtkxlCxabmK8qbWE8ddArGxNMoQ6pTDHfx8awZMpbS6gx8UO8Y1RCptWi+PGF18FMajhGCys4Jq97MeLU4L4OeCVTArO1IigrmM4jSVGHU6u71gaVq4nwFCVewm3npjFvTmHY5TG69x1YYmPuhdMb7PGdS1Cka11A3566jjoa8fIUmRT1OeG2/1vOSOTlfmH9ROppjQIqOzM5eRjsI0eOgD0zh6Vc/sMX/8fa8rX3vRPa/lx0AuwD334e7OgRzKdUKaP2LvLGdCDKQ9v+IxNgx5RnLm5RCY4lp8/ikhtVKvVUJP2qJKjtUnYYku2w/iekvEe8PmxLczrrqYS0Romnb0w5wVqbXgr1VTHlnPITtDUpT18mg/1ocikX0oFxt/cVXd66DA0Hlo1RDZSiKIqiKErf0MRGOaEAAB6WSURBVAcoRVEURVGUHtEHKEVRFEVRlB7Z3jxQoZH6sNPLtM5j3PyRyMUdHxxBTVN4AOP7R0m/Ei2j7qCUc0HNCdIGnKJUKplRzK80Snmi5qZRj3XV03YMkL7KZLAeWGkEtQLBCsZ4F+pOW5CrY+6loQT1DsUJ1EC16rivcNHXLWE8eD9poOIXX8F+JNSvGOO+5+5xYzRCj933ePlntlrLEgahlItuTJMW5v9Km+48kjrlAZnHMWguYi28hSra4z/m9CmTP4f1zab+3yjYXGdpL1A6gmNQnkBNY6uGPhX4NbYon1FMmqitxhgjIRQI41w36+seTtx7L9icN85gOiapnnZ1Oj/3+/8T2mbf/T6wf/kd6GfN7+N9Wrj3ONiJl1enQgmryqTPFNJF3nPsbrC/EXg1/Ui7wtqUAaoF2VxWDdRuwtJ8Xq+i3SB7seL9doQ4h6cZzAsVkyboOumTq15uwwnSJpfoiSMf428e7+vUVedXYRZvrAdPoPaQ80IVMqSJSvGcC979ny+gP8eejivoIOvVN1CKoiiKoig9og9QiqIoiqIoPaIPUIqiKIqiKD2yrRqo7PCgHPvwT63ZP/jmM9Be+taza8tHZjFemY4Pgi2LmKspoDwlA7HTPeWzGN9sLWG+n9PnXsN9lfFYR8bHwZ708qvMXsMaaVeontoc5X1KxzBue23Jabf2U1y6lGK9qYGjR8GulbC9+b0XXFsDx2OYNA2tDMae0wOoX6nW8dl6yatjVqiilijybNNBO9I/nF4mm0OdWC7nxiRrMHY/dv0q2NU5jLdXDI5J+bDbd2UAr01xEnVzcqFLl+9AJh/GHEYtqnMVr6A+rTjg9IEh1WmLuP7WFmOtlYTrynmEgfP/RgPPY2IctUUf/dmfBfvrX/8K2BfPuvxu+Qjv2a9940mwr86ij/7S+z4E9sPDmP+u4OnKWqRNaZDNOXne/a73gP38qVNry19/8mvQViN9ZiGPWs+mSqB2FUdG8P565vQlsC9evAZ2uOh+hwo59NFgHOe6qIia4svXpsBeqbr7aqKEfrLP4m9v2kQts6Hf5sqcs4MyaoBXKqhZLWRw/smTlitr0P/nF9y+R6mOpL+nTr9o+gZKURRFURSlR/QBSlEURVEUpUe2NYS3UqnLn3zXK4GRxdd7lR9/+9oyvhTsLwWyf3IT+6ouYhqDN78RyzwUaxjSS0r4qrDpfSMZUukRS5+Bz57B17BNej06uOBCDZWIwillPO4y9St/CMOUNUq3sOilhWjN4raXXnGfWjcb+Kq/3wRiJO+VIWg1MZxYb7rQ5cw0vqYOXsTxCysYd4iGKG1E2YV+MysYEq0toy1Skr3GI299I9h1Cokly/h63nqpMQYpTB6EW1sCqFeCwE2NmSy9xKdP+t/7rsfAvnTuHNhf/YoL09ViHJPRYZQX/OD558A+dxFDen/jY38N7AMH3GfiC2fOQFtCY5y77xjY1Wm8jx+997615ddnzkNbIUufsgfbG3JVeiNOMOw8R2kmZhcxVJbx0r80Kcwc0NxnqjjHLy+gnQTut+biNJYym2+gHcc4ZyxTuh2bce94mnW8dy5dwPm82MJ+XMhiGHP/BP7GDXqhSkvpcKzv3x1iePoGSlEURVEUpUf0AUpRFEVRFKVHuj5AGWOOGGOeNMa8aIx5wRjzydW/jxpjvmqMeWX1/yPd9qXsXdSPlM2iPqT0A/UjpV9sRAMVi8jft9b+wBhTFpHvG2O+KiIfE5GvW2s/bYz5lIh8SkT+wdZ1dXdy6gyWWshGOKQnx7HchZ3EgGrN+0R6hT7trsUYD84toGbBLGF7JXb7Sg3GsZfqpFlokhYrxhhwPkadh1jv09UEzyHfdGUAArtuTZO++JExInnvsb84gJ/UGi92beYwdr944QrYmSaecyPEvhe9kjF5SilRoX2L2XsaqCzJlsqDQ2BHZSrP4n1GHGVxPO3Gsl/0dS7yM25Y6oDxyp4EVMsh5NoOCfrN+38Cy7NUPW3G888+C23PPIeap8ESloWprKB25d/8+r8De+KQS6nylohSCzx7Gmzzow+D/fA9D4Jd9q7PiX13QdsjDz8K9qUqpgCZufSS3Eb0zY+aicj5ee/6G/SNXOQmKxNQiRTyuSJ9hn/fXfjb8cLLLt3O8y+fgzZL+h9DOqarizhftUKcNxsF5zvVDP425Ci9Dt8rNSrDlFp3nueuoZ9EKfWDdJNt95Z1dot0SksreN81aqjrqtG1mKP0O0VPXxWEeA6HvPvKbKaUi7X2qrX2B6vLyyLykogcFpGPiMgTq6s9ISIf7bYvZe+ifqRsFvUhpR+oHyn9oicNlDHmmIi8WUS+IyLj1toffiJyTUTG19nmE8aYp4wxT1Wr/OWSshfp1Y98H6rRV4/K3mSzc9EcJVFV9iab9aN6TX/T9jIbfoAyxpRE5L+LyN+1FtOJ2hvv9G76Et5a+xlr7Ulr7cmBAU4goOw1bsWPfB8qDA1ws7LH6MdcNDo6erNVlD1EP/woX9DftL3MhvJAGWMycsPRfsda+4XVP08ZYyastVeNMRMicn2rOrmbuX/iENiXqRTD9AzmIqotTYA9kXGx6DnKhfNMgnHtkRrGfAst/NdPbsHFqudz+Gy8PIRx7EH6h9PCeeynBPigkvNKNyQGY8lTVy+vLbcoR45PP/zIWCtB6sbBJng8I67t8DDqSQ684SGwT33jKbArFYyxr3i6gYGjqCFrLOO6QpWG9gKzly+DPWzvBXuggDoLX2dnqaxCsr52DujnXJSm6x+zo+6B9BKsCTl0EO/xX/74x9eWz57FslH/9Ff+KdhnXsZcTgN5vA9jypPz+uuvry0XJnEuOjuDwzD1/PfA/rn3/ATYH/zgj60tf/QjfxHagiJq/A4azMnzgtxe9MuPgiCQsvdiIOGSWd58GBoqLRKhPUz/ONw/gg/5xaI3R1PJrHoTtUTXpzCTYsPisUJy8FbLbW+aXOIIfa5F92qziXNwNnLzZkq63oTLC9H8zf7ta2rTAHVd1QTPaYXKl6WkTeT8VdNeGZnSCP5WhMOufE2rQ3myjXyFZ0Tkt0TkJWvtr3pNXxKRx1eXHxeRL3bbl7J3UT9SNov6kNIP1I+UfrGRN1CPicgvisjzxpgfVv/9RyLyaRH5PWPMx0XkvIj8pa3ponKHoH6kbBb1IaUfqB8pfaHrA5S19puCxYl93t/f7ih3KupHymZRH1L6gfqR0i+2tRbenciBBGO2Q15tKhGREYtamWMrGIfNNFx89ekatn2rgXkt9udQsPiuHApvDouLERcbGFtOZ3Ffl5sYH85OHgC7kMeYcOLlpKpnUf/Q8rQutpN4pA+kVqThjZklzZWtu/FuLeLYD1qM7ZfyGMFuLlB8fnZlbTlH8faE6hTuRQ1U2kDfN3Uck2prHmzr5XSp0rXYaCKo3UBA+XyEcr8J5W9LYnevHb/rKLR97K/8Itj/7td/HexLFzF32UAR54Cx0tja8rVaBdqWZigHzyDqmL7y/W+DfW/Fbf/Az6MGqjmJ80Gtfvtcr60ktanUmm7eTkkvE3saHs6RZ6j2qZnD+f+5Ct4/i94XyCsrK9DWoDpxTfo60KakebI0b3p5/ww9W1ZJK9iiS28F2/3TSi3ui1WHrFOyCc8LXjtpxizlIwxIX2U4fxvl/cvmXL6roIB1UBdqbt1kMxooRVEURVEUBdEHKEVRFEVRlB7RByhFURRFUZQeUQ3UJpk5jXlbikMYSx0bxvpgQxXMz3H92vm15SzFpUuku2lmUMPwQh7z7LwoTsNwXxbrGR2qU928ZAH7XSBXoDxSYl38famG8fZa3mmvUq5l1GdCE8hg5MY4ymCekWbq+hmMYF+mzqMmZIni77MV1JBAfhPS++QDPO6ZCubRuuuxH8Fjv3wW7MWXzq0tH/yxN0ObHcXrHNdJN0d1rnybcxu1Aozf52K0q9/CDD7BYVc/dejRB6Bt4StPg50lXYWtocagyv32fLJOeoTI7Oy/5TiXk292k/V11URZ5zsxXZ/3vQ/r5jG/8Zn/CPali5c23I8s5ZCyNMZXKjgHvOTV5btrBe+FzEd+CuylSdRy7VWMMRJk3fUOyI98OyUNVGUZ6xxOXUe9W9rAe+TV1y6sLS8v4fVJSDsUk81199p82m8mH2VNVEr6RUM53Zri5saIts1mMZ8e/14YeqcTeI8ofJsZmtsCw7U30UyovmvG0+76eQ5FBGrwdULfQCmKoiiKovSIPkApiqIoiqL0iIbwNkm+iK/Jl6/NgL24jK8Nk30jYF/Nude0QQ5fQb4pvw/sMxQe/NriNNjRkNv+1QjDbB/OlcE+QGnv5194GWwZnwQzO+CetVODr3ivVlzYrNWhPEY/CEwguax79Rq36HWzl+ohLGCoKzyKYYf5770Edn4Yr82kt37W4rXhV/Wjo2NgD+bQL8ITd+H2p13Jn1ILb8M0iyG8ZoPCS/ylr/e6nkNjCYVi7XVMZxG1cGfNgjvPxWkMedoMvn6vXcZw9PRFtPMDeB7ihfhi6mcmxH3vNBzS6wUOpdnIjWkQ08p0nJ/5aQyV5XIYpv+1X/s1sGdm3XxjIww7DO/HUiAl8smp61ip5JnEfRr/2IVz2M/f/F0wfw+7JQ/99LtlL2ItRbzoeqbeh/sBxZSiEP3kQA7nq5UKp6WY9fbbOcSU4XAXx+w41OiFdzlkx6GxIEB5COO7f0DvaDgyxqHHbAF91IRubuTwd4bmowyNH6fUadCcE3rbc1vgpYXoVPJJ30ApiqIoiqL0iD5AKYqiKIqi9Ig+QCmKoiiKovSIaqA2yb4H7wd7cQY1UM0mxpovWBzyc1mnazL7UKeUJy3M9UH81LJG9UP2HXLlFmYi1F6dvo7ii/dSWZikjp/UPnP+AtiXBlxs+v53PwZtj7zj7WvLA//5d2QrSWwqFa90QtIgQZD3CW1i8JyzZdSQJUUavyIKOwaG3XhWFrF0Qr1Fn+GXMHbfJG0Ra8Oy3if9tZlFaAv2YeqLVgX1bIUi+kXdKwFE8jTJkJ5hYAn7lQjqBuott359HnVyzQb2I2zi+M5NoWaqVKJr46V+KAzgWK/QZ/M7TWcJFH8SzpoREnqE9Hm1R9r2+TlewPe8B7VF2QzOH5994om15VdfxZQqGUq1MUkawOwwaqRePvXs2vKTFlMcvJt0XZPks3uVIBDJ+7eQxXHKhK4xCvHajdCcMTCEc/JsHjU+x5ePrS0vVvHejGm+yUakDyK9UCtdP81BTOWxMpQuJ8rinBGQf0deGo+Ij8saTU4tQDqmhjdvsu6Lzymk+45v4RLdD/722Rzuy1+1UxoTfQOlKIqiKIrSI/oApSiKoiiK0iP6AKUoiqIoitIjqoHaJOX7joMdHDsIdo20Mz+YRjuZfMvacrGEsealKpXCIElJSqVf5r2cPxkqs/HcMD4rDycYa15YwHj8DwTjxcP3Pri2/MDP/HloO3KfK1uS4ZT4fSZNEllZnF+zsynGrrNZd/w0Rs1OhWL7UQE1UQWDfa94pUjqTdQcZMpUJoOu3eJKFeyEyjJkyk7HdPUa5k/KDeO1GaDSLZZ0X6Gnuxgso46uMjcPdnINtSs2xSlgMHFjskT5pwxpCEpZHK8q5XKq0hiUhj3NGeXASaOtLQHUFRY9Gb+pc04o1kBx2QxfExVQmRf0SJE0Zk0f8q53of4wX3DX4POf/xy0zcyhJq0wgD47kUf/v3T23Nry8w30m/sTnIvua+I5YyGjPYQVSb25ljVs1YYbt0YT56M2kU6CmsI0wvtr/J571pZLDbweMflNSJq1hPRDcVvpImfbhPeF972hfRsux+LljQpDziFF932M5xyTJirn9Suk+SUTsQYKzLb8TW3aRL/fXErKL7vT4fbXN1CKoiiKoig9og9QiqIoiqIoPaIPUIqiKIqiKD2iGqhNMjSGNdCiOmplCgOoSQmwvJ1ggBWDrSWKH6dTqJVpXsCY+sGxcddWQf3J1SbmefpyBTUOE/e/Aex3v/PHwC54+WeWSLmR1L2YeXrrNcQ2gjFGsp6OJB9QThIvzt3WlTyumzuKerUs5VJpePsqDmGdvOIbUE+SkAaqSXqHKIP1o3LHD7vlabyu2RT/XdNqYr9qBvUPxUGnLfI1MSIijRg1T+kQ5pgqDaF/Fva53EBBHfV64f4JsKPRA2BXxzGvVj7E8TYDTrPQJB8yBsdzq7FiUfdBmhCzrnGTfXWrmwfaC8plQzqONj1VlzxRb3rkkbXle+5BPeYzzz4P9u9/4X+CHdG9M+rlF8u18NrXKMFYprC1NS9vF+IklbkFN7faNsGMs5utmFpw3Yjme7/mpwhqIVPSRbKGL8MaKPKbOmmPQPNDsG5J2Cb8fE2sO2rTJZEmkHNK+ecVkE6pbV3WOFGuq4AlUKDt4lqB6zfBPtZvUhRFURRFUW6GPkApiqIoiqL0iD5AKYqiKIqi9IhqoDbJ4YljYC8vL4Fdq6IWiWsBxV5cPKa4NGsrioOow8kHqBupzrpjhRmMD9dqlH8jxHpqJ71cTiIiiWBfplZcjb+7wiPQFs5f9jbkzDb9JTCB5AvuvFPLAWpPAyVc3wjdffxHHwJ7jHQAQcbtK0f/1ogH0GY9Q46OVaf8JjLscvAcKKN2JZ9FvdRyBevEcS2qMOPZdNzB8XGw04Oo+0pIi9T09A2DQ6jvy5fQJ3IHhsFebKD+b5jy2NicG8/5Kmrwgm5Coy2GtUfI1un6OE9OQroN7lfI6Wq8ro2OYm27k295FOxv/PE3wT796mtgxzk3Nx0yeO+UativarC1WsfbBWNEwtDPA8X5xFxbSImKjHTOkWTadDlufgo5b5Fw7rHOedba0q75jsR6QNY8cc051gDaDQqIRCRt0yV1cPA2l2PdVuf3QS3SgYVgYkcST3toO+h69Q2UoiiKoihKj+gDlKIoiqIoSo/oA5SiKIqiKEqPqAZqk+zfjzqRchnrS9WqqAup1VHPUvNqGtXrmN+nQXqpkTxermNHj4H9/Jln15bDCOPr9QXMS/S2MdTCjF25Dna1fhXs/Q+7vEUTS6jzmnn66bXlmDRfW4HxAuesnfHD1Ry6tqQxy+VRaxTS+rFX/y6gnRWpLlOT9FMx7SuiunG+mKVIPhMZvHYHD6HmzJBuptp0fhPm8JxaReoXaQ7iFDVQgZcsJYrw31e5Bu4rIU1HlgYwk8E8Nql14z8QYVu0I2mF7E2WbtBRE8UaEMPNvO36+WZa5JNpm96C647RsfxtKWdUmeoifvTDHwL7M7/5m2CfO7ewtlyr4dyTp5+KVpHyEO1RUmul6deha6upaG+2+MOtweJ6djHNKf727GPscTHlW0o4zxP1E+o3dqsh1y3vmV3XaNNmRQHP39Qvry9JWxufNU8idJ9SX/zcWDyekCNO1kffQCmKoiiKovRI1wcoY0zeGPNdY8yzxpgXjDG/svr348aY7xhjXjXG/FdjjP6TRFkX9SNls6gPKf1A/UjpFxsJ4TVE5H3W2hVjTEZEvmmM+T8i8vdE5N9Yaz9vjPkNEfm4iPz6FvZ1V1IsYDqAXIghlEIOUw00Yko9UHchLw7hNRtk1zHc8tY3vQXbWy7kdPrMGWgbKWMKhHsnJsE+MDWH+8rj69BC7M5z5jvfhrZWzrXFtXVDeP3xIytivKhHRKUwEu+Vb9qk8AiniYgxnCr0CXIh526PhEqzSIiv25tU6kKoHEuG/CD21m/Rq2eTwdsyy2E5CtVkvbQGtQb3kz6VJps/pW556RbSDL7WLgzROVB4mvu92MTxrXtj2PYVfH1D6S/6NxfZ9nDBRumc8qA9wpF615c/+ebSFjeJJW782BQjsrTvR96IqUr+5l//ZbC//H//z9ry4DSG8IMLOD+cn74E9jP/4QmwV7x0GnXy34DmzMNHj4L9nvf/ONhve89jYI+OeOWGeDw8+wtf/IqsQ9/8yKZWGv483eHac1/bPtlvC8Txzqy32Nl3U9qWQ3ptWIgPdty27a1LW4jPl1jwunzcDv0QnM9527bx7JYKJaDSL37neDy9dbkEzHq7uCn2Bj8sjJRZ/c+KyPtE5L+t/v0JEflot30pexf1I2WzqA8p/UD9SOkXG9JAGWNCY8wzInJdRL4qImdFZMHaNVXoJRE5vM62nzDGPGWMeapK/2JV9ha36ke+D9WWK9ys7CH6NRfNzc3dbBVlj9AvP+IogbK32NADlLU2sda+SUQmReRHReSBjR7AWvsZa+1Ja+3JgYFC9w2UO5Zb9SPfhwr0xZqyt+jXXMRZu5W9Rb/8KJvLd99AuWPpKY2BtXbBGPOkiLxDRIaNMdHqE/ukiFzuvPWdyb5BHMJmA/UpDfwaWOotDKgWs277Zh41i40GxmUbdYzhjgzg+oPvd1qBNz98D7SFKW67v4lvA5tLmMZgYAQ/gc6X3bN2LaByNXV3kimVorgZm/Yj6/rSquIAZ0M3JtkAJ7ckwH8txilqohqkw2nV3LVqVFDbFWZRe1U+tA/sUnEQ7CiD/3ioxW78Z5axrAlV3ZHFKzNgh6SdiAruPA2lr8ixRowC+oZ0AaH/GT19+tygz4Z5TBISNDQT1GP5+qxSAccnCbv7jc+mfciYNj0YNq8vfGj/hLzbv0O9cWSpBZfvYF1HW8mO9fuVtH1t3lkTcvIklnp5+E2utNHiIr6h+/5n/xvYz3z+82BfzKGfGU8LWhzEkj8n7r0P7A/81AfAfvPb3gp2obT+P5o6amw2UB2oH79p/mFaNKf4jQGlJ2nXv6FzhHRv+peTt+UUFqxLatOKEf7+2jJp0KZtaqo2fZW7Kl33lXL5Mmr3Ug0EVMco5dQMREL9ais5440JtwXe2HeSm23kK7z9xpjh1eWCiHxARF4SkSdF5OdWV3tcRL7YbV/K3kX9SNks6kNKP1A/UvrFRt5ATYjIE8aYUG48cP2etfZ/GWNeFJHPG2P+uYg8LSK/tYX9VG5/1I+UzaI+pPQD9SOlL3R9gLLWPicib77J31+TG7FjRemK+pGyWdSHlH6gfqT0C9MWF9zKgxkzLSLnRWSfiMx0WX0n0H5tnPX6dNRau3+rDqo+dMvcbv1SP9J+9cLN+rWlPiSy5keVmxx7N3A7XavdQM9z0bY+QK0d1JinrLUnt/3AXdB+bZyd7tNOH389tF+9sdP92unjr4f2qzd2sl86Jr1xJ/VLa+EpiqIoiqL0iD5AKYqiKIqi9MhOPUB9ZoeO2w3t18bZ6T7t9PHXQ/vVGzvdr50+/npov3pjJ/ulY9Ibd0y/dkQDpSiKoiiKcjujITxFURRFUZQe0QcoRVEURVGUHtnWByhjzE8bY04bY141xnxqO49N/fhtY8x1Y8wp72+jxpivGmNeWf3/yA7064gx5kljzIvGmBeMMZ/cDX0zxuSNMd81xjy72q9fWf37cWPMd1av5381xmS77atP/VE/6tyvXedH6kMd+7Lr/Gg3+tDq8dWPbt6PXedDq324s/3IWrst/4lIKCJnReRuEcmKyLMi8tB2HZ/68h4ReVRETnl/+1ci8qnV5U+JyL/cgX5NiMijq8tlETkjIg/tdN/kRgnI0upyRkS+IyJvF5HfE5GfX/37b4jI31Q/Uj9SH7r9/Wg3+pD60e3lQ3vBj7azw+8QkT/07H8oIv9wJ5xt9fjHyNlOi8iEd9FP71TfvD59UW4Uutw1fRORARH5gYi8TW5kbY1udn3Vj3b+Wu1WP1Ifuv38aLf5kPrR7edDd6IfbWcI77CIXPTsS6t/2y2MW2uvri5fE5HxneyMMeaY3KjX9B3ZBX0zxoTGmGdE5LqIfFVu/MtrwVobr66yXddT/agHdpMfqQ/1xK7xo93kQ6v9UT/aGDt+rXzuRD9SEflNsDceP3csv4MxpiQi/11E/q61dslv26m+WWsTa+2bRGRSbhTcfGC7+3C7oX6EqA/dGjvpR7vNh1aPq37UIzoXtdMPP9rOB6jLInLEsydX/7ZbmDLGTIiIrP7/+k50whiTkRuO9jvW2i/spr6JiFhrF0TkSbnxenPYGBOtNm3X9VQ/2gC72Y/UhzbEjl+r3exDIupHG2BXXKs72Y+28wHqeyJy76rKPSsiPy8iX9rG43fjSyLy+Ory43IjVrutGGOMiPyWiLxkrf3V3dI3Y8x+Y8zw6nJBbsSwX5IbTvdz29wv9aMu7EY/Uh/qmZ2+53edD632S/1o4+hctH6/+uNH2yzW+qDcUOGfFZF/vIOisc+JyFURacmNOOfHRWRMRL4uIq+IyNdEZHQH+vUuufEq8zkReWb1vw/udN9E5I0i8vRqv06JyD9Z/fvdIvJdEXlVRH5fRHLqR+pH6kO3vx/tRh9SP7q9fGgv+JGWclEURVEURekRFZEriqIoiqL0iD5AKYqiKIqi9Ig+QCmKoiiKovSIPkApiqIoiqL0iD5AKYqiKIqi9Ig+QCmKoiiKovSIPkApiqIoiqL0yP8HWFQuobOPZl4AAAAASUVORK5CYII=\n",
"text/plain": [
""
]
},
"metadata": {
"needs_background": "light",
"tags": []
},
"output_type": "display_data"
}
],
"source": [
"# Mixup Visualization\n",
"if cutout:\n",
" plot_mixed_images(batch_X[:4])"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {}
},
"outputs": [],
"source": [
"# Torchvision Visualization\n",
"if torchvision_transforms:\n",
" plot_mixed_images(batch_X[:4])"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"---\n",
"# Model"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"## Architecture: ResNet\n",
"\n",
"ResNet is a family of network architectures whose main property is that the network is organised as a stack of _residual blocks_. Residual blocks consist of a stack of layers whose output is added the input, making a _shortcut connection_.\n",
"\n",
"See the [original paper](https://arxiv.org/abs/1512.03385) for more details.\n",
"\n",
"ResNet is just a popular choice out of many others, but data augmentation works well in general. We just picked ResNet for illustration purposes."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" ResNet model in PyTorch\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"execution": {},
"tags": [
"hide-input"
]
},
"outputs": [],
"source": [
"# @markdown ResNet model in PyTorch\n",
"\n",
"class BasicBlock(nn.Module):\n",
" \"\"\"ResNet in PyTorch.\n",
" Reference:\n",
" [1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun\n",
" Deep Residual Learning for Image Recognition.\n",
" arXiv:1512.03385\n",
" \"\"\"\n",
" expansion = 1\n",
"\n",
" def __init__(self, in_planes, planes, stride=1):\n",
" super(BasicBlock, self).__init__()\n",
" self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)\n",
" self.bn1 = nn.BatchNorm2d(planes)\n",
" self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=1, padding=1, bias=False)\n",
" self.bn2 = nn.BatchNorm2d(planes)\n",
"\n",
" self.shortcut = nn.Sequential()\n",
" if stride != 1 or in_planes != self.expansion*planes:\n",
" self.shortcut = nn.Sequential(\n",
" nn.Conv2d(in_planes, self.expansion*planes, kernel_size=1, stride=stride, bias=False),\n",
" nn.BatchNorm2d(self.expansion*planes)\n",
" )\n",
"\n",
" def forward(self, x):\n",
" out = F.relu(self.bn1(self.conv1(x)))\n",
" out = self.bn2(self.conv2(out))\n",
" out += self.shortcut(x)\n",
" out = F.relu(out)\n",
" return out\n",
"\n",
"class Bottleneck(nn.Module):\n",
" expansion = 4\n",
"\n",
" def __init__(self, in_planes, planes, stride=1):\n",
" super(Bottleneck, self).__init__()\n",
" self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=1, bias=False)\n",
" self.bn1 = nn.BatchNorm2d(planes)\n",
" self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)\n",
" self.bn2 = nn.BatchNorm2d(planes)\n",
" self.conv3 = nn.Conv2d(planes, self.expansion*planes, kernel_size=1, bias=False)\n",
" self.bn3 = nn.BatchNorm2d(self.expansion*planes)\n",
"\n",
" self.shortcut = nn.Sequential()\n",
" if stride != 1 or in_planes != self.expansion*planes:\n",
" self.shortcut = nn.Sequential(\n",
" nn.Conv2d(in_planes, self.expansion*planes, kernel_size=1, stride=stride, bias=False),\n",
" nn.BatchNorm2d(self.expansion*planes)\n",
" )\n",
"\n",
" def forward(self, x):\n",
" out = F.relu(self.bn1(self.conv1(x)))\n",
" out = F.relu(self.bn2(self.conv2(out)))\n",
" out = self.bn3(self.conv3(out))\n",
" out += self.shortcut(x)\n",
" out = F.relu(out)\n",
" return out\n",
"\n",
"\n",
"class ResNet(nn.Module):\n",
" def __init__(self, block, num_blocks, num_classes=10):\n",
" super(ResNet, self).__init__()\n",
" self.in_planes = 64\n",
"\n",
" self.conv1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1, bias=False)\n",
" self.bn1 = nn.BatchNorm2d(64)\n",
" self.layer1 = self._make_layer(block, 64, num_blocks[0], stride=1)\n",
" self.layer2 = self._make_layer(block, 128, num_blocks[1], stride=2)\n",
" self.layer3 = self._make_layer(block, 256, num_blocks[2], stride=2)\n",
" self.layer4 = self._make_layer(block, 512, num_blocks[3], stride=2)\n",
" self.linear = nn.Linear(512*block.expansion, num_classes)\n",
"\n",
" def _make_layer(self, block, planes, num_blocks, stride):\n",
" strides = [stride] + [1]*(num_blocks-1)\n",
" layers = []\n",
" for stride in strides:\n",
" layers.append(block(self.in_planes, planes, stride))\n",
" self.in_planes = planes * block.expansion\n",
" return nn.Sequential(*layers)\n",
"\n",
" def forward(self, x):\n",
" out = F.relu(self.bn1(self.conv1(x)))\n",
" out = self.layer1(out)\n",
" out = self.layer2(out)\n",
" out = self.layer3(out)\n",
" out = self.layer4(out)\n",
" out = F.avg_pool2d(out, 4)\n",
" out = out.view(out.size(0), -1)\n",
" out = self.linear(out)\n",
" return out\n",
"\n",
"\n",
"def ResNet18():\n",
" return ResNet(BasicBlock, [2, 2, 2, 2])\n",
"\n",
"\n",
"def ResNet34():\n",
" return ResNet(BasicBlock, [3, 4, 6, 3])\n",
"\n",
"\n",
"def ResNet50():\n",
" return ResNet(Bottleneck, [3, 4, 6, 3])"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"## Model setup and test"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"-----> verify if model is run on random data\n",
"model loaded\n",
"Using 1 GPUs.\n",
"Using CUDA..\n"
]
}
],
"source": [
"# load the Model\n",
"net = ResNet18()\n",
"print('-----> verify if model is run on random data')\n",
"y = net(Variable(torch.randn(1,3,32,32)))\n",
"print('model loaded')\n",
"\n",
"result_folder = './results/'\n",
"if not os.path.exists(result_folder):\n",
" os.makedirs(result_folder)\n",
"\n",
"logname = result_folder + net.__class__.__name__ + '_' + '.csv'\n",
"\n",
"if use_cuda:\n",
" net.cuda()\n",
" net = torch.nn.DataParallel(net)\n",
" print('Using', torch.cuda.device_count(), 'GPUs.')\n",
" cudnn.benchmark = True\n",
" print('Using CUDA..')"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"---\n",
"# Training"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"## Loss function and Optimizer\n",
"\n",
"We use the cross entropy loss, commonly used for classification, and stochastic gradient descent (SGD) as optimizer, with momentum and weight decay."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {}
},
"outputs": [],
"source": [
"# optimizer and criterion\n",
"\n",
"def mixup_criterion(y_a, y_b, lam):\n",
" '''\n",
" - Mixup criterion\n",
" - https://github.com/hongyi-zhang/mixup\n",
" '''\n",
" return lambda criterion, pred: lam * criterion(pred, y_a) + (1 - lam) * criterion(pred, y_b)\n",
"\n",
"criterion = nn.CrossEntropyLoss() # only for test data\n",
"optimizer = optim.SGD(net.parameters(), lr=base_learning_rate, momentum=0.9, weight_decay=1e-4)"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"## Train and test loops"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {}
},
"outputs": [],
"source": [
"# Training & Test functions\n",
"def train(epoch, alpha, use_cuda=False):\n",
" print('\\nEpoch: %d' % epoch)\n",
" net.train()\n",
" train_loss = 0\n",
" correct = 0\n",
" total = 0\n",
" for batch_idx, (inputs, targets) in enumerate(trainloader):\n",
" if use_cuda:\n",
" inputs, targets = inputs.cuda(), targets.cuda()\n",
" optimizer.zero_grad()\n",
" if mixup:\n",
" # generate mixed inputs, two one-hot label vectors and mixing coefficient\n",
" inputs, targets_a, targets_b, lam = mixup_data(inputs, targets, alpha, use_cuda)\n",
" inputs, targets_a, targets_b = Variable(inputs), Variable(targets_a), Variable(targets_b)\n",
" outputs = net(inputs)\n",
" loss_func = mixup_criterion(targets_a, targets_b, lam)\n",
" loss = loss_func(criterion, outputs)\n",
" else:\n",
" inputs, targets = Variable(inputs), Variable(targets)\n",
" outputs = net(inputs)\n",
" loss = criterion(outputs, targets)\n",
"\n",
" loss.backward()\n",
" optimizer.step()\n",
"\n",
" train_loss += loss.item()\n",
" _, predicted = torch.max(outputs.data, 1)\n",
" total += targets.size(0)\n",
" if mixup:\n",
" correct += lam * predicted.eq(targets_a.data).cpu().sum() + (1 - lam) * predicted.eq(targets_b.data).cpu().sum()\n",
" else:\n",
" correct += predicted.eq(targets.data).cpu().sum()\n",
"\n",
" if batch_idx % 500 == 0:\n",
" print(batch_idx, len(trainloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'\n",
" % (train_loss/(batch_idx+1), 100.*correct/total, correct, total))\n",
" return (train_loss/batch_idx, 100.*correct/total)\n",
"\n",
"\n",
"def test(epoch, use_cuda=False):\n",
" global best_acc\n",
" net.eval()\n",
" test_loss = 0\n",
" correct = 0\n",
" total = 0\n",
" with torch.no_grad():\n",
" for batch_idx, (inputs, targets) in enumerate(testloader):\n",
" if use_cuda:\n",
" inputs, targets = inputs.cuda(), targets.cuda()\n",
" # inputs, targets = Variable(inputs, volatile=True), Variable(targets)\n",
" outputs = net(inputs)\n",
" loss = criterion(outputs, targets)\n",
"\n",
" test_loss += loss.item()\n",
" _, predicted = torch.max(outputs.data, 1)\n",
" total += targets.size(0)\n",
" correct += predicted.eq(targets.data).cpu().sum()\n",
"\n",
" if batch_idx % 200 == 0:\n",
" print(batch_idx, len(testloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'\n",
" % (test_loss/(batch_idx+1), 100.*correct/total, correct, total))\n",
"\n",
" # Save checkpoint.\n",
" acc = 100.*correct/total\n",
" if acc > best_acc:\n",
" best_acc = acc\n",
" checkpoint(acc, epoch)\n",
" return (test_loss/batch_idx, 100.*correct/total)"
]
},
{
"cell_type": "markdown",
"metadata": {
"execution": {}
},
"source": [
"### Auxiliary functions\n",
"\n",
"* `checkpoint()`: Store checkpoints of the model\n",
"* `adjust_learning_rate()`: Decreases the learning rate (learning rate decay) at certain epochs of training."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" `checkpoint` and `adjust_learning_rate` functions\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"cellView": "form",
"execution": {},
"tags": [
"hide-input"
]
},
"outputs": [],
"source": [
"# @markdown `checkpoint` and `adjust_learning_rate` functions\n",
"def checkpoint(acc, epoch):\n",
" # Save checkpoint.\n",
" print('Saving..')\n",
" state = {\n",
" 'net': net.state_dict(),\n",
" 'acc': acc,\n",
" 'epoch': epoch,\n",
" 'rng_state': torch.get_rng_state()\n",
" }\n",
" if not os.path.isdir('checkpoint'):\n",
" os.mkdir('checkpoint')\n",
" torch.save(state, './checkpoint/ckpt.t7')\n",
"\n",
"\n",
"def adjust_learning_rate(optimizer, epoch):\n",
" \"\"\"decrease the learning rate at 100 and 150 epoch\"\"\"\n",
" lr = base_learning_rate\n",
" if epoch <= 9 and lr > 0.1:\n",
" # warm-up training for large minibatch\n",
" lr = 0.1 + (base_learning_rate - 0.1) * epoch / 10.\n",
" if epoch >= 100:\n",
" lr /= 10\n",
" if epoch >= 150:\n",
" lr /= 10\n",
" for param_group in optimizer.param_groups:\n",
" param_group['lr'] = lr"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\n",
"Epoch: 0\n",
"0 391 Loss: 2.443 | Acc: 10.938% (14/128)\n",
"0 79 Loss: 1.531 | Acc: 46.094% (59/128)\n",
"Saving..\n",
"Epoch: 0 | train acc: 31.604000091552734 | test acc: 44.2599983215332\n",
"\n",
"Epoch: 1\n",
"0 391 Loss: 1.619 | Acc: 39.844% (51/128)\n",
"0 79 Loss: 1.199 | Acc: 60.156% (77/128)\n",
"Saving..\n",
"Epoch: 1 | train acc: 47.03200149536133 | test acc: 54.41999816894531\n",
"\n",
"Epoch: 2\n",
"0 391 Loss: 1.301 | Acc: 53.906% (69/128)\n",
"0 79 Loss: 1.013 | Acc: 61.719% (79/128)\n",
"Saving..\n",
"Epoch: 2 | train acc: 56.257999420166016 | test acc: 62.599998474121094\n",
"\n",
"Epoch: 3\n",
"0 391 Loss: 1.036 | Acc: 64.062% (82/128)\n",
"0 79 Loss: 0.909 | Acc: 69.531% (89/128)\n",
"Saving..\n",
"Epoch: 3 | train acc: 62.43199920654297 | test acc: 65.6500015258789\n",
"\n",
"Epoch: 4\n",
"0 391 Loss: 0.839 | Acc: 68.750% (88/128)\n",
"0 79 Loss: 0.859 | Acc: 70.312% (90/128)\n",
"Saving..\n",
"Epoch: 4 | train acc: 67.0 | test acc: 69.08999633789062\n",
"\n",
"Epoch: 5\n",
"0 391 Loss: 0.922 | Acc: 64.844% (83/128)\n",
"0 79 Loss: 0.660 | Acc: 76.562% (98/128)\n",
"Saving..\n",
"Epoch: 5 | train acc: 70.1259994506836 | test acc: 72.52999877929688\n",
"\n",
"Epoch: 6\n",
"0 391 Loss: 0.833 | Acc: 65.625% (84/128)\n",
"0 79 Loss: 0.616 | Acc: 78.125% (100/128)\n",
"Saving..\n",
"Epoch: 6 | train acc: 73.45999908447266 | test acc: 73.45999908447266\n",
"\n",
"Epoch: 7\n",
"0 391 Loss: 0.686 | Acc: 75.000% (96/128)\n",
"0 79 Loss: 0.533 | Acc: 81.250% (104/128)\n",
"Saving..\n",
"Epoch: 7 | train acc: 75.99600219726562 | test acc: 75.91000366210938\n",
"\n",
"Epoch: 8\n",
"0 391 Loss: 0.626 | Acc: 78.125% (100/128)\n",
"0 79 Loss: 0.458 | Acc: 82.031% (105/128)\n",
"Saving..\n",
"Epoch: 8 | train acc: 78.42400360107422 | test acc: 79.11000061035156\n",
"\n",
"Epoch: 9\n",
"0 391 Loss: 0.465 | Acc: 85.938% (110/128)\n",
"0 79 Loss: 0.465 | Acc: 87.500% (112/128)\n",
"Saving..\n",
"Epoch: 9 | train acc: 80.72599792480469 | test acc: 80.37000274658203\n",
"\n",
"Epoch: 10\n",
"0 391 Loss: 0.509 | Acc: 81.250% (104/128)\n",
"0 79 Loss: 0.523 | Acc: 79.688% (102/128)\n",
"Epoch: 10 | train acc: 82.16400146484375 | test acc: 79.25\n",
"\n",
"Epoch: 11\n",
"0 391 Loss: 0.423 | Acc: 82.031% (105/128)\n",
"0 79 Loss: 0.610 | Acc: 78.125% (100/128)\n",
"Epoch: 11 | train acc: 83.96199798583984 | test acc: 79.68000030517578\n",
"\n",
"Epoch: 12\n",
"0 391 Loss: 0.221 | Acc: 89.844% (115/128)\n",
"0 79 Loss: 0.467 | Acc: 82.812% (106/128)\n",
"Saving..\n",
"Epoch: 12 | train acc: 85.61799621582031 | test acc: 80.88999938964844\n",
"\n",
"Epoch: 13\n",
"0 391 Loss: 0.427 | Acc: 85.938% (110/128)\n",
"0 79 Loss: 0.522 | Acc: 82.812% (106/128)\n",
"Saving..\n",
"Epoch: 13 | train acc: 87.21199798583984 | test acc: 81.54000091552734\n",
"\n",
"Epoch: 14\n",
"0 391 Loss: 0.216 | Acc: 93.750% (120/128)\n",
"0 79 Loss: 0.386 | Acc: 86.719% (111/128)\n",
"Epoch: 14 | train acc: 88.08000183105469 | test acc: 81.44999694824219\n"
]
}
],
"source": [
"# start training\n",
"if not os.path.exists(logname):\n",
" with open(logname, 'w') as logfile:\n",
" logwriter = csv.writer(logfile, delimiter=',')\n",
" logwriter.writerow(['epoch', 'train loss', 'train acc',\n",
" 'test loss', 'test acc'])\n",
"\n",
"for epoch in range(start_epoch, end_apochs):\n",
" adjust_learning_rate(optimizer, epoch)\n",
" train_loss, train_acc = train(epoch, alpha, use_cuda=use_cuda)\n",
" test_loss, test_acc = test(epoch, use_cuda=use_cuda)\n",
" with open(logname, 'a') as logfile:\n",
" logwriter = csv.writer(logfile, delimiter=',')\n",
" logwriter.writerow([epoch, train_loss, train_acc.item(),\n",
" test_loss, test_acc.item()])\n",
" print(f'Epoch: {epoch} | train acc: {train_acc} | test acc: {test_acc}')"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" epoch | \n",
" train loss | \n",
" train acc | \n",
" test loss | \n",
" test acc | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 0 | \n",
" 1.932130 | \n",
" 31.604000 | \n",
" 1.535233 | \n",
" 44.259998 | \n",
"
\n",
" \n",
" | 1 | \n",
" 1 | \n",
" 1.446863 | \n",
" 47.032001 | \n",
" 1.262779 | \n",
" 54.419998 | \n",
"
\n",
" \n",
" | 2 | \n",
" 2 | \n",
" 1.212518 | \n",
" 56.257999 | \n",
" 1.069593 | \n",
" 62.599998 | \n",
"
\n",
" \n",
" | 3 | \n",
" 3 | \n",
" 1.051850 | \n",
" 62.431999 | \n",
" 0.996476 | \n",
" 65.650002 | \n",
"
\n",
" \n",
" | 4 | \n",
" 4 | \n",
" 0.928131 | \n",
" 67.000000 | \n",
" 0.898354 | \n",
" 69.089996 | \n",
"
\n",
" \n",
"
\n",
"