{ "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "execution": {}, "id": "view-in-github" }, "source": [ "\"Open   \"Open" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "# Transfer Learning \n", "\n", "**By Neuromatch Academy**\n", "\n", "__Content creators:__ [Jama Hussein Mohamud](https://engmubarak48.github.io/jmohamud/index.html) & [Alex Hernandez-Garcia](https://alexhernandezgarcia.github.io/)\n", "\n", "__Production editors:__ Saeed Salehi, Spiros Chavlis\n", "\n" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "---\n", "# Objective\n", "\n", "One desired capability for machines is the ability to transfer the knowledge (features) learned on one domain to another This can potentially save compute time, enable training when data is scarce, and even improve performance. Unfortunately, there is no single recipe for transfer learning and instead multiple options are possible and much remains to be well understood. In this project, you will explore how transfer learning works in different scenarios. " ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "---\n", "# Setup" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [], "source": [ "# imports\n", "import os\n", "import gc\n", "import csv\n", "import glob\n", "import torch\n", "import multiprocessing\n", "\n", "import numpy as np\n", "import pandas as pd\n", "import torch.nn as nn\n", "import matplotlib.pyplot as plt\n", "\n", "import torch.optim as optim\n", "import torch.nn.functional as F\n", "import torch.backends.cudnn as cudnn\n", "from torch.autograd import Variable\n", "\n", "import torchvision\n", "import torchvision.transforms as transforms" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Set random seed\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " Executing `set_seed(seed=seed)` you are setting the seed\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "execution": {}, "tags": [ "hide-input" ] }, "outputs": [], "source": [ "# @title Set random seed\n", "\n", "# @markdown Executing `set_seed(seed=seed)` you are setting the seed\n", "\n", "# for DL its critical to set the random seed so that students can have a\n", "# baseline to compare their results to expected results.\n", "# Read more here: https://pytorch.org/docs/stable/notes/randomness.html\n", "\n", "# Call `set_seed` function in the exercises to ensure reproducibility.\n", "import random\n", "import torch\n", "\n", "def set_seed(seed=None, seed_torch=True):\n", " if seed is None:\n", " seed = np.random.choice(2 ** 32)\n", " random.seed(seed)\n", " np.random.seed(seed)\n", " if seed_torch:\n", " torch.manual_seed(seed)\n", " torch.cuda.manual_seed_all(seed)\n", " torch.cuda.manual_seed(seed)\n", " torch.backends.cudnn.benchmark = False\n", " torch.backends.cudnn.deterministic = True\n", "\n", " print(f'Random seed {seed} has been set.')\n", "\n", "# In case that `DataLoader` is used\n", "def seed_worker(worker_id):\n", " worker_seed = torch.initial_seed() % 2**32\n", " np.random.seed(worker_seed)\n", " random.seed(worker_seed)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Set device (GPU or CPU)\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "execution": {}, "tags": [ "hide-input" ] }, "outputs": [], "source": [ "# @title Set device (GPU or CPU)\n", "\n", "# inform the user if the notebook uses GPU or CPU.\n", "\n", "def set_device():\n", " device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", " if device != \"cuda\":\n", " print(\"WARNING: For this notebook to perform best, \"\n", " \"if possible, in the menu under `Runtime` -> \"\n", " \"`Change runtime type.` select `GPU` \")\n", " else:\n", " print(\"GPU is enabled in this notebook.\")\n", "\n", " return device" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "### Random seeds\n", "\n", "If you want to obtain reproducible results, it is a good practice to set seeds for the random number generators of the various libraries" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Random seed 2021 has been set.\n", "GPU is enabled in this notebook.\n" ] } ], "source": [ "set_seed(seed=2021)\n", "device = set_device()" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "### Training hyperparameters\n", "\n", "Here we set some general training hyperparameters such as the learning rate, batch size, etc. as well as other training options such as including data augmentation (`torchvision_transforms`)." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [], "source": [ "# hyper-parameters\n", "use_cuda = torch.cuda.is_available()\n", "best_acc = 0 # best test accuracy\n", "start_epoch = 0 # start from epoch 0 or last checkpoint epoch\n", "batch_size = 128\n", "max_epochs = 15 # Please change this to 200\n", "max_epochs_target = 10\n", "base_learning_rate = 0.1\n", "torchvision_transforms = True # True/False if you want use torchvision augmentations" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "---\n", "# Data" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "## Source dataset\n", "\n", "We will train the source model using CIFAR-100 data set from PyTorch, but with small tweaks we can get any other data we are interested in.\n", "\n", "Note that the data set is normalised by substracted the mean and dividing by the standard deviation (pre-computed) of the training set. Also, if `torchvision_transforms` is `True`, data augmentation will be applied during training." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " Download and prepare Data\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "execution": {}, "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "==> Preparing data..\n", "Downloading https://www.cs.toronto.edu/~kriz/cifar-100-python.tar.gz to ./CIFAR100/cifar-100-python.tar.gz\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "3dd2d265bd4a44de80738813df1a1b1e", "version_major": 2, "version_minor": 0 }, "text/plain": [ "HBox(children=(FloatProgress(value=0.0, max=169001437.0), HTML(value='')))" ] }, "metadata": { "tags": [] }, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Extracting ./CIFAR100/cifar-100-python.tar.gz to ./CIFAR100\n", "Files already downloaded and verified\n" ] } ], "source": [ "# @markdown Download and prepare Data\n", "print('==> Preparing data..')\n", "def percentageSplit(full_dataset, percent = 0.0):\n", " set1_size = int(percent * len(full_dataset))\n", " set2_size = len(full_dataset) - set1_size\n", " final_dataset, _ = torch.utils.data.random_split(full_dataset, [set1_size, set2_size])\n", " return final_dataset\n", "\n", "\n", "# CIFAR100 normalizing\n", "mean = [0.5071, 0.4866, 0.4409]\n", "std = [0.2673, 0.2564, 0.2762]\n", "\n", "# CIFAR10 normalizing\n", "# mean = (0.4914, 0.4822, 0.4465)\n", "# std = (0.2023, 0.1994, 0.2010)\n", "\n", "# torchvision transforms\n", "transform_train = transforms.Compose([])\n", "if torchvision_transforms:\n", " transform_train.transforms.append(transforms.RandomCrop(32, padding=4))\n", " transform_train.transforms.append(transforms.RandomHorizontalFlip())\n", "\n", "transform_train.transforms.append(transforms.ToTensor())\n", "transform_train.transforms.append(transforms.Normalize(mean, std))\n", "\n", "transform_test = transforms.Compose([\n", " transforms.ToTensor(),\n", " transforms.Normalize(mean, std),\n", "])\n", "\n", "trainset = torchvision.datasets.CIFAR100(\n", " root='./CIFAR100', train=True, download=True, transform=transform_train)\n", "\n", "testset = torchvision.datasets.CIFAR100(\n", " root='./CIFAR100', train=False, download=True, transform=transform_test)" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "### CIFAR-100\n", "\n", "CIFAR-100 is a data set of 50,000 colour (RGB) training images and 10,000 test images, of size 32 x 32 pixels. Each image is labelled as 1 of 100 possible classes. \n", "\n", "The data set is stored as a custom `torchvision.datasets.cifar.CIFAR` object. You can check some of its properties with the following code:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Object type: \n", "Training data shape: (50000, 32, 32, 3)\n", "Test data shape: (10000, 32, 32, 3)\n", "Number of classes: 100\n" ] } ], "source": [ "print(f\"Object type: {type(trainset)}\")\n", "print(f\"Training data shape: {trainset.data.shape}\")\n", "print(f\"Test data shape: {testset.data.shape}\")\n", "print(f\"Number of classes: {np.unique(trainset.targets).shape[0]}\")" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "## Data loaders\n", "\n", "A dataloader is an optimized data iterator that provides functionality for efficient shuffling, transformation and batching of the data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Dataloader\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "execution": {}, "tags": [ "hide-input" ] }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "----> number of workers: 2\n" ] } ], "source": [ "##@title Dataloader\n", "num_workers = multiprocessing.cpu_count()\n", "\n", "print(f'----> number of workers: {num_workers}')\n", "\n", "trainloader = torch.utils.data.DataLoader(\n", " trainset, batch_size=batch_size, shuffle=True, num_workers=num_workers)\n", "testloader = torch.utils.data.DataLoader(\n", " testset, batch_size=batch_size, shuffle=False, num_workers=num_workers)" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "## Architecture: ResNet\n", "\n", "ResNet is a family of network architectures whose main property is that the network is organised as a stack of _residual blocks_. Residual blocks consist of a stack of layers whose output is added the input, making a _shortcut connection_.\n", "\n", "See the [original paper](https://arxiv.org/abs/1512.03385) for more details.\n", "\n", "ResNet is just a popular choice out of many others, but data augmentation works well in general. We just picked ResNet for illustration purposes." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### ResNet model in PyTorch\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "cellView": "form", "execution": {}, "tags": [ "hide-input" ] }, "outputs": [], "source": [ "# @title ResNet model in PyTorch\n", "\n", "class BasicBlock(nn.Module):\n", " \"\"\"ResNet in PyTorch.\n", " Reference:\n", " [1] Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun\n", " Deep Residual Learning for Image Recognition. arXiv:1512.03385\n", " \"\"\"\n", "\n", " expansion = 1\n", "\n", " def __init__(self, in_planes, planes, stride=1):\n", " super(BasicBlock, self).__init__()\n", " self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)\n", " self.bn1 = nn.BatchNorm2d(planes)\n", " self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=1, padding=1, bias=False)\n", " self.bn2 = nn.BatchNorm2d(planes)\n", "\n", " self.shortcut = nn.Sequential()\n", " if stride != 1 or in_planes != self.expansion*planes:\n", " self.shortcut = nn.Sequential(\n", " nn.Conv2d(in_planes, self.expansion*planes, kernel_size=1, stride=stride, bias=False),\n", " nn.BatchNorm2d(self.expansion*planes)\n", " )\n", "\n", " def forward(self, x):\n", " out = F.relu(self.bn1(self.conv1(x)))\n", " out = self.bn2(self.conv2(out))\n", " out += self.shortcut(x)\n", " out = F.relu(out)\n", " return out\n", "\n", "\n", "class Bottleneck(nn.Module):\n", " expansion = 4\n", "\n", " def __init__(self, in_planes, planes, stride=1):\n", " super(Bottleneck, self).__init__()\n", " self.conv1 = nn.Conv2d(in_planes, planes, kernel_size=1, bias=False)\n", " self.bn1 = nn.BatchNorm2d(planes)\n", " self.conv2 = nn.Conv2d(planes, planes, kernel_size=3, stride=stride, padding=1, bias=False)\n", " self.bn2 = nn.BatchNorm2d(planes)\n", " self.conv3 = nn.Conv2d(planes, self.expansion*planes, kernel_size=1, bias=False)\n", " self.bn3 = nn.BatchNorm2d(self.expansion*planes)\n", "\n", " self.shortcut = nn.Sequential()\n", " if stride != 1 or in_planes != self.expansion*planes:\n", " self.shortcut = nn.Sequential(\n", " nn.Conv2d(in_planes, self.expansion*planes, kernel_size=1, stride=stride, bias=False),\n", " nn.BatchNorm2d(self.expansion*planes)\n", " )\n", "\n", " def forward(self, x):\n", " out = F.relu(self.bn1(self.conv1(x)))\n", " out = F.relu(self.bn2(self.conv2(out)))\n", " out = self.bn3(self.conv3(out))\n", " out += self.shortcut(x)\n", " out = F.relu(out)\n", " return out\n", "\n", "\n", "class ResNet(nn.Module):\n", " def __init__(self, block, num_blocks, num_classes=100):\n", " super(ResNet, self).__init__()\n", " self.in_planes = 64\n", "\n", " self.conv1 = nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=1, bias=False)\n", " self.bn1 = nn.BatchNorm2d(64)\n", " self.layer1 = self._make_layer(block, 64, num_blocks[0], stride=1)\n", " self.layer2 = self._make_layer(block, 128, num_blocks[1], stride=2)\n", " self.layer3 = self._make_layer(block, 256, num_blocks[2], stride=2)\n", " self.layer4 = self._make_layer(block, 512, num_blocks[3], stride=2)\n", " self.linear = nn.Linear(512*block.expansion, num_classes)\n", "\n", " def _make_layer(self, block, planes, num_blocks, stride):\n", " strides = [stride] + [1]*(num_blocks-1)\n", " layers = []\n", " for stride in strides:\n", " layers.append(block(self.in_planes, planes, stride))\n", " self.in_planes = planes * block.expansion\n", " return nn.Sequential(*layers)\n", "\n", " def forward(self, x):\n", " out = F.relu(self.bn1(self.conv1(x)))\n", " out = self.layer1(out)\n", " out = self.layer2(out)\n", " out = self.layer3(out)\n", " out = self.layer4(out)\n", " out = F.avg_pool2d(out, 4)\n", " out = out.view(out.size(0), -1)\n", " out = self.linear(out)\n", " return out\n", "\n", "\n", "def ResNet18():\n", " return ResNet(BasicBlock, [2, 2, 2, 2])\n", "\n", "\n", "def ResNet34():\n", " return ResNet(BasicBlock, [3, 4, 6, 3])\n", "\n", "\n", "def ResNet50():\n", " return ResNet(Bottleneck, [3, 4, 6, 3])" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "#### Test on random data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "-----> verify if model is run on random data\n", "model loaded\n", "Using 1 GPUs.\n", "Using CUDA..\n" ] } ], "source": [ "# Load the Model\n", "net = ResNet18()\n", "print('-----> verify if model is run on random data')\n", "y = net(Variable(torch.randn(1,3,32,32)))\n", "print('model loaded')\n", "\n", "result_folder = './results/'\n", "if not os.path.exists(result_folder):\n", " os.makedirs(result_folder)\n", "\n", "logname = result_folder + net.__class__.__name__ + '_pretrain' + '.csv'\n", "\n", "if use_cuda:\n", " net.cuda()\n", " net = torch.nn.DataParallel(net)\n", " print('Using', torch.cuda.device_count(), 'GPUs.')\n", " cudnn.benchmark = True\n", " print('Using CUDA..')" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "## Set up training" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "### Set loss function and optimizer\n", "\n", "We use the cross entropy loss, commonly used for classification, and stochastic gradient descent (SGD) as optimizer, with momentum and weight decay." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [], "source": [ "# Optimizer and criterion\n", "\n", "criterion = nn.CrossEntropyLoss()\n", "optimizer = optim.SGD(net.parameters(), lr=base_learning_rate, momentum=0.9, weight_decay=1e-4)" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "### Train and test loops" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [], "source": [ "# Training & Test functions\n", "\n", "def train(net, epoch, use_cuda=True):\n", " print('\\nEpoch: %d' % epoch)\n", " net.train()\n", " train_loss = 0\n", " correct = 0\n", " total = 0\n", " for batch_idx, (inputs, targets) in enumerate(trainloader):\n", " if use_cuda:\n", " inputs, targets = inputs.cuda(), targets.cuda()\n", "\n", " optimizer.zero_grad()\n", " inputs, targets = Variable(inputs), Variable(targets)\n", " outputs = net(inputs)\n", " loss = criterion(outputs, targets)\n", " loss.backward()\n", " optimizer.step()\n", "\n", " train_loss += loss.item()\n", " _, predicted = torch.max(outputs.data, 1)\n", " total += targets.size(0)\n", " correct += predicted.eq(targets.data).cpu().sum()\n", "\n", " if batch_idx % 500 == 0:\n", " print(batch_idx, len(trainloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'\n", " % (train_loss/(batch_idx+1), 100.*correct/total, correct, total))\n", " return (train_loss/batch_idx, 100.*correct/total)\n", "\n", "\n", "def test(net, epoch, outModelName, use_cuda=True):\n", " global best_acc\n", " net.eval()\n", " test_loss, correct, total = 0, 0, 0\n", " with torch.no_grad():\n", " for batch_idx, (inputs, targets) in enumerate(testloader):\n", " if use_cuda:\n", " inputs, targets = inputs.cuda(), targets.cuda()\n", "\n", " outputs = net(inputs)\n", " loss = criterion(outputs, targets)\n", "\n", " test_loss += loss.item()\n", " _, predicted = torch.max(outputs.data, 1)\n", " total += targets.size(0)\n", " correct += predicted.eq(targets.data).cpu().sum()\n", "\n", " if batch_idx % 200 == 0:\n", " print(batch_idx, len(testloader), 'Loss: %.3f | Acc: %.3f%% (%d/%d)'\n", " % (test_loss/(batch_idx+1), 100.*correct/total, correct, total))\n", "\n", " # Save checkpoint.\n", " acc = 100.*correct/total\n", " if acc > best_acc:\n", " best_acc = acc\n", " checkpoint(net, acc, epoch, outModelName)\n", " return (test_loss/batch_idx, 100.*correct/total)" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "### Auxiliary functions\n", "\n", "* `checkpoint()`: Store checkpoints of the model\n", "* `adjust_learning_rate()`: Decreases the learning rate (learning rate decay) at certain epochs of training." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [], "source": [ "# checkpoint & adjust_learning_rate\n", "def checkpoint(model, acc, epoch, outModelName):\n", " # Save checkpoint.\n", " print('Saving..')\n", " state = {\n", " 'state_dict': model.state_dict(),\n", " 'acc': acc,\n", " 'epoch': epoch,\n", " 'rng_state': torch.get_rng_state()\n", " }\n", " if not os.path.isdir('checkpoint'):\n", " os.mkdir('checkpoint')\n", " torch.save(state, f'./checkpoint/{outModelName}.t7')\n", "\n", "def adjust_learning_rate(optimizer, epoch):\n", " \"\"\"decrease the learning rate at 100 and 150 epoch\"\"\"\n", " lr = base_learning_rate\n", " if epoch <= 9 and lr > 0.1:\n", " # warm-up training for large minibatch\n", " lr = 0.1 + (base_learning_rate - 0.1) * epoch / 10.\n", " if epoch >= 100:\n", " lr /= 10\n", " if epoch >= 150:\n", " lr /= 10\n", " for param_group in optimizer.param_groups:\n", " param_group['lr'] = lr" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "### Train the model\n", "\n", "This is the loop where the model is trained for `max_epochs` epochs." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Epoch: 0\n", "0 391 Loss: 4.748 | Acc: 0.781% (1/128)\n", "0 79 Loss: 3.545 | Acc: 12.500% (16/128)\n", "Saving..\n", "Epoch: 0 | train acc: 8.527999877929688 | test acc: 13.369999885559082\n", "\n", "Epoch: 1\n", "0 391 Loss: 3.597 | Acc: 16.406% (21/128)\n", "0 79 Loss: 3.157 | Acc: 23.438% (30/128)\n", "Saving..\n", "Epoch: 1 | train acc: 18.392000198364258 | test acc: 21.829999923706055\n", "\n", "Epoch: 2\n", "0 391 Loss: 2.932 | Acc: 26.562% (34/128)\n", "0 79 Loss: 2.450 | Acc: 39.844% (51/128)\n", "Saving..\n", "Epoch: 2 | train acc: 27.016000747680664 | test acc: 31.079999923706055\n", "\n", "Epoch: 3\n", "0 391 Loss: 2.649 | Acc: 35.938% (46/128)\n", "0 79 Loss: 2.134 | Acc: 39.844% (51/128)\n", "Saving..\n", "Epoch: 3 | train acc: 35.84000015258789 | test acc: 35.70000076293945\n", "\n", "Epoch: 4\n", "0 391 Loss: 2.153 | Acc: 41.406% (53/128)\n", "0 79 Loss: 1.911 | Acc: 49.219% (63/128)\n", "Saving..\n", "Epoch: 4 | train acc: 42.827999114990234 | test acc: 43.619998931884766\n", "\n", "Epoch: 5\n", "0 391 Loss: 1.878 | Acc: 50.000% (64/128)\n", "0 79 Loss: 2.149 | Acc: 43.750% (56/128)\n", "Saving..\n", "Epoch: 5 | train acc: 48.87200164794922 | test acc: 45.380001068115234\n", "\n", "Epoch: 6\n", "0 391 Loss: 1.814 | Acc: 51.562% (66/128)\n", "0 79 Loss: 1.847 | Acc: 46.875% (60/128)\n", "Saving..\n", "Epoch: 6 | train acc: 53.59000015258789 | test acc: 50.310001373291016\n", "\n", "Epoch: 7\n", "0 391 Loss: 1.514 | Acc: 56.250% (72/128)\n", "0 79 Loss: 1.568 | Acc: 51.562% (66/128)\n", "Saving..\n", "Epoch: 7 | train acc: 57.35200119018555 | test acc: 54.209999084472656\n", "\n", "Epoch: 8\n", "0 391 Loss: 1.194 | Acc: 62.500% (80/128)\n", "0 79 Loss: 1.403 | Acc: 59.375% (76/128)\n", "Saving..\n", "Epoch: 8 | train acc: 60.61600112915039 | test acc: 57.20000076293945\n", "\n", "Epoch: 9\n", "0 391 Loss: 1.124 | Acc: 69.531% (89/128)\n", "0 79 Loss: 1.339 | Acc: 64.844% (83/128)\n", "Saving..\n", "Epoch: 9 | train acc: 63.55400085449219 | test acc: 58.900001525878906\n", "\n", "Epoch: 10\n", "0 391 Loss: 1.013 | Acc: 72.656% (93/128)\n", "0 79 Loss: 1.225 | Acc: 66.406% (85/128)\n", "Epoch: 10 | train acc: 65.91999816894531 | test acc: 58.83000183105469\n", "\n", "Epoch: 11\n", "0 391 Loss: 0.971 | Acc: 64.844% (83/128)\n", "0 79 Loss: 1.491 | Acc: 63.281% (81/128)\n", "Epoch: 11 | train acc: 68.05000305175781 | test acc: 57.560001373291016\n", "\n", "Epoch: 12\n", "0 391 Loss: 1.028 | Acc: 70.312% (90/128)\n", "0 79 Loss: 1.358 | Acc: 63.281% (81/128)\n", "Saving..\n", "Epoch: 12 | train acc: 69.99600219726562 | test acc: 60.099998474121094\n", "\n", "Epoch: 13\n", "0 391 Loss: 0.699 | Acc: 82.812% (106/128)\n", "0 79 Loss: 1.299 | Acc: 63.281% (81/128)\n", "Saving..\n", "Epoch: 13 | train acc: 71.9739990234375 | test acc: 60.220001220703125\n", "\n", "Epoch: 14\n", "0 391 Loss: 0.768 | Acc: 75.781% (97/128)\n", "0 79 Loss: 1.182 | Acc: 74.219% (95/128)\n", "Saving..\n", "Epoch: 14 | train acc: 73.69400024414062 | test acc: 62.90999984741211\n" ] } ], "source": [ "# Start training\n", "outModelName = 'pretrain'\n", "if not os.path.exists(logname):\n", " with open(logname, 'w') as logfile:\n", " logwriter = csv.writer(logfile, delimiter=',')\n", " logwriter.writerow(['epoch', 'train loss', 'train acc', 'test loss', 'test acc'])\n", "\n", "for epoch in range(start_epoch, max_epochs):\n", " adjust_learning_rate(optimizer, epoch)\n", " train_loss, train_acc = train(net, epoch, use_cuda=use_cuda)\n", " test_loss, test_acc = test(net, epoch, outModelName, use_cuda=use_cuda)\n", " with open(logname, 'a') as logfile:\n", " logwriter = csv.writer(logfile, delimiter=',')\n", " logwriter.writerow([epoch, train_loss, train_acc.item(), test_loss, test_acc.item()])\n", " print(f'Epoch: {epoch} | train acc: {train_acc} | test acc: {test_acc}')" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "## Transfer learning\n", "### Re-use the trained model to improve training on a different data set" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "### Delete variables from the previous model" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [], "source": [ "# delete the backbone network\n", "delete = True\n", "if delete:\n", " del net\n", " del trainset\n", " del testset\n", " del trainloader\n", " del testloader\n", " gc.collect()" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "#### Target dataset\n", "\n", "We will now use CIFAR-10 as _target_ data set. Again, with small tweaks we can get any other data we are interested in.\n", "\n", "CIFAR-10 is very similar to CIFAR-100, but it contains only 10 classes instead of 100." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "==> Preparing target domain data..\n", "Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./CIFAR10/cifar-10-python.tar.gz\n" ] }, { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "ef171f0d74cc467ab4b78cd32bba5071", "version_major": 2, "version_minor": 0 }, "text/plain": [ "HBox(children=(FloatProgress(value=0.0, max=170498071.0), HTML(value='')))" ] }, "metadata": { "tags": [] }, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "\n", "Extracting ./CIFAR10/cifar-10-python.tar.gz to ./CIFAR10\n", "Files already downloaded and verified\n" ] } ], "source": [ "# Target domain Data\n", "print('==> Preparing target domain data..')\n", "\n", "# CIFAR10 normalizing\n", "mean = (0.4914, 0.4822, 0.4465)\n", "std = (0.2023, 0.1994, 0.2010)\n", "num_classes = 10\n", "lr = 0.0001\n", "\n", "# torchvision transforms\n", "transform_train = transforms.Compose([])\n", "if torchvision_transforms:\n", " transform_train.transforms.append(transforms.RandomCrop(32, padding=4))\n", " transform_train.transforms.append(transforms.RandomHorizontalFlip())\n", "\n", "transform_train.transforms.append(transforms.ToTensor())\n", "transform_train.transforms.append(transforms.Normalize(mean, std))\n", "\n", "transform_test = transforms.Compose([\n", " transforms.ToTensor(),\n", " transforms.Normalize(mean, std),\n", "])\n", "\n", "trainset = torchvision.datasets.CIFAR10(\n", " root='./CIFAR10', train=True, download=True, transform=transform_train)\n", "\n", "testset = torchvision.datasets.CIFAR10(\n", " root='./CIFAR10', train=False, download=True, transform=transform_test)" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "#### Select a subset of the data\n", "\n", "To simulate a lower data regime, where transfer learning can be useful.\n", "\n", "Choose percentage from the trainset. Set `percent = 1.0` to use the whole train data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "size of the new trainset: 30000\n" ] } ], "source": [ "percent = 0.6\n", "\n", "trainset = percentageSplit(trainset, percent = percent)\n", "print('size of the new trainset: ', len(trainset))" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "#### Dataloaders\n", "\n", "As before" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "----> number of workers: 2\n" ] } ], "source": [ "# Dataloader\n", "num_workers = multiprocessing.cpu_count()\n", "\n", "print(f'----> number of workers: {num_workers}')\n", "\n", "trainloader = torch.utils.data.DataLoader(\n", " trainset, batch_size=batch_size, shuffle=True, num_workers=num_workers)\n", "testloader = torch.utils.data.DataLoader(\n", " testset, batch_size=batch_size, shuffle=False, num_workers=num_workers)" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "### Load pre-trained model\n", "\n", "Load the checkpoint of the model previously trained on CIFAR-100" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " ===> loading pretrained model from: /content/checkpoint/pretrain.t7\n", "Best Accuracy: tensor(62.9100)\n", "Load pretrained model with msg: \n" ] } ], "source": [ "model = ResNet18()\n", "\n", "checkpointPath = '/content/checkpoint/pretrain.t7'\n", "\n", "print(' ===> loading pretrained model from: ', checkpointPath)\n", "if os.path.isfile(checkpointPath):\n", " state_dict = torch.load(checkpointPath)\n", " best_acc = state_dict['acc']\n", " print('Best Accuracy:', best_acc)\n", " if \"state_dict\" in state_dict:\n", " state_dict = state_dict[\"state_dict\"]\n", " # remove prefixe \"module.\"\n", " state_dict = {k.replace(\"module.\", \"\"): v for k, v in state_dict.items()}\n", " for k, v in model.state_dict().items():\n", " if k not in list(state_dict):\n", " print('key \"{}\" could not be found in provided state dict'.format(k))\n", " elif state_dict[k].shape != v.shape:\n", " print('key \"{}\" is of different shape in model and provided state dict'.format(k))\n", " state_dict[k] = v\n", " msg = model.load_state_dict(state_dict, strict=False)\n", " print(\"Load pretrained model with msg: {}\".format(msg))\n", "else:\n", " raise Exception('No pretrained weights found')" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "### Freeze model parameters\n", "\n", "In transfer learning, we usually do not re-train all the weights of the model, but only a subset of them, for instance the last layer. Here we first _freeze_ all the parameters of the model, and we will _unfreeze_ one layer below." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [], "source": [ "# Freeze the model parameters, you can also freeze some layers only\n", "\n", "for param in model.parameters():\n", " param.requires_grad = False" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "### Loss function, optimizer and _unfreeze_ last layer" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [], "source": [ "num_ftrs = model.linear.in_features\n", "model.linear = nn.Linear(num_ftrs, num_classes)\n", "\n", "model.to(device)\n", "\n", "criterion = nn.CrossEntropyLoss()\n", "optimizer = torch.optim.SGD(\n", " model.linear.parameters(),\n", " lr=lr,\n", " momentum=0.9,\n", " weight_decay=1e-4,\n", ")" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "#### Check number of parameters\n", "\n", "We can calculate the number of total parameters and the number of trainable parameters, that is those that will be updated during training. Since we have freezed most of the parameters, the number of training parameters should be much smaller." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total Parameters: 11173962 Trainable parameters: 5130\n" ] } ], "source": [ "total_params = sum(p.numel() for p in model.parameters())\n", "trainable_total_params = sum(p.numel() for p in model.parameters() if p.requires_grad)\n", "\n", "print('Total Parameters:', total_params, 'Trainable parameters: ', trainable_total_params)" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "### Train the target model" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Epoch: 0\n", "0 235 Loss: 2.302 | Acc: 16.406% (21/128)\n", "0 79 Loss: 0.630 | Acc: 79.688% (102/128)\n", "Saving..\n", "Epoch: 0 | train acc: 71.086669921875 | test acc: 75.08999633789062\n", "\n", "Epoch: 1\n", "0 235 Loss: 0.757 | Acc: 72.656% (93/128)\n", "0 79 Loss: 0.619 | Acc: 79.688% (102/128)\n", "Saving..\n", "Epoch: 1 | train acc: 75.86000061035156 | test acc: 76.54000091552734\n", "\n", "Epoch: 2\n", "0 235 Loss: 0.666 | Acc: 75.781% (97/128)\n", "0 79 Loss: 0.640 | Acc: 78.125% (100/128)\n", "Saving..\n", "Epoch: 2 | train acc: 77.04000091552734 | test acc: 76.55000305175781\n", "\n", "Epoch: 3\n", "0 235 Loss: 0.579 | Acc: 81.250% (104/128)\n", "0 79 Loss: 0.577 | Acc: 79.688% (102/128)\n", "Saving..\n", "Epoch: 3 | train acc: 77.56999969482422 | test acc: 77.2300033569336\n", "\n", "Epoch: 4\n", "0 235 Loss: 0.661 | Acc: 78.125% (100/128)\n", "0 79 Loss: 0.613 | Acc: 76.562% (98/128)\n", "Saving..\n", "Epoch: 4 | train acc: 77.6866683959961 | test acc: 77.44999694824219\n", "\n", "Epoch: 5\n", "0 235 Loss: 0.627 | Acc: 80.469% (103/128)\n", "0 79 Loss: 0.626 | Acc: 80.469% (103/128)\n", "Epoch: 5 | train acc: 78.163330078125 | test acc: 77.37999725341797\n", "\n", "Epoch: 6\n", "0 235 Loss: 0.602 | Acc: 77.344% (99/128)\n", "0 79 Loss: 0.607 | Acc: 78.125% (100/128)\n", "Saving..\n", "Epoch: 6 | train acc: 78.42333221435547 | test acc: 78.02999877929688\n", "\n", "Epoch: 7\n", "0 235 Loss: 0.537 | Acc: 75.781% (97/128)\n", "0 79 Loss: 0.608 | Acc: 79.688% (102/128)\n", "Saving..\n", "Epoch: 7 | train acc: 78.49333190917969 | test acc: 78.1500015258789\n", "\n", "Epoch: 8\n", "0 235 Loss: 0.578 | Acc: 75.781% (97/128)\n", "0 79 Loss: 0.650 | Acc: 76.562% (98/128)\n", "Epoch: 8 | train acc: 78.15333557128906 | test acc: 76.83000183105469\n", "\n", "Epoch: 9\n", "0 235 Loss: 0.583 | Acc: 77.344% (99/128)\n", "0 79 Loss: 0.616 | Acc: 77.344% (99/128)\n", "Saving..\n", "Epoch: 9 | train acc: 78.66999816894531 | test acc: 78.20999908447266\n" ] } ], "source": [ "outModelName = 'finetuned'\n", "logname = result_folder + model.__class__.__name__ + f'_{outModelName}.csv'\n", "\n", "if not os.path.exists(logname):\n", " with open(logname, 'w') as logfile:\n", " logwriter = csv.writer(logfile, delimiter=',')\n", " logwriter.writerow(['epoch', 'train loss', 'train acc', 'test loss', 'test acc'])\n", "\n", "for epoch in range(start_epoch, max_epochs_target):\n", " adjust_learning_rate(optimizer, epoch)\n", " train_loss, train_acc = train(model, epoch, use_cuda=use_cuda)\n", " test_loss, test_acc = test(model, epoch, outModelName, use_cuda=use_cuda)\n", " with open(logname, 'a') as logfile:\n", " logwriter = csv.writer(logfile, delimiter=',')\n", " logwriter.writerow([epoch, train_loss, train_acc.item(), test_loss, test_acc.item()])\n", " print(f'Epoch: {epoch} | train acc: {train_acc} | test acc: {test_acc}')" ] }, { "cell_type": "markdown", "metadata": { "execution": {} }, "source": [ "## Plot results" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
epochtrain losstrain acctest losstest acc
000.81969071.0866700.71326075.089996
110.68094075.8600010.67443176.540001
220.65024577.0400010.67588376.550003
330.63855577.5700000.65277677.230003
440.63050077.6866680.66642877.449997
\n", "
" ], "text/plain": [ " epoch train loss train acc test loss test acc\n", "0 0 0.819690 71.086670 0.713260 75.089996\n", "1 1 0.680940 75.860001 0.674431 76.540001\n", "2 2 0.650245 77.040001 0.675883 76.550003\n", "3 3 0.638555 77.570000 0.652776 77.230003\n", "4 4 0.630500 77.686668 0.666428 77.449997" ] }, "execution_count": 24, "metadata": { "tags": [] }, "output_type": "execute_result" } ], "source": [ "# title plot results\n", "results = pd.read_csv(f'/content/results/ResNet_{outModelName}.csv', sep =',')\n", "results.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Average Accuracy over 10 epochs: 77.0\n", "best accuraccy over 10 epochs: 78.20999908447266\n" ] } ], "source": [ "train_accuracy = results['train acc'].values\n", "test_accuracy = results['test acc'].values\n", "\n", "print(f'Average Accuracy over {max_epochs_target} epochs:', sum(test_accuracy)//len(test_accuracy))\n", "print(f'best accuraccy over {max_epochs_target} epochs:', max(test_accuracy))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "execution": {} }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAEWCAYAAABhffzLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3dd3xV9f348dc7mwxWAjLCRkCGshURq+JCEfcEd4utrdX+Wkf7VVs77dDapdZVqyAuwFVXB8oUZKMykiiBBIQkJJAEst+/Pz4ncAnZyb03N/f9fDzuI/ecc88573uSvO+5n/M574+oKsYYY8JHRLADMMYYE1iW+I0xJsxY4jfGmDBjid8YY8KMJX5jjAkzlviNMSbMWOIPMSLynojcGOw4TNsnIkNFZL2IFIrI94MdT7CJyEci8s1gx9EWWOIPABEp8nlUicghn+mZTdmWqk5T1X82Yd8xIpIrIhk++6wUkRKf6Z804z09LyK/bMTrRES+FJEvmroP02L3AItUNUlV/9zSjYnISBH5wPt7OuYGIC+x+v5dbW3pPo1/WOIPAFVNrH4AO4CLfObNrX6diET5YfenA+tVdZBPDEuA7/nE8Gs/7Nd3/92BgSIywY/7OYafjqff+CHefsDnrRhLOfAqcGs9q/r+XQ1tzr6N/1niDyIROUNEskTkXhH5GviHiHQRkXdEJEdE8r3nqT7rHP66KiI3ichSEfmD99qvRGRajd1cALzbQBy3iMhmbxsfiEg/b76IyB9FZK+IHBCRTd5Z32xgJnCPd2b3dj2bvxF404vhqCYqERkhIv8WkX0isqf6m4eIRIrIT7xvKYUiskZE+ohIfxFR36RUy/FY5sWcB/xMRAaJyP9EJM87U50rIp191u8jIgu8450nIn/1viXtE5FRPq/rLiIHRaRbHcfwW94xLBSRL0RkrDdfRWSwz+sOf1Oq4/e/WUSm+7w+youtenuniMhyESkQkQ0ickYd8fwPOBP4q/c7GiIinUTkBW97mSJyv4hE1HXsam5TVbeq6rM088OkRnwRInKf9zvOE5FXRaSrt6z69zxbRHaJyG4R+ZHPurEi8pi3bJf3PNZn+cXimrgOeNs/32fX/bz3WSgiH4pIirdOnIjM8WIpEJFPReS4lr7PNktV7RHAB7AdONt7fgZQAfwWiAU6AMnA5UA8kAS8Brzhs/5HwDe95zfhzsK+BUQC3wF2AeLz+i3A0Box+G7jYiAdOAGIAu4HlnvLzgPWAJ0B8V7T01v2PPDLBt5rPHAA9+FzOZALxHjLkoDdwA+BOG/6ZG/Z3cAmYKi335O849IfUCCqnuNRAdzhvZcOwGDgHO/4dgMWA495r48ENgB/BBK8OE7zlj0O/NZnP3cCb9fxPq8EsoEJXryDgX7eMgUG+7z28HGr4/f/IDDX5/UXApu9572BPO94RnjvKw/oVkdch4+NN/0C7kM4yTuW24Bb6zp29fxeBwNax/5yvN/zMuCMerZxJ/AJkOq9978D87xl1b/ned7vZZS33er/m59763b3fqfLgV94yyYC+71jE+Eds2E+8WUAQ7xj/RHwsLfsNuBt3N9sJDAO6BjsfOG3PBTsAMLtwbGJvwyIq+f1o4F8n+nD/8zeP2u6z7J47x+mhzc9yHd5Hdt4r/qf35uOAA7imgnO8pLDKUBEjW08T8OJf5b3DxuFS6r7gUu9ZdcC6+pYbytwcS3zqxNCfYl/RwMxXVK9X2BSdXy1vO5kXLOceNOrgavq2OYHwJ11LGso8R/1+8cl1UIg3pueCzzoPb8XeLGWfd9Yx759j02kt6/hPstvAz5q7LGrEaPWccyScIn8Ru99DKpjG5uBqT7TPXEnMVE+v+dhPst/BzzrPc8ALvBZdh6w3Xv+d+CP9RyP+32mbwfe957fgvsAObExxyDUH9bUE3w5qlpSPSEi8SLyd++r+AHcGWpnEYmsY/2vq5+o6kHvaaL38wJcYq9PP+BP3tfbAmAf7qy1t6r+D/gr8Ddgr4g8JSIdm/DebgReVdUK7z3O50hzTx/cP3Bt6lvWkJ2+EyJynIi8LCLZ3vGcA6T47CdTVStqbkRVV+I+AM8QkWG4ZPeWH+I96vevqum4pHiRiMQDM4CXvMX9gCurf1fe7+s0XNJsSAoQDWT6zMvEnRFXO+rYNZWqrlTVQlUtVdcBYRnub7A2/YCFPu9jM1AJ+Dav+MaTCfTynvfi2PdRvayh38XXPs8PcuR/5UXch+jLXvPR70Qkup7thDRL/MFXs3fED3FNHCerakfcxVFwybipGmzfx/1z3aaqnX0eHVR1OYCq/llVxwHDcV+R764j7qOIuy5xFjBLRL722rCvAC7w2lV3AgPriWlQLfOLvZ/xPvN61HhNzbh+7c0b5R3PWRw5ljuBvlL3RdV/eq+/HnjdN0E3Ml5wyaUp8YJr4rgW1wz3hfdhUL2fF2v8rhJU9eE69u0rF3dG3c9nXl9cE1V9sbSEUvff7U5gWo33EqeqvvH0qRHrLu/5Lo59H9XL6vtd1B2oarmqPqSqw4FTgenADU3dTqiwxN/2JAGHgALvYtdPm7MR72xxIrCogZc+CfxYREZ463USkSu95xNE5GTvzKcYKAGqvPX2UHfiBpcst+E+xEZ7jyFAFi6pvQP0FJG7vIt1SSJysrfuM8AvROR4cU4UkWRVzcElqlniLgDfQsP/5ElAEbBfRHpz5IMLYBXuOsPDIpLgXeCb7LN8DnApLvm/UM8+ngF+JCLjvHgHi3eBHFgPXOfFez7wjQbiBXgZOBd3zeYln/lzcN8EzvO2F+ddIE6tdSs+VLUS1yPnV96x7gf8P2+bjeK9tzggxpuOq76oKiKdvbjixF2Qnok7aXm/js096cVS3ZGgm4hcXOM1D3jfgEcANwOvePPnAfd766TgrotUv49ngZtFZKp3Abm3942tofd2poiM8r5ZH8B9SFY1sFrIssTf9jyGu/CUi7uAVdc/TkPOAlbUc5YKgKouxF1cfNlrCvkMqO4Z1BF4GsjHfZ3OA37vLXsWGO59VX+jlk3fCDyuql/7PnD/8DeqaiHuAtxFuK/fabheKACP4pLUh7h/wmdxxwTchey7vVhG4Npl6/MQMBZ3feFfwAKf917p7X8wrj0/C7jaZ/lOYC3uzHVJXTtQ1deAX+GSdCHwBtDVW3ynt48CXE+o2o5Vze3tBlbgzjxf8Zm/E/ct4Ce4axM7cceisf/Hd+A+wL8ElnrxPtfIdcGdZR/iSK+eQ7jrMeCakX7JkYu7dwCXqOq2Orb1J1zT2YciUoj7Wz+5xms+xnU8+C/wB1X90Jv/S9w1l424TgBrvXmo6irch8Qfcb/zjzn620FdegCv4/7eNnvrvdiI9UJS9YUr086IyOPAZ6r6eLBjCWUi8hywS1XvD3Ys4UJE+gNfAdG1XX8xLRdSN7iYJlmP655mmslLQJcBY4IbiTGty5p62ilVfcprMjDNICK/wDV7/V5Vvwp2PMa0Jr819YjIUHzaJ3EXAh/EXWx8EteNajswU1UP+CUIY4wxxwhIG793pTwbd/HmdeBHqvqx1ytjgKo+4PcgjDHGAIFL/OcCP1XVySKyH+isqioifYAPvL6zdUpJSdH+/fv7PU5jjGlP1qxZk6uqx9SXCtTF3WtwfW/BdQW7GNet7UqOvkmjVv3792f16tX+i84YY9ohEcmsbb7fL+6KSAzutvPXvFm3ALeLyBrczTVldaw3W0RWi8jqnJwcf4dpjDFhIxC9eqYBa1V1D4CqblHVc70yAPOoo66G1ytlvKqO79at1kq4xhhjmiEQif9ajjTzICLdvZ8RuBLATwYgBmOMMR6/tvGLSALutvzbfGZfKyLf9Z4vAP7RnG2Xl5eTlZVFSUm9FQlCXlxcHKmpqURHt9tCgcaYAPNr4lfVYtwAGr7z/oSr09EiWVlZJCUl0b9/f0SaU7iy7VNV8vLyyMrKYsCAAcEOxxjTToTsnbslJSUkJye326QPICIkJye3+281xpjACtnED7TrpF8tHN6jMSawQjrxG2NMe5W+t5BfvvMF+cW19nhvEUv8zVRQUMDjjze94vEFF1xAQUGBHyIyxoS6kvJKFq7L4qonV3D2o4t5fvl21mTmt/p+rCxzM1Un/ttvv/2o+RUVFURF1X1Y3323oZEQjTHhZtueQuat2sGCtdnsP1ROv+R47ps2jMvHptItKbbV92eJv5nuu+8+MjIyGD16NNHR0cTFxdGlSxe2bNnCtm3buOSSS9i5cyclJSXceeedzJ49GzhSfqKoqIhp06Zx2mmnsXz5cnr37s2bb75Jhw4dGtizMaY9OFRWybubdjNv1Q5WZ+YTHSmcN6IH103syykDk4mI8N/1vXaR+B96+3O+2NW6lZ2H9+rITy8aUefyhx9+mM8++4z169fz0UcfceGFF/LZZ58d7nb53HPP0bVrVw4dOsSECRO4/PLLSU4+qmcraWlpzJs3j6effpqrrrqK+fPnM2vWrFZ9H8aYtmXL1weYt3IHC9ZlU1hSwYCUBH5ygTu7T05s/bP72rSLxN8WTJw48ai+9n/+859ZuHAhADt37iQtLe2YxD9gwABGjx4NwLhx49i+fXvA4jXGH8orqygqqaCo1OdRUkFhaQXFPs/da8opLq30pssPv7Zn5w5MHpTMqYNTGNO3M7FRkcF+Wy12sKyCdza6s/t1OwqIiYzg/JE9uHZiX04Z2DXgvffaReKv78w8UBISEg4//+ijj/jPf/7DihUriI+P54wzzqi1L35s7JFP98jISA4dOhSQWI2pqaKyirziMgq9pF1cWnH4eXVS9k3eRT7Li0uPTJdWVDVqf4mxUe4RF3X4efekOOJjI8nIKeavi9L58//SiYuOYEL/rkwenMLkQSkM79WRSD82gbS2L3YdYN6qHbyxLpvC0goGdUvg/gtP4LKxqXRNiAlaXO0i8QdDUlIShYWFtS7bv38/Xbp0IT4+ni1btvDJJ58EODpj6lZRWUXa3iI2Ze/ns+z9bMzaz+bdBxpM2jFREST5JOuE2Ch6dIw7krzjokiMObI8KS6KxNhoEmIjDz9PjIsiPjqywfbr/YfKWfllHssz8liWnsvD720BoFOHaCYNTGby4GQmD05hQEpCm7vXpbi0gnc27uKlVTvZsLOAmKgILhzVk2sn9mVC/y5tIl5L/M2UnJzM5MmTGTlyJB06dOC44447vOz888/nySef5IQTTmDo0KGccsopQYzUhLOKyirSc4rYlLWfTdnusXn3AUrKXZJPiIlkRO9OXH9KPwZ0SyApLpokL6kfSd5uOiYqcL2/O3WI5twRPTh3RA8A9h4oOfwhsDwjj/c//xqAnp3iOHVQyuEPguM6xgUsxpo+y97PvFU7eHP9LopKKzi+eyIPTh/OZWN70zk+eGf3tQnICFwtNX78eK05EMvmzZs54YQTghRRYIXTezXN55vkP8vez8Y6kvyo3p04MbUTI3t3YkBygl97j/iDqpKZd5BlGbksS89lRUYe+QfLARjULYHJg1M4dVAKkwYm0ynev8UNi0oreHvDLuat2sHGrP3ERkVw4Yk9uW5iX8b1C/7ZvYisUdXxNefbGb8xIaiisoqMnGJ3Fp9VwKbs/XxRM8n36sTMk/sxqrdL8gNTQi/J10ZE6J+SQP+UBGae3I+qKuWL3QdYnpHLsvQ8XludxQsrMokQGNm70+FvBBP6dyUuunUuFG/K2s9Lq3bw1vpsissqGXpcEj+7aDiXjkn1+4dNa7DEb0wbV1mlpB/VJl9wVJKPj4lkZK9OXDexH6NSOzKqdycGpCSG1EXQloiIEEZ6H26zTx9EWUUV63cWeM1CuTyz5Eue/DiDmMgIxvbrzGmDUzh1cAon9u5EVGTjm68KS8p5c707u/981wHioiO46MReXHtyX8b06Rz0s/umsMRvTBtSWaVk1GiT/2LXAQ6VVwIuyY/o1ZFrJ/blxNROYZfkGyMmKoKJA7oycUBXfnDOEIpLK1j11T6WpeeyLCOPP3y4DT7cRlJsFCcP7Op9I0hhyHGJxyRvVWVD1n7mrdzBWxt2cai8kmE9kvjFxSOYMbo3nTr48ey+rBj+/VM48ycQ37VVN22J35ggyiksZUlaDhu9dvnPfZJ8h2iX5K+Z2IdRXtv8wG6W5JsqITaKM4d158xh3QHIKyplxZd5LEvPY3lGLv/ZvBeAlMRYTh3kegyN69eFFRl5vLRqJ5t3H6BDdCQzTnJn9yeldvL/2X1FKbw8E776GIacB8ef06qbt8RvTICVlFfyn817WLA2m4+35VBZpYeT/NUT+hy++NquknxVJRzMg6K9ULTH/Sze6zO9B4pyoDgHTpgOFz4KEf65cSs5MZbpJ/Zi+om9AMjKP8jy9DzvYnEeb23Ydfi1w3t25JeXjOTi0b1IigtQ231lBcy/Fb5cBBf/rdWTPljiNyYgVJXVmfksWJvFOxt3U1hSQY+Occw+fSDTT+zJsB6hdWMSAKpwKP9I8i7OOTqJ+yb44hzQWu4TiOoAScdBQndIHgQpx8Oa590Z78V/81vy95XaJZ6rJsRz1YQ+qCppe4tYk5nPiF7ueklA2+6rquCtO2Dz23D+wzDGPyVc/Jb4RWQo8IrPrIHAg8BHuAHW44AK4HZVXeWvOPyloKCAl1566ZjqnI3x2GOPMXv2bOLj4/0QmWlLMvOKWbA2mwXrsti57xDxMZGcP7IHl49N5ZSByW0v2atCaWEdSdw3uXtn61Xlx24jMsYl8sTu0CkVeo+BxOPcI6Gb99xbHpMINRPrx7+DRb9ySf+iv0BE4O4fEBGGHJfEkOOSArbPw1Th/ftgw0twxk/glO/4bVd+S/yquhUYDSAikUA2sBB4GnhIVd8TkQuA3wFn+CsOf6mrLHNjPPbYY8yaNcsSfzu1/2A5/9q0mwVrs1idmY8ITB6Uwg/OHsJ5I3qQENvGvmjnZcCSR2D7UpfMK2opHSKRXtL2Enb34e5ndYL3TeZxnY9N5k3xjXugqgI+/i1ERMGFfwxo8g+aRb+GVX+HU77rjoEfBeovcCqQoaqZIqJAR29+J2BX3au1Xb5lmc855xy6d+/Oq6++SmlpKZdeeikPPfQQxcXFXHXVVWRlZVFZWckDDzzAnj172LVrF2eeeSYpKSksWrQo2G/FtILyyioWb8thwdps/r15D2UVVQzunsi95w/jkjG96NmpDZbbzsuAxX+Aja+4s/Sh50PH3kcn8eoz9Q5dA5t8z/ixS/5LHnHJ/4I/tOzDpK1b/hdY/DsYcz2c9yu/v9dAJf5rgHne87uAD0TkD7gRwE6tbQURmQ3MBujbt2/9W3/vPvh6U2vF6vQYBdMernOxb1nmDz/8kNdff51Vq1ahqsyYMYPFixeTk5NDr169+Ne//gW4Gj6dOnXi0UcfZdGiRaSkpLRuzCagVJXPdx1g/tos3lq/i7ziMromxHDdxL5cPjaVkb07ts2+3Ucl/GjXpHDq911be1shAmc9AJXlsPzPLvmf/3D7TP5rnocP74fhl8BFfwrIe/R74heRGGAG8GNv1neAH6jqfBG5CngWOLvmeqr6FPAUuJIN/o6zJT788EM+/PBDxowZA0BRURFpaWlMmTKFH/7wh9x7771Mnz6dKVOmBDlS0xq+3l/CG+uzWbA2i217ioiJjODs4d25bEwq3xjajegm3BQUUPu+dAl/w8su4Z/8bZh8Z9tK+L5E4Jyfux5Bn/zNJf9zf9m+kv9n8+Htu2DwOXDZ0wG5mA2BOeOfBqxV1T3e9I3And7z14BnWr6Hus/MA0FV+fGPf8xtt912zLK1a9fy7rvvcv/99zN16lQefPDBIERoWupgWQUffP41C9ZmszQ9F1UY168Lv7p0JNNH9Wrbt+nv+xIWPwIb5oVGwvcl4po+qipgxV9dYjz7ofaR/Ld9AAtmQ99JcNULEBW4Qm6BSPzXcqSZB1yb/jdwvXvOAtICEEOr8y3LfN555/HAAw8wc+ZMEhMTyc7OJjo6moqKCrp27cqsWbPo3LkzzzzzzFHrWlNP21ZVpXzyZR7z12bz/me7KS6rJLVLB+44czCXjk1lQEpCwxsJpmMS/m1ewu8R7MiaRgSm/dYl/2V/gohoOOv+0E7+25fCqzfAcSPhulcgJrAdPfya+EUkATgH8D0V/hbwJxGJAkrw2vFDjW9Z5mnTpnHdddcxadIkABITE5kzZw7p6encfffdREREEB0dzRNPPAHA7NmzOf/88+nVq5dd3G2D0vcWsXBdFgvXZrNrfwmJsVFMP7EXl43tzYT+Xdt+obN9X8GSP8D6EE/4vkTcBd6qCvfeIqPhjPuCHVXzZK+Bl66Gzv1g1gKI69jwOq3MyjKHgHB6r8GSX1zG2xt3MX9tNht2FhAhcPqQblw2NpVzhx/XalUd/co34UdEwfhb4LS7Qjvh11R9g9P6OXDm/fCNu4MdUdPs3Qz/mAaxHeGW96FjL7/uzsoyG1NDaUUli7bksGBtFou27qW8UjmhZ0fuv/AEZozuRfek4A3q0ST5272LtvNcf/uJs90ZfseewY6s9UVEwIw/uzP/Rb+EyCg47QfBjqpx9n0FL1wCkbFww5t+T/r1scRv2p2qKqXgUDm5RaXkFpaSU1RKblHZ4elcb3p7XjGFJRV0S4rlplP7c+mYVIb3CvzX7marmfAnfBMm39U+E76viEi45HHQSvjPz9y3m1PvCHZU9TuwC164GCpL4eb3oOuAoIYT0olfVdtmP+lWFApNcYFQWaXkH6xO3mXkFJWQW+imDyd2L6nvKy6jourY4xYdKSQnxJKSFENKYiwje3fkvBE9OG1wSpPqsgdd/nZ3Y9P6l8Ir4fuKiIRLnnRn/h/e75K/H0sctEhxnjvTP7gPbnwTuge/2TZkE39cXBx5eXkkJye32+SvquTl5REXFyJNDk1UUVnFvuKyYxJ3rs8Zek6he76vuJRacjkxURF0S4wlJTGGnp3iGNW70+HEXv3o5k136hAd2n8r+ZleG76X8Mff6trwg9hkEFSRUa7ve1WFq3ETEQUTvxXsqI5Wsh/mXAYFmTBrPvQeF+yIgBBO/KmpqWRlZZGTkxPsUPwqLi6O1NTUYIfRKvYfLOf1tVm8uT6brPxD5B8so7YvNHHREYeTdmqXeMb07XxUIk9JjKFbUiwpSbEkxUaFdjJvjPxM7wx/riX8miKj4fLn4LWb4N0fuW8C428JdlRO2UF46RrY8xlc8xL0Py3YER0Wsok/OjqaAQOC205mGlY9gtGcTzJ5e8MuSiuqGN2nM9NG9nBJPCmWbok+Z+hJsSTERLb/ZN4YRyX8CK+Xzg8s4dcUFQNX/gNeuR7e+YE78x97Q3Bjqihz/fR3rIArnnWDqbQhIZv4TdtWXFrBWxt2MeeTTD7fdYCEmEiuGJfKzJP7hdYF1GAo2OES/ro5RxL+5LugU+9gR9Z2RcW6u19fmQlvfd8l/9HXBSeWqkpY8C1I/7ervTPy8uDEUQ9L/KZVbf26kLkrM1m4NpvC0gqG9Ujil5eM5JIxvUkMdjnikgOQs8X1pc7Z4s6oo+MgNsl7dDzyPCbx2HmxiW6+v+qpHE74c90NS+Nudmf4lvAbJzoOrp4D866BN253zWInXR3YGFTh7TvhizdcXaFxNwV2/41kid+0WGlFJe9/9jVzPsnk0+35xERFMH1UT2ae0pexfbsEvtmmrBhytnoJfjPs9ZL9gawjr4mOhy793UhPZUVu8JHyg43b/uEPhaSjnx/+kKhlXm2vi4p1Cb5gByx51DvDF5csLOE3T3QHuGYevHQVvPFt9yE96orA7FvV9TBa9yKcfneb7mJqid80W2ZeMS+t2sFrq7PYV1xG/+R4/u+CE7h8XCpdEwJQcKq8BHK3eWfxX7gEn7PZncnjXTWOjIWUIdDvVOg+zA0g0m2Yu12+Zn35ygooK3QfAqWFUOp9IJQeODKvrJZ5pYVQnHv0fK1sOP6IKPcBUFromnTG3egl/PZxMT9oYuJd/Zu5V7oiaBFRMOIS/+/349+5QnITb4Mz/8//+2sBS/ymSSoqq/jflr3MWbmDxdtyiIwQzjnhOGae0pfJg1L8U8emogz2ZRxJ7nu/cMl+35dHxnGNiILk46HXGBg90yX37sPdWX1kI//MI6OgQxf3aAlVKD9Ux4dE0bEfGrGJri++JfzWE5MA170Kcy53A5dHRMIJF/lvf588AR/92v3thcC4ASFbq8cE1p4DJby8aicvf7qD3ftL6NExjmsn9uXqCX3o0amV7jOorID8r1yzzOFmms2Ql+76aoM7M+460N0E0+0E97P7CdB1UEDL2poQUXLA9aPftR6ufhGGTmv9faybA29+132wXPF84080AsBq9Zgmq6pSlmXkMveTHfx78x4qq5TTh3TjoRkjOGtY9+bf7VpVBQXbjzTN7PXa4XO3uVvaq3Xp75L70Au8RD/MNdtEt88b2owfxHV0N069cInrXnn1XBhybutt//M3XNG4QWfB5c+2qaRfn9CI0gRUfnEZr6/JYu7KTLbnHaRrQgzfnDKA6yb2pV9yC2rQlxbByidg+V+hpODI/I6pLrEPOuPIWXy3oe7rujEtFdcJrl8IL8yAV2bBtfNg8NSWbzf9PzD/m5A60fUmiopt+TYDxBK/AdyNVmt35DP3kx28s2k3ZRVVTOjfhR+cM4TzR/YgNqoFXRjLS2DNP1xBsYO5MGSa+8pdneDjOrXeGzGmNh06w/VvuOT/8nXu4u/AM5q/vcwV8PIs12HguldC7iTFEn+YKyqt4I112cz5JJMtXxeSGBvFNRP6MPPkfgztkdSyjVdWuMqRHz3sulL2nwJTfwp9JrRO8MY0RXxXuP5N+OdFrpTCzNdgQDPGwd613nUX7ZQKsxa6D5UQY4k/TH2x6wBzV2byxrpsissqGdGrI7+5bBQzTupFQktvtKqqcjewLPqVuzDbexxc8reWnWEZ0xoSkl0t/H9Od8l71nzX1bexcra5i8VxneCGNyCxm/9i9SO/JX4RGQq84jNrIPAgMAkY6s3rDBSo6mh/xWGOKCmv5N1Nu5nzSSZrdxQQGxXBRSf1YtYp/TgptVPLb7RShbR/w/9+Dl9vcu311zHRVh4AAB4WSURBVLzkLs628e5tJowkdoMb34bnL3R9/WctgL4nN7xefqarqS+R7sMjhLvf+i3xq+pWYDSAiEQC2cBCVX2s+jUi8giw318xmCO27SnkpudWsWt/CQO7JfDA9OFcPrY3neNbqQtk5nL4789dUaou/eHSp9wdk/4qb2BMSyR2P5L851zuzt5Tj+n1eETh1y7plxfDTe9C8qDAxeoHgWrqmQpkqGpm9Qxxp5dXAWcFKIawtXr7Pm55/lPioiN58daJnDY4pfXKKOxaD//7hevhkNgDLnwUxlxvfepN25fUwyX/f1wAL17qzuJ7jz32dQf3ueVFe91reowMfKytLFDDDl0DzKsxbwqwR1XTaltBRGaLyGoRWd3ea+77038372HmMytJSYxl/ndOZcrx3Von6edsdf2in/oGZK+Bc34B318HE261pG9CR8decNM77m7tFy9xJzK+Sgth7hWQl+G6gbaTjgl+v3NXRGKAXcAIVd3jM/8JIF1VH2loG3bnbvO8tnon9y3YxIheHfnHTRNITmyFfsYFO1wvnQ3zXKGzSd91D+uSaUJZfqZr9ikrct8Ceoxy3ZDnXuGaMa9+EYZdGOwomyyYd+5OA9bWSPpRwGVA2xiHrJ1RVf6++Esefm8LU45P4YlZ41peErlor+uHv/o5VzbhlNtdQbGElNYJ2phg6tLvSJv/Cxe7G74W/Qa2L4XLngrJpF+fQCT+azm2medsYIuqZtXyetMCVVXKr9/dzDNLv+Kik3rxyJUnERPVgha9Q/mw/C+uCFVFKYy9Hk6/x0oGm/an64Ajyf+pM1wBwAsfhROvCnZkrc6viV9EEoBzgNtqLKqtzd+0UHllFfe8vpGF67K56dT+PDh9ePOrZZYVw8onYdmfXKGrUVfAGT8O+d4MxtQreRDc+A68er2rtDnh1mBH5Bd+TfyqWgwk1zL/Jn/uNxwdLKvgO3PW8vG2HO4+byi3nzGoeRdxK0phzfOuWad4ryuvcNb97aIngzGNkjIYbl8R7Cj8yu7cbQfyi8u4+flP2ZhVwMOXjeKaiX2bvpHKCtj4srtwu3+nK69wzVzoM7H1AzbGBJUl/hCXXXCIG55dyc78QzwxaxznjejRtA1UVcHmt1x5hdxt0GsszPiLK69gd9sa0y5Z4g9h2/YUcsOzqyguq+DFWyZy8sBjWtXqpgrp/3XlFXZvcLXur54Dw6ZbwjemnbPEH6LWZO7jludXExsVwau3TeKEnh0bv3LmCq+8wnI39uylf4dRV1p5BWPChCX+QFN13cTqfdT/mhXpOfzsrU2clBTDby8bSc/IbNizs5Zt1NhOWTGs+jukfeiVV3gExtxgd9oaE2Ys8ftbeYkb9Sf9P0Dr3CU9CfggEjgIzGniynGd4eyHYOJsiIlvlXiMMaHFEr8/VVXCgm9C+r9h4m2uHohEeA/xeV7bQ455zUfbcnl74x4GH9eRm08bSFxMdCO34zPd8yQrr2BMmLPE7y+q8N69sPltOO/Xrp5NM1VVKb95bzNPr/+K6SeewS1XndSyoRCNMWHNEr+/LH0UPn0aJn2vRUm/vLKKe1/fyIJ12dw4qR8/vWhE8+/GNcYYLPH7x/qXXK+ZUVe6csXNdLCsgtvnruWjrTn86NwhfPfMwa1XR98YE7Ys8be2tP/Am99zN0Bd/DhENK9AWn5xGbf881M27CzgN5eN4trm3I1rjDG1sMTfmrLXuMFJjhsOV73Y7G6SuwoOccNzq9ix7yCPzxzH+SObeDeuMcbUwxJ/a8nLgLlXQUIyzJwPcU24ocpH2p5CbnhuFUUlFbxwy0ROacrduMYY0wiW+FtDUY4bsFmrYNZCSDquWZtZk5nPLc9/SkxUBK/cNonhvZr34WGMMfWxxN9SpUXw0pVQ+LUbuzNlcLM2s2jLXr4zdw09Osbx4q0n06er3VxljPEPS/wtUVnu2vR3b4RrXoLUY4a2bJT5a7K4Z/5GTuiZxPM3TySlNcbGNcaYOljiby5VeOsOyPgvXPRnGHp+szbz1OIMfv3uFiYPTubJWeNIiotu5UCNMeZoLRiMtX4iMlRE1vs8DojIXd6yO0Rki4h8LiK/81cMfvXfn8OGeXDGT2DcjU1evXps3F+/u4ULT+zJczdNsKRvjAkIv53xq+pWYDSAiEQC2cBCETkTuBg4SVVLRaS7v2Lwm5VPuTtzx90E37inyauXV1Zx7/yNLFibzQ3e3biRdjeuMSZAAtXUMxXIUNVMEfk98LCqlgKo6t4AxdA6vngT3rsHhl4AFzzS5EFLDpZV8N25a1m0NYf/d84Q7jjL7sY1xgSW35p6argGmOc9HwJMEZGVIvKxiEyobQURmS0iq0VkdU5OToDCbEDmcpj/LUidAJc/C5FN+9wsOFjGzGdW8vG2HH596Si+P/V4S/rGmIDze+IXkRhgBvCaNysK6AqcAtwNvCq1ZD9VfUpVx6vq+G7duvk7zIbt3QzzroEu/eC6V5pcy35XwSGueHIFn+86wOMzx3LdyVaCwRgTHIFo6pkGrFXVPd50FrBAVRVYJSJVQArQRk7ra7E/y92gFdUBZs2H+K5NWj19byHXP+vuxv3nzROZNMjuxjXGBE8gmnqu5UgzD8AbwJkAIjIEiAFyAxBH8xzKhzlXQGkhzHodOjf9TP1Hr22kvLKKl287xZK+MSbo/Jr4RSQBOAdY4DP7OWCgiHwGvAzc6J39tz3lJfDyTMhLh6vnQI9RTd7EvuIyNmQVcMOk/ozoZSNfGWOCz69NPapaDCTXmFcGzPLnfltFVSUs+BZkLnMXcgd+o1mbWZaeiypMOT6llQM0xpjmCVSvntByeNjEt9ywiaOuaPamlqTl0DEuihNTO7digMYY03yW+GvTSsMmqipL03KZPDjFbtAyxrQZlvhraqVhEwEycorZtb+E06yZxxjThjSY+EXkIhEJjw+IVho2sdqSNNdD9fTj28B9CMYY42lMZrsaSBOR34nIMH8HFDStNGyir6VpufRPjrfa+saYNqXBxK+qs4AxQAbwvIis8MopJPk9ukBppWETfZVVVLHiyzxr5jHGtDmNastQ1QPA67h+9z2BS4G1InKHH2MLjFYaNrGmdTvyOVhWyRRr5jHGtDGNaeOfISILgY+AaGCiqk4DTgJ+6N/w/Mx32MSZrzV72MTaLEnLJTJC7E5dY0yb05gbuC4H/qiqi31nqupBEbnVP2EFQCsNm1iXJem5jO7TmY42uIoxpo1pTFPPz4BV1RMi0kFE+gOo6n/9EpW/+Q6bOP2PzR42sS4FB8vYmFVgd+saY9qkxiT+14Aqn+lKjpRYDk0tHDaxIcvS86xMgzGmzWpM4o/y6usAh2vttLyvY7C0cNjExliankNSbBQnWZkGY0wb1JjEnyMiM6onRORi2nIZ5fq0cNjExlBVFm/LZdKgZKIiw+O+N2NMaGlMZvo28BMR2SEiO4F7gdv8G5YftHDYxMbanneQ7IJDTBli3TiNMW1Tg9lPVTOAU0Qk0Zsu8ntUra2FwyY2RXWZhimDrX3fGNM2Neq0V0QuBEYAcdXD46rqz/0YV+tp4bCJTbV4Wy59unagX7KVaTDGtE2NuYHrSVy9njsAAa4E+vk5rtbRCsMmNkV5ZRWffJnHlOO7Ucv48cYY0yY0po3/VFW9AchX1YeAScCQhlYSkaEist7ncUBE7hKRn4lIts/8C1r6Jur07j0tGjaxqdbvLKCotMKaeYwxbVpjmnpKvJ8HRaQXkIer11MvVd0KjAYQkUggG1gI3Iy7E/gPzYq4Kc75uRs9q5nDJjbVkrRcIgROHWSJ3xjTdjUm8b8tIp2B3wNrAQWebuJ+pgIZqpoZ0CaQjj3dI0CWpOVwYmpnOsVbmQZjTNtVb1OPNwDLf1W1QFXn49r2h6nqg03czzXAPJ/p74nIRhF5TkS61LHv2SKyWkRW5+TkNHF3gbf/UDkbdhZwut2ta4xp4+pN/KpaBfzNZ7pUVfc3ZQciEgPM4EiZhyeAQbhmoN3AI3Xs+ylVHa+q47t1a/t94ldk5FKlcJqVYTbGtHGNubj7XxG5XJrfRjMNWKuqewBUdY+qVnofKk8DE5u53TZlcVouCTGRjOlrZRqMMW1bYxL/bbiz9VKvZ06hiBxowj6uxaeZR0R8G90vBT5rwrbarKVpuUwalEK0lWkwxrRxjblzt9lDLIpIAnAOR5d4+J2IjMZdJN5OKJZ/qCEzr5gd+w5y62kDgh2KMcY0qMHELyKn1za/5sAsdbymGEiuMe/6RkcXIpakuZp1VobZGBMKGtOd826f53G4Nvk1wFl+iSgELUnLoXfnDgxISQh2KMYY06DGNPVc5DstIn2Ax/wWUYipqKxieUYeF47qaWUajDEhoTlXIrOAE1o7kFC1IWs/hSUVTLFunMaYENGYNv6/4C7EgvugGI27g9fgmnlE4NRByQ2/2Bhj2oDGtPGv9nleAcxT1WV+iifkLE3L5cTeneiSELqjURpjwktjEv/rQImqVoIruCYi8ap60L+htX0HSspZt7OAb39jYLBDMcaYRmvUnbtAB5/pDsB//BNOaPkkI4/KKrX2fWNMSGlM4o/zHW7Re27DS+H678fHRDK2b6115owxpk1qTOIvFpGx1RMiMg445L+QQseStBxOGZhMTJSVaTDGhI7GtPHfBbwmIrtwQy/2wA3FGNZ27jvI9ryD3Hhq/2CHYowxTdKYG7g+FZFhwFBv1lZVLfdvWG2flWkwxoSqxgy2/l0gQVU/U9XPgEQRud3/obVtS9Nz6NkpjkHdEoMdijHGNEljGqe/paoF1ROqmg98y38htX2VVcqy9DxOG5xiZRqMMSGnMYk/0ncQFm/g9LC+W2lT9n72HypnyhDrxmmMCT2Nubj7PvCKiPzdm74NeM9/IbV9S7a5MYAnW5kGY0wIakzivxeYDXzbm96I69kTtpak5TKyd0eSE2ODHYoxxjRZg0093ti4K3GjZU3E1eHf7N+w2q6i0grW7si3u3WNMSGrzsQvIkNE5KcisgX4C7ADQFXPVNW/NrRhERkqIut9HgdE5C6f5T8UERWRkOoP+UlGHhVVypTBIRW2McYcVl9TzxZgCTBdVdMBROQHjd2wqm7FlXCuviCcDSz0pvsA5+J9mISSpem5xEVHMK6/lWkwxoSm+pp6LgN2A4tE5GkRmYq7c7c5pgIZqprpTf8RuIcjdf5DxuK0HE4ekExsVGSwQzHGmGapM/Gr6huqeg0wDFiEK93QXUSeEJFzm7ifa4B5ACJyMZCtqhvqW0FEZovIahFZnZOT08Td+Ud2wSG+zCm2u3WNMSGtMRd3i1X1JW/s3VRgHa6nT6OISAwwA1fvJx74CfBgI/b7lKqOV9Xx3bq1jQupS9PcB9Dp1n/fGBPCmlRWUlXzvYQ8tQmrTQPWquoeYBAwANggIttxHyRrRSQkuocuTsvluI6xHN/dyjQYY0JXY/rxt9S1eM08qroJ6F69wEv+41U1NwBxtIgr05DL1GHHWZkGY0xI82sheRFJAM4BFvhzP4Hw+a79FBwst/Z9Y0zI8+sZv6oWA3XWNVDV/v7cf2uqLsM82frvG2NCnA0d1UhL0nI4oWdHuiVZmQZjTGizxN8IxaUVrMnM53Rr5jHGtAOW+Bth1Vf7KK9Uq89jjGkXLPE3wuK0HGKjIhhvZRqMMe2AJf5GWJqWy8QBXYmLtjINxpjQZ4m/Abv3HyJtb5F14zTGtBuW+Buw1OvGae37xpj2whJ/A5ak5ZKSGMuwHknBDsUYY1qFJf56VFUpS9NzmXJ8ipVpMMa0G5b46/HF7gPsKy6z9n1jTLtiib8e1WUaTrMyDcaYdsQSfz2WpucwrEcS3TvGBTsUY4xpNZb463CorJJPv8q3s31jTLtjib8OK7/Ko6yyiik22pYxpp2xxF+HpWm5xERFMLF/12CHYowxrcoSfx2WpOUyoX8XOsRYmQZjTPtiib8Wew+UsHVPod2ta4xpl/w2ApeIDAVe8Zk1EHgQNyLXxUAVsBe4SVV3+SuO5rBunMaY9sxviV9VtwKjAUQkEsgGFgL5qvqAN//7uA+Db/srjuZYmp5LckIMw3t2DHYoxhjT6vw65q6PqUCGqmbWmJ8AaIBiaJSqKmVJWi6TB6cQEWFlGowx7U+gEv81wLzqCRH5FXADsB84s7YVRGQ2MBugb9++AQjR2fJ1IblFpVamwRjTbvn94q6IxAAzgNeq56nq/6lqH2Au8L3a1lPVp1R1vKqO79YtcBdZl6bnAFaG2RjTfgWiV880YK2q7qll2Vzg8gDE0GhL0nI5vnsiPTpZmQZjTPsUiMR/LUc38xzvs+xiYEsAYmiUkvJKVn21z872jTHtml/b+EUkATgHuM1n9sNeV88qIJM21KPn0+37KK2osvZ9Y0y75tfEr6rFuH77vvPaVNOOr6VpuURHCicPtDINxpj2y+7c9bE4LZdx/boQHxOozk7GGBN4lvg9OYWlbN59wNr3jTHtniV+z7J0V6bhdEv8xph2zhK/Z3FaDl3ioxnRy8o0GGPaN0v8gKqy1Mo0GGPChCV+YNueIvYWWpkGY0x4sMQPLElzZRpOs/Z9Y0wYsMSPK9MwqFsCvTt3CHYoxhjjd2Gf+EvKK1n5VZ514zTGhI2wT/xrM/MpKbcyDcaY8BH2iX9xWi5REcLJA5MbfrExxrQDYZ/4l6bnMLZfFxJjrUyDMSY8hHXizysq5bPsA0yxQdWNMWEkrBP/Uq9Mw5QhdmHXGBM+wjvxp+XSqUM0o3p3CnYoxhgTMGGb+FWVJWm5TB6cTKSVaTDGhJGwTfwZOUV8faDE+u8bY8KO37qyeMMrvuIzayDwINAbuAgoAzKAm1W1wF9x1GXxNte+f5pd2DXGhBm/nfGr6lZVHa2qo4FxwEFgIfBvYKSqnghsA37srxjqszQ9lwEpCfTpGh+M3RtjTNAEqqlnKpChqpmq+qGqVnjzPwFSAxTDYaUVlazIyLOzfWNMWApU4r8GmFfL/FuA92pbQURmi8hqEVmdk5PTqsGszSzgUHmllWkwxoQlvyd+EYkBZgCv1Zj/f0AFMLe29VT1KVUdr6rju3Vr3QuwS9NziIwQJg2yMg3GmPATiDoF04C1qrqneoaI3ARMB6aqqgYghqMsSctlTJ/OJMVFB3rXxhgTdIFo6rkWn2YeETkfuAeYoaoHA7D/o+QXl7Epe7914zTGhC2/Jn4RSQDOARb4zP4rkAT8W0TWi8iT/oyhpmUZuajCada+b4wJU35t6lHVYiC5xrzB/txnQ5ZsyyUpLoqTUq1MgzEmPIXVnbuqytL0XE4dlExUZFi9dWOMOSysst+XucVkFxyy9n1jTFgLq8S/NM2VaTjdEr8xJoyFVeJfkpZD367x9E22Mg3GmPAVNom/vLKKFRl5dreuMSbshU3iX7ejgOIyK9NgjDFhk/iXpOUQITBpkCV+Y0x4C6PEn8voPp3p1MHKNBhjwltYJP79B8vZmFXAadabxxhjwiPxL8/IpUrhdGvfN8aY8Ej8i9NySYyN4qQ+nYMdijHGBF27T/yqypK0HCYNSibayjQYY0z7T/yZeQfJyj9k3TiNMcbT7hP/kjQ3bKPV5zHGGCcMEn8uqV060N/KNBhjDNDOE3+FT5kGEQl2OMYY0ya068S/IauAwtIKa+Yxxhgffkv8IjLUG1qx+nFARO4SkStF5HMRqRKR8f7aP8DibbmIwKmDkht+sTHGhAm/Db2oqluB0QAiEglkAwuBeOAy4O/+2ne1Xp3juHJcKp3jY/y9K2OMCRl+HXPXx1QgQ1Uzq2cEos396gl9uXpCX7/vxxhjQkmg2vivAeY1ZQURmS0iq0VkdU5Ojp/CMsaY8OP3xC8iMcAM4LWmrKeqT6nqeFUd362bXZw1xpjWEogz/mnAWlXdE4B9GWOMaUAgEv+1NLGZxxhjjP/4NfGLSAJwDrDAZ96lIpIFTAL+JSIf+DMGY4wxR/Nrrx5VLQaSa8xbiOvWaYwxJgja9Z27xhhjjmWJ3xhjwoyoarBjaJCI5ACZDb6wdilAbiuGE+rseBxhx+JodjyO1h6ORz9VPaY/fEgk/pYQkdWq6teaQKHEjscRdiyOZsfjaO35eFhTjzHGhBlL/MYYE2bCIfE/FewA2hg7HkfYsTiaHY+jtdvj0e7b+I0xxhwtHM74jTHG+LDEb4wxYaZdJ34ROV9EtopIuojcF+x4gkVE+ojIIhH5whv28s5gx9QWiEikiKwTkXeCHUuwiUhnEXldRLaIyGYRmRTsmIJFRH7g/Z98JiLzRCQu2DG1tnab+L3hHv+GKws9HLhWRIYHN6qgqQB+qKrDgVOA74bxsfB1J7A52EG0EX8C3lfVYcBJhOlxEZHewPeB8ao6EojEDSTVrrTbxA9MBNJV9UtVLQNeBi4OckxBoaq7VXWt97wQ90/dO7hRBZeIpAIXAs8EO5ZgE5FOwOnAswCqWqaqBcGNKqiigA4iEoUbI3xXkONpde058fcGdvpMZxHmyQ5ARPoDY4CVwY0k6B4D7gGqgh1IGzAAyAH+4TV9PeOVVA87qpoN/AHYAewG9qvqh8GNqvW158RvahCRRGA+cJeqHgh2PMEiItOBvaq6JtixtBFRwFjgCVUdAxQDYXlNTES64FoGBgC9gAQRmRXcqFpfe0782UAfn+lUb15YEpFoXNKfq6oLGnp9OzcZmCEi23FNgGeJyJzghhRUWUCWqlZ/C3wd90EQjs4GvlLVHFUtxw0idWqQY2p17TnxfwocLyIDvAHfrwHeCnJMQSEigmu/3ayqjwY7nmBT1R+raqqq9sf9XfxPVdvdWV1jqerXwE4RGerNmgp8EcSQgmkHcIqIxHv/N1Nphxe6/ToCVzCpaoWIfA/4AHdl/jlV/TzIYQXLZOB6YJOIrPfm/URV3w1iTKZtuQOY650kfQncHOR4gkJVV4rI68BaXG+4dbTD0g1WssEYY8JMe27qMcYYUwtL/MYYE2Ys8RtjTJixxG+MMWHGEr8xxoQZS/ymTRERFZFHfKZ/JCI/a6VtPy8iV7TGthrYz5VehctF/t5Xjf3eJCJ/DeQ+TWiyxG/amlLgMhFJCXYgvryCXY11K/AtVT3TX/EY0xKW+E1bU4G7YeYHNRfUPGMXkSLv5xki8rGIvCkiX4rIwyIyU0RWicgmERnks5mzRWS1iGzzavZU1+X/vYh8KiIbReQ2n+0uEZG3qOVOVhG51tv+ZyLyW2/eg8BpwLMi8vta1rnbZz8PefP6e3Xw53rfFF4XkXhv2VSvcNomEXlORGK9+RNEZLmIbPDeZ5K3i14i8r6IpInI73ze3/NenJtE5Jhja8JLu71z14S0vwEbqxNXI50EnADsw915+oyqTvQGnbkDuMt7XX9cye5BwCIRGQzcgKvCOMFLrMtEpLoi41hgpKp+5bszEekF/BYYB+QDH4rIJar6cxE5C/iRqq6usc65wPHe/gV4S0ROx5UJGArcqqrLROQ54Hav2eZ5YKqqbhORF4DviMjjwCvA1ar6qYh0BA55uxmNq75aCmwVkb8A3YHeXn15RKRzE46raYfsjN+0OV7l0BdwA2I01qfeuAOlQAZQnbg34ZJ9tVdVtUpV03AfEMOAc4EbvHIWK4FkXIIGWFUz6XsmAB95xbwqgLm4mvb1Odd7rMOVBBjms5+dqrrMez4H961hKK5g2DZv/j+9fQwFdqvqp+COlxcDwH9Vdb+qluC+pfTz3udAEfmLiJwPhG1lVuPYGb9pqx7DJcd/+MyrwDtZEZEIIMZnWanP8yqf6SqO/juvWaNEcWffd6jqB74LROQMXIni1iLAb1T17zX207+OuJrD9zhUAlGqmi8iJwHnAd8GrgJuaeb2TTtgZ/ymTVLVfcCruAul1bbjmlYAZgDRzdj0lSIS4bX7DwS24gr5fccrXY2IDGnEQCSrgG+ISIq4YT6vBT5uYJ0PgFu8cREQkd4i0t1b1leOjHN7HbDUi62/1xwFrtDex978niIywdtOUn0Xn70L5RGqOh+4n/AtuWw8dsZv2rJHgO/5TD8NvCkiG4D3ad7Z+A5c0u4IfFtVS0TkGVxz0FqvFG8OcEl9G1HV3SJyH7AIdyb/L1V9s4F1PhSRE4AVbjcUAbNwZ+ZbcWMhP4dronnCi+1m4DUvsX8KPKmqZSJyNfAXEemAa98/u55d98aNrlV9ovfj+uI07Z9V5zQmyLymnneqL74a42/W1GOMMWHGzviNMSbM2Bm/McaEGUv8xhgTZizxG2NMmLHEb4wxYcYSvzHGhJn/DxO2CLfi6wB9AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light", "tags": [] }, "output_type": "display_data" } ], "source": [ "figureName = 'figure' # change figure name\n", "\n", "plt.plot(results['epoch'].values, train_accuracy, label='train')\n", "plt.plot(results['epoch'].values, test_accuracy, label='test')\n", "plt.xlabel('Number of epochs')\n", "plt.ylabel('Accuracy')\n", "plt.title(f'Train/Test Accuracy curve for {max_epochs} epochs')\n", "plt.savefig(f'/content/results/{figureName}.png')\n", "plt.legend()\n", "plt.show()" ] } ], "metadata": { "accelerator": "GPU", "colab": { "collapsed_sections": [], "include_colab_link": true, "name": "transfer_learning", "provenance": [], "toc_visible": true }, "kernel": { "display_name": "Python 3", "language": "python", "name": "python3" }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "name": "python" } }, "nbformat": 4, "nbformat_minor": 0 }