Open In Colab   Open in Kaggle

Tutorial 1: Introduction to Continual Learning

Week 3, Day 4: Continual Learning

By Neuromatch Academy

Content creators: Keiland Cooper, Diganta Misra, Gido van de Ven, Andrea Cossu, Vincenzo Lomonaco

Content reviewers: Arush Tagade, Jeremy Forest, Siwei Bai, Kelson Shilling-Scrivo

Content editors: Gagana B, Anoop Kulkarni, Spiros Chavlis

Production editors: Deepak Raya, Spiros Chavlis

Post-Production team: Gagana B, Spiros Chavlis

Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs


Tutorial Objectives

In this tutorial, we’ll dive head-first into the exciting field of continual learning (CL). CL has gained increasing attention in recent years, and for good reason. CL is positioned as a problem across sub-disciplines, from academia to industry, and may promise to be a major pathway towards strong artificial intelligence (AI). As datasets get bigger and AI gets smarter, we’re expecting more and more cognitive capabilities from our machines.

We have a few specific objectives for this tutorial:

  • Introduce major CL concepts

  • Introduce the most common strategies to aid CL

  • Utilize benchmarks and evaluation metrics

  • Explore present day applications of CL

Tutorial slides

These are the slides for the videos in this tutorial. If you want to locally download the slides, click here.


Setup

First, let’s load in some useful packages and functions. We’ll primarily be using PyTorch as our neural network framework of choice. Be sure to run all the cells below so the code runs properly.

Install dependencies

# @title Install dependencies
!pip install seaborn --quiet

!pip install git+https://github.com/NeuromatchAcademy/evaltools --quiet
from evaltools.airtable import AirtableForm

# Generate airtable form
atform = AirtableForm('appn7VdPRseSoMXEG','W3D4_T1','https://portal.neuromatchacademy.org/api/redirect/to/9c55f6cb-cdf9-4429-ac1c-ec44fe64c303')
# Imports
import numpy as np
from scipy.stats import norm
import matplotlib.pyplot as plt
import math
import pandas as pd

import torch
import torch.nn as nn
import torch.optim as optim
import torch.nn.functional as F

import torchvision.datasets as datasets
import torchvision.transforms as transforms

Figure settings

# @title Figure settings
import ipywidgets as widgets       # Interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/content-creation/main/nma.mplstyle")

Plotting functions

# @title Plotting functions

def plot_mnist(data, nPlots=10):
  """
  Plot MNIST-like data

  Args:
    data: torch.tensor
      MNIST like data to be plotted
    nPlots: int
      Number of samples to be plotted aka Number of plots

  Returns:
    Nothing
  """
  plt.figure(figsize=(12, 8))
  for ii in range(nPlots):
    plt.subplot(1, nPlots, ii + 1)
    plt.imshow(data[ii, 0], cmap="gray")
    plt.axis('off')
  plt.tight_layout
  plt.show()


def multi_task_barplot(accs, tasks, t=None):
  """
  Plot accuracy of multiple tasks

  Args:
    accs: list
      List of accuracies per task
    tasks: list
      List of tasks

  Returns:
    Nothing
  """
  nTasks = len(accs)
  plt.bar(range(nTasks), accs, color='k')
  plt.ylabel('Testing Accuracy (%)', size=18)
  plt.xticks(range(nTasks),
            [f"{TN}\nTask {ii + 1}" for ii, TN in enumerate(tasks.keys())],
            size=18)
  plt.title(t)
  plt.show()


def plot_task(data, samples_num):
  """
  Plots task accuracy

  Args:
    data: torch.tensor
      Data of task to be plotted
    samples_num: int
      Number of samples corresponding to data for task

  Returns:
    Nothing
  """
  plt.plot(figsize=(12, 6))
  for ii in range(samples_num):
    plt.subplot(1, samples_num, ii + 1)
    plt.imshow(data[ii][0], cmap="gray")
    plt.axis('off')
  plt.show()

Helper functions

# @title Helper functions

def load_mnist(mnist_train, mnist_test, verbose=False, asnumpy=True):
  """
  Helper function to load MNIST data
  Note: You can try an alternate implementation with torchloaders

  Args:
    mnist_train: np.ndarray
      MNIST training data
    mnist_test: np.ndarray
      MNIST test data
    verbose: boolean
      If True, print statistics
    asnumpy: boolean
      If true, MNIST data is passed as np.ndarray

  Returns:
   X_test: np.ndarray
      Test data
    y_test: np.ndarray
      Labels corresponding to above mentioned test data
    X_train: np.ndarray
      Train data
    y_train: np.ndarray
      Labels corresponding to above mentioned train data
  """

  x_traint, t_traint = mnist_train.data, mnist_train.targets
  x_testt, t_testt = mnist_test.data, mnist_test.targets

  if asnumpy:
    # Fix dimensions and convert back to np array for code compatability
    # We aren't using torch dataloaders for ease of use
    x_traint = torch.unsqueeze(x_traint, 1)
    x_testt = torch.unsqueeze(x_testt, 1)
    x_train, x_test = x_traint.numpy().copy(), x_testt.numpy()
    t_train, t_test = t_traint.numpy().copy(), t_testt.numpy()
  else:
    x_train, t_train = x_traint, t_traint
    x_test, t_test = x_testt, t_testt

  if verbose:
    print(f"x_train dim: {x_train.shape} and type: {x_train.dtype}")
    print(f"t_train dim: {t_train.shape} and type: {t_train.dtype}")
    print(f"x_train dim: {x_test.shape} and type: {x_test.dtype}")
    print(f"t_train dim: {t_test.shape} and type: {t_test.dtype}")

  return x_train, t_train, x_test, t_test


def permute_mnist(mnist, seed, verbose=False):
    """
    Given the training set, permute pixels of each
    image.

    Args:
      mnist: np.ndarray
        MNIST Data to be permuted
      seed: int
        Set seed for reproducibility
      verbose: boolean
        If True, print statistics

    Returns:
      perm_mnist: List
        Permutated set of pixels for each incoming image
    """

    np.random.seed(seed)
    if verbose: print("Starting permutation...")
    h = w = 28
    perm_inds = list(range(h*w))
    np.random.shuffle(perm_inds)
    perm_mnist = []
    for set in mnist:
        num_img = set.shape[0]
        flat_set = set.reshape(num_img, w * h)
        perm_mnist.append(flat_set[:, perm_inds].reshape(num_img, 1, w, h))
    if verbose: print("done.")
    return perm_mnist

Set random seed

Executing set_seed(seed=seed) you are setting the seed

# @title Set random seed

# @markdown Executing `set_seed(seed=seed)` you are setting the seed

# for DL its critical to set the random seed so that students can have a
# baseline to compare their results to expected results.
# Read more here: https://pytorch.org/docs/stable/notes/randomness.html

# Call `set_seed` function in the exercises to ensure reproducibility.
import random
import torch

def set_seed(seed=None, seed_torch=True):
  """
  Function that controls randomness. NumPy and random modules must be imported.

  Args:
    seed : Integer
      A non-negative integer that defines the random state. Default is `None`.
    seed_torch : Boolean
      If `True` sets the random seed for pytorch tensors, so pytorch module
      must be imported. Default is `True`.

  Returns:
    Nothing.
  """
  if seed is None:
    seed = np.random.choice(2 ** 32)
  random.seed(seed)
  np.random.seed(seed)
  if seed_torch:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

  print(f'Random seed {seed} has been set.')


# In case that `DataLoader` is used
def seed_worker(worker_id):
  """
  DataLoader will reseed workers following randomness in
  multi-process data loading algorithm.

  Args:
    worker_id: integer
      ID of subprocess to seed. 0 means that
      the data will be loaded in the main process
      Refer: https://pytorch.org/docs/stable/data.html#data-loading-randomness for more details

  Returns:
    Nothing
  """
  worker_seed = torch.initial_seed() % 2**32
  np.random.seed(worker_seed)
  random.seed(worker_seed)

Set device (GPU or CPU). Execute set_device()

# @title Set device (GPU or CPU). Execute `set_device()`
# especially if torch modules used.

# Inform the user if the notebook uses GPU or CPU.

def set_device():
  """
  Set the device. CUDA if available, CPU otherwise

  Args:
    None

  Returns:
    Nothing
  """
  device = "cuda" if torch.cuda.is_available() else "cpu"
  if device != "cuda":
    print("GPU is not enabled in this notebook. \n"
          "If you want to enable it, in the menu under `Runtime` -> \n"
          "`Hardware accelerator.` and select `GPU` from the dropdown menu")
  else:
    print("GPU is enabled in this notebook. \n"
          "If you want to disable it, in the menu under `Runtime` -> \n"
          "`Hardware accelerator.` and select `None` from the dropdown menu")

  return device
SEED = 2021
set_seed(seed=SEED)
DEVICE = set_device()
Random seed 2021 has been set.
GPU is not enabled in this notebook. 
If you want to enable it, in the menu under `Runtime` -> 
`Hardware accelerator.` and select `GPU` from the dropdown menu

Data-loader MNIST dataset

# @title Data-loader MNIST dataset
import tarfile, requests, os
from torchvision import transforms
from torchvision.datasets import MNIST

name = 'MNIST'
fname = name + '.tar.gz'
url = 'https://www.di.ens.fr/~lelarge/MNIST.tar.gz'

if not os.path.exists(name):
  print('\nDownloading and unpacking MNIST data. Please wait a moment...')
  r = requests.get(url, allow_redirects=True)
  with open(fname, 'wb') as fh:
    fh.write(r.content)
  with tarfile.open(fname) as tar:
    tar.extractall('./')  # Specify which folder to extract to
  os.remove(fname)
  print('\nDownloading MNIST completed.')
else:
  print('MNIST has been already downloaded.')


# Load the Data
mnist_train = MNIST('./', download=False,
                    transform=transforms.Compose([transforms.ToTensor(), ]),
                    train=True)
mnist_test = MNIST('./', download=False,
                    transform=transforms.Compose([transforms.ToTensor(), ]),
                   train=False)
Downloading and unpacking MNIST data. Please wait a moment...
Downloading MNIST completed.

Section 0: Overview

Video 0: Overview of the Session and Introduction to Continual Learning


Section 1: The sequential learning problem: catastrophic forgetting

Time estimate: ~20mins

Video 1: Introduction to catastrophic forgetting

Here, we’ll explore catastrophic forgetting first hand - a key barrier preventing continual learning in neural networks. To do so, we’ll build a simple network model and try our best to teach it the trusty MNIST dataset

Section 1.1: A brief example of catastrophic forgetting

Let’s define a simple CNN that can perform fairly well on MNIST. We’ll also load in some training and testing functions we wrote to load the data into the model and train/test it. We don’t need to get into the details how they work for now (pretty standard) but feel free to double click the cell if you’re curious!

class Net(nn.Module):
  """
  Simple MultiLayer CNN with following attributes and structure.
  nn.Conv2d(1, 10, kernel_size=5) # First Convolutional Layer
  nn.Conv2d(10, 20, kernel_size=5) # Second Convolutional Layer [add dropout]
  nn.Linear(320, 50) # First Fully Connected Layer
  nn.Linear(50, 10) # Second Fully Connected Layer
  """

  def __init__(self):
    """
    Initialize Multilayer CNN parameters

    Args:
      None

    Returns:
      Nothing
    """
    super(Net, self).__init__()
    self.conv1 = nn.Conv2d(1, 10, kernel_size=5)
    self.conv2 = nn.Conv2d(10, 20, kernel_size=5)
    self.conv2_drop = nn.Dropout2d()
    self.fc1 = nn.Linear(320, 50)
    self.fc2 = nn.Linear(50, 10)

  def forward(self, x):
    """
    Forward pass of network

    Args:
      x: np.ndarray
        Input data

    Returns:
      x: np.ndarray
        Output from final fully connected layer
    """
    x = F.relu(F.max_pool2d(self.conv1(x), 2))
    x = F.relu(F.max_pool2d(self.conv2_drop(self.conv2(x)), 2))
    x = x.view(-1, 320)
    x = F.relu(self.fc1(x))
    x = F.dropout(x, training=self.training)
    x = self.fc2(x)
    return x

Model Training and Testing Functions [RUN ME!]

# @title Model Training and Testing Functions [RUN ME!]

# @markdown `train(model, x_train, t_train, optimizer, epoch, device)`
def train(model, x_train, t_train, optimizer, epoch, device):
  """
  Train function

  Args:
    model: Net() type
      Instance of the multilayer CNN
    x_train: np.ndarray
      Training data
    t_train: np.ndarray
      Labels corresponding to the training data
    optimizer: torch.optim type
      Implements Adam algorithm.
    epoch: int
      Number of epochs
    device: string
      CUDA/GPU if available, CPU otherwise

  Returns:
    Nothing
  """
  model.train()

  for start in range(0, len(t_train)-1, 256):
    end = start + 256
    x = torch.from_numpy(x_train[start:end])
    if torch.cuda.is_available():
      x = x.type(torch.cuda.FloatTensor)
    else:
      x = x.type(torch.FloatTensor)
    y = torch.from_numpy(t_train[start:end]).long()
    x, y = x.to(device), y.to(device)

    optimizer.zero_grad()

    output = model(x)
    loss = F.cross_entropy(output, y)
    loss.backward()
    optimizer.step()
  print('Train Epoch: {} \tLoss: {:.6f}'.format(epoch, loss.item()))


# @markdown `test(model, x_test, t_test, device)`
def test(model, x_test, t_test, device):
    """
    Test function.

    Args:
      model: Net() type
        Instance of the multilayer CNN
      x_test: np.ndarray
        Test data
      t_test: np.ndarray
        Labels corresponding to the test data
      device: string
        CUDA/GPU if available, CPU otherwise

    Returns:
      Nothing
    """
    model.eval()
    correct, test_loss = 0, 0
    for start in range(0, len(t_test)-1, 256):
      end = start + 256
      with torch.no_grad():
        x = torch.from_numpy(x_test[start:end])
        if torch.cuda.is_available():
          x = x.type(torch.cuda.FloatTensor)
        else:
          x = x.type(torch.FloatTensor)
        y = torch.from_numpy(t_test[start:end]).long()
        x, y = x.to(device), y.to(device)
        output = model(x)
        test_loss += F.cross_entropy(output, y).item()  # Sum up batch loss
        pred = output.max(1, keepdim=True)[1]  # Get the index of the max logit
        correct += pred.eq(y.view_as(pred)).sum().item()

    test_loss /= len(t_train)
    print('Test set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(
        test_loss, correct, len(t_test),
        100. * correct / len(t_test)))
    return 100. * correct / len(t_test)
class simpNet(nn.Module):
  """
  Defines a simple neural network with following configuration:
  nn.Linear(28*28, 320) # First Fully Connected Layer
  nn.Linear(320, 10) + ReLU # Second Fully Connected Layer
  """

  def __init__(self):
    """
    Initialize SimpNet Parameters

    Args:
      None

    Returns:
      Nothing
    """
    super(simpNet,self).__init__()
    self.linear1 = nn.Linear(28*28, 320)
    self.out = nn.Linear(320, 10)
    self.relu = nn.ReLU()

  def forward(self, img):
    """
    Forward pass of SimpNet

    Args:
      img: np.ndarray
        Input data

    Returns:
      x: np.ndarray
        Output from final fully connected layer
    """
    x = img.view(-1, 28*28)
    x = self.relu(self.linear1(x))
    x = self.out(x)
    return x

Now let’s load our dataset: MNIST. We’ll also run a function we defined in the helper function cell above that permutes (scrambles) the images. This allows us to create additional datasets with similar statistics to MNIST on the fly. We’ll call the normal MNIST Task 1, and the permuted MNIST Task 2. We’ll see why in a second!

# Load in MNIST and create an additional permuted dataset
x_train, t_train, x_test, t_test = load_mnist(mnist_train, mnist_test,
                                              verbose=True)
x_train2, x_test2 = permute_mnist([x_train, x_test], 0, verbose=False)

# Plot the data to see what we're working with
print('\nTask 1: MNIST Training data:')
plot_mnist(x_train, nPlots=10)
print('\nTask 2: Permuted MNIST data:')
plot_mnist(x_train2, nPlots=10)
x_train dim: (60000, 1, 28, 28) and type: uint8
t_train dim: (60000,) and type: int64
x_train dim: (10000, 1, 28, 28) and type: uint8
t_train dim: (10000,) and type: int64
Task 1: MNIST Training data:
../../../_images/W3D4_Tutorial1_39_2.png
Task 2: Permuted MNIST data:
../../../_images/W3D4_Tutorial1_39_4.png

Great! We have our data. This commonly used task is typically called the “permuted MNIST task”, given the shuffling of the data. The permutations are the same across all images in the same task (all the permuted image in that task follow are permuted in the same way). This is useful as it allows you to create almost as many tasks as you would like out of the same dataset. While it may not be the best benchmark for CL, it is commonly used, and will serve our purposes well enough for illustration.

Now, let’s initialize and train our model on the standard MNIST dataset (Task 1) and make sure everything is working properly.

# Define a new model and set params
model = Net().to(DEVICE)
optimizer = optim.SGD(model.parameters(), lr=0.01, momentum=0.9)

# Train the model on MNIST
nEpochs = 3
print(f"Training model on {nEpochs} epochs...")
for epoch in range(1, nEpochs+1):
  train(model, x_train, t_train, optimizer, epoch, device=DEVICE)
  test(model, x_test, t_test, device=DEVICE)
Training model on 3 epochs...
Train Epoch: 1 	Loss: 0.877743
Test set: Average loss: 0.0003, Accuracy: 8384/10000 (84%)
Train Epoch: 2 	Loss: 0.763951
Test set: Average loss: 0.0002, Accuracy: 9267/10000 (93%)
Train Epoch: 3 	Loss: 0.960827
Test set: Average loss: 0.0001, Accuracy: 9402/10000 (94%)

Okay great! It seems we get decent accuracy on standard MNIST which means the model is learning our dataset. Now, a reasonable assumption is that, like humans, once the network learns something, it can aggregate its knowledge and learn something else.

First, let’s get a baseline for how the model performs on the dataset it was just trained on (Task 1) as well as to see how well it performs on a new dataset (Task 2).

# Test the model's accuracy on both the regular and permuted dataset

# Let's define a dictionary that holds each of the task
# datasets and labels
tasks = {'MNIST':(x_test, t_test),
         'Perm MNIST':(x_test2, t_test)}
t1_accs = []
for ti, task in enumerate(tasks.keys()):
  print(f"Testing on task {ti + 1}")
  t1_accs.append(test(model, tasks[task][0], tasks[task][1], device=DEVICE))

# And then let's plot the testing accuracy on both datasets
multi_task_barplot(t1_accs, tasks, t='Accuracy after training on Task 1 \nbut before Training on Task 2')
Testing on task 1
Test set: Average loss: 0.0001, Accuracy: 9402/10000 (94%)

Testing on task 2
Test set: Average loss: 0.0020, Accuracy: 894/10000 (9%)
../../../_images/W3D4_Tutorial1_43_3.png

As we saw before, the model does great on the Task 1 dataset it was trained on, but not so well on the new one. No worries! We haven’t taught it the permuted MNIST dataset yet! So let’s train the same task 1-trained-model on the new data, and see if we can get comparable performance between the two types of MNIST

# Train the previously trained model on Task 2, the permuted MNIST dataset
for epoch in range(1, 3):
  train(model, x_train2, t_train, optimizer, epoch, device=DEVICE)
  test(model, x_test2, t_test, device=DEVICE)

# Same data as before, stored in a dict
tasks = {'MNIST':(x_test, t_test),
         'Perm MNIST':(x_test2, t_test)}
# Test the model on both datasets, same as before
t12_accs = []
for ti, task in enumerate(tasks.keys()):
  print(f"Testing on task {ti + 1}")
  t12_accs.append(test(model, tasks[task][0], tasks[task][1], device=DEVICE))

# And then let's plot each of the testing accuracies after the new training
multi_task_barplot(t12_accs, tasks, t='Accuracy after training on Task 1 and then Training on Task 2')
Train Epoch: 1 	Loss: 1.818300
Test set: Average loss: 0.0008, Accuracy: 5744/10000 (57%)
Train Epoch: 2 	Loss: 1.703263
Test set: Average loss: 0.0006, Accuracy: 6988/10000 (70%)

Testing on task 1
Test set: Average loss: 0.0019, Accuracy: 804/10000 (8%)

Testing on task 2
Test set: Average loss: 0.0006, Accuracy: 6988/10000 (70%)
../../../_images/W3D4_Tutorial1_45_6.png

Hey! Training did the trick, task 2 (permuted MNIST) has great accuracy now that we trained the model on it. But something is wrong. We just saw that Task 1 (standard MNIST) had high accuracy before we trained on the new task. Why? Try to incorporate what you learned in the lecture to help explain the problem we’re seeing. You might also take a few seconds and think of what possible soultions you might like to try. In the next section, we’ll look into exactly that!


Section 2: Continual Learning strategies

Time estimate: ~25mins

Video 2: CL strategies

Split MNIST

For this section, we will again use the MNIST dataset, but we will now create 5 tasks by splitting the dataset up in such a way that each task contains 2 classes i.e.,

Task_1: Classes (0, 1); 
Task_2: Classes (2, 3); 
Task_3: Classes (4, 5); 
Task_4: Classes (6, 7); 
Task_5: Classes (8, 9);

This problem is called Split MNIST, and it is popular toy problem in the continual learning literature.

set_seed(seed=SEED)

# Specify which classes should be part of which task
task_classes_arr = [(0, 1), (2, 3), (4, 5), (6, 7), (8, 9)]
tasks_num = len(task_classes_arr)

# Divide the data over the different tasks
task_data_with_overlap = []
for task_id, task_classes in enumerate(task_classes_arr):
  train_mask = np.isin(t_train, task_classes)
  test_mask = np.isin(t_test, task_classes)
  x_train_task, t_train_task = x_train[train_mask], t_train[train_mask]
  x_test_task, t_test_task = x_test[test_mask], t_test[test_mask]
  # Convert the original class labels (i.e., the digits 0 to 9) to
  # "within-task labels" so that within each task one of the digits is labeled
  # as '0' and the other as '1'.
  task_data_with_overlap.append((x_train_task, t_train_task - (task_id * 2),
                                 x_test_task, t_test_task - (task_id * 2)))

# Display tasks
for sample in range(len(task_classes_arr)):
  print(f"Task: {sample + 1}")
  plot_task(task_data_with_overlap[sample][0], len(task_classes_arr))
Random seed 2021 has been set.
Task: 1
../../../_images/W3D4_Tutorial1_52_1.png
Task: 2
../../../_images/W3D4_Tutorial1_52_3.png
Task: 3
../../../_images/W3D4_Tutorial1_52_5.png
Task: 4
../../../_images/W3D4_Tutorial1_52_7.png
Task: 5
../../../_images/W3D4_Tutorial1_52_9.png

Naive strategy (“fine-tuning”)

First, let’s see what happens if we simply sequentially train a deep neural network on these tasks in the standard way.

Let’s start by defining our network. As is common in the continual learning literature, we will use a “multi-headed layout”. This means that we have a separate output layer for each task to be learned, but the hidden layers of the network are shared between all tasks.

class FBaseNet(nn.Module):
  """
  Base network that is shared between all tasks
  """

  def __init__(self, hsize=512):
    """
    Initialize parameters of base network

    Args:
      hsize: int
        Size of head in the multi-headed layout

    Returns:
      Nothing
    """
    super(FBaseNet, self).__init__()
    self.l1 = nn.Linear(784, hsize)

  def forward(self, x):
    """
    Forward pass of FBaseNet

    Args:
      x: np.ndarray
        Input data

    Returns:
      x: np.ndarray
        Outputs after passing x through first fully connected layer
    """
    x = x.view(x.size(0), -1)
    x = F.relu(self.l1(x))
    return x

class FHeadNet(nn.Module):
  """
  Output layer of FBaseNet which will be separate for each task
  """

  def __init__(self, base_net, input_size=512):
    """
    Initialize parameters of base network

    Args:
      input_size: int
        Size of input [default: 512]

    Returns:
      Nothing
    """
    super(FHeadNet, self).__init__()

    self.base_net = base_net
    self.output_layer = nn.Linear(input_size, 2)

  def forward(self, x):
    """
    Forward pass of FHeadNet

    Args:
      x: np.ndarray
        Input data

    Returns:
      x: np.ndarray
        Outputs after passing x through output layer
    """
    x = self.base_net.forward(x)
    x = self.output_layer(x)
    return x
# Define the base network (a new head is defined when we encounter a new task)
base = FBaseNet().to(DEVICE)
heads = []

# Define a list to store test accuracies for each task
accs_naive = []

# Set the number of epochs to train each task for
epochs = 3

# Loop through all tasks
for task_id in range(tasks_num):
  # Collect the training data for the new task
  x_train, t_train, _, _ = task_data_with_overlap[task_id]

  # Define a new head for this task
  model = FHeadNet(base).to(DEVICE)
  heads.append(model)

  # Set the optimizer
  optimizer = optim.SGD(heads[task_id].parameters(), lr=0.01)

  # Train the model (with the new head) on the current task
  train(heads[task_id], x_train, t_train, optimizer, epochs, device=DEVICE)

  # Test the model on all tasks seen so far
  accs_subset = []
  for i in range(0, task_id + 1):
    _, _, x_test, t_test = task_data_with_overlap[i]
    test_acc = test(heads[i], x_test, t_test, device=DEVICE)
    accs_subset.append(test_acc)
  # For unseen tasks, we don't test
  if task_id < (tasks_num - 1):
    accs_subset.extend([np.nan] * (4 - task_id))
  # Collect all test accuracies
  accs_naive.append(accs_subset)
Train Epoch: 3 	Loss: 0.000000
Test set: Average loss: 0.0000, Accuracy: 2113/2115 (100%)

Train Epoch: 3 	Loss: 0.443125
Test set: Average loss: 0.0031, Accuracy: 1930/2115 (91%)

Test set: Average loss: 0.0002, Accuracy: 1861/2042 (91%)
Train Epoch: 3 	Loss: 0.097207
Test set: Average loss: 0.0096, Accuracy: 1239/2115 (59%)

Test set: Average loss: 0.0011, Accuracy: 1498/2042 (73%)

Test set: Average loss: 0.0000, Accuracy: 1864/1874 (99%)

Train Epoch: 3 	Loss: 0.000244
Test set: Average loss: 0.0070, Accuracy: 893/2115 (42%)

Test set: Average loss: 0.0007, Accuracy: 1497/2042 (73%)

Test set: Average loss: 0.0004, Accuracy: 1152/1874 (61%)

Test set: Average loss: 0.0000, Accuracy: 1974/1986 (99%)
Train Epoch: 3 	Loss: 0.695176
Test set: Average loss: 0.0034, Accuracy: 1807/2115 (85%)

Test set: Average loss: 0.0004, Accuracy: 1042/2042 (51%)

Test set: Average loss: 0.0004, Accuracy: 1051/1874 (56%)

Test set: Average loss: 0.0004, Accuracy: 1091/1986 (55%)

Test set: Average loss: 0.0005, Accuracy: 1046/1983 (53%)

As you can see, whenever this network is trained on a new task, its performance on previously learned tasks drops substantially.

Now, let’s see whether we can use a continual learning strategy to prevent such forgetting.

Elastic Weight Consolidation (EWC)

EWC is a popular CL strategy which involves computing the importance of weights of the network relative to the task using the Fisher score and then penalizing the network for changes to the most important weights of the previous task.

It was introduced in the paper “Overcoming catastrophic forgetting in neural networks “.

For EWC, we need to define a new function to compute the fisher information matrix for each weight at the end of every task:

def on_task_update(task_id, x_train, t_train, model, shared_model, fisher_dict,
                   optpar_dict, device):
  """
  Helper function to accumulate gradients to further calculate fisher scores

  Args:
    task_id: int
      ID of the task to be updated
    x_train: np.ndarray
      Training data
    t_train: np.ndarray
      Corresponding ground truth of training data
    shared_model: FBaseNet instance
      Instance of the part of the model that is shared amongst all tasks
    fisher_dict: dict
      Dictionary with fisher values
    optpar_dict: dict
      Dictionary with optimal parameter values
    device: string
      CUDA/GPU if available, CPU otherwise

  Returns:
    Nothing
  """
  model.train()
  optimizer.zero_grad()

  # Accumulating gradients
  for start in range(0, len(t_train) - 1, 256):
    end = start + 256
    x = torch.from_numpy(x_train[start:end])
    if torch.cuda.is_available():
      x = x.type(torch.cuda.FloatTensor)
    else:
      x = x.type(torch.FloatTensor)
    y = torch.from_numpy(t_train[start:end]).long()
    x, y = x.to(device), y.to(device)
    output = model(x)
    loss = F.cross_entropy(output, y)
    loss.backward()

  fisher_dict[task_id] = {}
  optpar_dict[task_id] = {}

  # Gradients accumulated can be used to calculate fisher
  for name, param in shared_model.named_parameters():
    optpar_dict[task_id][name] = param.data.clone()
    fisher_dict[task_id][name] = param.grad.data.clone().pow(2)

We also need to modify our train function to add the new regularization loss:

def train_ewc(model, shared_model, task_id, x_train, t_train, optimizer,
              epoch, ewc_lambda, fisher_dict, optpar_dict, device):
  """
  Adding Regularisation loss to training function

  Args:
    model: FHeadNet instance
      Initates a new head network for task
    task_id: int
      ID of the task to be updated
    x_train: np.ndarray
      Training data
    t_train: np.ndarray
      Corresponding ground truth of training data
    shared_model: FBaseNet instance
      Instance of the part of the model that is shared amongst all tasks
    fisher_dict: dict
      Dictionary to store fisher values
    optpar_dict: dict
      Dictionary to store optimal parameter values
    device: string
      CUDA/GPU if available, CPU otherwise
    optimizer: torch.optim type
      Implements Adam algorithm.
    num_epochs: int
      Number of epochs
    ewc_lambda: float
      EWC hyperparameter

  Returns:
    Nothing
  """
  model.train()
  for start in range(0, len(t_train) - 1, 256):
    end = start + 256
    x = torch.from_numpy(x_train[start:end])
    if torch.cuda.is_available():
      x = x.type(torch.cuda.FloatTensor)
    else:
      x = x.type(torch.FloatTensor)
    y = torch.from_numpy(t_train[start:end]).long()
    x, y = x.to(device), y.to(device)

    optimizer.zero_grad()

    output = model(x)
    loss = F.cross_entropy(output, y)

    for task in range(task_id):
      for name, param in shared_model.named_parameters():
        fisher = fisher_dict[task][name]
        optpar = optpar_dict[task][name]
        loss += (fisher * (optpar - param).pow(2)).sum() * ewc_lambda

    loss.backward()
    optimizer.step()

  print(f"Train Epoch: {epoch} \tLoss: {loss.item():.6f}")

Now let’s train with EWC:

# Define the base network (a new head is defined when we encounter a new task)
base = FBaseNet().to(DEVICE)
heads = []

# Define a list to store test accuracies for each task
accs_ewc = []

# Set number of epochs
epochs = 2

# Set EWC hyperparameter
ewc_lambda = 0.2

# Define dictionaries to store values needed by EWC
fisher_dict = {}
optpar_dict = {}

# Loop through all tasks
for task_id in range(tasks_num):
    # Collect the training data for the new task
    x_train, t_train, _, _ = task_data_with_overlap[task_id]

    # Define a new head for this task
    model = FHeadNet(base).to(DEVICE)
    heads.append(model)

    # Set the optimizer
    optimizer = optim.SGD(heads[task_id].parameters(), lr=0.01)

    # Train the model (with the new head) on the current task
    for epoch in range(1, epochs+1):
        train_ewc(heads[task_id], heads[task_id].base_net, task_id, x_train,
                  t_train, optimizer, epoch, ewc_lambda, fisher_dict,
                  optpar_dict, device=DEVICE)
    on_task_update(task_id, x_train, t_train, heads[task_id],
                   heads[task_id].base_net, fisher_dict, optpar_dict,
                   device=DEVICE)

    # Test the model on all tasks seen so far
    accs_subset = []
    for i in range(0, task_id + 1):
        _, _, x_test, t_test = task_data_with_overlap[i]
        test_acc = test(heads[i], x_test, t_test, device=DEVICE)
        accs_subset.append(test_acc)
    # For unseen tasks, we don't test
    if task_id < (tasks_num - 1):
        accs_subset.extend([np.nan] * (4 - task_id))
    # Collect all test accuracies
    accs_ewc.append(accs_subset)
Train Epoch: 1 	Loss: 0.005017
Train Epoch: 2 	Loss: 0.001975
Test set: Average loss: 0.0004, Accuracy: 2108/2115 (100%)
Train Epoch: 1 	Loss: 0.486721
Train Epoch: 2 	Loss: 0.117900
Test set: Average loss: 0.0008, Accuracy: 1581/2115 (75%)

Test set: Average loss: 0.0002, Accuracy: 1847/2042 (90%)
Train Epoch: 1 	Loss: 0.098756
Train Epoch: 2 	Loss: 0.057696
Test set: Average loss: 0.0007, Accuracy: 1883/2115 (89%)

Test set: Average loss: 0.0002, Accuracy: 1665/2042 (82%)

Test set: Average loss: 0.0000, Accuracy: 1864/1874 (99%)
Train Epoch: 1 	Loss: 0.045494
Train Epoch: 2 	Loss: 0.032803
Test set: Average loss: 0.0013, Accuracy: 964/2115 (46%)

Test set: Average loss: 0.0004, Accuracy: 1691/2042 (83%)

Test set: Average loss: 0.0003, Accuracy: 1504/1874 (80%)

Test set: Average loss: 0.0000, Accuracy: 1978/1986 (100%)
Train Epoch: 1 	Loss: 0.953535
Train Epoch: 2 	Loss: 0.440366
Test set: Average loss: 0.0013, Accuracy: 1523/2115 (72%)

Test set: Average loss: 0.0011, Accuracy: 1551/2042 (76%)

Test set: Average loss: 0.0005, Accuracy: 1340/1874 (72%)

Test set: Average loss: 0.0003, Accuracy: 1691/1986 (85%)

Test set: Average loss: 0.0003, Accuracy: 1789/1983 (90%)

Plot Naive vs EWC results

seaborn library should be installed

# @title Plot Naive vs EWC results

# @markdown `seaborn` library should be installed
import seaborn as sns

fig, axes = plt.subplots(1, 3, figsize=(15, 6))
accs_fine_grid = np.array(accs_naive)
nan_mask = np.isnan(accs_naive)

sns.heatmap(accs_naive, vmin=0, vmax=100, mask=nan_mask, annot=True,fmt='.0f',
            yticklabels=range(1, 6), xticklabels=range(1, 6), ax=axes[0],
            cbar=False)
sns.heatmap(accs_ewc, vmin=0, vmax=100, mask=nan_mask, annot=True,fmt='.0f',
            yticklabels=range(1, 6), xticklabels=range(1, 6), ax=axes[1],
            cbar=False)

axes[0].set_ylabel('Tested on Task')

axes[0].set_xlabel('Naive')
axes[1].set_xlabel('EWC')

axes[2].plot(range(1, 6), np.nanmean(accs_naive, axis=1), linewidth=2.0)
axes[2].plot(range(1, 6), np.nanmean(accs_ewc, axis=1), linewidth=2.0)

axes[2].legend(['Naive', 'EWC'])
axes[2].set_ylabel('Accumulated Accuracy for Seen Tasks')
axes[2].set_xlabel('Task Number')
plt.show()
../../../_images/W3D4_Tutorial1_67_0.png

Section 3: Continual learning benchmarks

Time estimate: ~30mins

In this section, we will introduce different ways in which a continual learning problem could be set up.

Video 3: Benchmarks and different types of continual learning

As introduced in the above video, continual learning research certainly does not only use the MNIST dataset. But not to make things more complicated than necessary (and to make sure the examples run in an acceptable amount of time), we continue with the Split MNIST example for now. At the end of this notebook we will take a sneak peak at the CORe50 dataset.

Another point made in the video is that continual learning is not a unitary problem, but that there are different types (or ‘scenarios’) of continual learning:

  • Task-incremental learning: an algorithm must incrementally learn a set of clearly distinct tasks (the tasks are clearly distinct because the algorithm is always told which task it must perform)

  • Domain-incremental learning: an algorithm must learn the same kind of task but in different contexts or domains

  • Class-incremental learning: an algorithm must incrementally learn to distinguish between an increasing number of classes.

![table.png](data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAA1UAAACaCAYAAABBqWI6AAABQWlDQ1BJQ0MgUHJvZmlsZQAAKJFjYGASSCwoyGFhYGDIzSspCnJ3UoiIjFJgf8rAzsABhAYMeonJxQWOAQE+QCUMMBoVfLvGwAiiL+uCzHp6km/dG4WpjamHA15u3eLZg6keBXClpBYnA+k/QJyUXFBUwsDAmABkK5eXFIDYLUC2SBHQUUD2DBA7HcJeA2InQdgHwGpCgpyB7CtAtkByRmIKkP0EyNZJQhJPR2JD7QUBjmAjcydTUwMCTiUdlKRWlIBo5/yCyqLM9IwSBUdgCKUqeOYl6+koGBkYGTIwgMIbovqzGDgcGcVOIcSy7jMwmD9nYGDKRoglKzEwbD/JwCDYhhDTuAr0UgADw36bgsSiRLgDGL+xFKcZG0HYPEUMDKw//v//LMvAwL6LgeFv0f//v+f+//93CQMD800GhgOFAJiSX8NKAWiEAAAAVmVYSWZNTQAqAAAACAABh2kABAAAAAEAAAAaAAAAAAADkoYABwAAABIAAABEoAIABAAAAAEAAANVoAMABAAAAAEAAACaAAAAAEFTQ0lJAAAAU2NyZWVuc2hvdBbO2rcAAAHWaVRYdFhNTDpjb20uYWRvYmUueG1wAAAAAAA8eDp4bXBtZXRhIHhtbG5zOng9ImFkb2JlOm5zOm1ldGEvIiB4OnhtcHRrPSJYTVAgQ29yZSA1LjQuMCI+CiAgIDxyZGY6UkRGIHhtbG5zOnJkZj0iaHR0cDovL3d3dy53My5vcmcvMTk5OS8wMi8yMi1yZGYtc3ludGF4LW5zIyI+CiAgICAgIDxyZGY6RGVzY3JpcHRpb24gcmRmOmFib3V0PSIiCiAgICAgICAgICAgIHhtbG5zOmV4aWY9Imh0dHA6Ly9ucy5hZG9iZS5jb20vZXhpZi8xLjAvIj4KICAgICAgICAgPGV4aWY6UGl4ZWxYRGltZW5zaW9uPjg1MzwvZXhpZjpQaXhlbFhEaW1lbnNpb24+CiAgICAgICAgIDxleGlmOlVzZXJDb21tZW50PlNjcmVlbnNob3Q8L2V4aWY6VXNlckNvbW1lbnQ+CiAgICAgICAgIDxleGlmOlBpeGVsWURpbWVuc2lvbj4xNTQ8L2V4aWY6UGl4ZWxZRGltZW5zaW9uPgogICAgICA8L3JkZjpEZXNjcmlwdGlvbj4KICAgPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4KgLKj2AAAQABJREFUeAHsnQWYJEXShhN3dzvc3Tnc3f1wd3dbXA/nDrgDDneHxd1hF3dbYHF31/zjDf4osmuqu6tldizieWa6Kisr5auUiMiIzKGiUHByBBwBR8ARcAQcAUfAEXAEHAFHwBFoCoGhm3rLX3IEHAFHwBFwBBwBR8ARcAQcAUfAEVAEXKjyhuAIOAKOgCPgCDgCjoAj4Ag4Ao5ACwi4UNUCeP6qI+AIOAKOgCPgCDgCjoAj4Ag4AsOmEDzyyCPhxRdfTIP82hFwBBwBR8ARcAQcAUfAEXAEHAFHQBCYYIIJwqqrrtoBi6HSjSq22WabcPbZZ3eI5AGOgCPgCDgCjoAj4Ag4Ao6AI+AI9HUE5p133jBw4MAOMFSsVK299tphiimm6BDJAxwBR8ARcAQcAUfAEXAEHAFHwBHo6whMOumkhRBUrFQVxvBAR8ARcAQcAUfAEXAEHAFHwBFwBByBqgj4RhVVofEHjoAj4Ag4Ao6AI+AIOAKOgCPgCNRHwIWq+hh5DEfAEXAEHAFHwBFwBBwBR8ARcASqIuBCVVVo/IEj4Ag4Ao6AI+AIOAKOgCPgCDgC9RFwoao+Rh7DEXAEHAFHwBFwBBwBR8ARcAQcgaoIuFBVFRp/4Ag4Ao6AI+AIOAKOgCPgCDgCjkB9BFyoqo+Rx3AEHAFHwBFwBBwBR8ARcAQcAUegKgIuVFWFxh84Ao6AI+AIOAKOgCPgCDgCjkCrCPz666+tJtHt3684/LcdpX300UfDGWecEV577bUwwQQThPnmmy/MPPPM4aKLLgqXXXZZGGGEEdqRTZem8e2334Ytt9wyjDfeeOH000/v0rJ45r0LgWuuuSY89dRTHSo1yiijhOmmmy7MMsssYYYZZujwvDsFdEb/6Iw0uxNmXhZHwBFwBByB5hH47rvvwjHHHFORwGqrrhbmm3++ijC7eeCBB8Ltt99ut2GYYYYJhx9+eHbfFRe9bZ479NBDwwsvvBDefvvt8MYbb4Q77rgjzDPPPF0B7ZDLM7aRRMCIww47bFxuueXiq6++Gn/77bdI2FBDDRWnnHLKNubUtUldcvElUb6Q/g14bEDXFsZz71UIfPPNN/GEE07I2tdwww0XaW/33XdfPPLII+NYY40Vl1pqqfj8889323p3Rv/ojDS7LYBeMEfAEXAEHIGGEfj444/jXHPNlc2f+++/f2EaX3/9dZx00kmzeNNMM03sDrxcb5vnPv3007jwwgsrzhNOOGHht+htgW1bqfrggw+CNOAgglQQ5k+16oiGO+ywQ+jfv38Yfvjhh5yk2Mk5LbPsMmHBBRcMY445Zph7nrk7OTdPvi8hMNpoo+lqlNV57rnnDhtsuIHeLrbYYrryu+yyy4ZFF100PPnkk0GUFRa12/y2o39suOGGYYkllghbbbWV1qsdaXYbgLwgjoAj4Ag4Am1HYPzxxw/CpGfpvvfee9l1erH77rsHEcCyoL322qvqilYWaQhcdId5Lj/3tlLtcccdN/z444+ahAhXrSTVY95tm1B1zz33BNGya8XHGnOsCgBgAkUzUBHWk28w+3v44Yd7chW87N0Ygbvvvjsr3ZJLLpldc7HMMsuEaaedNrz++uth0003DZgwdDdqtX8ceOCB4dJLL60w5Wg1ze6GkZfHEXAEHAFHoL0IwGcOPfRfWwW8//77HTKAVx0wYECQlZPw7rvvavw111yzQ7yuCOjqea5o7m0FB8wZxapGk1h++eVbSarHvPtX62uxyN9//32Wwm677xZ++eWX7H6jjTZSH6QsILlAioWJ/PLLL5PQyss//vgj4Kv15ptvVj6QO7QSYmoYPvroI32G9gG7TZOOO7wgAW+99VZ49tln9V3ee+mll8LPP/9cEfXDDz9UxvX3338P999/f0V6lmdReUjkjUFvqK3uZ599VpGm3zgCZRC49957s2grrLBCdm0Xk002mV4i2GNHnqda/YVntGf6AO144ICB+df1vl6/bKZ/MIFZfl988UV47LHHKvJ+8cUXAwqYo48+Okw99dRBTB0zrWOrfc7eb2ScqCic3zgCjoAj4Ah0awSwioJ5Z4UEYp5K6aeffgo77rhj2G+//VSg4tmss86q/vFpPK7ZVIG5kjn2hx9+0DkT36CU6s1pFrdsPJun8rylhZedv5jfb7755vDII48E5vx6VGvuTd9tlLfFZw1ZAEF35ZVXTpPS61q8ikVulV8nHYRt81UHS3gPfL06hSSDttCg1wdFcabPbFTFVCl+9dVXVdMWISquttpqUcyb4jbbbBPFPDD++9//rogvAk3cY489ojjox1122UX9tXbaaSeNc+KJJ2r4qKOOqnmS/ymnnBLtXhz6K9LiRjbPiPPOO2+UDTSiMG9x11131XfxUxGhKt5www1xzjnnjOOMM46Gy+YaccUVV9RraRDxrLPOitNPP30ceeSRNYz4KV1++eVRmMEoHTZuu+22cfTRR49nnnlmGsWvHYGaCNAv8KOSzq7tkD6QJ9owz/l78MEHs8e1+guRzjvvvDjxxBPH7bffXvueTDxx4403zt7nola/bKZ/0K+WXnrpOMYYY2h5ZXUtiulFlA1r9F4mtCiDrpZBzP10HKBelJNn+JG10ueaGScqAPEbR8ARcAQcgR6BwHrrrRdF2RhlczSdX8Yee+yKcsNPwkvCK9ocynyUp0MOOSQyPy6++OJxpZVWigsttJDGJ7zsnFY2HnlX4y0bnb8eeugh5WEnmWSSeNxxx8UpppgiimVLXGONNfRPhMx8VfW+aO5lXwSjZnnbLbbYQnGbbbbZLCn9rcerEKlVfl1ckiL1wpeLfR3WXXfdSP1lo4zs219xxRUV5WrHDZrgttHJJ5+cFZYGCzMkkn2H9PlYOAby4XEsPPXUUzsAj8M+DZp07rrrriiaAmXM0k4iS7b6HObrwgsvjDR4mETeYcMMNsoweuaZZ5RRQ4DiY0FcE3eVVVaxaPGoo47SMATEgw8+WD8IcRDCoH79+ulzBCvKZIQgxYeDCYRgTnlvpJFGiiJpa5j/cwTqISA7ZGq7oe0wmBcRgwTP+WMCgcr0F5QJCDeiHdJ35p9//qy9ElCmXzbTP0Q7GEVTpeWlHx1//PFZn6cO1113nZZHtHBZvKuuukrD+Ndqn2tknMgy9QtHwBFwBByBHoMA85rsjKvzG/OMzZHMjdATTzwRZ5xxxihWGBEluT0Xy6aKOm633Xb6TPx5NVx2BFTejvhi4aRhZee0svFItNo8V3b+YvMq42lvvfVWLSdCJOVm7odH/fzzzzU8/6/a3Eu8VnhbNqgj/9122y3Lsgyv0i5+/eWXX86+8/nnnx/FXyxee+212eLL3nvvnZWrXRdtFaooFDuXGQMFmDPNNJMyfFZgGr5p2k0AQbqeacaZIkKZESs9vL/IIotokPhY6L0JNwQigRNn/fXXjwcccIDGswZI5zGiYdvHRXqHEIisnGgJjDbYYANNkzKiVUcwO/fcc+M777yjUdCEkCc7mhjddNNNGkbDRTsBvSWCFPH4Q+BzcgTKILDZZptl7SbtD/YuA5K1KwR7a2/1+gvtGAGfd2nP0AUXXBBtQinbL5vpH2KCkJVZtljVvMUcIgtDyILOPvtsDaOcNhES3mqfKztOkJeTI+AIOAKOQM9DgHlm1VVX1YIjENk8ibDB/DfHHHNEcTWJrJKwCsVzlIw2h/Ii1lKEYykB3wiZRVO6g3XZOa1sPPIpmucILzt/wa9S9qmmmorXlGy+Jo1aVG3ubYW3feWVV7JvYHwGZajHq7STX0eQsu9MvqxepYIWuy22m9ouVFFAmDbM+agMf+m2lqeddpqGsRLEdotFNHDgQF1p4l0+tvhTqVCEFgJQoMGDB2fpI5CxRSYaCNmRT8N33nnnLGlWsEgLKd6YNSR5wljR+uSTT7K4JnxNPvnkEek9T7YNpzGHdEgaMWltvvnmWXSkYcL4u/POO7Nwv3AEaiFg7Y92Yyuqafy0XWE6C5XpL8STM+O0PdIPMJdNqUy/JL6Vr2z/4J3DDjtM86Vviu8lQfHqq6/O+gcTHWQKEUwtUmqlzzUyTqR5+rUj4Ag4Ao5Az0Fg3333zVxIWF0y/qv/jf0j97byhHWHPZONn7IK4q6CYpxnpqTnIRYdhOGmYlR2Tisbj3Tz8xxhjcxfi4nLDeVkHoUQHhEOCUt5U32Y+1c097bK26IsJW/mfTPxL8OrtJNfN6FSNgDRNkC1DzroIC0XLgi1XJRyEJW+bYtQVSR8YIOJORygIvQYyVbkGrbAAgtYUIdfBCLe4w+bTHHMj/vss0+FRsGYQOJwFhZ05ZVXZu+ZWRQNy5ZErVMR18BmRcoI8yfLV5wZLTj7ReNhzx9//HENx6fFwmAUjWzZFQw648NZPv7bexBAiLK2lGrF0hramQ/Ew2wWKtNfiIf5g/kDImDRN4zK9Mtm+gfpY9dMeU2LSBjmAIQxsKHooCwMfISZwoJ4rfa5suMEeTk5Ao6AI+AI9EwE8IdHCIGwDmIu4Q+fKVxRZCc6fWZMNc/McokHRxxxhMbHp9l4WtkwIlPwy8YP+j7/ysxpjcQrmud4v5H5y9xvVl99dV7N6oN/WTVfKuJVm3tb5W0RWME4tS6rx6u0k1+nbiaowk/ZiiTuQpSL8z47g9oiVHHYWhGZhI9QBGFiZCtJLFVWI3MKRBuO8JIyf/YO/iYAAyNmUjCOaIT97W9/s2hRdvzQMMLxV4Hwd2ITCcLSVbT00FXZDSVLwy6OPfZYfQdtBnWBcCQkHf7wDzOaffbZNQy/MCdHoAwCNijSljbZZJMOr3A4obW1VEAp019YxYWsDZOOLX2X7ZfN9A/6pm1SYWa2mGLgT0kZTPuXDuAIf0ZW3mb7XNlxwvLzX0fAEXAEHIGehQAWTGxoZoRLic2VCEmpQGQKRJ6nfJ5tSobLihECGfHgF80csOycVjYeeRXNc4Q3On8xn2I5hSDDXgHwFPC7taja3NsKb8scb7y+mfdThnq8Sjv59aeffjprA7KjsELwxhtvZGFnnHFGLViaftayUIX0h6kfdop5MhtRW2JFI22rV+muY5gBpitDbGJBQ0YwMSIONpEQH4wNK4iz5ZZbWpRsqRNHQ0wGWWqEkSMef2byhKbCynHvvffG2267TTuMSdbpylqWuFyY8yO7FmJuSKOTQ+M0bWx0jXBmJD98Q2S7dgv2X0egJgI2gNJ22HkyJXz0zLYaM9h09bNef2EyQLNmZKtdJtCU7ZfN9I/77rtP+wL+i6Z0MP9Idtk0E2A2haHeKEkQ8riHWulzjYwTho3/OgKOgCPgCPQsBODF2ITACCGL+YS/NBzzc/MtZhUjJTNtX2eddTSYNGylA99+FJOyRXgsO6eVjUdmRfNco/PXDjvsoHyt+f+ndat1XW3ubYW3lfMzM/zhh/FVA896vEo7+XUzAcXFx1bqbNdHBG14D7Nyq4VPo89aPqdKhCndh17M3YKsKEkb/pOkwEEaVZAKBTHd08DRRhtNzwTghrOkxNcoiJQcROseRKr+80X5Lytces35UeL7pGfbiCCTxRHTvsA5N5DYguqv+EUF+Wh6LSAEkdCDrKAFMe/TMP6d879z9AwcaXQIkxouu4wEEaz0ngPhoJVWXkl/8/+ee+45DRINRJANBcJaa62lh7ESKIJlFl1Mm4J03CAfMMiGGVm4XzgC1RCg7XImhhFtF6Kt0Vc4jZw2LvbGesaarP5Y1Lr9ZdCgQeHJJ5/MzuyQ7VX1XWubZfqlCGZ6YCIvNtI/KDskE1bgtHvGBbF91zOoxEQ4O0/EzpAQX8sgxyYE2YJV32ulzzUyTmhm/s8RcAQcAUegRyEAL3feeecFYZSzck800USBuUR8ioIw9Fk41yIc6b1sN56FcyErVHrPXCm+y0GU83peIoHwc5xvNfVUU+t8TFi9Oa3s3EdaRfNcI/MX52iJQKJ8rJhBBnFvCeKOUnFeLPkUUbW5V5SoGr0Z3lZWibKsZLMLPeOLb1KPt28nvy6+2loG8OCgZ4jzaSFZlAkiiAexgNH7tv5rVArLx0ejjgYdO0XM/dhyGQmRpVhMfPI730mDzZzdpSIREz92GEmJjSPMqZ44pI9224gtHglnedHsJPnFzBCpFK28Saa8w/aZrExhhoQvB1p+/KzQnq+99tpqXnj99ddnknVqfmR58iuHymkc6squLkZI+mg/9txzzyiClq6wsfzs5AiUQYCVUtOS0a75YwmfFSXOVKBN0caqaaDq9RfbQpQ0cQLFvpw00YQZ1euXzfYPW+6nv3E+Fpoq6pTvYyzP00fRDKZnu7XS5xodJwwL/3UEHAFHwBHo/ghgZWHmfGx+hgkfYRBnhtrRHO+//35cYYUVsq20mWPh2XAZEQW9xr/llluyI3TwQ2Jbb5tD2PDBNhwrO6eVjUfmRfOc5V2Wz2WjDrPAMj6Ccuf5a61s8q/a3EuUZnlbNp/Cgs3mfXOXqcerkGc7+HWsc+zMWupgdNJJJylG4GI+6fasXb9DkZB8gKaJU4mR+tAIoElHupYl1iCMm2qbxRG9Q9qcVC2OeUEagMYZZphhOsShWLIlo4abRr1DpFwA6bKCVSR9yiYAWkYBWt/67LPPgtiaBhHYcqnUvmVFgbrmibQ4lVo6o2rh88/93hHoTARq9Zf33ntPTzRHc4dGTHwOgygzOhSnTL/s8FIuIO0frG6JiZ+eRo8WTba0DSI0BTRWRSTOwdp3GRdSStNMw5vtc7XGiTR9v3YEHAFHwBHomQgw15nFQ9kaiHI+iN9NtmrFe6zkiCJdV8LKzmll46XlqjbPpXG4zs9fWIhtvfXWQUzugvgvBfKW/QOCKGs1rrimKG+en1fTdKvNvcRpdp6Fx4bfHnHEEdOsdDWtHm/fLn69IuP/v5EFF+UzZFGl6HHLYS0LVS2XwBNwBByBXokAg7poB7Vuci6VClS9sqJeKUfAEXAEHIFej0DZOa1svHYAhumj7JIdZBUmyMYaWZLnnHOOClsIVeLPrIrV7KFfdBoCnSOqdVpxPWFHwBHoKQjIEQdaVFbGWKFycgQcAUfAEXAEeioCZee0svHagQMrUxDWXymx+gTtsssuLlClwHTy9TByJsyhnZyHJ+8IOAJ9DAE5PDFccMEFAXMKBn1MCMTGvGJDlz4GiVfXEXAEHAFHoIciUHZOKxuvXTCIj7Ka1Infs27KwQYNbNTGhmxyDleQXQHblZWnUwIBN/8rAZJHcQQcgcYQkEMTg2yEUfGSOA6HIv/Jikh+4wg4Ao6AI+AIdDMEys5pZeN1RvVMiZnuDtwZ+Xia1RFwoao6Nv7EEXAEHAFHwBFwBBwBR8ARcAQcgboIuE9VXYg8giPgCDgCjoAj4Ag4Ao6AI+AIOALVEXChqjo2/sQRcAQcAUfAEXAEHAFHwBFwBByBugi4UFUXIo/gCDgCjoAj4Ag4Ao6AI+AIOAKOQHUEXKiqjo0/cQQcAUfAEXAEHAFHwBFwBBwBR6AuAsOmMc4///xw5513pkF+7Qg4Ao6AI+AIOAKOgCPgCDgCjoAjIAhMO+20oehEqgqharTRRgvjjz++A+YIOAKOgCPgCDgCjoAj4Ag4Ao6AI5BDYKyxxsqF/HnrW6oXwuKBjoAj4Ag4Ao6AI+AIOAKOgCPgCJRDwH2qyuHksRwBR8ARcAQcAUfAEXAEHAFHwBEoRMCFqkJYPNARcAQcAUfAEXAEHAFHwBFwBByBcghU+FTFGMu95bEcAUfAEXAEHAFHwBFwBBwBR8AR6IMIDDXUUB1qXSFUbbvttuHss8/uEMkDHAFHwBFwBBwBR8ARcAQcAUfAEejrCMw777xh4MCBHWCo2Kjiiy++CF9//XWHSB7gCDgCjoAj4Ag4Ao6AI+AIOAKOQF9HYMQRRwwTTTRRBxgqhKoOTz3AEXAEHAFHwBFwBBwBR8ARcAQcAUegJgK+UUVNePyhI+AIOAKOgCPgCDgCjoAj4Ag4ArURcKGqNj7+1BFwBBwBR8ARcAQcAUfAEXAEHIGaCLhQVRMef+gIOAKOgCPgCDgCjoAj4Ag4Ao5AbQRcqKqNjz91BBwBR8ARcAQcAUfAEXAEHAFHoCYCLlTVhMcfOgKOgCPgCDgCjoAj4Ag4Ao6AI1AbAReqauPjTx0BR8ARcAQcAUfAEXAEHAFHwBGoiYALVTXh8YeOgCPgCDgCjoAj4Ag4Ar0Jgc8++yzw51QdgTvvvDN8/PHH1SP4kw4IDNshpImAPffcM7z33nul39x3333DXHPNVTp+UcSLL7449O/fP3t0xBFHhOmmmy67b+Ti/vvvD7vuumuYaaaZwiWXXBKGGmqoRl73uC0icPLJJ4fHHnssS4X7iSeeOLvvaxeffvpp+O9//xsGDx4cZplllrDwwguHUUcdVU/v3mSTTXo9HMcff3wYNGiQYtBIZZ999tlw8803h19//TUccsgh+uo333wTmBgYK/od1C9MPc3UjSRZGLcz0izMyAM7IPDKK6+Ea6+9NnDw4h577NHheXcLuOiii8KTTz4ZXn/99bDGGmuErbbaqlQRn3vuuXDdddeFcccdN+y44476zttvvx122WWXsNdee4VFFlmkVDq1IjXbz2ql2Z2e/fLLL+Hee+8NN910U1hppZXC8ssv352K52XpQgR22GGH8Pvvv4dvv/02zDvvvGH33XdvqTQ//vhjOO2008J9990Xpp122nDwwQdr300TfeCBB8KJJ56o/XnZZZdNHw2R608++SScffbZ4emnnw7vvPNO3TyZR2efffaw1FJL1Y07JCJcc8014YILLghDDz10WH311cNmm21WmC3zwphjjhn23nvvMNJIIxXG6dTA2AYSYSaOMsooUSoTH3/88SgCTpRC69/kk08e33zzzXjppZfGmWacScMEnJZz/e233+L888+f5SMNtuk0l1tuuSwd6RRNp9PXXxRBIAoD0TAMP/30U5xxxhmzb/DCCy80nEZveeGJJ56IwkhFYQDiZZddFv/zn//EOeaYI8pAEtdZZ53eUs2a9Zh55pl1PPn+++9rxksfiuAU//GPf2gbEgZKH8mEGY8++ug42WSTabhMJukrTV13RppNFaQPvsT3W3nllfVbbrrppt0egYMOOigK86blXH/99bUPv//++3XL/fDDD2v/Zw7dfvvts/iXX3651l2EqiyslYtm+lkr+Q3pd88999woTKFi9r///W9IZ+/5dVMEhDGPE0wwQYTvGHvsseOKK67YUklFyRbnnnvu2K9fvyiLBdreDjjggA5piuCmzzbffPMOzzo7gHlwlVVWibIAEvfbb7/sjzFguOGGi7LQkYXtv//+8cwzz4wvvfRSZxerdPqUB0yRLaacckrFEXmjiP72t7/pc+rUFRTakenUU08dTznllCwpkca1UkwKU0wxRRb++eefR1mBiFdddVUW1srFxhtvnOXTilB10kknxWGHHVaZLzqIU+MIPPjgg1G0AvHWW29t/GV5Y7XVVsu+ZV8WqhiU6SOiZc1w5JoBUbTTWVhvuUAJ89VXX1VU56OPPoqvvvpqRViZG8YXxhwTquwdWR3Q8HYIVZ2ZpqXtv9URkJUq/ZbdXagSrXAceeSRoykQYeAYI8sSyqm8UMW7Ax4bEH/++eeyyWTx7rrrruzaLprtZ/Z+md+i/l3tvaIyVotbNlxWDxRHF6rKItb74y299NJRVoq0ovCiH3zwQUuVpq/bfEV7p9/K6leHNAcOHBhHGGGEOKQV9wgXCCIsRKQkK3VxnHHGiXPOOWca3O2vmedlNVB5djDNE/VEgF1yySXzj4bIfVt8qmTyCDvttJO0pdokWoGw3nrr6bJr7ZjlnrbLTI+lX5ZD33rrrTDaaKOVy9xjZQi8MeiNIJrYwBK4U/MIYK4iAqWaNomQnyUkmqSAGdEPP/yQhfWGiy+//DKsu+664euvv66ojmgRmzLlldXyinTshvGp3dQZaba7jL0xPWFKekS1MAOiv1qbpNyY8ZYlq2d+jptv/vnC8MMPXzYZjXfCCScEWbXp8E6z/axDQlUCqvXvoujVylgUt5Ew76eNoNX74wpXrea45n6y9tprh4kmmqilio833njZfEW6o48+uvKSaaKY/2KOhknvYostlj5q6Zr61KIPP/ww/Pvf/1bzxmGGGaYi6hOPPxFEQGloXKpIoIkb5vqXX365iTf/egU5AhcREZ7U5PKvJ39eUc+dd945jD/++PlHQ+T+L86thezwZ8p/sGrJbbPNNuqzhA8WDYzfMcYYQ32simyeeX7FFVco40WDZZLCzlyk68IsSDNtaPPNN1+YdNJJC+MS+OijjwYanpFIt4GBGDtsIz7i4osvHvC9uueee9TfB3tOm/gsHr8wxdjOfvHFF0GWlTPfMTGBDM8880wWlfLTGbETFTOngK8MfjOQaD2CaO0CNrALLLBAWGGFFTScBnn33XfrNf/wO6J+xBWTEfUJW3XVVdWOFL+c66+/Prz77rtBVjnUbjh7US5g4G+55RYdYBhUeM9wopyU12ihhRbSyxtvvFG/F3FluVvD6CDYB4tZi94/8sgjykwQZvUp+60tv776i/AEwwT2MEFbbrllBgV9ZKONNsru0wvaBT4YYmobZOk7fZRdYz/+5BNPhl9+/UXbAm1XNNVhwgknDLJSpAMUkWnr2CyTJjbVEIqGfFuvlydtl/qMNdZY2gZpH7QZwiD8kmR1UicfmC/6nOXNc9oM/SOfr2j8g5hIBlkdb3oyBAvyNGLsopwQNvayEqDXosVryr+yHjYkXq8elA+GGtvwp556Svs2fkSMba+99lqYfvrp9fuIZjSIFrIhLOrlrZVP/pXJs9E2RHzKQfsDLzE1CfPMM0/WPsiTcYjxvpavLN/r+eef13EWfPJEOoynjMe0v3x7Ij7KNDA0x/UZZpghn0yHe8ZXysx3oA5G5AejYuMn7Zx06dcwW7WI/kbbHn+86szAH3/8oQpAsQDpkBT+ZsMNO1yYauqptD/T10QTH8QMMYjZpJaD+qeKw6J+Vq/vphkzj+DLCAPD+GNUr39bPH7rlbFMey2qe5qHXZPWd999Z7faHlI8yqaTJeAXPQoBfHUZW8WkvlPKzdzJeMAYTVujv5nPOPsAGI9VL3Ox+Amnn3668sg2Z6bvMM6IGWFYdNFFwxZbbJE+qrjmXcpS5KN+5113alx43iFFYA8/+8Ybb5SWGYrKho+kmPUr71v0nHmxSJ4oitv2MPk4badq5n+WEUvy0tjiWWedpUuhLMVKxXQZUiY/ixYx6ROmIooAE++4446IXTrxzjnnHI0jgojeE2bmf/ihCAMapfGqDSYmDrVIHFk1Lmnwh3kF5lbYb1oY9rLiLByF8crCRJiJMsFlSUsHittuu62afMiqQsSkkPfxh8EPQybZKB0gex8bV9K1PGSjDE1LNHaKDfa5mIvgXyNOeZGlWtLBRt/ewe9GNP0ax8JE4IkvvvhixCRTGBKNi2njlVdemZV18ODBijVLwvjwYB5FPvilQISlPk7i0ByxUxXGV9OTjhrFWVzj8g3TuOLUGDfccMNsSb3st3bzP4UzmkkrGB944IE1TX1op7Qb+g+Y0z6XWGKJiDlCSvhliXCuJrqHHnqomib8/e9/1+9OPEyUMDmkDQ16fZC+ShgmDITRlo3q5SmCvLZzyi8KjiiaQDUxIB3Ssz5z/vnnR2FINX18xejjogCIZ5xxRhSNvi7t006NZGKIotiIxx57bMQmHv9METKz9IhHHySfvPmfOPdruJn/McbQd4grzGC8+uqrLRvt/1NNNVXcYIMNKkwwswj/f5FPk+B62BCnXj3ATDY00L4rm25k34AxQKwBdKzi24lCRPs4fZuxNK0D+RRRvbzz72BmUTbPsm3o9ttvV+wZS7CTF41iZsqBnwM+R4xVfF8bbw8//PCsaCIA6XfD/I+xEJNjvqMocKI4YWfxuHjooYe0zWy99dY63jGXnHfeeRoHkx98nmiDmNVeeOGFiiNpgVM1MgzxaxINsPr1Yj4jAqC+gvk4PhPMD6RF3+T+uOOOq5akhlu7P/XUU7Vd25xoPlkiuMTtttsuigJMfQfTxETRpv6Wl1x8iY4Z+BAyPtPOaceUA3MZ+hiY004tP9qP9bOyfdfyZlzAHwXTf+ZmEUYzc8ei/l00F1crI3kY1rX6fLW6WxnhFai/mf9RV7BlvsM/w3yA66Vj6flvz0QA/gufW/q69Ut4J1FYZRWChxQBJRa10yxSiQvLA58+5hnGlmYInyH6F/01JeZQ+jVzvs2n6fOy1/QD5mlRapV9pS3x4EXgG1sl45WYp/K01lprxUZ8svPvt3KP5rPtVEuogvERjZ02bBgISFY39J7GjiOdkTF1PDdiMq0lVDGZ4Jci20DaK3V/YfzImz+EKsgYNMKYjBmUsdUUKTuLa0II8WFuiWt2nEzalqYJNKRhYZNMMokyq8ZYMvHCBIh2WoUX8odMkGQyhNJ0RZMfsUNH4ILhtLQRUBgYcDS0MCZ5IwQfws2RzxgVmEnrwMY0Eo/Ox6CEsGXpyeqVJaeTvYWnPlWNfGsXqv6EE8E5/ZYwQygUigiHUzEpyB7ddttt+n0YUIxgeBCIYfaMYKj4XgjTRsZ8mVBFOIwR8VKhqkyeMMq8J2YOyhjRb2BsCbP+RfqmFJFdzbjV/sW1KRuM2eMZExVKACMmLNKDSTeyPltPqCI+7RzmE8ZdzLQsCe1LYJ6GZQ+TC+sfJqjxqAw29epB/7YxBgUJkz7CNWMHk5+seET6PUw6/Z7vhRKJPlqP6uVd9H4jeZZtQzY/IBya/yRtnO/JeIgSAGJSXHDBBfUbiWmxhtlYJdrgiMKGcQ6/CL4jY6dYHmg8xn/8eVOGgckcJoI86Q9ioqd50g5QZnHPRJ338dME5R/hCDUovIxoc7LCpvmjFDBCEKA+/W/sb0FVf9nghzKkfRSGivdtowpZGYso6wiDOUxJdq1VIcnCwAXlImTzBd/GiP5YrZ+V7buMIbQn2iAEDrLSrBsAWD75/m3h+d+iMhKnTHutVXfSyAtVYm0Sl1lmmQ6Mc710SMup5yOAMgQ/oiJCyKJ/Mbe0QuLioumg9Ev7dDNpMg4wHxhfhhBF/29VoKLf0l+7wp8KPhdFPGNGK3TMMccozvfKwkhKzAGpHJE+GxLXbfGpkoZYmlgeNZtwM5HAlMdIVlnsUs2TuBGGLMiqjYYfdthhahKXRUou2E75yCOPVBO5RuwpU/8VS04mX7tUswaWWDETkh0Hs3CWMCG2uhWhR69nm202/cWkThgd3ZJSJgcNS9MkHXy5qJfs6qR2r5RdPrqaUrBsDFk9ZPfEDmkIA6LbXYJpag6JfxP28tJw1aSKF808j+19zYRQhChNk7JCfA/MCKEUE0w2MY/BfAYzNAizkXrUyLeul1ZfeY7JJCamIsjot8MWW3anDLIKGmRAzWDArBOzAr6xMIP6R/uk3YhwpX6LmFgdddRRQTTUFSY/IrRl6dhF0dajZr5pccrkSVzrz6Jl1+1lKRdtErr5lpv1t+gf8TBf5GiDPGH6i9mAkeWBaVczRF8UhlVNQYSJz5LA7AKT2yI8skgFF2WxqVcP+iJxIFnB037NuMD4g/kYps/07X322UfNNNkiftZZZw0DBgyo66taL++CajWUZxFm+TZEHtNMM41mhfmn7D6l12zbi6kjRFuHMAkVoUq/kc0V+kD+icJN7ebBAr8ITNAZO0WY0Sgi4KqpDaaT1j9oM5jYicCh/QHfBvoLJoYcC8K9aJWzMc7ysl9Z5VVTcbZGN+J9UU5pGcmzUcIPlTmN/pGaoeVNffnu1cx08N3CjFwYL81eFI8VJon5MtXqZ9avavVdzBllh7Cwz95/tkHSBwd8NziapF1Upr02UndRbqpZNccr0G5SaiSd9D2/7lkIYCYP31REoqhSkzubq4ri1Aqj/zFWy+YJGk2sRir6dK13qz0TBZTO8bhcYJYuyhEd9/GxzvtbVkujKBx/KlklHqL+VFYO3EkYo2TDEB03LbzR31lmnkVfSX20RFgMovTSv0bTa1f8tvhUNVIYmBnR7uo5NNiDDhwwMFx08UVZEinjSEPCrhRBiwkPHyUcWm0izl6SC9FYB9HKKzMk2oFsgrY4ou1VHyK755f4pNsIMXkYWVkffujhbEIzhoE4lKkawQxD+I2weQeEPTcEM2z2oMZMmACnEQr+maCaf2SCHAwHhFBlhCDImS8pMegUMd0Wx4Qtq7uFF/028q2L3u+rYQyWCNy0CxhM/PtkuVzbmJgvKSwM3DCI+HbwZySmUnqJnx7CGc9gulOCqWqGyuQJ41802BsTU+awxaL36auQaLcCtukwynavF038w5kV4ZUxZrfddtNyM1nZuUCNJFkWmzL1sPqXtb/H90tWZHSjmCIhxupRJm+LW++3bJ5F6Qw9VEddHgqYIp8jE9TSNk6a+N+lxCQtq4fq50Q4whSCiDE4hKHw4o+z34zIlzG4DOF/i+CXKrB4j3mKdPCHapTwQZWVGmWc0ndtPLe2wLP0Oo2L8M2ZU9SLb8w8uXiB4iR9p1p6RXnk+y5lpr3NNvufCkRLVzTodtmW3zLttWzdOX8SPkJMClUAzBewbDr59/y+ZyEAA44/ThHhp4gyoRm6qf9N4fQzTg9ioqubtqGkQdHWDkKwIj0UHvCrrQpUlMn8qVo5g4rN3SiXWAuUribzhvHv+FeJVYiO25zr1ShNOtmfeyXgw2qEggrlqyn/LXxI/g5xoYrKoY2FUWQFBC0hDbGIkDiZFHE6QyhgJYoBnb+8MzEbVMBI8sfgzgoQE50RQkm6UQThSOrtoNcHvZ4lk5/8swclLozhhFm2lSl7rWiys2eN/DKBG7GShpY1JbTD7aSy37qdefbUtGjj6XfGuRStKgIWDAYH3zGIwTzhJI9gy2GieQbT6s+mKZAJ1hbe7G+ZPJtNu9571FX8ELUPM3CyeQGMUiuEAAIzzqTFZjgcKIgjs60sN5J2WWw6ox5ly9mVeZctY7V4phSq9pxvyao+O0JBgwcP1tUsVvTaRYydKOzQSJvQQ9pck3cz84n10WYVHeQvZjC6wQqKFxhGNO3nn39+ofBA/FYJxRsEFp1JZdpr2bozlrKBFJr+dBMqK3/ZdCy+//Y8BOifCDp5pUgrNWFTF3b1w5Lnhhtu0LGAVWOojDVPmbwZ+7A0GmP0MbJNpVqd0+kLpFFt9btMuVB4ofhL+ex676HoEtNtjcZ7jNfNCkCjjTqapoOiFUKwZWxig7aupCEuVDEhsRud2NBrQ0SjaKZpeSAwh8CsBckc0wuxp9flQiYPNMwpYcIjTr5qBoHQdcghh6hJiMXBhMrMSyzMTB3svtlfOpQRO1E1SwggmHshGNJQinZsaTZtey8tK7seVmPILX4rv41861by6S3voiXDrCa/gx9mfigUaBsoGBCqrC2z8yPbkucJk1QbeFkBxdSqVSqT5+TJDmCt5pe+L35i2v9ZaYUBZVJoBzEhMnYwnjB5sUtaM1QWm86qR5kyd2XeZcrXjjhm2oOQ9dhjjxUmSd9opp3yDkwZ/dC0rWkG6diahte6tpU45rZmCZMXrA7EH0xNIjEnRzEnm5c0m2TN99A2Q8yz6aofYfQhGCUbewhrlsq017J1xxRL/JgDJl4oZeAPUiqbTvqOX/csBJg7IUyH20GY2WM59a9//atipdnGFsaZVon+xCIBFkIvvPiCWlQwlyPANdvHUFawqs4OiKnJcaNlZYUf3qQZYt5FmIL/NzeURtMZZdRRslcQpvod3C+zYMkedMHFX0s5bcychmCUXhOGXTMCFYRWLdXMa2Dyj4/O+2w3zrItgyyUX3GyV8QpVbdZ5J7BM2W82H4S5in9M/8ne7/ZXzTbprnE5C9dDSLNdAvXWnmYLwVxECRTYtvZPJbp87LX4hye+UvBkJuUz/swDOJEWDapwnhMThBLwo1868LE+lggWh+EqiIypt2YRmsrMAhsK50SaaCoMMaP75C2nXx83jWz1m++/Uv7jJ8BZO+WyVNfKPnPtPMoEWoRfR/tExpGeweBvR2EaRNmlky4+OWw8tcMlcGmM+tRr8xDIu8ybaheOZt9jhZaNqfQlQjSoJ+wCsSckBKMDgq6ZshMZfDJSglTODTW1cyK0rj5a2s31QQgG0/z76X3Vh/8zxAkmY8w/YWsv7TzDEF8DiFMDvOrVaykm8WF5V2vf1u8tIxl22utumshk3+sdHNWGH4vHCeSUiPppO/5dc9BAAYeS5xafF9ZS6NLL7lUlRhYSJkrhyHB2IMQlPr7NjNfMe+aQGUmfywckD6CFYqTZghlCONl6hvaTDrNvoMwyvE88OfNClTkbStkKKSwMpHNnJpe9Wq2LkXvdYpQxQqIkZlL2H3aEGDqEbBS51YmRnsfjSCrUhATtjkQ2rlJ+B4ZIbjAeOIsC5EPwlgax+Lmf1Oh5+tv/jyINH0vfZ5OIsacoqFkMw2ICYTJF4d3TIIwP6ERQdiQGmGekif8YUxzCYOHWRL4cF4RTB8CaGpikk5WRWWkU5rQZBMWZlO2sgHO2J8ywdAgEVpNy5LWE6YBAlNLz34JT8+RwYyKeuDz0si3TrFJbWRJv68QOGL2iUIgJc6B4fvDNBlDQx/gnjZGe6OfgD2DMEwNz1h1wcePyQQtNr98awSxPOEXAuHoD4OIqaGZ1+FLQvsqkydp2Le0854IM1MI6zOE2VkhtBXak22SYsKc/aJNQyvHBiv4RMA4GrMMk8w4ATF2QNZe9Ub+GZOX9h17xi8MFZMtK8Vl/ZjyaZbBpmw9rO8ZZlZWNIz0+bRf8czqbe9Z/PS3bN7pO1w3kmeZNkSan3/xpw18ntGgbvm+b9/S2oIp4ayNkR7EmCO7umk75942jcDPCkaflV7GVMY4tKQQaYJlWUYKZQU+AGxmQR8xMkbH8iTcxuNvv6tUeNg79svZXJjgIADi40efoW2ZBhiNsikRLc107CUdzI1sfKefsKGQrZphlojFA+lTXxgZE9QMU/slLcO1Vt9FucHYgqksWn9WeZk/CGM1iH4EVevf+jD5V1RG+mOZPl+r7mRhmPGdYcIYIzFBwuwXf26jeulYPP/tuQjQj+i/JsTna0L/wzSNsaIe7bvfvjp22DyQxmesZS7H54iVLPygTSGTxqt1XSRQWXzGBtxfECSsL9uzMr8ocBhHMYXtCmKcZAy2sbzZMhj/y6qd7EaabfDUbHpte08+XtuI7ZJl0tIzQ6SALFfpn2jjojiiaT4yIcTZZ589eyZSfhS/qew8Flnx0a1yicw2uZxXwvlO8hF0S2W2DpYJJEpDzbZmJx9xxo+i3Yqy21mUwTNLX5jLmvVjW0YxFcniizmDbjfLe1Z+fmVi1rOw2MrXwsWxUbeDJwO2qWW7Tspvz9km17YYZ/t1YdiyZ7L0qVtK5wvHFulpPLZPlkao0WT1Qc+9svRl0on//Oc/owwCupWthbM1OliwpaeF8csWt5BMorp1sDRqfQ5ebAvM+VYQ2zez3aa9KxO0bkMsQm0WxjPxidP4MhHrGSWEcTaW+ABpeNlvzZldKW6iSepw1pIm2Mv/cRYNWIumWbf9ZTtVYVS0fXJOEdtHpyRCsZ5LZd+RowpEiK84u8LS5NvQXmTHnci2wtynW6rL4BzXXHNNPauH9DhmgLNbaAdsESvCvWZdL0/xV9AzcUifrdHZ8lwms8j22YTRf8TfQ9MSP0etL/mxzbEISJEz0YTp0bjCpOsxA0TmjB5RrGgdKCdYWF9ke3MRzHRbcfKgP7MNOXHY/trS4/wryldEYF7mPBHadLU062FTph4cnWD9X1Ya9Qwg3hOBSfsb9QMvYfD13DoRFLLz6MBFVpuLqqdh9TDMv9honmXaEFuMMx9QD841YutvvhNbhxPGH9vvUw+2NhYlj4bJxgtRhG4tItt+i6CgZ8AwL9A+GXsZg1Pi3CoxVcvSJV9RTmgUxkdRQGTPyJ++Uo9EAIu0S+YJEcb1/ELZ5CAyBkLMTWxpbuVmrhPriJpbK5OmmKfrvMX4ybu8Q3+m33P0CPOCmM1reWnPbAFPXpAIMZExk23c+WPr+LQuHJ1B3wczMKWsRf2skb5L22CuJ12+GXM1ZUop379FUZk+rrjOl5GHZdprrbozzvCdKJ8IglEEqsiZNow1VmbmKvColU5FQf2mxyLAtvkcXVON6BP0kfTYj2pxZaVIxy+2By8i+i9jDzwg4xntrhGCp+N4B1FqVX2Nsa/R7d9JDx6DubArSFbvNP/8WF2rLKI0jLLA0CEK4yx8dJljKzq83IkBmPZ0CYnWvOKQRZgV0aJVnC8l/klaNpgtmDPxDemSsjaSKQILh/bKrn2NvFYRl4aPgIOwWatTVbzUxA2THMwok3o7SFZNCg9cK/Ot25F/T09DNPcV7YYBCMFGtOI1q8Z3FE10zYOCSUtWiTQdrmEqUqHKMmAAs/xg2uiXRVQmz6L38mEw4vTvMkTZUsZMNFVRtIFlXq0ZhzIgwFSra82XCx7Ww6az6lFQlA5BQyLvsm2oQ+EaDKCd0pZrEeMnAhQCbzuJtlKvzzWaH+3G5jj6HoJLGSIu9Xzrrbf0/aI5Q0zS29a+0zLRB5mrGLuKqJH+XVTGeu21TN2LypUPa1c6+XT9vnsgwPdFcVtGcdY9Stw5pUAAQdiDR+0K4juwONAIySpyFHNKPY+QsU2svPSw87ICcCN5tSPuUCQiDJaTI+AI9BEEMKHDLhv/IdHm95FaV68m9uqYhmBu6+QIOAKOgCPQ8xHA5BMzccxSMX/l7Dd8jUccccSeX7kma8CGDvgR2zEJTSYzRF/DFFmswtRPn43VRPmkZ1yJtVqnbObWauWGbTUBf98RcAR6FgJmi9yX9SnYYDPBYJvOzmliEtmzPqKX1hFwBBwBR6AqAmyGwI667ASKvy6HvfdlgQqgam3SURXILn7Afgr4U4trj/qFbrjhhnrgexcXq2r2LlRVhcYfOAK9DwGc0m2TBzauwNEWx+2+Ruzgxk6XDNjn/u/c7OyMvoaD19cRcAQcgd6IANvos9sjG6qIua5uZtMb69kX6iT+cNnGWd29vm7+192/kJfPEWgjAiyZM8EYIVCJo73d9plfcU4Pt95ya5hjzjkCA7aTI+AIOAKOQO9DQHwTg2z60vsq5jXqlgi4UNUtP4sXyhFwBBwBR8ARcAQcAUfAEXAEegoCnXJOVU+pvJfTEXAEHAFHwBFwBBwBR8ARcAQcgVYRcKGqVQT9fUfAEXAEHAFHwBFwBBwBR8AR6NMIuFDVpz+/V94RcAQcAUfAEXAEHAFHwBFwBFpFwIWqVhH09x0BR8ARcAQcAUfAEXAEHAFHoE8jULGl+u677x4uvvjiPg2IV94RcAQcAUfAEXAEHAFHwBFwBByBIgRmn332cNddd3V4VLH73+DBg8OHH37YIZIHOAKOgCPgCDgCjoAj4Ag4Ao6AI9DXEWCb/plnnrkDDBVCVYenHuAIOAKOgCPgCDgCjoAj4Ag4Ao6AI1ATAfepqgmPP3QEHAFHwBFwBBwBR8ARcAQcAUegNgIuVNXGx586Ao6AI+AIOAKOgCPgCDgCjoAjUBMBF6pqwuMPHQFHwBFwBBwBR8ARcAQcAUfAEaiNgAtVtfHxp46AI+AIOAKOgCPgCDgCjoAj4AjURMCFqprw+ENHwBFwBBwBR8ARcAQcAUfAEXAEaiPgQlVtfPypI+AIOAKOgCPgCDgCjoAj4Ag4AjURcKGqJjz+0BFwBBwBR8ARcAQcAUegNyHw2WefBf6cHIF2IjBsq4m99dZbYf/99w8xxg5JjTfeeGG66aYLM8wwQ1hkkUXCSCON1CFOdwy4//77w6677hpmmmmmcMkll4ShhhqqrcXccsstw+OPPx6OP/74sNxyy7U1bU+sNgIffPBB2GOPPbL2Ov/88+t97bc6/ymHbt9+++0dMqJ8M844Y4fwnhzAmHHNNdeEH3/8MfTr168nV6VHlP2XX34J9957b7jpppvCSiutFJZffvmGy81YNWjQoPDf//634XfLvvDcc8+F6667Low77rhhxx131NfefvvtsMsuu4S99tpL55B8WhdddFF48sknw+uvvx7WWGONsNVWW2mUd999N5x//vk6zk400UThgAMOCJNPPnn+db93BByBPojADjvsEH7//ffw7bffhnnnnTfsvvvuLaHAXHbaaaeF++67L0w77bTh4IMP1nEsTfSBBx4IJ554oo5tyy67bPqoS6+Ziy+44IIw9NBDh9VXXz1sttlmheWBbxpzzDHD3nvv3WN4+cKKdHagCEMt0/vvvx9FcEKq0j+ZvOILL7wQZSKPIpxE+Vhx0kknjWeffXb8448/Ws6vsxMQQSeri3SStmYnwlSW9qKLLtrWtPtaYsLgRWGoGq72//73v+wbrLbaag2/3xkvCOMbBw4cGEUJoWWbYoop4sMPPxxlsO6M7LoszcGDB8f1119f67jyyit3WTn6UsbnnntunH322RVz2n4zJCfHx1FGGSV+//33zbxe9x3augh7Wsbtt98+i3/55ZdrmAhVWZhdHHTQQVGYI72lTTHPMBeJIKb96OOPP45PP/10HGOMMeKmm25qr/mvI+AI9GEERICIE0wwQfzpp5/i2GOPHVdcccWW0Pjmm2/i3HPPHUVBGOeaay4dr0SJ0yFNEdz02eabb97hWVcFnHnmmZGyHnHEEXHKKafU8onwVFicv/3tb/p83333LXzugX8igMa+LbTNNtso4AhWiy22WEWahx9+ePZszTXXrHjWHW9OOumkOOyww8bJJpss0mHaSXRkGi8MwNFHH93OpPtUWg8++GCUlc946623NlxvBH5TAHQXocoqsdNOO2nZRFtkQT3+96677qqow1dffaV1dKGqApbsBsULGLWTRIuqmDcrVH300Ufx1VdfbWeROqSFgoR+mQpVRBrw2ID4888/V8T/5JNP4sgjjxxFy6rhjKuMCRCCFkKk0fPPPx8pf3ejfL/oTuXrjDaYr193rn++rH7fexBYeumlo6wUaYWuuuqqKNYrLVWOscjGRlOay+pXhzRRmo4wwgix3Yr6Dhk1GfD5559HWWVT3pey5um3336LCIZLLrlk/pHfJwi0zadqmGGGkfmwmDDxEU2nPrz22muDrFgVR+wmoSwFv/POOwEzpdFGG62tpZJOpaYq7733nppNtjXxPpLYG4PeCKKZVvOx3lZl2gc04ogj9oqqnXDCCUFWSirqYnWsCPQbReDLL78M6667bvj666/biogIIC2lJ5pdNeVuKZE6L1u7yJtbzzf/fGH44YeveBszmx9++CHI6pmG8+7CCy+s1yJcVZinzDLLLIHydycq6hfdpXyd1QbT+nXn+qfl9OvehYDwvmouLCtKWrG11147YB7cCpmbC2mQ7uijj668Y5om5smYzWHeLIsO6aO2XDNfvPzyyy2lJat24eSTTw4iPKkpYz4xePydd945jD/++PlHfp8g0LJPVZJWzUtZEQgvvviixjnwwAPD1ltvXRF/4ICB4f4H7g/Y///9738PIg1nzx999NGAz4mRaBqU6bj55psDE8Cqq66aCW34Q91zzz1hwgknDLLM2oE5RZihYfMrZiHaCVIfg3xelAOGBH8EIxrf4osvHiyviSeeOMjKQjCmwOLlf2n4d999dxZMGRdccMHwzDPPhDfffDMLX2ihhfT6xhtv1HJSP1lezp7bxXfffRdE2xfExCVMP930YaWVV9I68RxsRLurUfFlW2GFFTQf6oEfF3bEEHjfcsstOtAwuJCXmGrqszwWreAumhwtq2h1wgILLKDlsfwbwZaBA3tkMfPRMj7yyCPKXBE26qijqq/UzTfdHJ5+5ukgpkrqR4GvBVj3ZqJt4bsmBQsAADr+SURBVJMiprdBlukLqyra/PDEE0+EqaeeunAioS/B0GI3/dRTT6lPIcIdE9Frr70Wpp9++vDrr7+qn4qsthamkWYsWsAgJlpBVqTUIZj+UaSkoJ1SLspubS9Nh/xpP1988YX2g3r9zN7Fzv35554PM840o44Dww03nD3S33p4yGpRIA5tB3xfeumlMM888wRLh3LRd2Hs8R0tojLfxd6TVfHAOIkyh2/BuMNYg627Ed8BP4BZZ521g6BBnHp1tnSoF+OHUbVvY8/5ZcyEgUjxpz+Dx1hjjRXwY6JfMlYZRun7Rde0J779+ONVn6jFZFyVXGISq21RNKrZeAlmOJsjdIl1gY4FPKfdmhM6ZTOlX5m2BP60b3NkxyfYqN73LINH2X5hefLLOP3ss88qQ0M/KSJ8RGRVTn1FZptttsJvUK989dpgNfxod4y3RrQR2oC1Mb4PzCZUr/6vvPJKGG7Y4cJUU0+lDF7ZtmR5+68jUA0B/EIZW+eYY45qUVoKZ6xmvmCcZoylHyCoPPbYY+Hiiy8unN9ayvD/X6ZO8HhvvPFGNtY1ky4+t2Khpbxa0fuyEteUT25RWr02TAbJthAmGwKS/uXN/8ig/439s+fEk4+v+QqDoD4WMnDGyy67LN52221qA7/UUktFbOIhEV6iTCTZ+//5z3/UR0sGag3DDAQTEfy3xhlnnCwe9q0y0Wga/MMERhp5POuss3QJliVgyjLnnHNGYdg0Hn5g+H9ZXUgXfxfsTi0M+1lxpI4yWWdh8803X11/Meoqm1Rk79gStDAVUTYjyMLFKTxiv0q9yBNsZIUvqwcX4vSoZoQsZYtgEYXJiuBh5j38YmLI+yL0xQsvvFCXdbkX4SOy1DtY/FuoO+aIlEGcvKM4icc777xT82oX7qKVVNyxOcZEhzzEIVK/TaPY8u1SrGgnG264oS7hf/rpp2r2I4KqtgfshfEDAcczzjgjw687m//tueee+s222267rLy1LsCPdk9bAgfa5BJLLBGFecpekwE+ihIgHnvssRF78plmnClutNFGWXsVJUMUwVOxEmE8YrpAO+EbYY5IfxBFRxThO4pApu2IfnT11VdneeQv6E8bbLCBpoNJgSgdIt8Dwk+M9DH/I2ySSSbRNk47l80FKpJ66KGHtOyihNG2KgJfPO+88yri5G/w28RuXZyFIz45mCtQfqN6eMiGIerfQ7uhfKKdy8wisMXHb+fKK69UHG0MwMQ5pTLfJY3PNXUXwVWxWWeddRQzM1s77LDD4lprraX1wQSDctDvjerV+ZxzztF0bXyg79Nm6IvY01fzTaQe9B1ZBdLvznvQ9ddfH/EJ5ZvRfkTjm429tJ8yvrOW7qmnnqrt0cZj85MSISLSD0TZE//xj39ovphj820Zb2lDvMP9cccdp/hxzXcTxkbDuTfTnFptCRMgfLTAXzZV0vGSNk4etJd637MsHrX6hVaw4B/m6Ph9nHLKKTpX4r9sZo9EB+t//vOf2k/AlPbBd2W8xWQHKlu+Wm2wFn6i7IqY9oOXCN5RFHKKO31flGjxjjvu0HLUqr8oHCPt/pKLL4mieFXz+/y8p4n4P0egQQREaa1jCH3bxg2xCIiiPMxSgqfaYostWjYVtjzwZcVPFN5rSBD9DP6oVdp4440VI3jEPDEHdZZfbT6vnnqP5q8tVE+ogmmnMdvfFVdcofnSiAmjIRoxQRPGhhFGm2yySfYuzCM+B2ICkoUxiZAmE4wN7qQBIwiJ1iCKpkzjwwRAMCVWHphZI8ufZwhVEO9bXBg7mBMmLNEOZOEmjFg6Rb9M0JYODIGR7HBVEc4gkGImq1cWVQUIHCxJh4kOMnwQroxggIkDkyEaiCi7NEZZiVDmCMYQgYTn5ngoGlq9n2qqqZSJSNMlXjO4Uz7ypAxgCNlGBcZkN4otjBbl4S/1qTLBF0HRCKGTePhfibmQBvcmoWq//faLYlZg1VWlBPVl8DOibyE4GzHYEwfhAYKhtHYsK5U60cDU0MZFM6/COowSjCtKikGvD1LFR9p+Le30l3TJB+EqJROqEPZhAumzpEmbhnk3QqnChh2UwYiJA0aeb1iNUM7gD2mEPw6Tm1E9PIgnuzRp2RHGLC8YQ+oD441iB2KCQYhHuKJeRmW+i8VNf60fs9mCEYoe8mWjH6NpppmmQlCsV+e8UIXCZJlllqnLQDDGURYUSZTBhCrKgbBJGEo0xjXiIvwSZuOmlTf/yyYzeZ9VHKR513yqUJLI7n4aZkKVpYOCgLgo6/LEXCArihXB9doSwprNJ5QLRRD3MBjMNWW+Z1k8qvWLigL//w39g/ZqykHGSvoJQrURAne+/eErAj7MK0Zly1fUBuvhRx58f9mtVJV5+LGJZUqUFYFs3LdyVKu/7LZbofxCCUq7dnIE2oUAG96geC8ihCz6DPNjK7TeeutpOiguGVeGFMFroXA2PqvZfI855hgtP/NOSihKUj45febXfyHwl12JtKbOpLzZz2+//qYmPaIZ02zTbaPN/4otpjGxg4SR0l/+ifZRzdwwnTP7e+w88UXgHhMdI5Z7IZZlMUGAzNQOMygjM03kHjOSPKX5Ty7mFyIM6jIrW14bsfRaj9J00rhpnrLph5qfYEaDiSKE6Y0RW9iLxk/LabbB66y9TsA8Bl8jI0sT3wNs2GVjDN22e8CAAWpOaaaIIkTpK2ZbDD6yG5eGpeVtBvcjjzxSTXb4PiyFQ2aTe+mll3bIoxVsRXDT9IQBCMKo6zVL2RDmKZhv9SbC3ArTAvqOMID6h5kTOMuKr5oBUV/aCKYBRtbuMamD+O5ZO1pnnSBCaeC70cYx2REmVX1S9tlnH+1HU08ztZqf0Y4wOWqWMMEQBlr7LGlSBkxZzWxVGFk1l8AU0epH2TEZE2araraYMMkKT2bCQL8nH6N6eBBPhBaNjjmejUeihFDTSB5su+22+pw2xziE+YWNK2W/iyZQ4h/9hXHGTHZ5RVb31CTOXq9XZ4vHr6yyqZ9b//796/oa0Z4wJ+V4iTxZO2J7YrYRJq6NPzffcnM+enZPXxRBQOOm84IwIVkcLmh3qRl4xcMGb+q1JcohwqH2Hcw5hXnQe9EyB7At08+axaNaVTA/ZKzfZ+8/+x3x6Nv//ve/9cgP7pkHZCVLTeZTP0x8RTABFMVV5t/RSvnq4UdZ+P5szUyfoH+wLT7+lDbuE6cWMU+xvbOsCmo02bGx15tt18LDn7UfAUzkMWMvIlEahtNPPz0bw4ri1Aqj3TJvyiYPGk0sRgrN3Wul0coz3EboM7hpMB81S7PMPIu+mvpoMc+zNwJ/TrUR6Cg91I7f9FPRTlW8O/c8cyvjbsxvOrmm1zD3THZFBMOPECUyYsVjE54ItPSJC8OGkCVmKwEfrosuvih7z+JlASUv0gnD0sD/S8w1KlLALr8WI1gRObkxwcjS5pF1Wmx37eyvlVdZOfBXRJTR/AJEO61RZJk4i8rEywYiKTH4iMlYGpRdN4I79vEQ/inmu2bMZz0htAjbrBAFFzAg+GvBBON3IuZpastsUVMMLawn/9IOEDBEo69/VhcxRdNLfBwQisRsSO9Fg6V23QgoEPdGppwo8mmyOOmvaPvUzwgGGV+2Zij1FeJ9fBMpM0IxjDzCFIy1tXfiwCjyx+YD1Qjliqy+ah9EEcBkia+gURk8hh6qo76J8ppfiKXFr/VBvgNU9rto5BL/EGiwyYfol/iksJGOMZ+E16szcSDO3UOBJCtLpZld3rP2wbVRUZhtCGH+TBY3/cUPkvkgVaTx3MbtNN30Ok2j0euybYlvjN9YSmW/Z1FZy+CR5pVegxM+XLPNPlsaHGTVLrunbMTBDzlPKAFoL6LBViVdK+Urix8MK30P5ktW+VRBky9XtXsx/dOzG+nb9FHm0GpzULU0PNwRqIUAggJ+Q0WEUhoFUTN0U/+bwulnnB7EiiaIybwqZVCuNUOM7Sh1UAyWJeZjU/4hUKHk4nw/zstqlCad7E+fenxTjVCCoZg1Jb+F+29HBIaYUMUhjkZ8GBy70ZgapVq29DpdobG4zf6ikWdTBFaC0EbSATqDYFZxYO8ssk0aEFSapVTIFTMunQjTtGzVJw1r5tqYK5h/W5mydIomeXvWzC8MEULBbrvtppuVwHywCpdqXJpJt7u+w8obgiIHnrKhQTUiDquUtEkGR/oezHV3p8GDB+vqDCtkjRAbbbBRC0wdyoMbbrhB6wuTCXUWHqbcKftdGqkTzPEhhxyiB+DCsLLDXbryWq/OlheCK4dQiklmxeY79nxI/Io5pWbDysaQombbEuXrjO9Zpt58c4iVsmokJtv6iBWrPNkqfSPMWT4Nu28Ev8022yyg2GGuZd7H+qAMidmRKsPoszC+rHqef/75DQn/ZfLxOH0TAfoIgg6WGO0iNn5hVz82tmGeQTHEaivULO+Kkg4+Jq90rFVmlI8o2SHeE1PcpgWg0Ub9c8drU7oiMDIWiV9mrSL4s/9HYIgJVWKfmYGOJA9DnWoqzeSHSGi/jaot1drzsr9odVdZZZUgfhHaAdC8mXBSNo2y8RAazTzI3qm2I5s9b+SXzsNEy85KCAwpjmXTYRAwYmWtFlNu8Zr5RZBlO1FWIOikMHWdRZi8oe0EF/HZUOEZxrq3EatwrFzASEPsAMlKRZ6MoRH/KjUBks0I1EQHpronECtgtkKTL6/VLR/OPX0dkzyYd9mEQbeHpe+TFqtcnY1H2e9SVPaiMMrNjp0cRVH0ncvU2dLFJAXTQVbvELAR1IY02cqeCQRDIv9m2xJla/f3LFtftM8QK1b5lVkEeBgn23XSVv+L0s6/WxSnXlgj+KGsxAwKk0HZmCkzw62XByZGWE0wZov/V5BNZnQlHosDJ0egVQTYuQ5KzahbSRN+A7eKf/3rXxU8mCkR0oWERvJhpRxz42YIAQ/+E/7W3DoaTWeUUf88poL3EKb6HdwvmHVLo2n1xfgdbVw6AQUEAKR4SA5l1Mmca3HAzSasdNXF7EGR+llFaQexKoZABaEFa/cqSVpGBBRxZK/4k00U0igtXad+XGbKZAnaFrZ2X+0XcxEzLYQpN60E8dHmYDLSDjJfHdISp/OKJDFjMu1+xYMGbsynB20s54tRfzQ1aOJ7KyEooHgwbGGOEbRSwhQSpQFCNxomtHO2MpCajaXvtPva8kuVJI3kgUIFwUg2Wah4jcnq0EMPrQhLb2STCTWJgnlnhQ7TVsqA5nxI4FHmu6TlTa8NM5QQRrKzmypRUjNoGOqUatU5jcc1mHCmE4wvxykMaTJ8qjHL1qfbWa5m2xJlsPLW6meNlNW+cb1+gSkzJLvBdlitwjwIKwDZBVHNFfE9zqeHqTvmh3ZER9kyWvnSNlgWP9lYQ31WWF3GRw6/3Xz/tfTz5bU+jUIERQL8AcejODkC7UAAQQMLHBRr1chMuKs9t/BLL7lUhX+O58krtekr8Fbms8w7Q2LORcjjGB6Ups0KVJTVVshQeskOwEE2rGp61Yv0+hq1TahK/WNYEjViCRTtKEITH4hJ3AZVBnwkayg1l5Nd7zSM1R5rsKnQZQwkDLQx5Ti5GqWChV1jfmaEEIGAJVtRW5D6cbBiA9k7XH/9zZ+HcKb5p89T0wwrF+9VI+zfjaqlY3Eoswk79su7MEWmPUV7jUYQZhGTJ7SEtupn5eE+b9+LhtO03tQbe1m+DR0ITb5pW9J6W3qUuyzumIGYZhphAOYP7NFgojlCuE3zqIaJ5U39TTvLtez4qKYmaFKM0cS8C+dtnKbZ7MQIMx7aiZkkEp7mbfG68tfKlvYhKw9lxeYb8xh842CWYECoF6ZthIMHZo+kwzP8E/GBg7nBjwZmxZgchBNWESFrx3mTBbCEuUr7D/Hxe4LsPb3J/WODBVYmEYzAncEehtmYqbTP8qoxcfacdg1hGw4TSfumDdE+bdzQCLl/lDU9YBwFAm0Qm/OyeHz+xZ/25PnJkDKmtuZkbf3V6lPmu+SKnN3a+Sm0Z/o8PqVmx06bZtJnFZYJj/bAd6Bv16oziVu/Ih6TJu0EkxEcm/EvrUdWN/slvim/bLwhzNpP2l8JT4mNhNiAgnbBigTv015NO8v4b/OBlTsd/0jLwr/9rlKZwLhEGX/4/q/5gPhl2hLvgU+esSr7PcviUa1fUM6UUIRwxhvCEdp1vj/jM2GsNmIFQHumT9AWEL6MaJMoU5grzA+wbPmK2mAZ/GAiWWk66qijtBiye5/2fxQ86apktfpjNmV9nzGLjZpSiwqrm/86As0gwJhiG+oUvc9YhBUQ80w92ne/fXWssPk6jU+fhEfBN4qVrPPOO0/n5zROZ1zTR5kXW10wsHmYhRCUJKZU6owy98o0ZRJqicTsQLfmFsk8CkDZH9ujsq0tZ5awpbJoiavmwzklIiTomSSi4dKtz0VrlZ11wlbOMjH8lbacsyMrE7pNeJonW4MLM6Jb9Vq4CAcaJpOynmFk4SKsRXHyzc7kkVUxPceK7STF1CHLS0wndEthYVCzMNIQJk/PipJOmIWLo2PF2TH5CrONuWhJsviyEUMURkK3rmabXCubTCS6jbNtPW7h4guWJcm5MtTBnokGRs+KkUlJ49i2nvacfG0LbUtEmAjdMlg6oaYjzJZ+M7bChdqBO+mIEFtx9hfbc8sAwKNsS2orJ7/1sBXmIHJWC3E5h4pzWSC2ubZ2wvckHc45sbSFSYmyWUkFbsJs6/kvmkAX/hMBJ4ovmG5VTnn5FqLx0jbLt6Od2XfifCAjEYj1XCp7Rv3pQyIMWRTtV7Q1YVT0uAH6jrVbtlwWRij7PiJ4ZHiIwBRpc5SH9IU5isIs67lG4E44xxcIU5/llb+gT5KvCFe6VS3jBdu2866VVQStKGZoulU04WyDTzyIbcTZApdw/mSAjyKM57OpuJcVBR0DOBpBfDKi7OAX2a7bSBQPem5aNTzYppt8yI92xrbOYJYeG8HW4dSb7XcZY4i7uJwFJkKQZlPmu1h50l/O76P/gzdbnguzrFvcE0YenOnFGVn0Te7Z/luEk1irzmDAOEZ8+gBHT3AGCekTxnbc9CH6VZ4I49w8EcA0Lt+OrXsZzzmDiPfZbp2xRZgW3eadMNoX+VYjEVz0yAzaOW0JDK3/cj4ax14wbojZpuZB/mxzTnnYatswF8sHfY+2CvacS0X+4Ec/EEE+K0KttiRKKT3Hhnf541tzTpZRve/ZKB75fmH55H+plygRtA9RLr4VOOSJs50425BtnEXIUmzTc+QaKV9RGyS/WviRPm2UPgBWkGxSlM13HAGQbn9fVH8REjU+2+Xzx/EJ6TfQRP2fI9AkAvCkjA/ViHGOearecRC8z5zC3MBYWESMZcxbjM/MFUVnPhW912wYZ/HR/0SxXDoJUcjp0SH5FxhfRZFX0V/zcfy+OgJD8UgG6y4nNOKcGP/rL7+GOeeas2IL9XYWjh0AsQ9HYwGhJWUHJZZs0aD1NEIDiHYYTWa6wUcj9UCriRYHTT7a684gmhmraWhB0FQL09NyNqQnzJUu6VtirJzQjtDy8p0h/A1YXUDziZa+NxLfkHZAu8ZsNk+sfLGboq16oJUHE3Nmz8dv570wWeqzl+7m2Ej6tB204KzOCrNe91VWNegLrJbw7UXZkdXbXh5SeNT7Llae9JexkG+Z+mGCwZtvvBnYet6Ivs+KBd+7TJ3tve70Cz6Y7jL+UgdWitD0dhY12pby5Wjme+bTsPtG+gX9lc0i6N+s4lQjVi0ZW8vu4lktnaI2SNxW8Uvzy9ef788YwQo68zLa/nbME2meft03EaBtMfdhpdEb/aypHxtxNOKzjlm8KN91V2qsP1hZY6UL64jrrr0uzDf/fH2zsbRY624jVLVYD3/dEXAEHAFHwBFwBBwBR8ARUAUNQhTmspga4+OHn3GzyufeBimKCxSOKLXYBwATYs64wlyxEeGst+HSan2G2O5/rRbU33cEHAFHwBFwBBwBR8ARcATqIcCmDfggY7HC6gsHwLtA9RdqrArja40PJCvhG264YeDQcKfWEPCVqtbw87cdAUfAEXAEHAFHwBFwBLoRApjDie+TboqDSan4v/Za8/9uBHufL4oLVX2+CTgAjoAj4Ag4Ao6AI+AI9D4E8LOWTZF6X8W8Rt0SARequuVn8UI5Ao6AI+AIOAKOgCPgCDgCjkBPQaB3boXWU9D3cjoCjoAj4Ag4Ao6AI+AIOAKOQI9HwIWqHv8JvQKOgCPgCDgCjoAj4Ag4Ao6AI9CVCLhQ1ZXoe96OgCPgCDgCjoAj4Ag4Ao6AI9DjEXChqsd/Qq+AI+AIOAKOgCPgCDgCjoAj4Ah0JQIV51T169cvXHHFFV1ZHs/bEXAEHAFHwBFwBBwBR8ARcAQcgW6JwGyzzRauvvrqDmWrEKrWWWedMNdcc3WI5AGOgCPgCDgCjoAj4Ag4Ao6AI+AI9HUExh133EIIfEv1Qlg80BFwBBwBR8ARcAQcAUfAEXAEHIFyCLhPVTmcPJYj4Ag4Ao6AI+AIOAKOgCPgCDgChQi4UFUIiwc6Ao6AI+AIOAKOgCPgCDgCjoAjUA4BF6rK4eSxHAFHwBFwBBwBR8ARcAQcAUfAEShEwIWqQlg80BFwBBwBR8ARcAQcAUfAEXAEHIFyCLhQVQ4nj+UIOAKOgCPgCDgCjoAj4Ag4Ao5AIQIuVBXC4oGOgCPgCDgCjoAj4Ag4Ao6AI+AIlEPAhapyOHksR8ARcAQcAUfAEXAEHIFegMBnn30W+HPqnQh8+eWX4ZZbbhnilas4/LdduT/wwAN60vBzzz0Xhh9++DDRRBOFv/3tb2HRRRcNCy20UFh77bW1sieffHJ47LHHsmy5n3jiibP7rry4//77w6677hpmmmmmcMkll4ShhhqqK4vT5/Luzm2jMz4GA8ANN9zQIelRRhklTD/99Po3wggjdHhuAW+//XbYZZddwl577RUWWWQRC+703+OPPz4MGjQo/Pe//21bXp2RZtsKl0voxx9/DLfddlu45pprwr/+9a8w1lhj5WL4bVe1zb6E/DfffBPuvPPO0L9//9DvoH5h6mmm1uo/++yz4eabbw6//vprOOSQQ4YYJH2lX8DjXHfddYGDQHfccceW8O2qb9VSoXvoyzvssEP4/fffw7fffhvmnXfesPvuu7dUE9r7aaedFu67774w7bTThoMPPljbRJoofPGJJ56o7WTZZZdNH3XpNXPXBRdcEIYeeuiw+uqrh80226ywPHvssUcYc8wxw9577x1GGmmkwjidGXjjjTeG22+/Pbz44ovhhx9+qJvVzz//HM4444y68doeIbaRpGHF7bffPsrHiXPMMUf88MMPs9RvuummONlkk+kzEbQ0/KeffoozzjhjlErp3wsvvJDF7+qL5ZZbLiuXdJSuLk6PzV+Ehbj//vs3XP7u3DYarkyJF2SAj7R/UTpou5tgggmiMEFxn332iRtssEGccsop4+STTx4POuigCDZ5uvzyy/U9Earyjzr1fuaZZ44i+MXvv/++bfl0RpptK1wuoeOOOy7yrRjD0vEuF61P33ZV2+wroAtjGI8++midX2mHTz/9tFZdhKz4j3/8Q9vmSiutNETh6Av94uGHH47LL7+84gvf0wp15bdqpdw98V0RIHTMZh4de+yx44orrthSNUShEeeee+7Yr1+/ONdcc2l7OOCAAzqkKYKbPtt88807POuqgDPPPDNS1iOOOEJ5DMYPEZ4KiyMLI1r+fffdt/B5ZwW+9tpr+o222267uN9++2V/22yzTZTFjrjwwgtnYTw/8sgjowiKUYSqzipSzXRDzacNPqRyfBQa6qefftrh7bfeeivKSlQ0oYoIq622mr7De91JqDrppJPisMMOqxMVncapOQSYdOaff/6mXu6ubaOpypR86ZhjjtH+sMoqq1S88dtvv8Wzzz47ympVnGqqqeKtt95a8ZybAY8NGOIDyUcffRRfffXVDmVpJaAz0syX5/HHH49fffVVPrip+7XWWku/WbuEqnaWrakKdcJLXdE2O6EaQyTJZr//Vlttpe3QhCoK+/nnn2vYkBaqyLvd/YI0uxs9+eSTim+rQhX16spv1d1w7czyLL300lFWijSLq666Kn7wwQctZffJJ59kcyB9F15WVr86pDlw4ECdv7urkp72J6tsyvdS1jzBgyAYLrnkkvlHnXb/5ptvKg+O8Jeno446SrGWleL8oy69b5tPFaYvDz30kLSnEFZYYYUOS5+ETzHFFOGcc87hstsTy8HvvPNOEEEwjDbaaN2+vN2xgJiiYRrlVB4BzGUhluJTGmaYYYIwTeHUU08NMtAE0UAHzKpSmm/++dTcNg3r7GtZpQnTTTddW7PpjDTTAmJque6664avv/46DW76erjhhmv63fyL7S5bPv2uuu+KttlVdW0l31a+/8gjj9wha8yHu4ra2S+6qg718jWT7Ha4B3Tlt6pXz97yXLjtIIJwkBUlrRKuKLintELjjTdeNgeS7uijj658Y5rm66+/rmZzmIoutthi6aO2XDOXvfzyyy2lJYshAbcLEZ7UlDGfGDzIzjvvHMYff/z8ow734FyPysSRVSflv+F38oSLDmOMCHn5R1163xafql9++SUceuihWUVkhSG7zl8gcK266qr54ML79957T+2V+R1jjDG0I8jKR4e4+KKIZB0mmWSSMN988wXsnLfYYguNx7tXXHGFMlA0eAYu7J/nnHPODulYwKOPPhpE62y3+tGwJX3mmWeUobUH+IdB2HqSD/WSZWB7nP3KClzAnvaLL74IstScdWiYY9I0okx0UGxcxZwqbLLJJmHUUUfVx7IaEO66664gWpGwwAILqODKAzrT3XffbUmoTxoYEFfME9QnjHJhAyurh+H6668P7777bpCVELUlzl78/4tq+TRa9912200FAJKljNdee20Q888sz7LfNl++vn6/7bbbhkceeSRceOGFgetUaP3jjz9UEYDyIk+vvPJKGG7Y4cJUU0+lg2ae4aG90x5pO/lnpIVyQUwQ1bEX594ZZpghy4JvSbs1BoMHsgoUxLwiTDjhhPr9X3rppTDPPPNkaTOg0qboj0VCWVGatH3Kht8Sbfj999/X/lZUXsogZgNqMz/rrLNmwia+J4xP1AcGFkaUySQVYmmvjCFibqm+oKTVCNHfRxpxpMyvJf9utfSLykZdKacR5QXnNExW1NXWnTjmeA2uqd17tTwtXX7rxWkU/zTtorZJG+Ab4TOIz49oebWNVWNyWmmj5MXYxhjMGJ22VcrZansljVbxK/r++bZZ1KbJuyzhR5K2HRgl8wPEvwQ/BGicccap60dMWs8//7z6psw222xZ365Wlnr9ot4YVQ/f/BjFuML3Nkr7SbW61suDtGirTzzxRBh/vPrMpeWd/tZrx2nc9JrxlHynnnrqQkEAv57nn3s+zDjTjGHEEUfs8D3qPa/XRyhLvTTS8nbHa/x/af/intIpxWMeYc6jn/K9GGfMP/ziiy8Ok046aafkS53grd94441An26WZEVb+TT4xyJijC7iwfNxZTUwiFlhWGqppfKP9J50ENDSPRWKIrK4Ie4AHcYi5hP6wiyzzKJCbNG7XRYmHallsiVPqYQux4lQUjrNaiZe4vSnS6VnnXVWZLmU5VrSF8EjykCZpS8CQ5RGHEVYiNJ41S4U8yhIBJkowlDcbLPN4h133BHXX399TUNWy7L3iy7uvffeKI1f45InpiuQfMQKHzBxqI/YmQqjo3GFuYsiPGRJYrMrjK8+v+iiiyImhaSHvxl28CxtyuYdWT7YxWObazjKRhma1gknnKBYYLP74IMPRhEKozgURvxwSEecLrN3MLcTLbzGsXTASJz7ogzG6v9COKaNV155ZVZWLmrl00jdpYNrGSx/vs+GG24Y+aZQ2W9brW1oIr30nziy6rek7tUIkwWwlQE7ikIjioNzxN5YmFH1oUjfE4E7rrPOOvGSiy+JBx54oC6lp2100OuD4sorrxz33HPPKKtgcZpppomivYviCKpmEfhwCdMbZfOLKIKc5kneohWL4gSq9sy0pcGDB2u24kiqfgb0CZbsZeDMTArwPRJBSNvdTDPOFIWZ03ocfvjh+i51KUqTvk0/oX+x1E/5hOnTdzGzkAE2rXI87LDD1PQIXx7MFchXBFGNc/7552t9qAO4MDZgbgiRP32OsYb2SvmWWGKJiHlHLbJxBb8IEWi1XKSPf9zHH3+cvVov/aKyyWp5ZiKNaQbmRox/5itD36KeRvgvgr2NwfXy5L16cRrF38rCb1HbxMxkp5120jH273//e5QdmnRsoh3Rpq+++uo0idhsGxXGRtMRC4q4+OKLx6233lrnD+aE8847T5+10l6tkO3Cr+j7W9us1aatHLJRjba91PyPOYi2aOZ/tB3mCMJE41yBNfMccyc+nNSpGtHf/vnPfyqm9Ff6GHMS8xMmQkZl+0W9MaoWvphuVRujcCfgm1NXXA7wszAiT8YF+hG+4LXysHf4tfGJsXKjjTbK+BLm4HpUqx3zbv5bWXq0Y9rvscceG/EHYuwkbxv3+MVPRzZH0LEA8yz6lVG958Sr1Ud4XiYN4nVXgtfiWzOP0R4Y4+GTnnrqqazI8IuijM/mg+xBgxeWx7nnnqt9jXlzSJAo2yP8cqu08cYbK0aM03nCpLeM/zQuAaLUiMyJecK0EF60yMQwH7faPWMV3xH+ortRW3yq/vOf/2gFqSR/xmCVqWwR48zgIkuompbZS8IUWfowgJDsaqRh2Mga0XnY/AKC4eIdY6gI23TTTWM9oYp4MG6WnwlVhNvExTM6JvkhcFhcGCkjGDTCzQaVCcDimUDzv//9LwuTlTYVvGBiicdAyWCHMx6MErhANlmZnWmarqwYRNEyqMDFQGz5gTMTtKwYZGGyKmFFLZVPI3UXjV+WT+pTVfbbUrCitpEVuJdelBGq0u8NA4X/IkI735qJIyXZvVIZAQtDoL3sssv0lsERRh2Fg5GsOGs6CNj4EqLQIF02mSGMewZdBlwxP8yUAGmftzowscPYQORBOrRtxguI/BdccEEVXmBsYMiqpcngyftiOhFhMohrDFPaP1GIEA//MyMExZTJkBVgjUNeKeHkKjsbZUGyCqjxmEhqkfVHFB0IM4wJa665pr7LGGFUJv2isiHUMR6mdvr0I8YLWTG05PUXAYjvY1QmzzJxyuJv+dpvtbbJ+ACjy3jFpgYoiGA6xSJBx1V7v5U2is8cQi2CLvkZwXwgoFvbbLa9WnrtxK/o+5dp05TFxudaQhXxECDozygNUJ4Y8Q0YD9Iwe5b+IuDxLn3WyBQ9lMGobL+oNUaRVi18a41R5jPJ5gHMobK6bUVTIUFWsLM61MrDXpIdThU38jTCqZ/xpp5PVb12THr0adIyAdjygElnoyIjmHXioRCAGM/5nkY46CM4G9V7XqaP1EvD8uruv2zkhEKuiBCywBV8W6H11ltP00HwTdtKK2mWeRdeEd7X+MQy7xTFMb9uxp2UmNuM907Dq12bYJXyF8zVrQpU5MdmFHwrkw+qlaErwtsiVLFLGRW0P1neLl2XIsaZQR/NF+kx4UF0fEvfBgy0RoSx2yC7pDFwQWg+IQYanrObGFoICObPJlMNqPLPJgTeT5k2G0QJTzWqMAOETS47tEHkA9NAmJjCaRj/YLJmn332bKcwtBjE4Q9GDIKJQvtMfUyrCGNgZJMngy0E42JpsFmIERpECzdGmmdsJEI4TJlRmXzK1p00qwlVZb8taRS1DcJ7MxmDR91rkbU3m1hZAeKb5oUq2o0swWcbWPBdbLCkfYhJXkU2MCKseiF8G7F6kGfe7ZlptVKhSsxxtSxoVo1g2FghQKhKiUmOcrOSalSU5imnnKLxxJTXokW0zbyLhtaIdBDixbTQglQQS9t6EePKihD1ZEIFH/7QshEmpnQVGvgs4f+/sLGCVRkj6mu7JYmJVCybflHZSJOJGsbQVl8Ig5Gj/qxeGbHyZhrAMnmWiUPaZfG3cqS/1dommxaJ+UYaVVc+adu24tFqG0UxxZjI97Q/ViHBjZVbqJX22m78ir5/mTZNPWxeqCdUERerCDDAesKIHXpTgdzC01+UKXyfoh3TxARQhVWUClCZfkG8WmNUWXxrjVGXXnqp1pWVNCM2+jEBsEweCJq011TpQlqsdIBjPaGqTDuuJlShnE1XwsSXRPM0yw+EPZQE6aoAbdqo3vMyfaReGpZXd/+F/0KRV0RiPhpPP/30ukqFoncJQ5hNd9FDYT6kiTmX8S615mq0DP1v7K/tC/7aiPGYxQtTVFh4vV8EKxQ18CkIZayE2/xU791az/mOtPkhKbTWKk/6rC0+VfgspCQrImpXmoY1ci1gBZkY9PwbzrYaOGBguOjii7IkZClar8V8J4jkrHa+Yo4QsFnlXgYwfY4vkXQS3ddeNNzqzyTadrXRJIJMqh0OBxPmoSlHQuy1ISvbww89HESA0DDRlOsv/6RxZdf5C9EyaBD+KcJE6TV25pA05syWVSYtDcN+thbZpgf5OOALSUPIHrWST77uWaIFF2W/bcGrHvT/CMjgr/4b3OKnBFVzlKaPcO4Ttse0bXz6ZAVT38HPT4QcvbZ/wjAFWQG1W/3FTpw2WURF+Q49VOUmG7xHGjjw5sn8fvD3MypKsyiMDS0g8yPimnPlzE4bvyjRoKufmfVF4hSRDPTqK0E50rKIaaJGFwVHYfnTtFIHXuorgrGeXYVjNL6R+GI0m74wbeobKuYd+j3Jl3Nt6OOyKqff7LvvvtMxU1a0tFhl6lQmDt+tLP4pHnZd9K49y//iyyOCv47pYNZqGxWmV31oqacR/j/80SegVtrrkMCv2TZt9S36lZXHIAKVzo/4v/KNZLW77llL1JfvIyu/HZLFf4I+Jxpz9Y2zCLX6Bd+g1hhVFt9aYxRzqWwDrfWTVTatqzC8QXYP0yKWyQMfELEQCDZHW91sjq3Xxsu2Y0s3/WXchkToUh5HrAWyey7YdIf6MbaLgBRk5bfCb73e8zJ9pF4aWqAe8I/NHPAbKiIR7gPnVzVDN/W/KZx+xulBFJJBTJuVD8XvtxkSIV/fh48uS4yb+B5B+FeJIKPnVnJeVqM06WR/+n2JAiV7lX6Dzxb8QSOEX6Moa4K4GKgvN/sb2PzUSDppXHhs/KmQO7rjJnJtEaryIOGI2aojIM7KfAzZi16d2WmseWLDCdHs64GnoknSQQ+hSlaigmicg2imdAMLBkQECA5BxMmfPxztEUrSjSJIXyT8fDZN3b8+6PXsvZRJywJLXhjDCEMmGreKt+oN5BWR69wMqXwoRplvW6e4ffrxo488qvXnQG0c2WuRLOXrZg0w40wmoj0O4ruhDrS0f1nBqfX6EHuWCvitZgpjx0GnspKrY4D4IQY2yqhFPGewZofFepjWSid9xsYeEAJdq+mjFEIAZlJCgSSrLgEmgEmFHaVEa61jHgyqUZk8y8Sx9Lrit9U2KiuoOomLJUNbi2/tdUjh10ybrlVhBFYxhdf5kzbFoZ848dNnahEKHahonhTLEH1WjxlM+wUv1Bqj2oEvAhd1ZScxsWIJbDAlGu5sg5wyecBTQM1uAtBKO2ZckpVF5VVgbmFUL7nkEi0P/9hEC95GVhkD4zwbd/HcNgmo97xMH6mXRlaYbnxBm0XQqbVJWaPFZ/MeDsOlTYM7QjYbjUFsttQMoWhkQwvabVmCFxb/Wo3Oe7Ky1LAAZHmNNuqfu10jxEMIjIw/YlpuURr6BXex3NC+g0KmVXri8Sd0/BHzylaT6pT3y3+1GtnD3LEjnVGt1Rji2IRk8fO/MCFoXdDO0jjQpBRpF9iJBmGLQRGAbcVE/JVUmwbDMWDAAD0t2gZypHgGHkgO+NUd1NhFzf7YWacdZPmRFjskNUu2Exaacho5DJ/9iV17s8l2eG9I5VP223YooAdkCFx3/XV6TZutRwxmrDwhWLD7pZiVBjF/0tfoH7TNov5ou4DVS7+7PWeVCsZQzCCDmKtmu/7VKydMA1Rt16P89vX10kufg3s70ocxhCkQ06XAijsr7QiBYhqtEx7MMZpqozJ5lolj6XXFb6ttFOHBVi7z5W/lm1paQwK/Ztu0lbHaL8wgu0SyYoVgjja5HsHQQ2YtURTfVgCLnqVh9Auo1hjVLnxZPYBZFX/OcN5552VjIPmXycNW1E2o5L1GqJV2LD6dQVwNVKFiqxFp3sypYtKmymTZqCawYs3uvjDCUL3nZfpIvTTS8nTXa5TrUH4RoNnysvOubOoSxB8viC9yNteIC4gm2ez4glUIOwYynpf9wyKL1WdWuVhNgmcWs9GmqjbKqKNk79GG+h3cT3noLLCBC8Yu+A34jv79++tKoJipNpBCx6h33HmHBsK/d0dqi1BFxRio6JwQAzQCTxEh2SOA1WpwCEXi3KavI0xVW5FBG80yPo0Y0wUYRyYJBBCYRVbLYBjFTl338Gdwgmx1CgaEQTb9wyykHQRjZ6YBCJmYDqTEwFeGbOIhLnVMCdOmIoY4jVP2ujPzMZNIVuwYJMp827Ll7mvxmDSZYMVuOnBdjxjsISZdY87uueceDWP5HCaBVY6U0MpiQtITiVUctNCs7BihmEnJtM2ME0bW/tEEs91ySuDM9u2NEia1slOXjkNl0y8qm+WLACUb1ugETjzMwtZYYw1d+ZUd0HSrZZg3ozJ5lolj6XXFb6ttlNU92nP+fETmH+sbrdSr3fgVff8ybbqZOmA+i2kczKb4gqiAXi8djlyA6eOMmPwcz0oXadpRI9XSSvsFcew7FI1R7cKXcqGohaFj/GTF3qhMHhaHd4sIwbAWNduOMVdjhYDVFWsbCDgpYa3DCgCCHytaHF/Ct8HSB6r3vEwfqZdGWp7ueo2gwfhZi8cra1V06SWX6sogvG7eJBQ8UfBzhINR/ptZeDt/EfI43gdTU/FdajppWyGDN2AFGx67UbM/Mk8FKgRZrMMQrHbccUfth80WEP6FlbleL1QBGAcAo/VBWmafepiblFhGXGLxJRRUk+ZZOTIyG05M3YzQHMOEp1I3mllMDCFMByw+whorTdiXIlBx4JqtSokjazaQ1hv0STcVer7+5q8ly7ROtpRJ/rZUar+sVMnuZCSlQh5L8WiYWVXDFIUOAKX1Zxk+T/h0mJaMiY/BDTy23HJLnQwROFNTjJRRTOtg5UYIszKmk2KZfCwNylir7jxHwBbHXi7VzBIBEF+3dAWk3rdNsbG2oQn24n/2LfOMPe0GU1hWKGAIwDNVNti3tm9rEGGKYN8ZfzbO6LFVVL45gz8rOjDsTBSsfpCP9TfMamnf1SYbnkP2y/XnX/xpi52fSGib+e9o7Sh9367tlzStLaTtx8wrUqxs8Ef7TpnFwVoFR3wSGTdY6THTZHwTwIvz3BgTYOrAmb7KuIECgBUvTGN5Vo3sO5gJLfEoGxPImf85U5mhsukXlc3yhZllIoEJRjsKwWhh8icbWHTwByiTZ5k45FMWf+LmqahtomihPdjYbe/wjSAba1pto4aTbEqgfgqYSTGOomBjpQZqpb22G7+i71+mTVMPa382hhBmeFo/IywlBBoYTSwVypyhg9AOfvQnfDWNSB/mH6befCfL9AverzVGlcGXcaLWGGVlpA0wfjAGmsKTZ2XywHdVdvFVAZ0VAdIBb1YUIHw8TFmrAbl/Zdpx0bcCb8Zt2ZRH+ziMqikIUAzA41B3/CqN8HeDZ7BVrXrPy/SRemlY3t35l++Dr5EJp/my8l0xoWOMqEf77rev4m59Lo3PN2NFFz6Y+ZoFBzPFTOO1+5rvyJhm/a7Z9I2HxJxRNqvIzlVtJL28QGXvInCaYEWbbpSYF0hbNqqo+h0bTbPt8YXJbiuJYKA73LEbD9uACzOiW4SK/X9cZpllKnb+YAciGdzYLUH/RIOgO98Jo6M75Fk420SKM2m2RTrvsPONNCDdQpItGtkdh62P2TVQtElaJ7Z95UwHzrghrjCTWjZhMmvWmS0lRSjIyiXmDFoudnMR07ssnPREC5rtcmTlFaZU02fHFHY3S+vIjoTsPASxW096HpZMnpHdn/LEFulpPGGudJtZ4rGzlkzEWZlkANYzRGRg0HM4rEzsusLZQmzzaWH8pvnVyqfRulM2dqJhK2jy4RuxE0zZb1utbZBubyR2fJRBMftmIuxE0W7qn2h59JwSEXwqdrUzHPhuYu6hONP+ZSDPtgoWZinSr/gW/LGd9P+1d3ZHjcNQGM3O7EN64YXtgDrYiigitMDbUgNFMNsHw8Deox0ZJ5Flx3bAOEczjIkt6+dY/7a+21apQyUKVatcJuIN63t8/5+CprygJpivoXCV741B9jt22oiP69hCQ9IV5aBY1U3nUAxEdRIpf+7N4SCFHpObpLQXiyvp/E3I/6MQVAoTVTIUhLgfO274iw4yyaRzLjrC99gnltIcq5GpnnMexT9MFyDZzW/qHvWV/FF3o/NJbVJWNEK1FLtUnMc/ZRdV0ZgAZNTFI6pGPCPSSN6wwRSr4nuqfNw4JPyutOWIaTPiLVz+mY48J+z3ldyQOPv8nML/MA2lsklZp42EMazjTWCytxeD9caOHkqolBHc2DKa04K8PjLKxMcf5TMmpunylPIak/EUxpz8Ss+/r0xzHWW7XBdR/+KZxaJB6u/IM/0ZaofU20NHnT/Vlg5271C3pD/hucVkv+l3c/hD60VfG1XjW2ujcjraR/pK+qFDV4sj+41FmpTPWMlP5ZS2C/MQtBOYU4i3Ftlr8Vgrx7VnhRor4yn6duoF7SltHs8VBcN4u57aNszA0A6iHItSb3Z91/FXqyNcHxIG/pbskO5H6bDL0ffQF7bVnrv8wpj+jT6v5CgXtDn0OfQJsZhY8jbbOco0fRpjzqEuJvGNunb7HsbWjEVpG8c6+j/G7F2O9GZTQ11+SudjsTOVe5R/l+r4fOwsLlZykv2mmKWnhxOvEk+Oh86iLSHMgDxebTYGNfMgnfM04DRMMZNt4kHKGBcrOmkgFp8dNNc+859YSUtGe7PU7Ji4GdghrUtB7RvkjQk/3zN3PBgnpuM7dH3P9tC/v8cRYAGBZ/o36h/lv6vsYCOIRnYNjjySn7aj7tEmZRef66R2If9uH5lkMflq+29f7/qfDo06Gqv2XV7S+b7wa2kjgFIHzaCw5vri5N4hfmpxnPvalDJKmaAd6uM0JQ9z8Ss9/yFlekzaiQuTI/ShY1ysxjeT3677++rF0DZqDr6lutNO95A48JPHEqS9PeZoh9X1/5hyTNtMvNnF24RkpoHfeZGYdLDY1PY35HoOs1ZH+uLIYSz1SPpZ3D518WCp+TlMF/ljgf0UhzFsFm/jq5c0LuCFSHz1MHhieUpcc/nFBEw2PTRXmHOH84MAY8VDJwEJSEACEpDABRFgny6fRfFZuU4CayLAJ4t8KokAC/sqETNjb+x2u11TNkfnhU9YUZBFrRPxs5iMp2078SKk2boxOvAz3MhUhf1Un/Ep5ZTk/5xys/dKQAISkIAEJPB9CLBPAlUv9kGjDIrohE4CayOAaAN2p9jfzb5Z1KSdUH08ZXQG2NeEvTb2JaLSF5+QfnhY2H/sFVv6hApkvqlaWMExORKQgAQkIIFzEUCZFgO9DKrud/eb29+354rKcCXwZQQQbMEAOwIjiHpguzQr231Zoox49QScVK3+EZtBCUhAAhKQwH8CKDk+/nncXP+6TrL8cpHAmgmgGJcVKdecT/O2DAJOqpbxHEyFBCQgAQlIQAISkIAEJPBNCcxm/Peb5t9kS0ACEpCABCQgAQlIQAISmETASdUkfN4sAQlIQAISkIAEJCABCVw6ASdVl14CzL8EJCABCUhAAhKQgAQkMImAk6pJ+LxZAhKQgAQkIAEJSEACErh0Ant2qu7u7jYPDw+XzsT8S0ACEpCABCQgAQlIQAISOCJwdXW12e12R+f3JlXPz8+bp6enI0+ekIAEJCABCUhAAhKQgAQkcOkE3t7eigj2JNVfXl42r6+vRY+elIAEJCABCUhAAhKQgAQkcMkEMCS93W6PEOxNqo6uekICEpCABCQgAQlIQAISkIAEqgQUqqji8aIEJCABCUhAAhKQgAQkIIE6ASdVdT5elYAEJCABCUhAAhKQgAQkUCXgpKqKx4sSkIAEJCABCUhAAhKQgATqBP4BArUR9czHrs0AAAAASUVORK5CYII=)

These continual learning scenarios can be distinguished from each other based on whether task identity information (or the task label) is provided to the algorithm, and – if it is not – whether task identity must be inferred (see schematic in the lecture slides).

In this paper you can find more details about these different types of continual learning.

Task-incremental Split MNIST versus class-incremental Split MNIST

Now, let’s get back to our Split MNIST example. To start with, let’s identify according to which scenario Split MNIST was performed in the previous section.

Recall that the Split MNIST problem consists of five tasks, whereby each task contains two digits. In the previous section, the model was set-up in such a way that it had a separate output layer for each of these tasks (this is typically called a ‘multi-headed output layer’). At test time, the model then used the output layer of the task to which the example to be classified belonged. This means that it was assumed that the model always knows which task it must perform, so this was an example of task-incremental learning.

In the continual learning literature. this variant of Split MNIST is also referred to as multi-headed Split MNIST. However, although a multi-headed output layer is probably the most common way to use task identity information, it is not the only way (for example, see this paper).

Now, let’s reorganize the Split MNIST problem to set it up as a class-incremental learning problem. That is, task information is no longer provided to the model; and the model must be able to decide itself to which task a test sample belongs. This means that, after all tasks have been learned, the model must now choose between all ten digits. This variant of Split MNIST is also referred to as single-headed Split MNIST.

# Load the MNIST dataset
x_train, t_train, x_test, t_test = load_mnist(mnist_train, mnist_test,
                                              verbose=True)

# Define which classes are part of each task
classes_per_task = [(0, 1), (2, 3), (4, 5), (6, 7), (8, 9)]

# Divide the MNIST dataset in tasks
task_data = []
for _, classes_in_this_task in enumerate(classes_per_task):

  # Which data-points belong to the classes in the current task?
  train_mask = np.isin(t_train, classes_in_this_task)
  test_mask = np.isin(t_test, classes_in_this_task)
  x_train_task, t_train_task = x_train[train_mask], t_train[train_mask]
  x_test_task, t_test_task = x_test[test_mask], t_test[test_mask]

  # Add the data for the current task
  task_data.append((x_train_task, t_train_task, x_test_task, t_test_task))

# In contrast to the task-incremental version of Split MNIST explored in the
# last section, now task identity information will not be provided to the model
x_train dim: (60000, 1, 28, 28) and type: uint8
t_train dim: (60000,) and type: int64
x_train dim: (10000, 1, 28, 28) and type: uint8
t_train dim: (10000,) and type: int64

Example: EWC on the class-incremental version of Split MNIST

Let’s now try the EWC method on this class-incremental version of Split MNIST.

# Define the model and the optimzer
model = Net().to(DEVICE)
optimizer = optim.SGD(model.parameters(), lr=0.001)

# Set 'lambda', the hyperparameter of EWC
ewc_lambda = 0.2

# Define dictionaries to store values needed by EWC
fisher_dict = {}
optpar_dict = {}

# Prepare list to store average accuracies after each task
ewc_accs = []

# Loop through all tasks
for id, task in enumerate(task_data):

  # Collect training data
  x_train, t_train, _, _ = task

  # Training with EWC
  print("Training on task: ", id)
  for epoch in range(1, 2):
    train_ewc(model, model, id, x_train, t_train, optimizer, epoch,
              ewc_lambda, fisher_dict, optpar_dict, device=DEVICE)

  on_task_update(id, x_train, t_train, model, model, fisher_dict,
                 optpar_dict, device=DEVICE)

  # Evaluate performance after training on this task
  avg_acc = 0
  for id_test, task in enumerate(task_data):
    print(f"Testing on task: {id_test}")
    _, _, x_test, t_test = task
    acc = test(model, x_test, t_test, device=DEVICE)
    avg_acc = avg_acc + acc

  print(f"Avg acc: {avg_acc / len(task_data)}")
  ewc_accs.append(avg_acc / len(task_data))
Training on task:  0
Train Epoch: 1 	Loss: 0.226513
Testing on task: 0
Test set: Average loss: 0.0000, Accuracy: 2109/2115 (100%)

Testing on task: 1
Test set: Average loss: 0.0064, Accuracy: 0/2042 (0%)

Testing on task: 2
Test set: Average loss: 0.0072, Accuracy: 0/1874 (0%)

Testing on task: 3
Test set: Average loss: 0.0083, Accuracy: 0/1986 (0%)

Testing on task: 4
Test set: Average loss: 0.0083, Accuracy: 0/1983 (0%)

Avg acc: 19.94326241134752
Training on task:  1
Train Epoch: 1 	Loss: 1.445262
Testing on task: 0
Test set: Average loss: 0.0015, Accuracy: 178/2115 (8%)

Testing on task: 1
Test set: Average loss: 0.0004, Accuracy: 1767/2042 (87%)

Testing on task: 2
Test set: Average loss: 0.0024, Accuracy: 1/1874 (0%)

Testing on task: 3
Test set: Average loss: 0.0030, Accuracy: 0/1986 (0%)

Testing on task: 4
Test set: Average loss: 0.0039, Accuracy: 0/1983 (0%)

Avg acc: 19.00044968254241
Training on task:  2
Train Epoch: 1 	Loss: 1.526267
Testing on task: 0
Test set: Average loss: 0.0020, Accuracy: 84/2115 (4%)

Testing on task: 1
Test set: Average loss: 0.0015, Accuracy: 172/2042 (8%)

Testing on task: 2
Test set: Average loss: 0.0004, Accuracy: 1690/1874 (90%)

Testing on task: 3
Test set: Average loss: 0.0032, Accuracy: 0/1986 (0%)

Testing on task: 4
Test set: Average loss: 0.0043, Accuracy: 0/1983 (0%)

Avg acc: 20.515235179052148
Training on task:  3
Train Epoch: 1 	Loss: 2.382280
Testing on task: 0
Test set: Average loss: 0.0016, Accuracy: 194/2115 (9%)

Testing on task: 1
Test set: Average loss: 0.0014, Accuracy: 481/2042 (24%)

Testing on task: 2
Test set: Average loss: 0.0014, Accuracy: 325/1874 (17%)

Testing on task: 3
Test set: Average loss: 0.0013, Accuracy: 914/1986 (46%)

Testing on task: 4
Test set: Average loss: 0.0018, Accuracy: 3/1983 (0%)

Avg acc: 19.248787692590746
Training on task:  4
Train Epoch: 1 	Loss: 2.575902
Testing on task: 0
Test set: Average loss: 0.0018, Accuracy: 319/2115 (15%)

Testing on task: 1
Test set: Average loss: 0.0015, Accuracy: 143/2042 (7%)

Testing on task: 2
Test set: Average loss: 0.0016, Accuracy: 89/1874 (5%)

Testing on task: 3
Test set: Average loss: 0.0015, Accuracy: 254/1986 (13%)

Testing on task: 4
Test set: Average loss: 0.0013, Accuracy: 1171/1983 (59%)

Avg acc: 19.73526967505204

That didn’t work well…

The model only correctly predicts the classes from the last task it has seen, all earlier seen classes seem to be forgotten.

You might wonder whether the reason that EWC performed so badly in the above example is because we chose an unsuitable value for the hyperparameter lambda. Although we don’t have time to demonstrate this, there are no values of lambda that would lead to good performance.

In general, parameter regularization based methods, such as EWC, have been found not to work well on class-incremental learning problems.

Replay

As discussed in the lecture of the previous section, another popular continual learning strategy is replay. Let’s see whether replay works better on the class-incremental learning version of Split MNIST than EWC did.

One implementation of replay is to simply store all data from previously seen tasks, and to then, whenever a new task must be learned, mix in that stored data with the training data of the new task.

To achieve this form of replay, let’s define the following function for shuffling multiple datasets (e.g., the data from previous tasks with the data from the current task) together:

def shuffle_datasets(dataset, seed, in_place=False):
  """
  Shuffle a list of two (or more) datasets.

  Args:
    dataset: np.ndarray
      Dataset
    seed: Integer
      A non-negative integer that defines the random state.
    in_place: boolean
      If True, shuffle datasets in place

  Returns:
    Nothing
  """

  np.random.seed(seed)
  rng_state = np.random.get_state()
  new_dataset = []
  for x in dataset:
    if in_place:
      np.random.shuffle(x)
    else:
      new_dataset.append(np.random.permutation(x))
    np.random.set_state(rng_state)

  if not in_place:
    return new_dataset

Note that this form of replay is somewhat extreme, as it stores all the training data from previous tasks. In practice, replay is often implemented in ways that store less data, for example either by using relatively small memory buffers (see this paper) or by learning a generative model to then generate the data to be replayed (see this paper or this paper).

Example: Test replay on the class-incremental version of Split MNIST

Let’s try whether this replay strategy works better than EWC.

# Define the model and the optimizer
model = Net().to(DEVICE)
optimizer = optim.SGD(model.parameters(), lr=0.01)

# Prepare list to store average accuracies after each task
rehe_accs = []

# Loop through all tasks
for id, task in enumerate(task_data):

  # Collect training data
  x_train, t_train, _, _ = task

  # Add replay
  for i in range(id):
    past_x_train, past_t_train, _, _ = task_data[i]
    x_train = np.concatenate((x_train, past_x_train))
    t_train = np.concatenate((t_train, past_t_train))

  x_train, t_train = shuffle_datasets([x_train, t_train], seed=SEED)

  # Training
  print(f"Training on task: {id}")
  for epoch in range(1, 3):
    train(model, x_train, t_train, optimizer, epoch, device=DEVICE)

  # Evaluate performance after training on this task
  avg_acc = 0
  for id_test, task in enumerate(task_data):
    print(f"Testing on task: {id_test}")
    _, _, x_test, t_test = task
    acc = test(model, x_test, t_test, device=DEVICE)
    avg_acc = avg_acc + acc

  print(f"Avg acc: {avg_acc / len(task_data)}")
  rehe_accs.append(avg_acc/len(task_data))
Training on task: 0
Train Epoch: 1 	Loss: 0.180061
Train Epoch: 2 	Loss: 0.012716
Testing on task: 0
Test set: Average loss: 0.0000, Accuracy: 2114/2115 (100%)

Testing on task: 1
Test set: Average loss: 0.0119, Accuracy: 0/2042 (0%)

Testing on task: 2
Test set: Average loss: 0.0132, Accuracy: 0/1874 (0%)

Testing on task: 3
Test set: Average loss: 0.0141, Accuracy: 0/1986 (0%)

Testing on task: 4
Test set: Average loss: 0.0110, Accuracy: 0/1983 (0%)

Avg acc: 19.990543735224584
Training on task: 1
Train Epoch: 1 	Loss: 0.402671
Train Epoch: 2 	Loss: 0.205235
Testing on task: 0
Test set: Average loss: 0.0000, Accuracy: 2102/2115 (99%)

Testing on task: 1
Test set: Average loss: 0.0000, Accuracy: 1992/2042 (98%)

Testing on task: 2
Test set: Average loss: 0.0040, Accuracy: 0/1874 (0%)

Testing on task: 3
Test set: Average loss: 0.0048, Accuracy: 0/1986 (0%)

Testing on task: 4
Test set: Average loss: 0.0039, Accuracy: 0/1983 (0%)

Avg acc: 39.38735259317917
Training on task: 2
Train Epoch: 1 	Loss: 0.532893
Train Epoch: 2 	Loss: 0.392877
Testing on task: 0
Test set: Average loss: 0.0000, Accuracy: 2084/2115 (99%)

Testing on task: 1
Test set: Average loss: 0.0000, Accuracy: 1988/2042 (97%)

Testing on task: 2
Test set: Average loss: 0.0000, Accuracy: 1817/1874 (97%)

Testing on task: 3
Test set: Average loss: 0.0026, Accuracy: 0/1986 (0%)

Testing on task: 4
Test set: Average loss: 0.0024, Accuracy: 0/1983 (0%)

Avg acc: 58.569638110341316
Training on task: 3
Train Epoch: 1 	Loss: 0.637280
Train Epoch: 2 	Loss: 0.496460
Testing on task: 0
Test set: Average loss: 0.0000, Accuracy: 2068/2115 (98%)

Testing on task: 1
Test set: Average loss: 0.0000, Accuracy: 1977/2042 (97%)

Testing on task: 2
Test set: Average loss: 0.0000, Accuracy: 1793/1874 (96%)

Testing on task: 3
Test set: Average loss: 0.0000, Accuracy: 1887/1986 (95%)

Testing on task: 4
Test set: Average loss: 0.0018, Accuracy: 0/1983 (0%)

Avg acc: 77.05748490353808
Training on task: 4
Train Epoch: 1 	Loss: 0.605949
Train Epoch: 2 	Loss: 0.674918
Testing on task: 0
Test set: Average loss: 0.0000, Accuracy: 2064/2115 (98%)

Testing on task: 1
Test set: Average loss: 0.0000, Accuracy: 1966/2042 (96%)

Testing on task: 2
Test set: Average loss: 0.0000, Accuracy: 1821/1874 (97%)

Testing on task: 3
Test set: Average loss: 0.0000, Accuracy: 1873/1986 (94%)

Testing on task: 4
Test set: Average loss: 0.0001, Accuracy: 1793/1983 (90%)

Avg acc: 95.15347301254934

And finally, let’s compare the performance of EWC and Replay on the class-incremental version of Split MNIST in a plot:

Plot EWC vs. Replay

# @title Plot EWC vs. Replay
plt.plot([1, 2, 3, 4, 5], rehe_accs, '-o', label="Replay")
plt.plot([1, 2, 3, 4, 5], ewc_accs, '-o', label="EWC")
plt.xlabel('Tasks Encountered', fontsize=14)
plt.ylabel('Average Accuracy', fontsize=14)
plt.title('CL Strategies on Class-incremental version of Split MNIST',
          fontsize=14);
plt.xticks([1, 2, 3, 4, 5])
plt.legend(prop={'size': 16})
plt.show()
../../../_images/W3D4_Tutorial1_92_0.png

Exercise 3: Identify the continual learning scenario of the permuted MNIST example from Section 1

What type of ‘scenario’ was the permuted MNIST problem that was introduced in Section 1? Was it task-incremental, domain-incremental or class-incremental? Try to motivate your answer.

Click for solution


Section 4: Evaluation of continual learning algorithms

Time estimate: ~20mins

Understanding how your CL algorithm is performing is key to gain insights on its behavior and to decide how to improve it.

Here, we will focus on how to build some of the most important CL metrics!

Video 4: Continual Learning Evaluation

We will now simulate the data for T tasks and record all the accuracy values in a single TxT matrix.

Section 4.1: Average Accuracy

The Average Accuracy (ACC) metric computes the average accuracy over all tasks.

def ACC(result_matrix):
  """
  Average Accuracy metric

  Args:
    result_matrix: np.ndarray
      TxT matrix containing accuracy values in each (i, j) entry.
      (i, j) -> test accuracy on task j after training on task i

  Returns:
    acc: float
      Average accuracy over all tasks
    final_accs:  np.ndarray
      Accuracies after training extracted from result matrix
  """

  final_accs = abs(result_matrix[-1, :])  # Take accuracies after final training
  acc = np.mean(final_accs)  # Compute average

  return acc, final_accs

Section 4.2: Backward Transfer

The Backward Transfer (BWT) metric of task i computes the accuracy on task i after training on last task minus the accuracy on task i after training on task i.

To get the average BWT you have to average across all tasks.

Negative BWT expresses the amount of forgetting suffered by the algorithm.

def BWT(result_matrix):
  """
  Backward Transfer metric

  Args:
    result_matrix: np.ndarray
      TxT matrix containing accuracy values in each (i, j) entry.
      (i, j) -> test accuracy on task j after training on task i

  Returns:
    task_bwt: np.ndarray
      BWT for each task
    average_bwt: float
      Average BWT across all tasks
  """

  final_accs = result_matrix[-1, :]  # Take accuracies after final training
  # Accuracies on task i right after training on task i, for all i
  training_accs = np.diag(result_matrix)
  task_bwt = final_accs - training_accs  # BWT for each task
  average_bwt = np.mean(task_bwt)  # Compute average

  return average_bwt, task_bwt

Coding Exercise 4.2: Evaluate your CL algorithm

You should replace the ellipses, i.e., ..., with your code. You should train 5 tasks and your result_matrix is [0,1] values from a randomly sampled distribution.

This is the only cell you have to modify :)

Hint: Use np.random.choice with weight values p = [0.1, 0.9].

def make_result_matrix(T):
  """
  Create a TxT matrix with values between 0 and 1 to
  be used to compute the metrics.

  Args:
    T : int
      The number of tasks

  Returns:
    result_matrix : numpy.array
      A TxT matrix
  """
  #################################################
  ## TODO for students: update the value of the parameter ##
  raise NotImplementedError("Student exercise: implement the mask")
  #################################################
  distribution = np.random.rand(T**2)
  # Create a random mask
  mask = ...
  distribution[mask] = np.nan

  result_matrix = []
  count = 0
  for j in range(T):
    temp = []
    for i in range(T):
      temp.append(distribution[count])
      count += 1
    result_matrix.append(temp)
  result_matrix = np.array(result_matrix)

  return result_matrix


# Add event to airtable
atform.add_event('Coding Exercise 4.2: Evaluate your CL algorithm')

set_seed(seed=SEED)
T = len(rehe_accs)  # Number of tasks
## Uncomment below to test you function
# result_matrix = make_result_matrix(T)
# print(result_matrix)
Random seed 2021 has been set.

Click for solution

[[0.60597828 0.73336936 0.13894716 0.31267308        nan]
 [0.12816238 0.17899311 0.75292543 0.66216051 0.78431013]
 [0.0968944  0.05857129 0.96239599 0.61655744 0.08662996]
 [0.56127236 0.61652471 0.96384302 0.57430429 0.37116085]
 [0.45214524 0.20185025 0.56930512 0.19509597 0.58370402]]

Think! 4.2: Performance metrics

  1. Why we choose a specific number of performance metrics even if we have access to numerous metrics?

  2. Why does the result matrix have nan values?

Student Response

# @title Student Response
from ipywidgets import widgets


text=widgets.Textarea(
   value='Type your answer here and click on `Submit!`',
   placeholder='Type something',
   description='',
   disabled=False
)

button = widgets.Button(description="Submit!")

display(text,button)

def on_button_clicked(b):
   atform.add_answer('q1' , text.value)
   print("Submission successful!")


button.on_click(on_button_clicked)

Click for solution

You don’t need to modify the next cell, just execute it to see metrics in action!

if result_matrix is None or T is None:
  raise ValueError("You should fill the values of `result_matrix`, and `T` first.")

print(f"\nResult matrix shape: {result_matrix.shape}")
print(f"Result matrix values:\n {result_matrix}")
# result_matrix : [value for value in result_matrix if not math.isnan(value)]
# Print(result_matrix)
# {rint Average Accuracy metric
acc, final_accs = ACC(result_matrix)
print(f"\nACC: {acc}")
print(f"Accuracies for each task: {final_accs}")

# Print Backward Transfer metric
bwt, bwt_task = BWT(result_matrix)
print(f"\nBWT: {bwt}")
print(f"BWT for each task: {bwt_task}")
print()

Expected Output:

Result matrix shape: (5, 5)
Result matrix values:
 [[0.60597828 0.73336936 0.13894716 0.31267308        nan]
 [0.12816238 0.17899311 0.75292543 0.66216051 0.78431013]
 [0.0968944  0.05857129 0.96239599 0.61655744 0.08662996]
 [0.56127236 0.61652471 0.96384302 0.57430429 0.37116085]
 [0.45214524 0.20185025 0.56930512 0.19509597 0.58370402]]
[0.45214524 0.20185025 0.56930512 0.19509597 0.58370402]

ACC: 0.4004201200681699
Accuracies for each task: [0.45214524 0.20185025 0.56930512 0.19509597 0.58370402]

BWT: -0.18065501708115667
BWT for each task: [-0.15383304  0.02285714 -0.39309087 -0.37920832  0.        ]

Section 5: Continual Learning Applications

Time estimate: ~10mins

Continual Learning with deep architectures may help us develop sustainable AI systems that can efficiently improve their skills and knowledge over time, adapting to ever-changing environments and learning objectives. In this section we will discuss about intriguing real-world applications that would highly benefit from recent advances in Continual Learning.

Video 5: Continual Learning Applications

CORe50 is an interesting real-world video dataset composed of 50 domestic objects belonging to 10 different categories and specifically designed for Continual Learning. You can find more information about the dataset and benchmark in its official website.

Here we will use the Avalanche library to automatically download and use this dataset. Avalanche allows you to explore more challenging datasets and tasks to bring your continual learning algorithms into the real-world!

Download the repo and import the modules

# @title Download the repo and import the modules
import sys, requests, tarfile, os
# !pip install git+https://github.com/ContinualAI/avalanche.git --quiet

# We download from OSF to ensure stability of the download.
REPO_PATH = "avalanche"
url = "https://osf.io/k3gp6/download"

if not os.path.exists(REPO_PATH):
  r = requests.get(url, allow_redirects=True)
  fname = REPO_PATH + '.tar.gz'
  with open(fname, 'wb') as fd:
    fd.write(r.content)

!pip install avalanche.tar.gz --quiet

Download the data

# @title Download the data

import zipfile, io, os

# To ensure that the data will be downloaded, we use our space on OSF.
# Original locations:
# Data core50: http://vps.continualai.org/data/core50_32x32.zip
# Other modules:
# 1. https://vlomonaco.github.io/core50/data/batches_filelists_NICv2.zip
# 2. https://vlomonaco.github.io/core50/data/batches_filelists.zip
# 3. https://vlomonaco.github.io/core50/data/paths.pkl
# 4. https://vlomonaco.github.io/core50/data/LUP.pkl
# 5. https://vlomonaco.github.io/core50/data/labels.pkl
# 6. https://vlomonaco.github.io/core50/data/labels2names.pkl

fname = "core50"
urls = ["https://osf.io/x3zvr/download",
        "https://osf.io/37nbk/download",
        "https://osf.io/sy86n/download",
        "https://osf.io/8hds7/download"]


for url in urls:
  r = requests.get(url, allow_redirects=True)
  z = zipfile.ZipFile(io.BytesIO(r.content))
  z.extractall(fname)

The scenario “New Instances” (scenario=ni) corresponds to the previously introduced Domain-Incremental setting and it’s based on the idea of encountering images of the same classes for every incremental batch of data (or experience if you will). The mini=True option downloads data 32x32 instead of the original 128x128.

Note: We download the data from OSF. Due to space limitations on OSF, if you want to use the 128x18 dataset, you have to download it via setting mini=False.

Original locations:

  • 32x32 data: http://vps.continualai.org/data/core50_32x32.zip

  • 128x128 data: http://bias.csr.unibo.it/maltoni/download/core50/core50_128x128.zip

from avalanche.benchmarks.classic import CORe50
benchmark = CORe50(dataset_root='core50', scenario="ni", mini=True)
Loading labels...
Loading LUP...
Loading labels names...
Files already downloaded and verified
for exp in benchmark.train_stream:
  print(exp.classes_in_this_experience)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]

Explore the challenging CORe50 scenarios!

CORe50 offers a number of interesting preset scenarios already implemented and available to you through Avalanche.

You could further try to explore the different scenarios offered (like the challenging NICv2-391) and possibly even apply what you’ve previously learned (like a replay approach) to get the best accuracy you can!

See the CORe50 help menu below.

help(CORe50)
Help on function CORe50 in module avalanche.benchmarks.classic.core50:

CORe50(*, scenario: str = 'nicv2_391', run: int = 0, object_lvl: bool = True, mini: bool = False, train_transform: Union[Any, NoneType] = Compose(
    ToTensor()
    RandomHorizontalFlip(p=0.5)
    Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
), eval_transform: Union[Any, NoneType] = Compose(
    ToTensor()
    Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])
), dataset_root: Union[str, pathlib.Path] = None)
    Creates a CL benchmark for CORe50.
    
    If the dataset is not present in the computer, this method will
    automatically download and store it.
    
    This generator can be used to obtain the NI, NC, NIC and NICv2-* scenarios.
    
    The benchmark instance returned by this method will have two fields,
    `train_stream` and `test_stream`, which can be iterated to obtain
    training and test :class:`Experience`. Each Experience contains the
    `dataset` and the associated task label.
    
    The task label "0" will be assigned to each experience.
    
    The benchmark API is quite simple and is uniform across all benchmark
    generators. It is recommended to check the tutorial of the "benchmark" API,
    which contains usage examples ranging from "basic" to "advanced".
    
    :param scenario: CORe50 main scenario. It can be chosen between 'ni', 'nc',
        'nic', 'nicv2_79', 'nicv2_196' or 'nicv2_391.'
    :param run: number of run for the benchmark. Each run defines a different
        ordering. Must be a number between 0 and 9.
    :param object_lvl: True for a 50-way classification at the object level.
        False if you want to use the categories as classes. Default to True.
    :param mini: True for processing reduced 32x32 images instead of the
        original 128x128. Default to False.
    :param train_transform: The transformation to apply to the training data,
        e.g. a random crop, a normalization or a concatenation of different
        transformations (see torchvision.transform documentation for a
        comprehensive list of possible transformations). Defaults to None.
    :param eval_transform: The transformation to apply to the test data,
        e.g. a random crop, a normalization or a concatenation of different
        transformations (see torchvision.transform documentation for a
        comprehensive list of possible transformations). Defaults to None.
    :param dataset_root: Absolute path indicating where to store the dataset
        and related metadata. Defaults to None, which means that the default
        location for
        'core50' will be used.
    
    :returns: a properly initialized :class:`GenericCLScenario` instance.

Summary

Well, you did it! Congratulations on making it through your (first?) Continual Learning codebase. As mentioned, this is only the tip of the iceberg, and there’s a lot more you can dig into if you want to explore.

If you do want to explore, one of the best places to learn more is ContinualAI.org. There, you can interact with a large portion of the continual learning community, and find resources such as a database of relevant papers, lectures from researchers discussing their papers, additional tutorials, and much more. You might also be interested in Avalanche, the largest library for continual learning.

MILA also has a wonderful website with open Continual Learning course materials, by Dr. Irina Rish.

Further questions? Feel free to reach out to Keiland!