Bonus Tutorial: Facial recognition using modern convnets#
Week 2, Day 3: Modern Convnets
By Neuromatch Academy
Content creators: Laura Pede, Richard Vogg, Marissa Weis, Timo Lüddecke, Alexander Ecker
Content reviewers: Arush Tagade, Polina Turishcheva, Yu-Fang Yang, Bettina Hein, Melvin Selim Atay, Kelson Shilling-Scrivo
Content editors: Roberto Guidotti, Spiros Chavlis
Production editors: Anoop Kulkarni, Roberto Guidotti, Cary Murray, Gagana B, Spiros Chavlis
Notebook is based on an initial version by Ben Heil
Tutorial Objectives#
In this tutorial you will learn about:
An application of modern CNNs in facial recognition.
Ethical aspects of facial recognition.
These are the slides for the videos in this tutorial. If you want to download locally the slides, click here.
Setup#
Install dependencies#
Install facenet - A model used to do facial recognition
Show code cell source
# @title Install dependencies
# @markdown Install `facenet` - A model used to do facial recognition
!pip install facenet-pytorch --quiet
!pip install Pillow --quiet
Install and import feedback gadget#
Show code cell source
# @title Install and import feedback gadget
!pip3 install vibecheck datatops --quiet
from vibecheck import DatatopsContentReviewContainer
def content_review(notebook_section: str):
return DatatopsContentReviewContainer(
"", # No text prompt
notebook_section,
{
"url": "https://pmyvdlilci.execute-api.us-east-1.amazonaws.com/klab",
"name": "neuromatch_dl",
"user_key": "f379rz8y",
},
).render()
feedback_prefix = "W2D3_T2_Bonus"
# Imports
import glob
import torch
import numpy as np
import sklearn.decomposition
import matplotlib.pyplot as plt
from PIL import Image
from torchvision import transforms
from torchvision.utils import make_grid
from torchvision.datasets import ImageFolder
from facenet_pytorch import MTCNN, InceptionResnetV1
Set random seed#
Executing set_seed(seed=seed) you are setting the seed
Show code cell source
# @title Set random seed
# @markdown Executing `set_seed(seed=seed)` you are setting the seed
# For DL its critical to set the random seed so that students can have a
# baseline to compare their results to expected results.
# Read more here: https://pytorch.org/docs/stable/notes/randomness.html
# Call `set_seed` function in the exercises to ensure reproducibility.
import random
import torch
def set_seed(seed=None, seed_torch=True):
"""
Function that controls randomness. NumPy and random modules must be imported.
Args:
seed : Integer
A non-negative integer that defines the random state. Default is `None`.
seed_torch : Boolean
If `True` sets the random seed for pytorch tensors, so pytorch module
must be imported. Default is `True`.
Returns:
Nothing.
"""
if seed is None:
seed = np.random.choice(2 ** 32)
random.seed(seed)
np.random.seed(seed)
if seed_torch:
torch.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.cuda.manual_seed(seed)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
print(f'Random seed {seed} has been set.')
# In case that `DataLoader` is used
def seed_worker(worker_id):
"""
DataLoader will reseed workers following randomness in
multi-process data loading algorithm.
Args:
worker_id: integer
ID of subprocess to seed. 0 means that
the data will be loaded in the main process
Refer: https://pytorch.org/docs/stable/data.html#data-loading-randomness for more details
Returns:
Nothing
"""
worker_seed = torch.initial_seed() % 2**32
np.random.seed(worker_seed)
random.seed(worker_seed)
Set device (GPU or CPU). Execute set_device()#
Show code cell source
# @title Set device (GPU or CPU). Execute `set_device()`
# especially if torch modules used.
# Inform the user if the notebook uses GPU or CPU.
def set_device():
"""
Set the device. CUDA if available, CPU otherwise
Args:
None
Returns:
Nothing
"""
device = "cuda" if torch.cuda.is_available() else "cpu"
if device != "cuda":
print("WARNING: For this notebook to perform best, "
"if possible, in the menu under `Runtime` -> "
"`Change runtime type.` select `GPU` ")
else:
print("GPU is enabled in this notebook.")
return device
SEED = 2021
set_seed(seed=SEED)
DEVICE = set_device()
Random seed 2021 has been set.
WARNING: For this notebook to perform best, if possible, in the menu under `Runtime` -> `Change runtime type.` select `GPU`
Section 1: Face Recognition#
Time estimate: ~12mins
Section 1.1: Download and prepare the data#
Download Faces Data#
Show code cell source
# @title Download Faces Data
import requests, zipfile, io, os
# Original link: https://github.com/ben-heil/cis_522_data.git
url = 'https://osf.io/2kyfb/download'
fname = 'faces'
if not os.path.exists(fname+'zip'):
print("Data is being downloaded...")
r = requests.get(url, stream=True)
z = zipfile.ZipFile(io.BytesIO(r.content))
z.extractall()
print("The download has been completed.")
else:
print("Data has already been downloaded.")
Data is being downloaded...
The download has been completed.
Video 1: Face Recognition using CNNs#
Submit your feedback#
Show code cell source
# @title Submit your feedback
content_review(f"{feedback_prefix}_Face_Recognition_using_CNNs_Video")
One application of large CNNs is facial recognition. The problem formulation in facial recognition is a little different from the image classification we’ve seen so far. In facial recognition, we don’t want to have a fixed number of individuals that the model can learn. If that were the case then to learn a new person it would be necessary to modify the output portion of the architecture and retrain to account for the new person.
Instead, we train a model to learn an embedding where images from the same individual are close to each other in an embedded space, and images corresponding to different people are far apart. When the model is trained, it takes as input an image and outputs an embedding vector corresponding to the image.
To achieve this, facial recognitions typically use a triplet loss that compares two images from the same individual (i.e., “anchor” and “positive” images) and a negative image from a different individual (i.e., “negative” image). The loss requires the distance between the anchor and negative points to be greater than a margin \(\alpha\) + the distance between the anchor and positive points.
Section 1.2: View and transform the data#
A well-trained facial recognition system should be able to map different images of the same individual relatively close together. We will load 15 images of three individuals (maybe you know them - then you can see that your brain is quite well in facial recognition).
After viewing the images, we will transform them: MTCNN (github repo) detects the face and crops the image around the face. Then we stack all the images together in a tensor.
Display Images#
Here are the source images of Bruce Lee, Neil Patrick Harris, and Pam Grier
Show code cell source
# @title Display Images
# @markdown Here are the source images of Bruce Lee, Neil Patrick Harris, and Pam Grier
train_transform = transforms.Compose((transforms.Resize((256, 256)),
transforms.ToTensor()))
face_dataset = ImageFolder('faces', transform=train_transform)
image_count = len(face_dataset)
face_loader = torch.utils.data.DataLoader(face_dataset,
batch_size=45,
shuffle=False)
dataiter = iter(face_loader)
images, labels = next(dataiter)
# Show images
plt.figure(figsize=(15, 15))
plt.imshow(make_grid(images, nrow=15).permute(1, 2, 0))
plt.axis('off')
plt.show()
Image Preprocessing Function#
Show code cell source
# @title Image Preprocessing Function
def process_images(image_dir: str, size=256):
"""
This function returns two tensors for the
given image dir: one usable for inputting into the
facenet model, and one that is [0,1] scaled for
visualizing
Parameters:
image_dir: string
The glob corresponding to images in a directory
size: int
Size [default: 256]
Returns:
model_tensor: torch.tensor
A image_count x channels x height x width
tensor scaled to between -1 and 1,
with the faces detected and cropped to the center
using mtcnn
display_tensor: torch.tensor
A transformed version of the model
tensor scaled to between 0 and 1
"""
mtcnn = MTCNN(image_size=size, margin=32)
images = []
for img_path in glob.glob(image_dir):
img = Image.open(img_path)
# Normalize and crop image
img_cropped = mtcnn(img)
images.append(img_cropped)
model_tensor = torch.stack(images)
display_tensor = model_tensor / (model_tensor.max() * 2)
display_tensor += .5
return model_tensor, display_tensor
Now that we have our images loaded, we need to preprocess them. To make the images easier for the network to learn, we crop them to include just faces.
bruce_tensor, bruce_display = process_images('faces/bruce/*.jpg')
neil_tensor, neil_display = process_images('faces/neil/*.jpg')
pam_tensor, pam_display = process_images('faces/pam/*.jpg')
tensor_to_display = torch.cat((bruce_display, neil_display, pam_display))
plt.figure(figsize=(15, 15))
plt.imshow(make_grid(tensor_to_display, nrow=15).permute(1, 2, 0))
plt.axis('off')
plt.show()
Section 1.3: Embedding with a pretrained network#
We load a pretrained facial recognition model called FaceNet. It was trained on the VGGFace2 dataset which contains 3.31 million images of 9131 individuals.
We use the pretrained model to calculate embeddings for all of our input images.
resnet = InceptionResnetV1(pretrained='vggface2').eval().to(DEVICE)
# Calculate embedding
resnet.classify = False
bruce_embeddings = resnet(bruce_tensor.to(DEVICE))
neil_embeddings = resnet(neil_tensor.to(DEVICE))
pam_embeddings = resnet(pam_tensor.to(DEVICE))
Think! 1.3: Embedding vectors#
We want to understand what happens when the model receives an image and returns the corresponding embedding vector.
What are the height, width and number of channels of one input image?
What are the dimensions of one stack of images (e.g. bruce_tensor)?
What are the dimensions of the corresponding embedding (e.g. bruce_embeddings)?
What would be the dimensions of the embedding of one input image?
Hints:
You can double click on a variable name and hover over it to see the dimensions of tensors.
You do not have to answer the questions in the order they are asked.
Submit your feedback#
Show code cell source
# @title Submit your feedback
content_review(f"{feedback_prefix}_Embedding_vectors_Discussion")
We cannot show 512-dimensional vectors visually, but using Principal Component Analysis (PCA) we can project the 512 dimensions onto a 2-dimensional space while preserving the maximum amount of data variation possible. This is just a visual aid for us to understand the concept. Note that if you would like to do any calculation, like distances between two images, this would be done with the whole 512-dimensional embedding vectors.
embedding_tensor = torch.cat((bruce_embeddings,
neil_embeddings,
pam_embeddings)).to(device='cpu')
pca = sklearn.decomposition.PCA(n_components=2)
pca_tensor = pca.fit_transform(embedding_tensor.detach().cpu().numpy())
num = 15
categs = 3
colors = ['blue', 'orange', 'magenta']
labels = ['Bruce Lee', 'Neil Patrick Harris', 'Pam Grier']
markers = ['o', 'x', 's']
plt.figure(figsize=(8, 8))
for i in range(categs):
plt.scatter(pca_tensor[i*num:(i+1)*num, 0],
pca_tensor[i*num:(i+1)*num, 1],
c=colors[i],
marker=markers[i], label=labels[i])
plt.legend()
plt.title('PCA Representation of the Image Embeddings')
plt.xlabel('PC 1')
plt.ylabel('PC 2')
plt.show()
Great! The images corresponding to each individual are separated from each other in the embedding space!
If Neil Patrick Harris wants to unlock his phone with facial recognition, the phone takes the image from the camera, calculates the embedding and checks if it is close to the registered embeddings corresponding to Neil Patrick Harris.
Section 2: Ethics – bias/discrimination due to pre-training datasets#
Time estimate: ~19mins
Popular facial recognition datasets like VGGFace2 and CASIA-WebFace consist primarily of caucasian faces. As a result, even state of the art facial recognition models substantially underperform when attempting to recognize faces of other races.
Given the implications that poor model performance can have in fields like security and criminal justice, it’s very important to be aware of these limitations if you’re going to be building facial recognition systems.
In this example we will work with a small subset from the UTKFace dataset with 49 pictures of black women and 49 picture of white women. We will use the same pretrained model as in Section 8 of Tutorial 1, see and discuss the consequences of the model being trained on an imbalanced dataset.
Video 2: Ethical aspects#
Submit your feedback#
Show code cell source
# @title Submit your feedback
content_review(f"{feedback_prefix}_Ethical_aspects_Video")
Section 2.1: Download the Data#
Run this cell to get the data#
Show code cell source
# @title Run this cell to get the data
# Original link: https://github.com/richardvogg/face_sample.git
url = 'https://osf.io/36wyh/download'
fname = 'face_sample2'
if not os.path.exists(fname+'zip'):
print("Data is being downloaded...")
r = requests.get(url, stream=True)
z = zipfile.ZipFile(io.BytesIO(r.content))
z.extractall()
print("The download has been completed.")
else:
print("Data has already been downloaded.")
Data is being downloaded...
The download has been completed.
Section 2.2: Load, view and transform the data#
black_female_tensor, black_female_display = process_images('face_sample2/??_1_1_*.jpg', size=150)
white_female_tensor, white_female_display = process_images('face_sample2/??_1_0_*.jpg', size=150)
We can check the dimensions of these tensors and see that for each group we have images of size \(150 \times 150\) and three channels (RGB) of 49 individuals.
Note: Originally, the size of images was \(200 \times 200\), but due to RAM resources, we have reduced it. You can change it back, i.e., size=200.
print(white_female_tensor.shape)
print(black_female_tensor.shape)
torch.Size([49, 3, 150, 150])
torch.Size([49, 3, 150, 150])
Visualize some example faces#
Show code cell source
# @title Visualize some example faces
tensor_to_display = torch.cat((white_female_display[:15],
black_female_display[:15]))
plt.figure(figsize=(12, 12))
plt.imshow(make_grid(tensor_to_display, nrow = 15).permute(1, 2, 0))
plt.axis('off')
plt.show()
Section 2.3: Calculate embeddings#
We use the same pretrained facial recognition network as in section 8 to calculate embeddings. If you have memory issues running this part, go to Edit > Notebook settings and check if GPU is selected as Hardware accelerator. If this does not help you can restart the notebook, go to Runtime -> Restart runtime.
resnet.classify = False
black_female_embeddings = resnet(black_female_tensor.to(DEVICE))
white_female_embeddings = resnet(white_female_tensor.to(DEVICE))
We will use the embeddings to show that the model was trained on an imbalanced dataset. For this, we are going to calculate a distance matrix of all combinations of images, like in this small example with \(n=3\) (in our case \(n=98\)).
Calculate the distance between each pair of image embeddings in our tensor and visualize all the distances. Remember that two embeddings are vectors and the distance between two vectors is the Euclidean distance.
Function to calculate pairwise distances#
torch.cdist is used
Show code cell source
# @title Function to calculate pairwise distances
# @markdown [`torch.cdist`](https://pytorch.org/docs/stable/generated/torch.cdist.html) is used
def calculate_pairwise_distances(embedding_tensor):
"""
This function calculates the distance
between each pair of image embeddings
in a tensor using the `torch.cdist`.
Parameters:
embedding_tensor : torch.Tensor
A num_images x embedding_dimension tensor
Returns:
distances : torch.Tensor
A num_images x num_images tensor
containing the pairwise distances between
each to image embedding
"""
distances = torch.cdist(embedding_tensor, embedding_tensor)
return distances
Visualize the distances#
Show code cell source
# @title Visualize the distances
embedding_tensor = torch.cat((black_female_embeddings,
white_female_embeddings)).to(device='cpu')
distances = calculate_pairwise_distances(embedding_tensor)
plt.figure(figsize=(8, 8))
plt.imshow(distances.detach().cpu().numpy())
plt.annotate('Black female', (2, -0.5), fontsize=20, va='bottom')
plt.annotate('White female', (52, -0.5), fontsize=20, va='bottom')
plt.annotate('Black female', (-0.5, 45), fontsize=20, rotation=90, ha='right')
plt.annotate('White female', (-0.5, 90), fontsize=20, rotation=90, ha='right')
cbar = plt.colorbar()
cbar.set_label('Distance', fontsize=16)
plt.axis('off')
plt.show()
Exercise 2.1: Face similarity#
What do you observe? The faces of which group are more similar to each other for the Face Detection algorithm?
Submit your feedback#
Show code cell source
# @title Submit your feedback
content_review(f"{feedback_prefix}_Face_Similarity_Discussion")
Exercise 2.2: Embeddings#
What does it mean in real life applications that the distance is smaller between the embeddings of one group?
Can you come up with example situations/applications where this has a negative impact?
What could you do to avoid these problems?
Submit your feedback#
Show code cell source
# @title Submit your feedback
content_review(f"{feedback_prefix}_Embeddings_Discussion")
Lastly, to show the importance of the dataset which you use to pretrain your model, look at how much space white men and women take in different embeddings. FairFace is a dataset which is specifically created with completely balanced classes. The blue dots in all visualizations are white male and white female.
Adopted from Kärkkäinen and Joo, 2019, arXiv
Section 3: Within Sum of Squares#
Time estimate: ~10mins
We can try to put this observation in numbers. For this we work with the embeddings. We want to calculate the centroid of each group, which is the average of the 49 embeddings of the group. As each embedding vector has a dimension of 512, the centroid will also have this dimension.
Now we can calculate how far away the observations \(x\) of each group \(S_i\) are from the centroid \(\mu_i\). This concept is known as Within Sum of Squares (WSS) from cluster analysis.
where \(|| \cdot ||\) is the Euclidean norm.
The Within Sum of Squares (WSS) is a number which measures this variability of a group in the embedding space. If all embeddings of one group were very close to each other, the WSS would be very small. In our case we see that the WSS for the black females is much smaller than for the white females. This means that it is much harder for the model to distinguish two black females than to distinguish two white females. The WSS complements the observation from the distance matrix, where we observed overall smaller pairwise distances between black females.
Function to calculate WSS#
Show code cell source
# @title Function to calculate WSS
def wss(group):
"""
This function returns the sum of squared distances
of the N vectors of a
group tensor (N x K) to its centroid (1 x K).
Hints:
- to calculate the centroid, torch.mean()
will be of use.
- We need the mean of the N=49 observations.
If our input tensor is of size
N x K, we expect the centroid to be of
dimensions 1 x K.
Use the axis argument within torch.mean
Args:
group: torch.tensor
A image_count x embedding_size tensor
Returns:
sum_sq: torch.tensor
A 1x1 tensor with the sum of squared distances.
"""
centroid = torch.mean(group, axis=0)
distance = torch.linalg.norm(group - centroid.view(1, -1), axis=1)
sum_sq = torch.sum(distance**2)
return sum_sq
Let’s calculate the WSS for the two groups of our example.
Show code cell source
# @markdown Let's calculate the WSS for the two groups of our example.
print(f"Black female embedding WSS: {np.round(wss(black_female_embeddings).item(), 2)}")
print(f"White female embedding WSS: {np.round(wss(white_female_embeddings).item(), 2)}")
Black female embedding WSS: 34.21
White female embedding WSS: 44.59
Summary#
In this tutorial we have learned how to apply a modern convnet in real application such as facial recognition. However, as the state-of-the-art tools for facial recognition are trained mostly with caucasian faces, they fail or they perform much worse when they have to deal with faces from other races.