Open In Colab   Open in Kaggle

Bonus Tutorial: Multilingual Embeddings

Week 2, Day 5: Time Series And Natural Language Processing

By Neuromatch Academy

Content creators: Alish Dipani, Kelson Shilling-Scrivo, Lyle Ungar

Content reviewers: Kelson Shilling-Scrivo

Content editors: Kelson Shilling-Scrivo

Production editors: Gagana B, Spiros Chavlis


Based on Content from: Anushree Hede, Pooja Consul, Ann-Katrin Reuel


Tutorial objectives

Before we begin with exploring how RNNs excel at modelling sequences, we will explore some of the other ways we can model sequences, encode text, and make meaningful measurements using such encodings and embeddings.

Tutorial slides

These are the slides for the videos in this tutorial. If you want to locally download the slides, click here.


Setup

Install dependencies

There may be errors and/or warnings reported during the installation. However, they are to be ignored.

# @title Install dependencies

# @markdown There may be *errors* and/or *warnings* reported during the installation. However, they are to be ignored.
!pip install torch==1.8.0+cu111 torchvision==0.9.0+cu111 torchaudio==0.8.0 torchtext==0.9.0 -f https://download.pytorch.org/whl/torch_stable.html --quiet
!pip install python-Levenshtein --quiet

!pip install git+https://github.com/NeuromatchAcademy/evaltools --quiet
from evaltools.airtable import AirtableForm

# Generate airtable form
atform = AirtableForm('appn7VdPRseSoMXEG','W2D5_T1','https://portal.neuromatchacademy.org/api/redirect/to/9c55f6cb-cdf9-4429-ac1c-ec44fe64c303')

Install fastText

If you want to see the original code, go to repo: https://github.com/facebookresearch/fastText.git

# @title Install fastText
# @markdown If you want to see the original code, go to repo: https://github.com/facebookresearch/fastText.git

# !pip install git+https://github.com/facebookresearch/fastText.git --quiet

import os, zipfile, requests

url = "https://osf.io/vkuz7/download"
fname = "fastText-main.zip"

print('Downloading Started...')
# Downloading the file by sending the request to the URL
r = requests.get(url, stream=True)

# Writing the file to the local file system
with open(fname, 'wb') as f:
  f.write(r.content)
print('Downloading Completed.')

# opening the zip file in READ mode
with zipfile.ZipFile(fname, 'r') as zipObj:
  # extracting all the files
  print('Extracting all the files now...')
  zipObj.extractall()
  print('Done!')
  os.remove(fname)

# Install the package
!pip install fastText-main/ --quiet
# Imports
import fasttext
import numpy as np
import matplotlib.pyplot as plt
from tqdm.notebook import tqdm

Figure Settings

# @title Figure Settings
import ipywidgets as widgets
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/content-creation/main/nma.mplstyle")

Helper functions

# @title Helper functions
def cosine_similarity(vec_a, vec_b):
  """Compute cosine similarity between vec_a and vec_b"""
  return np.dot(vec_a, vec_b) / (np.linalg.norm(vec_a) * np.linalg.norm(vec_b))


def getSimilarity(word1, word2):
  v1 = ft_en_vectors.get_word_vector(word1)
  v2 = ft_en_vectors.get_word_vector(word2)
  return cosine_similarity(v1, v2)

Set random seed

Executing set_seed(seed=seed) you are setting the seed

# @title Set random seed

# @markdown Executing `set_seed(seed=seed)` you are setting the seed

# For DL its critical to set the random seed so that students can have a
# baseline to compare their results to expected results.
# Read more here: https://pytorch.org/docs/stable/notes/randomness.html

# Call `set_seed` function in the exercises to ensure reproducibility.
import random
import torch

def set_seed(seed=None, seed_torch=True):
  """
  Function that controls randomness.
  NumPy and random modules must be imported.

  Args:
    seed : Integer
      A non-negative integer that defines the random state. Default is `None`.
    seed_torch : Boolean
      If `True` sets the random seed for pytorch tensors, so pytorch module
      must be imported. Default is `True`.

  Returns:
    Nothing.
  """
  if seed is None:
    seed = np.random.choice(2 ** 32)
  random.seed(seed)
  np.random.seed(seed)
  if seed_torch:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

  print(f'Random seed {seed} has been set.')

# In case that `DataLoader` is used
def seed_worker(worker_id):
  """
  DataLoader will reseed workers following randomness in
  multi-process data loading algorithm.

  Args:
    worker_id: integer
      ID of subprocess to seed. 0 means that
      the data will be loaded in the main process
      Refer: https://pytorch.org/docs/stable/data.html#data-loading-randomness for more details

  Returns:
    Nothing
  """
  worker_seed = torch.initial_seed() % 2**32
  np.random.seed(worker_seed)
  random.seed(worker_seed)

Set device (GPU or CPU). Execute set_device()

# @title Set device (GPU or CPU). Execute `set_device()`

# Inform the user if the notebook uses GPU or CPU.

def set_device():
  """
  Set the device. CUDA if available, CPU otherwise

  Args:
    None

  Returns:
    Nothing
  """
  device = "cuda" if torch.cuda.is_available() else "cpu"
  if device != "cuda":
    print("WARNING: For this notebook to perform best, "
        "if possible, in the menu under `Runtime` -> "
        "`Change runtime type.`  select `GPU` ")
  else:
    print("GPU is enabled in this notebook.")

  return device
DEVICE = set_device()
SEED = 2021
set_seed(seed=SEED)

Section 1 : Multilingual Embeddings

Traditionally, word embeddings have been language-specific, with embeddings for each language trained separately and existing in entirely different vector spaces. But, what if we wanted to compare words in one language to another? Say we want to create a text classifier with a corpus of English and Spanish words.

We use the multilingual word embeddings provided in fastText. More information can be found here.

Training multilingual embeddings

We first train separate embeddings for each language using fastText and a combination of data from Facebook and Wikipedia. Then, we find a dictionary of common words between the two languages. The dictionaries are automatically induced from parallel data - datasets consisting of a pair of sentences in two languages with the same meaning.

Then, we find a matrix that projects the embeddings into a common space between the given languages. The matrix is designed to minimize the distance between a word \(x_i\) and its projection \(y_i\). If our dictionary consists of pairs \((x_i, y_i)\), our projector \(M\) would be:

(100)\[\begin{equation} M = \underset{W}{\operatorname{argmax}} \sum_i ||x_i - Wy_i||^2 \end{equation}\]

Also, the projector matrix \(W\) is constrained to e orthogonal, so actual distances between word embedding vectors are preserved. Multilingual models are trained by using our multilingual word embeddings as the base representations in DeepText and “freezing” them or leaving them unchanged during the training process.

After going through this, try to replicate the above exercises but in different languages!

# @markdown ### Download FastText English Embeddings of dimension 100
# @markdown This will take 1-2 minutes to run

import os, zipfile, requests

url = "https://osf.io/2frqg/download"
fname = "cc.en.100.bin.gz"

print('Downloading Started...')
# Downloading the file by sending the request to the URL
r = requests.get(url, stream=True)

# Writing the file to the local file system
with open(fname, 'wb') as f:
  f.write(r.content)
print('Downloading Completed.')

# opening the zip file in READ mode
with zipfile.ZipFile(fname, 'r') as zipObj:
  # extracting all the files
  print('Extracting all the files now...')
  zipObj.extractall()
  print('Done!')
  os.remove(fname)
# Load 100 dimension FastText Vectors using FastText library
ft_en_vectors = fasttext.load_model('cc.en.100.bin')
# @markdown ### Download FastText French Embeddings of dimension 100

# @markdown **Note:** This cell might take 2-4 minutes to run

import os, zipfile, requests

url = "https://osf.io/rqadk/download"
fname = "cc.fr.100.bin.gz"

print('Downloading Started...')
# Downloading the file by sending the request to the URL
r = requests.get(url, stream=True)

# Writing the file to the local file system
with open(fname, 'wb') as f:
  f.write(r.content)
print('Downloading Completed.')

# opening the zip file in READ mode
with zipfile.ZipFile(fname, 'r') as zipObj:
  # extracting all the files
  print('Extracting all the files now...')
  zipObj.extractall()
  print('Done!')
  os.remove(fname)
# Load 100 dimension FastText Vectors using FastText library
french = fasttext.load_model('cc.fr.100.bin')

First, we look at the cosine similarity between different languages without projecting them into the same vector space. As you can see, the same words seem close to \(0\) cosine similarity in other languages - so neither similar nor dissimilar.

hello = ft_en_vectors.get_word_vector('hello')
hi = ft_en_vectors.get_word_vector('hi')
bonjour = french.get_word_vector('bonjour')

print(f"Cosine Similarity between HI and HELLO: {cosine_similarity(hello, hi)}")
print(f"Cosine Similarity between BONJOUR and HELLO: {cosine_similarity(hello, bonjour)}")
cat = ft_en_vectors.get_word_vector('cat')
chatte = french.get_word_vector('chatte')
chat = french.get_word_vector('chat')

print(f"Cosine Similarity between cat and chatte: {cosine_similarity(cat, chatte)}")
print(f"Cosine Similarity between cat and chat: {cosine_similarity(cat, chat)}")
print(f"Cosine Similarity between chatte and chat: {cosine_similarity(chatte, chat)}")

First, let’s define a list of words that are in common between English and French. We’ll be using this to make our training matrices.

en_words = set(ft_en_vectors.words)
fr_words = set(french.words)
overlap = list(en_words & fr_words)
bilingual_dictionary = [(entry, entry) for entry in overlap]

We define a few functions to make our lives a bit easier: make_training_matrices takes in the source words, target language words, and the set of common words. It then creates a matrix of all the word embeddings of all common words between the languages (in each language). These are our training matrices.

The function learn_transformation then takes in these matrices, normalizes them, and performs SVD, which aligns the source language to the target and returns a transformation matrix.

def make_training_matrices(source_dictionary, target_dictionary,
                           bilingual_dictionary):
  source_matrix = []
  target_matrix = []
  for (source, target) in tqdm(bilingual_dictionary):
    # if source in source_dictionary.words and target in target_dictionary.words:
    source_matrix.append(source_dictionary.get_word_vector(source))
    target_matrix.append(target_dictionary.get_word_vector(target))
  # return training matrices
  return np.array(source_matrix), np.array(target_matrix)


# from https://stackoverflow.com/questions/21030391/how-to-normalize-array-numpy
def normalized(a, axis=-1, order=2):
  """Utility function to normalize the rows of a numpy array."""
  l2 = np.atleast_1d(np.linalg.norm(a, order, axis))
  l2[l2==0] = 1
  return a / np.expand_dims(l2, axis)


def learn_transformation(source_matrix, target_matrix, normalize_vectors=True):
  """
  Source and target matrices are numpy arrays, shape
  (dictionary_length, embedding_dimension). These contain paired
  word vectors from the bilingual dictionary.
  """
  # optionally normalize the training vectors
  if normalize_vectors:
    source_matrix = normalized(source_matrix)
    target_matrix = normalized(target_matrix)
  # perform the SVD
  product = np.matmul(source_matrix.transpose(), target_matrix)
  U, s, V = np.linalg.svd(product)
  # return orthogonal transformation which aligns source language to the target
  return np.matmul(U, V)

Now, we just have to put it all together!

source_training_matrix, target_training_matrix = make_training_matrices(ft_en_vectors, french, bilingual_dictionary)
transform = learn_transformation(source_training_matrix, target_training_matrix)

Let’s run the same examples as above, but this time, whenever we use French words, the matrix multiplies the embedding by the transpose of the transform matrix. That works a lot better!

hello = ft_en_vectors.get_word_vector('hello')
hi = ft_en_vectors.get_word_vector('hi')
bonjour = np.matmul(french.get_word_vector('bonjour'), transform.T)

print(f"Cosine Similarity between HI and HELLO: {cosine_similarity(hello, hi)}")
print(f"Cosine Similarity between BONJOUR and HELLO: {cosine_similarity(hello, bonjour)}")
cat = ft_en_vectors.get_word_vector('cat')
chatte = np.matmul(french.get_word_vector('chatte'), transform.T)
chat = np.matmul(french.get_word_vector('chat'), transform.T)

print(f"Cosine Similarity between cat and chatte: {cosine_similarity(cat, chatte)}")
print(f"Cosine Similarity between cat and chat: {cosine_similarity(cat, chat)}")
print(f"Cosine Similarity between chatte and chat: {cosine_similarity(chatte, chat)}")

Now, try a couple of your examples. Try some examples you looked at in Tutorial 1, Section 2.1, but with English and French. Does it work as expected?