Bonus Tutorial: Multilingual Embeddings

Week 3, Day 1: Time Series And Natural Language Processing

By Neuromatch Academy

Content creators: Alish Dipani, Kelson Shilling-Scrivo, Lyle Ungar

Content reviewers: Kelson Shilling-Scrivo

Content editors: Kelson Shilling-Scrivo

Production editors: Gagana B, Spiros Chavlis

Based on Content from: Anushree Hede, Pooja Consul, Ann-Katrin Reuel

---

Tutorial objectives

Before we begin with exploring how RNNs excel at modelling sequences, we will explore some of the other ways we can model sequences, encode text, and make meaningful measurements using such encodings and embeddings.

---

Setup

Install dependencies

There may be errors and/or warnings reported during the installation. However, they are to be ignored.

Show code cell source Hide code cell source

# @title Install dependencies
# @markdown There may be *errors* and/or *warnings* reported during the installation. However, they are to be ignored.
!pip install python-Levenshtein --quiet

Install and import feedback gadget

Show code cell source Hide code cell source

# @title Install and import feedback gadget

!pip3 install vibecheck datatops --quiet

from vibecheck import DatatopsContentReviewContainer
def content_review(notebook_section: str):
    return DatatopsContentReviewContainer(
        "",  # No text prompt
        notebook_section,
        {
            "url": "https://pmyvdlilci.execute-api.us-east-1.amazonaws.com/klab",
            "name": "neuromatch_dl",
            "user_key": "f379rz8y",
        },
    ).render()


feedback_prefix = "W3D1_T3_Bonus"

Install fastText

If you want to see the original code, go to repo: https://github.com/facebookresearch/fastText.git

Show code cell source Hide code cell source

# @title Install fastText
# @markdown If you want to see the original code, go to repo: https://github.com/facebookresearch/fastText.git

# !pip install git+https://github.com/facebookresearch/fastText.git --quiet

import os, zipfile, requests

url = "https://osf.io/vkuz7/download"
fname = "fastText-main.zip"

print('Downloading Started...')
# Downloading the file by sending the request to the URL
r = requests.get(url, stream=True)

# Writing the file to the local file system
with open(fname, 'wb') as f:
  f.write(r.content)
print('Downloading Completed.')

# opening the zip file in READ mode
with zipfile.ZipFile(fname, 'r') as zipObj:
  # extracting all the files
  print('Extracting all the files now...')
  zipObj.extractall()
  print('Done!')
  os.remove(fname)

# Install the package
!pip install fastText-main/ --quiet

Downloading Started...

Downloading Completed.
Extracting all the files now...
Done!

# Imports
import fasttext
import numpy as np
import matplotlib.pyplot as plt
from tqdm import tqdm

Figure Settings

Show code cell source Hide code cell source

# @title Figure Settings
import logging
logging.getLogger('matplotlib.font_manager').disabled = True

import ipywidgets as widgets
%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/content-creation/main/nma.mplstyle")

Helper functions

Show code cell source Hide code cell source

# @title Helper functions
def cosine_similarity(vec_a, vec_b):
  """Compute cosine similarity between vec_a and vec_b"""
  return np.dot(vec_a, vec_b) / (np.linalg.norm(vec_a) * np.linalg.norm(vec_b))


def getSimilarity(word1, word2):
  v1 = ft_en_vectors.get_word_vector(word1)
  v2 = ft_en_vectors.get_word_vector(word2)
  return cosine_similarity(v1, v2)

Set random seed

Executing set_seed(seed=seed) you are setting the seed

Show code cell source Hide code cell source

# @title Set random seed

# @markdown Executing `set_seed(seed=seed)` you are setting the seed

# For DL its critical to set the random seed so that students can have a
# baseline to compare their results to expected results.
# Read more here: https://pytorch.org/docs/stable/notes/randomness.html

# Call `set_seed` function in the exercises to ensure reproducibility.
import random
import torch

def set_seed(seed=None, seed_torch=True):
  """
  Function that controls randomness.
  NumPy and random modules must be imported.

  Args:
    seed : Integer
      A non-negative integer that defines the random state. Default is `None`.
    seed_torch : Boolean
      If `True` sets the random seed for pytorch tensors, so pytorch module
      must be imported. Default is `True`.

  Returns:
    Nothing.
  """
  if seed is None:
    seed = np.random.choice(2 ** 32)
  random.seed(seed)
  np.random.seed(seed)
  if seed_torch:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

  print(f'Random seed {seed} has been set.')

# In case that `DataLoader` is used
def seed_worker(worker_id):
  """
  DataLoader will reseed workers following randomness in
  multi-process data loading algorithm.

  Args:
    worker_id: integer
      ID of subprocess to seed. 0 means that
      the data will be loaded in the main process
      Refer: https://pytorch.org/docs/stable/data.html#data-loading-randomness for more details

  Returns:
    Nothing
  """
  worker_seed = torch.initial_seed() % 2**32
  np.random.seed(worker_seed)
  random.seed(worker_seed)

Set device (GPU or CPU). Execute set_device()

Show code cell source Hide code cell source

# @title Set device (GPU or CPU). Execute `set_device()`

# Inform the user if the notebook uses GPU or CPU.

def set_device():
  """
  Set the device. CUDA if available, CPU otherwise

  Args:
    None

  Returns:
    Nothing
  """
  device = "cuda" if torch.cuda.is_available() else "cpu"
  if device != "cuda":
    print("WARNING: For this notebook to perform best, "
        "if possible, in the menu under `Runtime` -> "
        "`Change runtime type.`  select `GPU` ")
  else:
    print("GPU is enabled in this notebook.")

  return device

DEVICE = set_device()
SEED = 2021
set_seed(seed=SEED)

WARNING: For this notebook to perform best, if possible, in the menu under `Runtime` -> `Change runtime type.`  select `GPU` 
Random seed 2021 has been set.

---

Section 1 : Multilingual Embeddings

Traditionally, word embeddings have been language-specific, with embeddings for each language trained separately and existing in entirely different vector spaces. But, what if we wanted to compare words in one language to another? Say we want to create a text classifier with a corpus of English and Spanish words.

We use the multilingual word embeddings provided in fastText. More information can be found here.

Training multilingual embeddings

We first train separate embeddings for each language using fastText and a combination of data from Facebook and Wikipedia. Then, we find a dictionary of common words between the two languages. The dictionaries are automatically induced from parallel data - datasets consisting of a pair of sentences in two languages with the same meaning.

Then, we find a matrix that projects the embeddings into a common space between the given languages. The matrix is designed to minimize the distance between a word \(x_i\) and its projection \(y_i\). If our dictionary consists of pairs \((x_i, y_i)\), our projector \(M\) would be:

(121)\[\begin{equation} M = \underset{W}{\operatorname{argmax}} \sum_i ||x_i - Wy_i||^2 \end{equation}\]

Also, the projector matrix \(W\) is constrained to e orthogonal, so actual distances between word embedding vectors are preserved. Multilingual models are trained by using our multilingual word embeddings as the base representations in DeepText and “freezing” them or leaving them unchanged during the training process.

After going through this, try to replicate the above exercises but in different languages!

Show code cell source Hide code cell source

# @markdown ### Download FastText English Embeddings of dimension 100
# @markdown This will take 1-2 minutes to run

import os, zipfile, requests

url = "https://osf.io/2frqg/download"
fname = "cc.en.100.bin.gz"

print('Downloading Started...')
# Downloading the file by sending the request to the URL
r = requests.get(url, stream=True)

# Writing the file to the local file system
with open(fname, 'wb') as f:
  f.write(r.content)
print('Downloading Completed.')

# opening the zip file in READ mode
with zipfile.ZipFile(fname, 'r') as zipObj:
  # extracting all the files
  print('Extracting all the files now...')
  zipObj.extractall()
  print('Done!')
  os.remove(fname)

Downloading Started...

Downloading Completed.
Extracting all the files now...

Done!

# Load 100 dimension FastText Vectors using FastText library
ft_en_vectors = fasttext.load_model('cc.en.100.bin')

Show code cell source Hide code cell source

# @markdown ### Download FastText French Embeddings of dimension 100

# @markdown **Note:** This cell might take 2-4 minutes to run

import os, zipfile, requests

url = "https://osf.io/rqadk/download"
fname = "cc.fr.100.bin.gz"

print('Downloading Started...')
# Downloading the file by sending the request to the URL
r = requests.get(url, stream=True)

# Writing the file to the local file system
with open(fname, 'wb') as f:
  f.write(r.content)
print('Downloading Completed.')

# opening the zip file in READ mode
with zipfile.ZipFile(fname, 'r') as zipObj:
  # extracting all the files
  print('Extracting all the files now...')
  zipObj.extractall()
  print('Done!')
  os.remove(fname)

Downloading Started...

Downloading Completed.
Extracting all the files now...

Done!

# Load 100 dimension FastText Vectors using FastText library
french = fasttext.load_model('cc.fr.100.bin')

First, we look at the cosine similarity between different languages without projecting them into the same vector space. As you can see, the same words seem close to \(0\) cosine similarity in other languages - so neither similar nor dissimilar.

hello = ft_en_vectors.get_word_vector('hello')
hi = ft_en_vectors.get_word_vector('hi')
bonjour = french.get_word_vector('bonjour')

print(f"Cosine Similarity between HI and HELLO: {cosine_similarity(hello, hi)}")
print(f"Cosine Similarity between BONJOUR and HELLO: {cosine_similarity(hello, bonjour)}")

Cosine Similarity between HI and HELLO: 0.7028388977050781
Cosine Similarity between BONJOUR and HELLO: 0.20523205399513245

cat = ft_en_vectors.get_word_vector('cat')
chatte = french.get_word_vector('chatte')
chat = french.get_word_vector('chat')

print(f"Cosine Similarity between cat and chatte: {cosine_similarity(cat, chatte)}")
print(f"Cosine Similarity between cat and chat: {cosine_similarity(cat, chat)}")
print(f"Cosine Similarity between chatte and chat: {cosine_similarity(chatte, chat)}")

Cosine Similarity between cat and chatte: -0.013087842613458633
Cosine Similarity between cat and chat: -0.02490561455488205
Cosine Similarity between chatte and chat: 0.6003134250640869

First, let’s define a list of words that are in common between English and French. We’ll be using this to make our training matrices.

en_words = set(ft_en_vectors.words)
fr_words = set(french.words)
overlap = list(en_words & fr_words)
bilingual_dictionary = [(entry, entry) for entry in overlap]

We define a few functions to make our lives a bit easier: make_training_matrices takes in the source words, target language words, and the set of common words. It then creates a matrix of all the word embeddings of all common words between the languages (in each language). These are our training matrices.

The function learn_transformation then takes in these matrices, normalizes them, and performs SVD, which aligns the source language to the target and returns a transformation matrix.

def make_training_matrices(source_dictionary, target_dictionary,
                           bilingual_dictionary):
  source_matrix = []
  target_matrix = []
  for (source, target) in tqdm(bilingual_dictionary):
    # if source in source_dictionary.words and target in target_dictionary.words:
    source_matrix.append(source_dictionary.get_word_vector(source))
    target_matrix.append(target_dictionary.get_word_vector(target))
  # return training matrices
  return np.array(source_matrix), np.array(target_matrix)


# from https://stackoverflow.com/questions/21030391/how-to-normalize-array-numpy
def normalized(a, axis=-1, order=2):
  """Utility function to normalize the rows of a numpy array."""
  l2 = np.atleast_1d(np.linalg.norm(a, order, axis))
  l2[l2==0] = 1
  return a / np.expand_dims(l2, axis)


def learn_transformation(source_matrix, target_matrix, normalize_vectors=True):
  """
  Source and target matrices are numpy arrays, shape
  (dictionary_length, embedding_dimension). These contain paired
  word vectors from the bilingual dictionary.
  """
  # optionally normalize the training vectors
  if normalize_vectors:
    source_matrix = normalized(source_matrix)
    target_matrix = normalized(target_matrix)
  # perform the SVD
  product = np.matmul(source_matrix.transpose(), target_matrix)
  U, s, V = np.linalg.svd(product)
  # return orthogonal transformation which aligns source language to the target
  return np.matmul(U, V)

Now, we just have to put it all together!

source_training_matrix, target_training_matrix = make_training_matrices(ft_en_vectors, french, bilingual_dictionary)

transform = learn_transformation(source_training_matrix, target_training_matrix)

Let’s run the same examples as above, but this time, whenever we use French words, the matrix multiplies the embedding by the transpose of the transform matrix. That works a lot better!

hello = ft_en_vectors.get_word_vector('hello')
hi = ft_en_vectors.get_word_vector('hi')
bonjour = np.matmul(french.get_word_vector('bonjour'), transform.T)

print(f"Cosine Similarity between HI and HELLO: {cosine_similarity(hello, hi)}")
print(f"Cosine Similarity between BONJOUR and HELLO: {cosine_similarity(hello, bonjour)}")

Cosine Similarity between HI and HELLO: 0.7028388977050781
Cosine Similarity between BONJOUR and HELLO: 0.5818598866462708

cat = ft_en_vectors.get_word_vector('cat')
chatte = np.matmul(french.get_word_vector('chatte'), transform.T)
chat = np.matmul(french.get_word_vector('chat'), transform.T)

print(f"Cosine Similarity between cat and chatte: {cosine_similarity(cat, chatte)}")
print(f"Cosine Similarity between cat and chat: {cosine_similarity(cat, chat)}")
print(f"Cosine Similarity between chatte and chat: {cosine_similarity(chatte, chat)}")

Cosine Similarity between cat and chatte: 0.4327278137207031
Cosine Similarity between cat and chat: 0.6866633296012878
Cosine Similarity between chatte and chat: 0.6003134846687317

Now, try a couple of your examples. Try some examples you looked at in Tutorial 1, Section 2.1, but with English and French. Does it work as expected?

Submit your feedback

Show code cell source Hide code cell source

# @title Submit your feedback
content_review(f"{feedback_prefix}_Multilingual_Embeddings_Bonus_Activity")