Load algonauts videos

Load algonauts videos#

By Neuromatch Academy

Content creators: Kshitij Dwivedi

Produtction editors: Spiros Chavlis

Terms of Use#

By using this colab to download the brain dataset (i.e., data including fMRI, images, labels), you agree to the following terms:

You will use the Datasets only for non-commercial research and educational purposes.
You will NOT distribute the Datasets or any parts thereof.
Massachusetts Institute of Technology make no representations or warranties regarding the datasets, including but not limited to warranties of non-infringement or fitness for a particular purpose.
You accept full responsibility for your use of the datasets and shall defend and indemnify Massachusetts Institute of Technology and International Business Machines Corporation, including its employees, officers and agents, against any and all claims arising from your use of the datasets, including but not limited to your use of any copyrighted images that you may create from the datasets.
You will treat people appearing in this data with respect and dignity.
This data comes with no warranty or guarantee of any kind, and you accept full liability.

Algonauts2021 dataset#

Here we will load fMRI responses (whole brain + 9 ROIs) to 1000 video clips of 3 seconds and visualize in glass brain using nilearn.

P.S. The dataset provided is part of Algonauts 2021 Challenge. Please check the details here if you are interested in participating.

Install dependencies#

# Imports
import os
import glob
import pickle

import numpy as np
import matplotlib.pyplot as plt
import nibabel as nib

from nilearn import datasets
from nilearn import plotting

fsaverage = datasets.fetch_surf_fsaverage()

During previous years, each user had to use this google form in order to receive the dropbox link with the corresponding dataset.

For this year you can use this link.

Important: You MUST set dropbox_link parameter dl=1, e.g., https://www.dropbox.com/s/myurl/participants_data.zip?dl=1

Enter the dropbox link#

Run the cell#

Dropbox link: https://www.dropbox.com/s/agxyxntrbwko7t1/participants_data.zip?dl=1
Data are already downloaded.
example.nii file is already downloaded.

#fMRI Data

The Algonauts dataset provides human brain responses to a set of 1,102 3-s long video clips of everyday events. The brain responses are measured with functional magnetic resonance imaging (fMRI). fMRI is a widely used brain imaging technique with high spatial resolution that measures blood flow changes associated with neural responses.

The training set consists of 1,000 video clips and the associated brain responses. The brain responses are provided here in two tracks corresponding to two independent tracks in the Algonauts challenge.

In the first track, brain responses provided are from selected voxels across the whole brain showing reliable responses to videos. The figure below shows the reliability values of different voxels in the brain.

In the second track, brain responses provided are from a set of specific regions of interest (ROIs) known to play a key role in visual perception. These ROIs start in early and mid-level visual cortex (V1, V2, V3, and V4) and extend into higher-level cortex that responds preferentially to all objects or particular categories (Body- EBA; Face - FFA, STS; Object - LOC; Scene - PPA). In the figure below we show the masks of the above mentioned ROIs for an example subject.

#Loading fMRI data The dataset contains 1,000 3-second videos + fMRI human brain data of 10 subjects in response to viewing videos from this set.

The ROI data is provided for 9 ROIs of the visual brain (V1, V2, V3, V4, LOC, EBA, FFA, STS, PPA) in a Pickle file (e.g. V1.pkl) that contains a num_videos x num_repetitions x num_voxels matrix. For each ROI, we selected voxels that showed significant split-half reliability.

The whole brain data is provided for selected voxels across the whole brain showing reliable responses to videos in a Pickle file (e.g. WB.pkl) that contains a num_videos x num_repetitions x num_voxels matrix.

In this section, we demonstrate how to load fMRI data for a given ROI.

Utility functions for data loading#

Show code cell source Hide code cell source

# @title Utility functions for data loading
def save_dict(di_, filename_):
  with open(filename_, 'wb') as f:
      pickle.dump(di_, f)

def load_dict(filename_):
  with open(filename_, 'rb') as f:
    u = pickle._Unpickler(f)
    u.encoding = 'latin1'
    ret_di = u.load()
    # print(p)
    # ret_di = pickle.load(f)
  return ret_di


def visualize_activity(vid_id,sub):
  # Setting up the paths for whole brain data
  fmri_dir = './participants_data_v2021'
  track = "full_track"

  # get the right track directory depending on whole brain/ROI choice
  track_dir = os.path.join(fmri_dir, track)

  # get the selected subject's directory
  sub_fmri_dir = os.path.join(track_dir, sub)

  #result directory to store nifti file
  results_dir = '/content/'

  # mapping the data to voxels and storing in a nifti file
  fmri_train_all,voxel_mask = get_fmri(sub_fmri_dir,"WB")
  visual_mask_3D = np.zeros((78,93,71))
  visual_mask_3D[voxel_mask==1]= fmri_train_all[vid_id,:]
  brain_mask = './example.nii'
  nii_save_path =  os.path.join(results_dir, 'vid_activity.nii')
  saveasnii(brain_mask,nii_save_path,visual_mask_3D)

  # visualizing saved nifti file
  plotting.plot_glass_brain(nii_save_path,
                          title='fMRI response',plot_abs=False,
                          display_mode='lyr',colorbar=True)


def get_fmri(fmri_dir, ROI):
  """This function loads fMRI data into a numpy array for to a given ROI.
  Parameters
  ----------
  fmri_dir : str
    path to fMRI data.
  ROI : str
    name of ROI.

  Returns
  -------
  np.array
    matrix of dimensions #train_vids x #repetitions x #voxels
    containing fMRI responses to train videos of a given ROI
  """

  # Loading ROI data
  ROI_file = os.path.join(fmri_dir, ROI + ".pkl")
  ROI_data = load_dict(ROI_file)
  # averaging ROI data across repetitions
  ROI_data_train = np.mean(ROI_data["train"], axis=1)
  if ROI == "WB":
    voxel_mask = ROI_data['voxel_mask']

    return ROI_data_train, voxel_mask

  return ROI_data_train

def saveasnii(brain_mask,nii_save_path,nii_data):
  img = nib.load(brain_mask)
  nii_img = nib.Nifti1Image(nii_data, img.affine, img.header)
  nib.save(nii_img, nii_save_path)

Loading fMRI data and inspecting dimensions#

../../_images/2fb1b6d3fa502d159853bdb26bb21cb9705c96228bfd0ec263da57b7c46e9c58.png

The above matrix shows each individual voxel’s (columns) response to 1000 videos (rows). To find out more details about how the fMRI responses were preprocessed please click here.

Visualize a given video (with id from 0-999)) and its corresponding brain activity

Visualize video#

Visualize corresponding brain response#

../../_images/f628c828c881bcf2ac53ee540d3d0f539623920417b9c3099b85e7a04bf58ae1.png

The values in the fMRI response above correspond to beta values from 5s to 9s after stimulus onset calculated using an Finite Impulse Response (FIR) model. The beta values were averaged across time and z-scored across videos.

References#

Radoslaw Martin Cichy, Kshitij Dwivedi, Benjamin Lahner, Alex Lascelles, Polina Iamshchinina, Monika Graumann, Alex Andonian, Apurva Ratan Murty, Kendrick Kay, Gemma Roig, Aude Oliva. (2021). The Algonauts Project 2021 Challenge: How the Human Brain Makes Sense of a World in Motion. arXiv, arXiv:2104.13714v1.
Radoslaw Martin Cichy, Gemma Roig, Alex Andonian, Kshitij Dwivedi, Benjamin Lahner, Alex Lascelles, Yalda Mohsenzadeh, Kandan Ramakrishnan, and Aude Oliva. (2019). The Algonauts Project: A Platform for Communication between the Sciences of Biological and Artificial Intelligence. arXiv, arXiv:1905.05675.