Material#
Course introduction and overview of NeuroAI#
Speaker#
Pierre Bellec, PhD TBD
Summary: NeuroAI research makes use of artificial neural networks (ANNs) developed by the AI community to model the activity of the brain or, conversely, uses observations of the brain to infer better, more effective AI models. This broad research topic encompasses a rich variety of application tasks, ranging from vision to language but also increasingly including domains such as working memory or control tasks. NeuroAI also incorporates a wide range of techniques, from established analyses predicting brain activity using ANNs (brain encoding) to inferring stimuli from brain activity using ANNs (brain decoding), as well as emerging topics such as end-to-end training of ANNs to mimic brain activity or crafting stimuli maximizing brain responses using ANNs.
This introduction will provide prominent examples of NeuroAI research studies to illustrate major application domains and techniques, and conceptualize how the different lectures of the course will help the audience get exposed to the most salient areas of the NeuroAI research space.
Materials#
Technical and interpretive issues for constructing brain models from AI systems#
Speaker#
Thomas Naselaris, PhD majored in Math and Philosophy as an undergraduate at Indiana University. He received a Ph.D. in Neuroscience under Apostolos Georgopoulos at the University of Minnesota. He completed a postdoctoral fellowship under Jack Gallant at University of California, Berkeley. He is currently an Associate Professor in the Department of Neuroscience at the University of Minnesota, and a member of the Medical Discovery Team on Optical Imaging and Brain Science at the Center for Magnetic Resonance Research. He is co-founder and currently an Executive Chair of the Conference on Cognitive Computational Neuroscience. The Naselaris lab studies the computations that make it possible to see and think. Taking inspiration from the tools and concepts of AI, we use computational models to bridge observations of brain activity at many spatial and temporal scales, from fMRI in humans to 2-photon microscopy in animal models. We are especially interested in the generative capabilities of the visual system. Much of life is spent imagining or dreaming of internal images that one has never actually observed. Why is the visual system so good at generating images, and how does this remarkable ability help us to see? We are addressing this question by monitoring the human brain as it engages complex, real-world scenery and as it calls upon memory to generate mental images.
Summary: The alignment of the intellectual goals of neuroscience and AI have led to a long and fruitful exchange of ideas. Over the years, neuroscience has provided AI with brain-inspired computational motifs that have been implemented and scaled to solve hard tasks. Recently, these implemented solutions to hard tasks have been converted into models that make highly accurate predictions of activity in sensing and acting brains.
This lecture will focus on technical and interpretive issues around constructing brain models out of AI systems. We will begin with a discussion of the data and compute requirements for building brain models based on AI systems, followed by a discussion of statistical techniques for linking high-dimensional AI systems with the high-dimensional datasets. We will then discuss some of the challenges of interpreting mappings between components of the brain and AI systems, exploring the tension between mechanistic interpretations (e.g., “this piece of a neural network is like that piece of a brain”) and agnostic interpretations (e.g., “this AI system includes representations that span the space of measured brain activities”).
Materials#
Brain encoding and decoding using biologically constrained DNNs#
Speaker#
Yu Zhang, PhD is a Researcher at Zhejiang Lab (https://en.zhejianglab.com/), working on bridging neuroscience research with artificial intelligence. Dr. Zhang received her Ph.D on medical imaging analysis from the Brainnetome center, Institute of Automation, Chinese Academy of Science in 2015, under the supervision of Prof. Tianzi Jiang. After graduation, she started her postdoctoral fellowship at McGill University and University de Montreal from 2015-2020. In 2018, Dr. Zhang received IVADO postdoc fellowship award funded for two years. In 2020, she joined the Zhejiang Lab at Hangzhou, China. Dr. Zhang has published over 30 scientific papers, with 2369 citations in total (such as eLife, Journal of Neuroscience, Cerebral Cortex, Neuroimage, Neurology etc.). Her current research interests include brain atlas, brain connectomes, encoding and decoding of brain cognition, modeling, simulating and modulating brain cognition using AI, and brain-inspired artificial intelligence.
Summary: A major goal in cognitive neuroscience research is to better understand the neural basis of cognitive functions. Artificial intelligence provides new avenues for modeling, simulating or even modulating the comprehensive processes of human cognition. By leveraging human brain atlas and connectome priors, we proposed a biologically-constrained GNN (BGNN) model to effectively combine local and distributed brain activities. The BGNN model learns multistage (temporally) and multilevel (spatially) latent representations transforming from sensory processing to representational abstraction (encoding phase) and predicts cognitive states using embedded representations at fine timescales (decoding phase). Moreover, it uncovered inter-subject aligned, behaviorally relevant neural representations underpinning cognitive processes and achieved better decoding of cognitive tasks. This approach has shown promising findings in representational learning of cognitive function and biologically meaningful interpretations of AI modeling of human cognition.
In this hand-on session, I will introduce how to build such a biologically-constrained AI model by incorporating human brain atlas and connectome priors, how to optimize the DNN architecture for brain encoding and decoding, and how to interpret the representations learnt from AI models.
Materials#
Functional specialisation in biological and artificial neural networks#
Speaker#
Shahab Bakhtiari, PhD is an Assistant Professor at University of Montreal. He received his undergraduate and graduate degrees in Electrical Engineering from University of Tehran. He then went on to earn a PhD in Neuroscience from McGill University where he studied the neural mechanisms underlying visual learning and perception in primates… After completing his PhD, he was a postdoctoral fellow at the Mila Quebec AI Institute, where he focused on research at the intersection of neuroscience and artificial intelligence. Shahab is currently leading the Systems Neuroscience and AI Lab (SNAIL) at University of Montreal. Research at SNAIL specifically focuses on visual perception and learning in both biological brains and artificial neural networks. We use deep learning as a computational framework to model learning and perception in the brain, and leverage our understanding of the nervous system to create more biologically-inspired artificial intelligence.
Summary: Functional specialization is a fundamental organizational characteristic of the nervous system. The brain consists of anatomical modules (areas and pathways), each with a specific specialization for carrying out particular processes. In the visual system, in particular, specialized pathways have developed to process visual input in distinct manners, facilitating the implementation of a wide range of intelligent behaviors by combining their outputs. Recent advances in Artificial Intelligence (AI) have led to machines capable of displaying diverse forms of visual intelligence, including tasks like object segmentation, categorization, and manipulation using visual input streams. The deep learning approach in AI not only enhanced machine vision but also provided a new means of understanding the visual system. Specifically, it enables us to model and grasp the functional properties of the visual system as emergent characteristics arising from a distributed network of neurons that are tuned for specific visual input statistics. This suggests the potential capability of this framework as an ideal foundation for investigating specialization in both biological and artificial visual systems.
In this talk, we will provide an overview of the recent works in developing artificial neural networks (ANN) that, first, possess specialized pathways, and second, their specialization closely resembles that of the visual cortex. We believe this represents the next phase of understanding the parallels between brains and ANNs, where the best model of the visual system also replicates its specialized pathways.
Materials#
Aligning representations across individual models#
Speaker#
Elizabeth DuPre, PhD is a Wu Tsai Interdisciplinary Postdoctoral Scholar, working in collaboration with Dr. Russell Poldrack and Dr. Scott Linderman. She has a background in cognitive and computational psychology, with a PhD in Neuroscience from McGill University and an MA in Developmental Psychology from Cornell University.
Currently, her research focuses on expanding our statistical toolkit for drawing inferences from high-dimensional, naturalistic datasets measured with modalities such as functional magnetic resonance imaging (fMRI). To do this, she is developing new methods and accompanying open source tools.
Summary: Computational neuroscience is focused on uncovering general organizational principles supporting neural activity and behavior; however, uncovering these principles relies on making appropriate comparisons across individuals. This presents a core technical and conceptual challenge, as individuals differ along nearly every relevant dimension: from the number of neurons supporting computation to the exact computation being performed. Similarly in artificial neural networks, multiple initializations of the same architecture—on the same data—may recruit non-overlapping hidden units, complicating direct comparisons of trained networks.
In this talk, I will introduce techniques for aligning representations in both brains and in machines. I will argue for the importance of considering alignment methods in developing a comprehensive science at the intersection of artificial intelligence and neuroscience that reflects our shared goal of understanding principles of computation. Finally, I will consider current applications and limitations of these techniques, discussing relevant future directions for this area.
Materials#
Multi-modal modeling, auditory - Language modeling beyond language modeling#
Speaker#
Mariya Toneva, PhD is faculty at the Max Planck Institute for Software Systems, where she leads the Bridging AI and Neuroscience group (BrAIN). Her research is at the intersection of Machine Learning, Natural Language Processing, and Neuroscience, with a focus on building computational models of language processing in the brain that can also improve natural language processing systems. Prior to MPI-SWS, Mariya was a C.V. Starr Fellow at the Princeton Neuroscience Institute, where she studied the role of episodic memory in language comprehension together with Ken Norman and Uri Hasson. She received her Ph.D. from Carnegie Mellon University in a joint program between Machine Learning and Neural Computation, where she was advised by Leila Wehbe and Tom Mitchell. Before beginning her graduate studies at CMU, she received a B.S. in both Computer Science and Cognitive Science from Yale University.
Summary: Language models that have been trained to predict the next word over billions of text documents have been shown to also significantly predict brain recordings of people comprehending language. Understanding the reasons behind the observed similarities between language in machines and language in the brain can lead to more insight into both systems. Additionally, the human language system integrates information from multiple sensory modalities which puts text-only language models at a fundamental disadvantage as cognitive models.
In this talk, we will discuss a series of recent works that make progress towards these questions along different dimensions. The unifying principle among these works that allows us to make scientific claims about why one black box (language model) aligns with another black box (the human brain) is our ability to make specific perturbations in the language model and observe their effect on the alignment with the brain. Building on this approach, these works reveal that the observed alignment is due to more than next-word prediction and word-level semantics and is partially related to joint processing of select linguistic information in both systems. Furthermore, we find that the brain alignment can be improved by training a language model to summarize narratives, and to incorporate auditory and visual information of a scene. Taken together, these works make progress towards determining the sufficient and necessary conditions under which language in machines aligns with language in the brain.
Materials#
Approaches for modeling brain responses to natural language using NLP#
Speaker#
Leila Wehbe, PhD is an assistant professor in the Machine Learning Department and the Neuroscience Institute at Carnegie Mellon University. Her work is at the interface of cognitive neuroscience and computer science. It combines naturalistic functional imaging with machine learning both to improve our understanding of the brain and to find insight for building better artificial systems. Previously, she was a postdoctoral researcher at UC Berkeley, working with Jack Gallant. She obtained her PhD from Carnegie Mellon University, where she worked with Tom Mitchell.
Summary: Over the past years a strong alignment has been observed between language model representations and brain activity recordings of individuals that process the same text. This has led to a lot of interest in using these models to understand more about how the brain comprehends language. However, this goal remains challenging due to the complexity of brain activity and the uninterpretable nature of network representations. We explain how to use “computational controls” to make targeted hypothesis tests using encoding models built on representations extracted from neural networks. We further show how to better learn these encoding models and how to interpret them to explore the tuning of different brain regions.
Materials#
Ethics in NeuroAI: In the Making of the NeuroAI Responsible#
Speaker#
Isil Poyraz Bilgin, PhD is a Postdoctoral research fellow at the CRUIGM CNeuroMod project and is supervised by Prof. Leila Wehbe and Prof. Pierre Bellec. Her work focuses on implementing encoding models to predict brain activities of processing the natural language using representations extracted from natural language models. Her main interest lies in developing optimizations to improve the predictive performance of the neural networks of language models with well-defined features of brain dynamics in processing naturalistic stimuli. She holds a bachelor’s degree in pure mathematics, MSc. in Information Technologies, MSc. in Neuroscience of Language and Ph.D. in Cybernetics. Her thesis work focuses on dynamic functional connectivity of the emergence of the neural representation of the novel semantic concepts in the human brain using simultaneous EEG and fMRI.
Summary: The recent decade witnessed a fast revolution of artificial intelligence and its integration in the biological sciences. AI-powered methodologies and tools serve as important facilitators in decoding brain data for clinical and commercial purposes. However, one of the core remaining challenges in building intelligent agents is responsibly handling the training data and resources.
NeuroAI cannot be considered as a separate discipline but a multidisciplinary effort of a crossroad between AI, computer science, neuroscience, psychology, linguistics, philosophy, law, and ethics. As researchers, industry professionals, and members of the community, we all are required to follow responsible practices in data use and protection, fair and transparent resource allocations, and elimination of biases and discrimination that are inherent in AI applications. In this talk, we will discuss the current and future ethical concerns the NeuroAI faces today, share the current initiatives in making the NeuroAI more responsible and fair, and help attendees understand necessary steps toward adopting ethical practices in future work in this area.