Multimodal Brain–Computer Interface
A foundation model that unifies EEG, ECoG, fMRI, and fNIRS in one shared representation, built in collaboration with Google DeepMind.
Overview
At Columbia’s Laboratory for Intelligent Imaging and Neural Computing (LIINC, PI Prof. Paul Sajda), I work on models that learn from the brain’s many signals at once. The aim is a single representation that spans very different ways of measuring neural activity, so that a decoder built for one task can draw on what the model has learned from all of them.
Why it’s hard
The brain is measured in fundamentally different ways. Electrophysiology (EEG, ECoG) reads electrical activity at millisecond resolution; hemodynamics (fMRI, fNIRS) reads slower blood-flow changes with different spatial structure. Each modality has its own hardware, sampling rates, and channel layouts, and no two datasets are collected the same way. Most neural decoders sidestep this by locking to a single modality and a single dataset — which is exactly what limits how well they generalize.
Approach
The core is a hierarchical transformer that brings electrophysiology and hemodynamics into one shared latent space, so signals that look nothing alike on the surface can inform the same representation. It is pretrained with self-supervision on large public datasets — the TUH EEG Corpus, SEED, and THINGS — and developed in collaboration with Google DeepMind. Much of the real work sits upstream of the model: standardizing heterogeneous acquisition protocols and channel counts so they can be learned jointly at all.
Related work
In addition to foundational representation learning, our collaborative work has explored adaptive extended-reality interfaces that integrate in-headset behavioral cues with EEG-derived cognitive state to augment human performance in real time (ACM UIST 2026).
Why it matters
A shared, pretrained representation makes decoding more robust and more general than a model tied to one modality and one dataset. It is also a building block for the assistive-communication systems I ultimately want to build — technology that turns neural activity into language for people who cannot speak.