Abstract
Real-time decoding of target variables from multiple simultaneously recorded neural time-series modalities, such as discrete spiking activity and continuous field potentials, is important across various neuroscience applications. However, a major challenge for doing so is that different neural modalities can have different timescales (i.e., sampling rates) and different probabilistic distributions, or can even be missing at some time-steps. Existing nonlinear models of multimodal neural activity do not address different timescales or missing samples across modalities. Further, some of these models do not allow for real-time decoding. Here, we develop a learning framework that can enable real-time recursive decoding while nonlinearly aggregating information across multiple modalities with different timescales and distributions and with missing samples. This framework consists of 1) a multiscale encoder that nonlinearly aggregates information after learning within-modality dynamics to handle different timescales and missing samples in real time, 2) a multiscale dynamical backbone that extracts multimodal temporal dynamics and enables real-time recursive decoding, and 3) modality-specific decoders to account for different probabilistic distributions across modalities. In both simulations and three distinct multiscale brain datasets, we show that our model can aggregate information across modalities with different timescales and distributions and missing samples to improve real-time target decoding. Further, our method outperforms various linear and nonlinear multimodal benchmarks in doing so.
Abstract (translated)
从多个同时记录的神经时间序列模态（如离散尖峰活动和连续场电位）中实时解码目标变量，在各种神经科学应用中都很重要。然而，实现这一目标的一个主要挑战是不同的神经模态可能具有不同的时间尺度（即采样率）以及不同的概率分布，并且在某些时间段内可能缺失数据。现有的多模态神经活动的非线性模型没有解决跨模态的不同时间尺度或丢失样本的问题。此外，其中一些模型不支持实时解码。在这里，我们开发了一个学习框架，该框架能够在不同时间和分布的多模态信息以及具有丢失样本的情况下进行实时递归解码的同时非线性聚合信息。 这个框架包括： 1. 一个多尺度编码器，它在学会了处理同种模态的动力学之后，能够非线性地汇总信息来应对不同的时间尺度和丢失样本，并实现实时操作。 2. 一个多尺度动力学骨干结构，用于提取多模态的动态特性并支持实时递归解码。 3. 具体针对不同概率分布跨模态的解码器。 在模拟环境以及三个具有不同时间尺度的真实脑数据集中，我们展示了该模型能够在存在不同时间尺度、不同分布和缺失样本的情况下汇总信息，从而提高目标变量的实时解码精度。此外，在执行上述任务时，我们的方法优于各种线性和非线性多模态基准模型。
URL
https://arxiv.org/abs/2512.12462
