多模态基础模型 颠覆级 有讲解视频
发表时间
2026-08-14
arXiv
2608.12944

收录解读

心脏信号(ECG、PPG、PCG)提供了同一心动周期的互补视角,但现有心脏基础模型往往仅针对单一模态,未利用跨传感器的共享生理信息。

本文提出CardioState-JEPA,一个基于生理感知的联合嵌入预测架构,将异质波形映射到共同token空间,通过共享Transformer编码器学习,并以预测掩码心脏状态为预训练目标;同时引入延迟对齐器处理电、机械、血流事件之间的时间偏移。

在25个下游任务上,冻结编码器相比最佳自监督信号基线,平均PPG分类AUROC提升8.2点,PCG杂音检测提升18.8点,ECG分类提升15.5点,并在若干ECG基准上达到或超过使用特权临床文本或监督标签训练的模型。

该方法展示了异质心脏信号可以相互监督一个统一的生理学基础模型,具有跨模态迁移价值;但尚未公布代码或大规模公共基准上的独立验证,且应用范围限于心脏信号。

原始摘要

Electrocardiography (ECG), photoplethysmography (PPG), and phonocardiography (PCG) provide complementary views of the same cardiac cycle, yet existing cardiac foundation models are trained for a single sensing modality, leaving the shared physiology across sensors unexploited. We introduce CardioState-JEPA, a cardiac foundation model to learn a single shared representation jointly across ECG, PPG, and PCG, built on a physiology-aware joint-embedding predictive architecture. The model maps heterogeneous waveforms into a common token space, processes them with a single shared Transformer encoder, and learns by predicting masked latent cardiac states, placing the pretraining target on shared physiology rather than sensor-specific waveform appearance. To handle the temporal offsets between electrical, mechanical, and hemodynamic events, cross-modal prediction uses a learned delay aligner that matches signals at the corresponding cardiac time. Because synchronized multi-sensor recordings are scarce, CardioState-JEPA first learns within-modality structure from abundant unimodal data and then uses paired data to align modalities in latent cardiac time. Evaluated as a frozen encoder across 25 downstream tasks spanning ECG, PPG, and PCG, our encoder improves average PPG classification by 8.2 AUROC points, PCG murmur detection by 18.8 AUROC points, and ECG classification by 15.5 AUROC points over the best self-supervised signal baseline and matches or exceeds cardiac models trained with privileged clinical text or supervised labels on several ECG benchmarks. These results establish that heterogeneous cardiac signals can mutually supervise a single foundation model of cardiac physiology.

解读视频

相关论文

链接