Abstract
The TRAIMA project (TRaitement Automatique des Interactions Multimodales en Apprentissage), conducted between March 2019 and June 2020, investigates the potential of automatic processing of multimodal interactions in educational settings. The project addresses a central methodological challenge in educational and interactional research: the analysis of verbal, paraverbal, and non-verbal data is currently carried out manually, making it extremely time-consuming and difficult to scale. TRAIMA explores how machine learning approaches could contribute to the categorisation and classification of such interactions. The project focuses specifically on explanatory and collaborative sequences occurring in classroom interactions, particularly in French as a Foreign Language (FLE) and French as a First Language (FLM) contexts. These sequences are analysed as inherently multimodal phenomena, combining spoken language with prosody, gestures, posture, gaze, and spatial positioning. A key theoretical contribution of the project is the precise linguistic and interactional definition of explanatory discourse as a tripartite sequence (opening, explanatory core, closure), drawing on discourse analysis and interactional linguistics. A substantial part of the research is devoted to the methodological foundations of transcription, which constitute a critical bottleneck for any form of automation. The report provides a detailed state of the art of existing transcription conventions (ICOR, Mondada, GARS, VALIBEL, Ferr{é}), highlighting their respective strengths and limitations when applied to multimodal classroom data. Through comparative analyses of manually transcribed sequences, the project demonstrates the inevitable variability and interpretative dimension of transcription practices, depending on theoretical positioning and analytical goals. Empirical work is based on several corpora, notably the INTER-EXPLIC corpus (approximately 30 hours of classroom interaction) and the EXPLIC-LEXIC corpus, which serve both as testing grounds for manual annotation and as reference datasets for future automation. Particular attention is paid to teacher gestures (kin{é}sic and proxemic resources), prosodic features, and their functional role in meaning construction and learner comprehension. The project also highlights the strategic role of the Techn{é}LAB platform, which provides advanced multimodal data capture (multi-camera video, synchronized audio, eye-tracking, digital interaction traces) and constitutes both a research infrastructure and a test environment for the development of automated tools. In conclusion, TRAIMA does not aim to deliver a fully operational automated system, but rather to establish a rigorous methodological framework for the automatic processing of multimodal pedagogical interactions. The project identifies transcription conventions, annotation categories, and analytical units that are compatible with machine learning approaches, while emphasizing the need for theoretical explicitness and researcher reflexivity. TRAIMA thus lays the groundwork for future interdisciplinary research at the intersection of didactics, discourse analysis, multimodality, and artificial intelligence in education.
Abstract (translated)
TRAIMA项目（2019年3月至2020年6月期间进行）旨在研究教育环境中自动处理多模态互动的潜力。该项目解决了一个在教育和互动研究中的核心方法论挑战：目前，对于言语、副语言以及非言语数据的分析都是手动完成的，这使得工作既耗时又难以扩展。TRAIMA探索了机器学习方法如何有助于此类交互的数据分类和归类。 项目特别关注课堂中解释性和协作性的序列，特别是在作为外语（FLE）和母语（FLM）背景下的法国语言教学环境中。这些序列被分析为多模态现象，结合口头言语、韵律、手势、姿态、目光接触以及空间定位等元素进行研究。项目的一个重要理论贡献在于精确地用三部分结构定义解释性话语：引入部分、核心阐释和结束部分，并借鉴了语篇分析与互动语言学的研究成果。 项目的重要一部分致力于转录方法论的基础，这是任何自动化形式的关键瓶颈之一。报告详述了现有转录惯例（如ICOR、Mondada、GARS、VALIBEL和Ferré）的现状，并突出显示这些惯例在应用于多模态教室数据时各自的优缺点与限制。 通过对比分析手动转录序列，项目展示了转录实践不可避免的变异性和解释维度，这取决于理论定位和分析目标。实证研究基于几个语料库，包括INTER-EXPLIC（大约30小时的课堂互动）和EXPLIC-LEXIC语料库，这些不仅用于测试人工注释方法，还作为未来自动化的参考数据集。 项目特别关注教师的手势（身体动作与空间距离资源）、韵律特征及其在意义构建和学习者理解中的功能角色。此外，项目强调了TechnéLAB平台的战略作用，该平台提供先进的多模态数据采集工具（包括多摄像头视频、同步音频、眼动追踪以及数字互动记录），并同时作为研究基础设施和自动化工具开发的测试环境。 最终，TRAIMA项目的目的是建立一个严格的自动处理多模态教育交互的方法论框架，而不是试图提供一套完全可操作的自动化系统。项目确定了与机器学习方法兼容的转录惯例、注释类别以及分析单位，并强调理论清晰度和研究者反思的重要性。因此，TRAIMA为跨学科研究奠定了基础，在这一领域中，教学法、语篇分析、多模态性及教育领域的人工智能相互交叉。
URL
https://arxiv.org/abs/2601.12844
