Abstract
Natural language interaction with sensing systems is crucial for enabling all users to comprehend sensor data and its impact on their everyday lives. However, existing systems, which typically operate in a Question Answering (QA) manner, are significantly limited in terms of the duration and complexity of sensor data they can handle. In this work, we introduce SensorChat, the first end-to-end QA system designed for long-term sensor monitoring with multimodal and high-dimensional data including time series. SensorChat effectively answers both qualitative (requiring high-level reasoning) and quantitative (requiring accurate responses derived from sensor data) questions in real-world scenarios. To achieve this, SensorChat uses an innovative three-stage pipeline that includes question decomposition, sensor data query, and answer assembly. The first and third stages leverage Large Language Models (LLMs) for intuitive human interactions and to guide the sensor data query process. Unlike existing multimodal LLMs, SensorChat incorporates an explicit query stage to precisely extract factual information from long-duration sensor data. We implement SensorChat and demonstrate its capability for real-time interactions on a cloud server while also being able to run entirely on edge platforms after quantization. Comprehensive QA evaluations show that SensorChat achieves up to 26% higher answer accuracy than state-of-the-art systems on quantitative questions. Additionally, a user study with eight volunteers highlights SensorChat's effectiveness in handling qualitative and open-ended questions.
Abstract (translated)
与感知系统进行自然语言交互对于使所有用户能够理解传感器数据及其对日常生活的影响至关重要。然而，现有的大多数基于问题回答（QA）方式的系统，在处理传感器数据的时间跨度和复杂性方面存在显著限制。在这项工作中，我们引入了SensorChat——首个专为长期多模态高维度传感数据分析设计的端到端问答系统，其中包括时间序列数据。SensorChat能够有效解答现实场景中的定性和定量问题，前者要求高层次推理，后者则需要从传感器数据中准确提取信息。 为了实现这一目标，SensorChat采用了一个创新性的三阶段管道流程：问题分解、传感器数据查询和答案组装。第一和第三阶段利用大型语言模型（LLMs）来促进直观的人机交互，并指导传感器数据查询过程。与现有的多模态LLM不同，SensorChat包含一个明确的查询阶段，能够精确地从长时间跨度的传感器数据中提取事实信息。 我们实施了SensorChat，并展示了其在云端服务器上的实时互动能力，同时也证明它经过量化处理后可在边缘平台独立运行。全面的问题回答评估表明，在定量问题上，SensorChat比最先进的系统高出26%的答案准确性。此外，一项针对八名志愿者的研究突显了SensorChat在处理定性和开放性问题方面的有效性。
URL
https://arxiv.org/abs/2502.02883
