标题： 多模态数据集用于增强工业任务监控和参与度预测 显示英文标题

标题： A Multimodal Dataset for Enhancing Industrial Task Monitoring and Engagement Prediction

摘要： 检测和解释动态工业工作流中的操作行为、参与度和物体交互仍然是人机协作研究中的重大挑战，尤其是在复杂的真实环境中。传统的单模态方法往往难以捕捉这些非结构化工业环境的复杂性。为了解决这一差距，我们提出一个新颖的多模态工业活动监测（MIAM）数据集，该数据集捕捉了现实中的装配和拆卸任务，有助于评估关键的元任务，如动作定位、物体交互和参与度预测。该数据集包含从22个会话中收集的多视角RGB、深度和惯性测量单元（IMU）数据，总计290分钟未剪辑视频，并对任务执行和操作员行为进行了详细标注。其独特之处在于多种数据模态的集成以及对真实、未剪辑工业工作流的重点关注——这对于推动人机协作和操作员监控的研究至关重要。此外，我们提出了一种多模态网络，融合RGB帧、IMU数据和骨骼序列以预测工业任务中的参与度。我们的方法提高了识别参与状态的准确性，为在动态工业环境中监控操作员表现提供了稳健的解决方案。数据集和代码可从https://github.com/navalkishoremehta95/MIAM/获取。 显示英文摘要

摘要： Detecting and interpreting operator actions, engagement, and object interactions in dynamic industrial workflows remains a significant challenge in human-robot collaboration research, especially within complex, real-world environments. Traditional unimodal methods often fall short of capturing the intricacies of these unstructured industrial settings. To address this gap, we present a novel Multimodal Industrial Activity Monitoring (MIAM) dataset that captures realistic assembly and disassembly tasks, facilitating the evaluation of key meta-tasks such as action localization, object interaction, and engagement prediction. The dataset comprises multi-view RGB, depth, and Inertial Measurement Unit (IMU) data collected from 22 sessions, amounting to 290 minutes of untrimmed video, annotated in detail for task performance and operator behavior. Its distinctiveness lies in the integration of multiple data modalities and its emphasis on real-world, untrimmed industrial workflows-key for advancing research in human-robot collaboration and operator monitoring. Additionally, we propose a multimodal network that fuses RGB frames, IMU data, and skeleton sequences to predict engagement levels during industrial tasks. Our approach improves the accuracy of recognizing engagement states, providing a robust solution for monitoring operator performance in dynamic industrial environments. The dataset and code can be accessed from https://github.com/navalkishoremehta95/MIAM/.