标题： 连续手语识别的分层子动作树 显示英文标题

标题： Hierarchical Sub-action Tree for Continuous Sign Language Recognition

摘要： 连续手语识别（CSLR）旨在将未剪辑的视频转录为词素，这些词素通常是文本单词。 最近的研究表明，由于训练数据不足，缺乏大型数据集和精确注释已成为CSLR的瓶颈。 为了解决这个问题，一些工作开发了跨模态解决方案来对齐视觉和文本模态。 然而，它们通常从词素中提取文本特征，而没有充分利用其知识。 在本文中，我们提出了分层子动作树（HST），称为HST-CSLR，以有效地将词素知识与视觉表示学习结合。 通过从大型语言模型中引入特定于词素的知识，我们的方法更有效地利用了文本信息。 具体来说，我们构建了一个HST用于文本信息表示，逐步对齐视觉和文本模态，并从树结构中受益以减少计算复杂性。 此外，我们施加了对比对齐增强以弥合两种模态之间的差距。 在四个数据集（PHOENIX-2014，PHOENIX-2014T，CSL-Daily和手语手势）上的实验证明了我们HST-CSLR的有效性。 显示英文摘要

摘要： Continuous sign language recognition (CSLR) aims to transcribe untrimmed videos into glosses, which are typically textual words. Recent studies indicate that the lack of large datasets and precise annotations has become a bottleneck for CSLR due to insufficient training data. To address this, some works have developed cross-modal solutions to align visual and textual modalities. However, they typically extract textual features from glosses without fully utilizing their knowledge. In this paper, we propose the Hierarchical Sub-action Tree (HST), termed HST-CSLR, to efficiently combine gloss knowledge with visual representation learning. By incorporating gloss-specific knowledge from large language models, our approach leverages textual information more effectively. Specifically, we construct an HST for textual information representation, aligning visual and textual modalities step-by-step and benefiting from the tree structure to reduce computational complexity. Additionally, we impose a contrastive alignment enhancement to bridge the gap between the two modalities. Experiments on four datasets (PHOENIX-2014, PHOENIX-2014T, CSL-Daily, and Sign Language Gesture) demonstrate the effectiveness of our HST-CSLR.