标题： 基于注意力感知的标记过滤加速视觉变换器模型 显示英文标题

标题： Speed-up of Vision Transformer Models by Attention-aware Token Filtering

摘要： 视觉Transformer（ViT）模型在图像嵌入提取方面取得了突破，为零样本图像分类等任务提供了最先进的性能。然而，这些模型计算负担沉重。本文提出了一种新的ViT模型加速方法，称为注意力感知标记过滤（ATF）。ATF由两个主要思想组成：一种新的标记过滤模块和一种过滤策略。标记过滤模块被引入到ViT模型的分词器和变换器编码器之间，而无需修改或微调变换器编码器。该模块过滤输入编码器的标记，使其能够动态保留特定对象类型区域中的标记，并保留静态接收高注意力的区域中的标记。这种过滤策略在过滤输入到变换器编码器的标记的同时保持了任务准确性。检索任务上的评估结果显示，ATF为ViT模型SigLIP提供了$2.8\times$倍速度提升，同时保持了检索召回率。 显示英文摘要

摘要： Vision Transformer (ViT) models have made breakthroughs in image embedding extraction, which provide state-of-the-art performance in tasks such as zero-shot image classification. However, the models suffer from a high computational burden. In this paper, we propose a novel speed-up method for ViT models called Attention-aware Token Filtering (ATF). ATF consists of two main ideas: a novel token filtering module and a filtering strategy. The token filtering module is introduced between a tokenizer and a transformer encoder of the ViT model, without modifying or fine-tuning of the transformer encoder. The module filters out tokens inputted to the encoder so that it keeps tokens in regions of specific object types dynamically and keeps tokens in regions that statically receive high attention in the transformer encoder. This filtering strategy maintains task accuracy while filtering out tokens inputted to the transformer encoder. Evaluation results on retrieval tasks show that ATF provides $2.8\times$ speed-up to a ViT model, SigLIP, while maintaining the retrieval recall rate.