Title: Design and Benchmarking of A Multi-Modality Sensor for Robotic Manipulation with GAN-Based Cross-Modality Interpretation Show Chinese title

Title: 基于GAN的跨模态解释的多模态传感器在机器人操作中的设计与基准测试

Abstract: In this paper, we present the design and benchmark of an innovative sensor, ViTacTip, which fulfills the demand for advanced multi-modal sensing in a compact design. A notable feature of ViTacTip is its transparent skin, which incorporates a `see-through-skin' mechanism. This mechanism aims at capturing detailed object features upon contact, significantly improving both vision-based and proximity perception capabilities. In parallel, the biomimetic tips embedded in the sensor's skin are designed to amplify contact details, thus substantially augmenting tactile and derived force perception abilities. To demonstrate the multi-modal capabilities of ViTacTip, we developed a multi-task learning model that enables simultaneous recognition of hardness, material, and textures. To assess the functionality and validate the versatility of ViTacTip, we conducted extensive benchmarking experiments, including object recognition, contact point detection, pose regression, and grating identification. To facilitate seamless switching between various sensing modalities, we employed a Generative Adversarial Network (GAN)-based approach. This method enhances the applicability of the ViTacTip sensor across diverse environments by enabling cross-modality interpretation. Show Chinese abstract

Abstract: 在本文中，我们介绍了创新传感器ViTacTip的设计和基准测试，该传感器在紧凑设计中满足了先进多模态传感的需求。ViTacTip的一个显著特点是其透明皮肤，其中包含一种“透皮”机制。该机制旨在接触时捕捉详细的物体特征，显著提高了基于视觉和接近感知的能力。同时，嵌入传感器皮肤中的仿生尖端被设计用来放大接触细节，从而大幅增强触觉和派生的力感知能力。为了展示ViTacTip的多模态能力，我们开发了一个多任务学习模型，能够同时识别硬度、材质和纹理。为了评估ViTacTip的功能并验证其通用性，我们进行了广泛的基准测试实验，包括物体识别、接触点检测、姿态回归和条纹识别。为了促进各种传感模式之间的无缝切换，我们采用了一种基于生成对抗网络（GAN）的方法。该方法通过实现跨模态解释，增强了ViTacTip传感器在多种环境中的适用性。