Title: Resource-Efficient Transformer Architecture: Optimizing Memory and Execution Time for Real-Time Applications Show Chinese title

Title: 资源高效Transformer架构：为实时应用优化内存和执行时间

Abstract: This paper describes a memory-efficient transformer model designed to drive a reduction in memory usage and execution time by substantial orders of magnitude without impairing the model's performance near that of the original model. Recently, new architectures of transformers were presented, focused on parameter efficiency and computational optimization; however, such models usually require considerable resources in terms of hardware when deployed in real-world applications on edge devices. This approach addresses this concern by halving embedding size and applying targeted techniques such as parameter pruning and quantization to optimize the memory footprint with minimum sacrifices in terms of accuracy. Experimental results include a 52% reduction in memory usage and a 33% decrease in execution time, resulting in better efficiency than state-of-the-art models. This work compared our model with existing compelling architectures, such as MobileBERT and DistilBERT, and proved its feasibility in the domain of resource-friendly deep learning architectures, mainly for applications in real-time and in resource-constrained applications. Show Chinese abstract

Abstract: 本文介绍了一种内存高效的变压器模型，旨在通过大幅减少内存使用和执行时间，同时不会损害模型性能接近原始模型的水平。 最近，提出了新的变压器架构，专注于参数效率和计算优化；然而，当这些模型部署到边缘设备上的现实应用时，通常需要相当大的硬件资源。 这种方法通过将嵌入大小减半并应用有针对性的技术（如参数剪枝和量化）来优化内存占用，在最小牺牲准确率的情况下解决了这一问题。 实验结果显示，内存使用减少了52％，执行时间减少了33％，比最先进的模型更高效。 这项工作将我们的模型与现有的令人信服的架构（如MobileBERT和DistilBERT）进行了比较，并证明了其在资源友好型深度学习架构领域的可行性，主要适用于实时和资源受限的应用场景。