标题： Twill：在异构移动边缘平台上的复合AI系统调度 显示英文标题

标题： Twill: Scheduling Compound AI Systems on Heterogeneous Mobile Edge Platforms

摘要： 复合人工智能（cAI）系统通过连接多个人工智能模型来解决复杂问题。 cAI系统通常由深度神经网络（DNNs）、转换器和大型语言模型（LLMs）组成，表现出高度的计算多样性和动态工作负载变化。 在移动边缘平台部署cAI服务在调度并发的DNN-转换器推理任务方面提出了重大挑战，这些任务以未知的顺序动态到达。 现有的移动边缘AI推理策略管理多DNN或仅转换器的工作负载，依赖于设计时的性能分析，无法处理cAI系统所需的DNN和转换器的并发推理。 在本工作中，我们解决了在异构移动边缘平台上调度cAI系统的挑战。 我们提出了Twill，一个运行时框架，通过任务亲和力感知的集群映射和迁移、优先级感知的任务冻结/解冻以及DVFS，在遵守功耗预算的同时最小化推理延迟，以处理cAI工作负载的并发推理请求。 我们在Nvidia Jetson Orin NX平台上实现了并部署了我们的Twill框架。 我们在当代DNN和LLMs上对Twill进行了评估，与最先进的边缘AI推理技术相比，平均推理延迟降低了54%，同时遵守功耗预算。 显示英文摘要

摘要： Compound AI (cAI) systems chain multiple AI models to solve complex problems. cAI systems are typically composed of deep neural networks (DNNs), transformers, and large language models (LLMs), exhibiting a high degree of computational diversity and dynamic workload variation. Deploying cAI services on mobile edge platforms poses a significant challenge in scheduling concurrent DNN-transformer inference tasks, which arrive dynamically in an unknown sequence. Existing mobile edge AI inference strategies manage multi-DNN or transformer-only workloads, relying on design-time profiling, and cannot handle concurrent inference of DNNs and transformers required by cAI systems. In this work, we address the challenge of scheduling cAI systems on heterogeneous mobile edge platforms. We present Twill, a run-time framework to handle concurrent inference requests of cAI workloads through task affinity-aware cluster mapping and migration, priority-aware task freezing/unfreezing, and DVFS, while minimizing inference latency within power budgets. We implement and deploy our Twill framework on the Nvidia Jetson Orin NX platform. We evaluate Twill against state-of-the-art edge AI inference techniques over contemporary DNNs and LLMs, reducing inference latency by 54% on average, while honoring power budgets.