Title: Q-learning for distributed routing in LEO satellite constellations Show Chinese title

Title: 基于Q学习的LEO卫星星座分布式路由

Abstract: End-to-end routing in Low Earth Orbit (LEO) satellite constellations (LSatCs) is a complex and dynamic problem. The topology, of finite size, is dynamic and predictable, the traffic from/to Earth and transiting the space segment is highly imbalanced, and the delay is dominated by the propagation time in non-congested routes and by the queueing time at Inter-Satellite Links (ISLs) in congested routes. Traditional routing algorithms depend on excessive communication with ground or other satellites, and oversimplify the characterization of the path links towards the destination. We model the problem as a multi-agent Partially Observable Markov Decision Problem (POMDP) where the nodes (i.e., the satellites) interact only with nearby nodes. We propose a distributed Q-learning solution that leverages on the knowledge of the neighbours and the correlation of the routing decisions of each node. We compare our results to two centralized algorithms based on the shortest path: one aiming at using the highest data rate links and a second genie algorithm that knows the instantaneous queueing delays at all satellites. The results of our proposal are positive on every front: (1) it experiences delays that are comparable to the benchmarks in steady-state conditions; (2) it increases the supported traffic load without congestion; and (3) it can be easily implemented in a LSatC as it does not depend on the ground segment and minimizes the signaling overhead among satellites. Show Chinese abstract

Abstract: 在低地球轨道（LEO）卫星星座（LSatCs）中的端到端路由是一个复杂且动态的问题。拓扑结构是有限大小的，动态且可预测的，从地球发出或经过空间段的流量高度不平衡，延迟主要由非拥塞路径中的传播时间以及拥塞路径中星际链路（ISLs）的排队时间主导。传统的路由算法依赖于与地面或其他卫星的过多通信，并且过于简化了对通往目的地路径链路的表征。我们将该问题建模为一个多智能体部分可观测马尔可夫决策问题（POMDP），其中节点（即卫星）仅与附近的节点进行交互。我们提出了一种分布式Q学习解决方案，该方案利用邻居的知识以及每个节点的路由决策的相关性。我们将我们的结果与两种基于最短路径的集中式算法进行比较：一种旨在使用最高数据速率的链路，另一种是已知所有卫星瞬时排队延迟的“全知”算法。我们提案的结果在各个方面都是积极的：（1）它在稳态条件下经历的延迟与基准相当；（2）它可以在不引起拥塞的情况下增加支持的流量负载；（3）它可以轻松地在LSatC中实现，因为它不依赖于地面段，并且最小化了卫星之间的信令开销。