Title: Optimal Sequential Decision-Making in Geosteering: A Reinforcement Learning Approach Show Chinese title

Title: 基于强化学习的地质导向中最优顺序决策方法

Abstract: Trajectory adjustment decisions throughout the drilling process, called geosteering, affect subsequent choices and information gathering, thus resulting in a coupled sequential decision problem. Previous works on applying decision optimization methods in geosteering rely on greedy optimization or approximate dynamic programming (ADP). Either decision optimization method requires explicit uncertainty and objective function models, making developing decision optimization methods for complex and realistic geosteering environments challenging to impossible. We use the Deep Q-Network (DQN) method, a model-free reinforcement learning (RL) method that learns directly from the decision environment, to optimize geosteering decisions. The expensive computations for RL are handled during the offline training stage. Evaluating DQN needed for real-time decision support takes milliseconds and is faster than the traditional alternatives. Moreover, for two previously published synthetic geosteering scenarios, our results show that RL achieves high-quality outcomes comparable to the quasi-optimal ADP. Yet, the model-free nature of RL means that by replacing the training environment, we can extend it to problems where the solution to ADP is prohibitively expensive to compute. This flexibility will allow applying it to more complex environments and make hybrid versions trained with real data in the future. Show Chinese abstract

Abstract: 在钻井过程中的轨迹调整决策，称为地质导向，会影响后续的选择和信息收集，从而导致一个耦合的顺序决策问题。 以往将决策优化方法应用于地质导向的研究依赖于贪心优化或近似动态规划（ADP）。 这两种决策优化方法都需要明确的不确定性和目标函数模型，使得在复杂和现实的地质导向环境中开发决策优化方法变得困难甚至不可能。 我们使用深度Q网络（DQN）方法，这是一种无需模型的强化学习（RL）方法，可以直接从决策环境中学习，以优化地质导向决策。 强化学习的昂贵计算是在离线训练阶段处理的。 用于实时决策支持的DQN评估只需几毫秒，比传统替代方案更快。 此外，对于两个之前发布的合成地质导向场景，我们的结果表明，RL能够实现与准最优ADP相当的高质量结果。 然而，RL的无模型性质意味着通过替换训练环境，我们可以将其扩展到ADP解计算成本过高的问题。 这种灵活性将允许将其应用于更复杂的环境，并在未来用真实数据训练混合版本。