标题： CRED：主动偏好学习中的反事实推理与环境设计 显示英文标题

标题： CRED: Counterfactual Reasoning and Environment Design for Active Preference Learning

摘要： 为了有效的现实世界部署，机器人应适应人类偏好，例如在配送路线中平衡距离、时间和安全性。 主动偏好学习（APL）通过呈现轨迹进行排序来学习人类奖励函数。 然而，现有方法常常难以探索完整的轨迹空间，并且在长时域任务中无法识别有信息量的查询。 我们提出了CRED，一种用于APL的轨迹生成方法，通过联合优化环境设计和轨迹选择来改进奖励估计。 CRED通过环境设计“想象”新的场景，并使用反事实推理——通过从当前信念中采样奖励并询问“如果这个奖励是真实偏好会怎样？”——来生成多样且有信息量的轨迹集进行排序。 在GridWorld和使用OpenStreetMap数据的真实世界导航中的实验表明，CRED提高了奖励学习并在不同环境中有效泛化。 显示英文摘要

摘要： For effective real-world deployment, robots should adapt to human preferences, such as balancing distance, time, and safety in delivery routing. Active preference learning (APL) learns human reward functions by presenting trajectories for ranking. However, existing methods often struggle to explore the full trajectory space and fail to identify informative queries, particularly in long-horizon tasks. We propose CRED, a trajectory generation method for APL that improves reward estimation by jointly optimizing environment design and trajectory selection. CRED "imagines" new scenarios through environment design and uses counterfactual reasoning -- by sampling rewards from its current belief and asking "What if this reward were the true preference?" -- to generate a diverse and informative set of trajectories for ranking. Experiments in GridWorld and real-world navigation using OpenStreetMap data show that CRED improves reward learning and generalizes effectively across different environments.