标题： 重新思考自主性：防止人工智能驱动的软件工程中的故障 显示英文标题

标题： Rethinking Autonomy: Preventing Failures in AI-Driven Software Engineering

摘要： 大型语言模型（LLMs）在软件工程中的集成已经革新了代码生成，通过提示软件和自主AI代理实现了前所未有的生产力。 然而，这种转变引入了重大风险，包括不安全的代码生成、幻觉输出、不可逆操作以及缺乏透明度和问责制。 像Replit数据库删除这样的事件凸显了建立强大安全和治理机制的紧迫需求。 本文全面分析了LLM辅助代码生成的固有挑战，如漏洞继承、过度信任、误解以及缺乏标准化的验证和回滚协议。 为了解决这些问题，我们提出了SAFE-AI框架，这是一种强调安全性、可审计性、反馈和可解释性的整体方法。 该框架整合了护栏、沙箱、运行时验证、风险感知日志记录、人机协同系统和可解释AI技术，以降低风险并促进信任和合规性。 我们引入了一种新的AI行为分类法，将建议性、生成性、自主性和破坏性行为进行分类，以指导风险评估和监督。 此外，我们识别了开放性问题，包括代码特定幻觉和自主性水平的标准化基准的缺乏，并提出了混合验证、语义护栏和主动治理工具的未来研究方向。 通过详细比较自主控制、提示工程、可解释性和治理框架，本文为软件工程中负责任的AI集成提供了路线图，符合欧盟人工智能法案和加拿大的AIDA等新兴法规，以确保安全、透明和可问责的AI驱动开发。 显示英文摘要

摘要： The integration of Large Language Models (LLMs) into software engineering has revolutionized code generation, enabling unprecedented productivity through promptware and autonomous AI agents. However, this transformation introduces significant risks, including insecure code generation, hallucinated outputs, irreversible actions, and a lack of transparency and accountability. Incidents like the Replit database deletion underscore the urgent need for robust safety and governance mechanisms. This paper comprehensively analyzes the inherent challenges of LLM-assisted code generation, such as vulnerability inheritance, overtrust, misinterpretation, and the absence of standardized validation and rollback protocols. To address these, we propose the SAFE-AI Framework, a holistic approach emphasizing Safety, Auditability, Feedback, and Explainability. The framework integrates guardrails, sandboxing, runtime verification, risk-aware logging, human-in-the-loop systems, and explainable AI techniques to mitigate risks while fostering trust and compliance. We introduce a novel taxonomy of AI behaviors categorizing suggestive, generative, autonomous, and destructive actions to guide risk assessment and oversight. Additionally, we identify open problems, including the lack of standardized benchmarks for code specific hallucinations and autonomy levels, and propose future research directions for hybrid verification, semantic guardrails, and proactive governance tools. Through detailed comparisons of autonomy control, prompt engineering, explainability, and governance frameworks, this paper provides a roadmap for responsible AI integration in software engineering, aligning with emerging regulations like the EU AI Act and Canada's AIDA to ensure safe, transparent, and accountable AI-driven development.