Title: Judging by Appearances? Auditing and Intervening Vision-Language Models for Bail Prediction Show Chinese title

Title: 仅凭外表？ 用于保释预测的审计和干预视觉-语言模型

Abstract: Large language models (LLMs) have been extensively used for legal judgment prediction tasks based on case reports and crime history. However, with a surge in the availability of large vision language models (VLMs), legal judgment prediction systems can now be made to leverage the images of the criminals in addition to the textual case reports/crime history. Applications built in this way could lead to inadvertent consequences and be used with malicious intent. In this work, we run an audit to investigate the efficiency of standalone VLMs in the bail decision prediction task. We observe that the performance is poor across multiple intersectional groups and models \textit{wrongly deny bail to deserving individuals with very high confidence}. We design different intervention algorithms by first including legal precedents through a RAG pipeline and then fine-tuning the VLMs using innovative schemes. We demonstrate that these interventions substantially improve the performance of bail prediction. Our work paves the way for the design of smarter interventions on VLMs in the future, before they can be deployed for real-world legal judgment prediction. Show Chinese abstract

Abstract: 大型语言模型（LLMs）已被广泛用于基于案件报告和犯罪历史的法律判决预测任务。 然而，随着大型视觉语言模型（VLMs）的可用性激增，法律判决预测系统现在可以利用罪犯的图像，而不仅仅是文本案件报告/犯罪历史。 以这种方式构建的应用程序可能导致无意的后果，并可能被恶意使用。 在本工作中，我们进行了一项审计，以研究独立VLMs在保释决定预测任务中的效率。 我们观察到，在多个交叉群体和模型\textit{错误地拒绝保释给值得的个体，且非常自信}中，性能都很差。 我们通过首先通过RAG管道包含法律先例，然后使用创新方案对VLMs进行微调，设计了不同的干预算法。 我们证明这些干预措施显著提高了保释预测的性能。 我们的工作为未来在VLMs上设计更智能的干预措施铺平了道路，在它们可用于现实世界的法律判决预测之前。