Paper Reading AI Learner

FairCode: Evaluating Social Bias of LLMs in Code Generation

2025-01-09 17:42:23
Yongkang Du, Jen-tse Huang, Jieyu Zhao, Lu Lin

Abstract

Large language models (LLMs) have demonstrated significant capability in code generation, drawing increasing attention to the evaluation of the quality and safety of their outputs. However, research on bias in code generation remains limited. Existing studies typically assess bias by applying malicious prompts or reapply tasks and dataset for discriminative models. Given that LLMs are often aligned with human values and that prior datasets are not fully optimized for code-related tasks, there is a pressing need for benchmarks specifically designed for evaluating code models. In this study, we introduce FairCode, a novel benchmark for evaluating bias in code generation. FairCode comprises two tasks: function implementation and test case generation, each evaluating social bias through diverse scenarios. Additionally, we propose a new metric, FairScore, to assess model performance on this benchmark. We conduct experiments on widely used LLMs and provide a comprehensive analysis of the results. The findings reveal that all tested LLMs exhibit bias. The code is available at this https URL.

Abstract (translated)

大型语言模型(LLM)在代码生成方面表现出显著的能力,这引起了对其输出质量和安全性的评估的广泛关注。然而,关于代码生成中的偏见的研究仍然有限。现有的研究通常通过应用恶意提示或重新使用用于区分模型的任务和数据集来评估偏见。鉴于LLM通常与人类价值观相一致,并且先前的数据集并未完全优化用于代码相关任务,因此迫切需要专门设计以评估代码模型的基准。在这项研究中,我们介绍了FairCode,这是一个新的基准测试工具,旨在评估代码生成中的偏见。FairCode包括两个任务:函数实现和测试用例生成,每个任务通过多种场景来评估社会偏见。此外,我们提出了一种新指标——FairScore,用于评估模型在该基准上的性能。我们在广泛使用的LLM上进行了实验,并提供了全面的结果分析。研究发现表明,所有测试的LLM都表现出某种程度的偏见。代码可在[此链接](https://example.com)获取。(请将示例链接替换为实际提供的链接)

URL

https://arxiv.org/abs/2501.05396

PDF

https://arxiv.org/pdf/2501.05396.pdf


Tags
3D Action Action_Localization Action_Recognition Activity Adversarial Agent Attention Autonomous Bert Boundary_Detection Caption Chat Classification CNN Compressive_Sensing Contour Contrastive_Learning Deep_Learning Denoising Detection Dialog Diffusion Drone Dynamic_Memory_Network Edge_Detection Embedding Embodied Emotion Enhancement Face Face_Detection Face_Recognition Facial_Landmark Few-Shot Gait_Recognition GAN Gaze_Estimation Gesture Gradient_Descent Handwriting Human_Parsing Image_Caption Image_Classification Image_Compression Image_Enhancement Image_Generation Image_Matting Image_Retrieval Inference Inpainting Intelligent_Chip Knowledge Knowledge_Graph Language_Model LLM Matching Medical Memory_Networks Multi_Modal Multi_Task NAS NMT Object_Detection Object_Tracking OCR Ontology Optical_Character Optical_Flow Optimization Person_Re-identification Point_Cloud Portrait_Generation Pose Pose_Estimation Prediction QA Quantitative Quantitative_Finance Quantization Re-identification Recognition Recommendation Reconstruction Regularization Reinforcement_Learning Relation Relation_Extraction Represenation Represenation_Learning Restoration Review RNN Robot Salient Scene_Classification Scene_Generation Scene_Parsing Scene_Text Segmentation Self-Supervised Semantic_Instance_Segmentation Semantic_Segmentation Semi_Global Semi_Supervised Sence_graph Sentiment Sentiment_Classification Sketch SLAM Sparse Speech Speech_Recognition Style_Transfer Summarization Super_Resolution Surveillance Survey Text_Classification Text_Generation Time_Series Tracking Transfer_Learning Transformer Unsupervised Video_Caption Video_Classification Video_Indexing Video_Prediction Video_Retrieval Visual_Relation VQA Weakly_Supervised Zero-Shot