Abstract
Research into community content moderation often assumes that moderation teams govern with a single, unified voice. However, recent work has found that moderators disagree with one another at modest, but concerning rates. The problem is not the root disagreements themselves. Subjectivity in moderation is unavoidable, and there are clear benefits to including diverse perspectives within a moderation team. Instead, the crux of the issue is that, due to resource constraints, moderation decisions end up being made by individual decision-makers. The result is decision-making that is inconsistent, which is frustrating for community members. To address this, we develop Venire, an ML-backed system for panel review on Reddit. Venire uses a machine learning model trained on log data to identify the cases where moderators are most likely to disagree. Venire fast-tracks these cases for multi-person review. Ideally, Venire allows moderators to surface and resolve disagreements that would have otherwise gone unnoticed. We conduct three studies through which we design and evaluate Venire: a set of formative interviews with moderators, technical evaluations on two datasets, and a think-aloud study in which moderators used Venire to make decisions on real moderation cases. Quantitatively, we demonstrate that Venire is able to improve decision consistency and surface latent disagreements. Qualitatively, we find that Venire helps moderators resolve difficult moderation cases more confidently. Venire represents a novel paradigm for human-AI content moderation, and shifts the conversation from replacing human decision-making to supporting it.
Abstract (translated)
社区内容审核研究通常假设审核团队以单一统一的声音进行管理。然而，最近的研究发现，审核人员彼此之间存在适度但令人担忧的分歧率。问题不在于这些根本性分歧本身。在审核中存在主观性是不可避免的，并且团队内部包含多元视角有明显的好处。相反，问题的核心是因为资源限制，审核决策最终由个别决策者做出。结果导致了决策的一致性缺失，这对社区成员来说是令人沮丧的。 为了解决这个问题，我们开发了Venire，一个基于机器学习的Reddit小组审议系统。Venire使用了一个训练过的机器学习模型来识别最有可能出现分歧的情况，并将这些情况快速提交给多人审查。理想情况下，Venire允许审核人员发现并解决那些原本可能被忽视的分歧。 我们通过三项研究设计和评估了Venire：一组与审核员的形式化访谈、两个数据集的技术评价以及一项使用思考出声法的研究，在这项研究中，审核员使用Venire来决定真实案例。定量分析显示，Venire能够提高决策的一致性并揭示潜在的分歧。定性分析表明，Venire有助于审核人员更自信地解决困难的审核案件。 Venire代表了人类与AI合作进行内容审核的新范式，并将对话从替代人为决策转变为支持它。
URL
https://arxiv.org/abs/2410.23448
