Abstract
Voice-enabled technology is quickly becoming ubiquitous, and is constituted from machine learning (ML)-enabled components such as speech recognition and voice activity detection. However, these systems don't yet work well for everyone. They exhibit bias - the systematic and unfair discrimination against individuals or cohorts of individuals in favour of others (Friedman & Nissembaum, 1996) - across axes such as age, gender and accent. ML is reliant on large datasets for training. Dataset documentation is designed to give ML Practitioners (MLPs) a better understanding of a dataset's characteristics. However, there is a lack of empirical research on voice dataset documentation specifically. Additionally, while MLPs are frequent participants in fairness research, little work focuses on those who work with voice data. Our work makes an empirical contribution to this gap. Here, we combine two methods to form an exploratory study. First, we undertake 13 semi-structured interviews, exploring multiple perspectives of voice dataset documentation practice. Using open and axial coding methods, we explore MLPs' practices through the lenses of roles and tradeoffs. Drawing from this work, we then purposively sample voice dataset documents (VDDs) for 9 voice datasets. Our findings then triangulate these two methods, using the lenses of MLP roles and trade-offs. We find that current VDD practices are inchoate, inadequate and incommensurate. The characteristics of voice datasets are codified in fragmented, disjoint ways that often do not meet the needs of MLPs. Moreover, they cannot be readily compared, presenting a barrier to practitioners' bias reduction efforts. We then discuss the implications of these findings for bias practices in voice data and speech technologies. We conclude by setting out a program of future work to address these findings -- that is, how we may "right the docs".
Abstract (translated)
语音驱动的技术已经变得非常普遍,其构成成分包括语音识别和语音活动检测等机器学习(ML)驱动组件。然而,这些系统并不一定适用于每个人。它们表现出偏见——对个体或群体整体进行系统性和不公平的歧视,以某些人为优势(Friedman & Nissembaum,1996)——跨越年龄、性别和口音等轴。机器学习依赖于大型数据集进行训练。数据集文档的设计旨在使机器学习从业者(MLP)更好地理解数据集的特征。然而, specifically, there is a lack of empirical research on voice dataset documentation. Additionally, while MLPs are frequently participants in fairness research, little attention is paid to those who work with voice data. Our work fills this gap by making an empirical contribution. Here, we combine two methods to form an exploration study. First, we conduct 13 semi-structured interviews, exploring the multiple perspectives of voice dataset documentation practice. Using open and axial coding methods, we explore MLP practices through the lens of roles and tradeoffs. Drawing from this work, we then randomly sample voice dataset documents (VDDs) for 9 voice datasets. Our findings then triangulate these two methods using MLP roles and trade-offs. We find that current VDD practices are inchoate, inadequate, and incommensurate. The characteristics of voice datasets arecodified in fragmented, disjoint ways that often do not meet the needs of MLPs. Moreover, they cannot be readily compared, presenting a barrier to practitioners' bias reduction efforts. We then discuss the implications of these findings for bias practices in voice data and speech technologies. We conclude by setting out a program of future work to address these findings——that is, how we may "right theDocs".
URL
https://arxiv.org/abs/2303.10721
