Abstract
Of the over 7,000 languages spoken in the world, commercial language identification (LID) systems only reliably identify a few hundred in written form. Research-grade systems extend this coverage under certain circumstances, but for most languages coverage remains patchy or nonexistent. This position paper argues that this situation is largely self-imposed. In particular, it arises from a persistent framing of LID as decontextualized text classification, which obscures the central role of prior probability estimation and is reinforced by institutional incentives that favor global, fixed-prior models. We argue that improving coverage for tail languages requires rethinking LID as a routing problem and developing principled ways to incorporate environmental cues that make languages locally plausible.
Abstract (translated)
在全球超过7000种语言中，商用语言识别（LID）系统仅能可靠地识别几百种语言的书面形式。研究级别的系统在特定情况下可以扩大这一覆盖范围，但对于大多数语言而言，覆盖率仍然支离破碎或根本不存在。本文认为这种情况主要是自我限制的结果。特别是，它源于将LID视为脱离上下文的文本分类这种持久的观点，这掩盖了先验概率估计的核心作用，并且受到倾向于全球性、固定先验模型的制度激励所强化。我们认为，要提高尾部语言（即使用较少的语言）的覆盖率，需要重新审视LID作为一个路由问题的本质，并开发出有原则的方法来整合环境线索，使各种语言在局部环境中变得合理可信。
URL
https://arxiv.org/abs/2602.08951
