Abstract
Oral histories are vital records of lived experience, particularly within communities affected by systemic injustice and historical erasure. Effective and efficient analysis of their oral history archives can promote access and understanding of the oral histories. However, Large-scale analysis of these archives remains limited due to their unstructured format, emotional complexity, and high annotation costs. This paper presents a scalable framework to automate semantic and sentiment annotation for Japanese American Incarceration Oral History. Using LLMs, we construct a high-quality dataset, evaluate multiple models, and test prompt engineering strategies in historically sensitive contexts. Our multiphase approach combines expert annotation, prompt design, and LLM evaluation with ChatGPT, Llama, and Qwen. We labeled 558 sentences from 15 narrators for sentiment and semantic classification, then evaluated zero-shot, few-shot, and RAG strategies. For semantic classification, ChatGPT achieved the highest F1 score (88.71%), followed by Llama (84.99%) and Qwen (83.72%). For sentiment analysis, Llama slightly outperformed Qwen (82.66%) and ChatGPT (82.29%), with all models showing comparable results. The best prompt configurations were used to annotate 92,191 sentences from 1,002 interviews in the JAIOH collection. Our findings show that LLMs can effectively perform semantic and sentiment annotation across large oral history collections when guided by well-designed prompts. This study provides a reusable annotation pipeline and practical guidance for applying LLMs in culturally sensitive archival analysis. By bridging archival ethics with scalable NLP techniques, this work lays the groundwork for responsible use of artificial intelligence in digital humanities and preservation of collective memory. GitHub: this https URL.
Abstract (translated)
口述历史是生活经验的重要记录，尤其在那些遭受系统性不公和历史遗忘的社区中。有效且高效地分析这些口述历史档案可以促进对口述史的理解与访问。然而，由于其非结构化格式、情感复杂性和高昂的注释成本，大规模分析仍然受到限制。本文提出了一种可扩展框架，用于自动标注日裔美国人拘留所口述历史的情感和语义。通过使用大语言模型（LLMs），我们构建了一个高质量的数据集，并在敏感的历史背景下评估了多个模型及提示工程策略。 我们的多阶段方法结合了专家注释、设计提示以及与ChatGPT、Llama和Qwen的模型评估。从15位叙述者中，我们将558句话标注为情感分类和语义分类，然后测试了零样本、少样本和RAG（检索增强生成）策略。 对于语义分类，ChatGPT取得了最高的F1分数（88.71%），其次是Llama（84.99%）和Qwen（83.72%）。在情感分析方面，Llama略微优于Qwen（82.66%）和ChatGPT（82.29%），所有模型表现出相似的结果。 我们使用最佳的提示配置注释了日裔美国人拘留所口述历史集合中1,002次采访中的92,191句话。研究结果表明，当通过精心设计的提示引导时，LLMs能够在大规模口述历史收藏上有效地执行语义和情感标注。 这项研究提供了一个可重复使用的注释管道以及在文化敏感档案分析中应用LLMs的实际指导原则。本工作将档案伦理与可扩展自然语言处理技术相结合，为数字人文科学领域中人工智能的负责任使用奠定了基础，并有助于集体记忆的保护。GitHub网址：[此链接](https://github.com/yourusername/repositoryname)（请用实际的GitHub地址替换链接）。
URL
https://arxiv.org/abs/2508.06729
