Abstract
The increasing prevalence of Large Language Models (LLMs) demands effective safeguards for their operation, particularly concerning their tendency to generate out-of-context responses. A key challenge is accurately detecting when LLMs stray from expected conversational norms, manifesting as topic shifts, factual inaccuracies, or outright hallucinations. Traditional anomaly detection struggles to directly apply within contextual semantics. This paper outlines our experiment in exploring the use of Representation Engineering (RepE) and One-Class Support Vector Machine (OCSVM) to identify subspaces within the internal states of LLMs that represent a specific context. By training OCSVM on in-context examples, we establish a robust boundary within the LLM's hidden state latent space. We evaluate out study with two open source LLMs - Llama and Qwen models in specific contextual domain. Our approach entailed identifying the optimal layers within the LLM's internal state subspaces that strongly associates with the context of interest. Our evaluation results showed promising results in identifying the subspace for a specific context. Aside from being useful in detecting in or out of context conversation threads, this research work contributes to the study of better interpreting LLMs.
Abstract (translated)
大型语言模型（LLMs）的日益普及要求有效的安全措施来保障其运行，尤其是在这些模型倾向于生成不符合上下文响应的情况下。一个关键挑战是准确检测出当LLM偏离预期对话规范时的情况，这可能表现为话题转变、事实错误或直接幻觉。传统的异常检测方法难以直接应用于语境化的语义中。本文概述了我们使用表示工程（RepE）和一类支持向量机（OCSVM）探索如何识别内部状态中的子空间，这些子空间代表特定上下文的实验。 通过在具体上下文中训练OCSVM，我们在LLM的隐藏状态潜在空间内建立了一个坚实的边界。我们的研究使用了两个开源的大规模语言模型——Llama和Qwen，并针对特定的上下文领域进行了评估。我们方法的核心在于识别出LLM内部状态子空间中的最佳层，这些层与感兴趣的上下文密切相关。 评估结果显示，在确定特定上下文的空间方面取得了令人鼓舞的结果。除了在检测是否符合上下文对话线程的应用中非常有用之外，这项研究还为更好地理解和解释大型语言模型做出了贡献。
URL
https://arxiv.org/abs/2601.12286
