Abstract
Identifying inputs that trigger specific behaviours or latent features in language models could have a wide range of safety use cases. We investigate a class of methods capable of generating targeted, linguistically fluent inputs that activate specific latent features or elicit model behaviours. We formalise this approach as context modification and present ContextBench -- a benchmark with tasks assessing core method capabilities and potential safety applications. Our evaluation framework measures both elicitation strength (activation of latent features or behaviours) and linguistic fluency, highlighting how current state-of-the-art methods struggle to balance these objectives. We enhance Evolutionary Prompt Optimisation (EPO) with LLM-assistance and diffusion model inpainting, and demonstrate that these variants achieve state-of-the-art performance in balancing elicitation effectiveness and fluency.
Abstract (translated)
识别能够触发特定行为或潜在特征的语言模型输入可能具有广泛的安全应用场景。我们研究了一类可以生成针对性强、语言流利的输入的方法，这些输入能激活特定的潜在特性或引发模型的行为。我们将这种方法形式化为上下文修改，并提出了ContextBench——一个基准测试集，其中包括评估核心方法能力和潜在安全应用的任务。我们的评估框架衡量了激发强度（即激活潜在特征或行为）和语言流畅性，强调了当前最先进的方法在平衡这些目标方面所面临的挑战。我们通过引入大模型辅助和扩散模型修复技术来增强进化提示优化（EPO），并证明这些变体在平衡激发有效性和流畅性方面的性能达到了最新水平。
URL
https://arxiv.org/abs/2506.15735
