Abstract
Prompt Recovery, reconstructing prompts from the outputs of large language models (LLMs), has grown in importance as LLMs become ubiquitous. Most users access LLMs through APIs without internal model weights, relying only on outputs and logits, which complicates recovery. This paper explores a unique prompt recovery task focused on reconstructing prompts for style transfer and rephrasing, rather than typical question-answering. We introduce a dataset created with LLM assistance, ensuring quality through multiple techniques, and test methods like zero-shot, few-shot, jailbreak, chain-of-thought, fine-tuning, and a novel canonical-prompt fallback for poor-performing cases. Our results show that one-shot and fine-tuning yield the best outcomes but highlight flaws in traditional sentence similarity metrics for evaluating prompt recovery. Contributions include (1) a benchmark dataset, (2) comprehensive experiments on prompt recovery strategies, and (3) identification of limitations in current evaluation metrics, all of which advance general prompt recovery research, where the structure of the input prompt is unrestricted.
Abstract (translated)
**翻译** 提示恢复，即从大型语言模型（LLM）的输出中重构提示信息，在LLMs广泛使用的情况下变得日益重要。大多数用户通过API访问LLMs，并且没有内部模型权重的支持，仅依赖于输出和logits，这使得提示恢复变得更加复杂。本文探讨了一个独特的任务——专注于重建用于风格转换和重新表述的提示信息，而不是传统的问答场景。我们介绍了一套利用LLM辅助创建的数据集，并采用多种方法确保数据质量。此外，还测试了零样本、少量样本、破解（jailbreak）、链式思维（chain-of-thought）以及针对性能不佳情况的新颖规范性提示回退等方法。我们的研究结果表明，单样本和微调策略能取得最佳效果，但同时也指出了传统句子相似度评估指标在评价提示恢复方面的不足之处。本文的贡献包括：(1) 提供了一个基准数据集；(2) 对多种提示恢复策略进行了全面实验；以及 (3) 发现了当前评估指标存在的局限性。所有这些都进一步推进了一般性的未受限输入提示结构的提示恢复研究。
URL
https://arxiv.org/abs/2504.04373
