Abstract
Cloud systems generate large, heterogeneous log data containing critical infrastructure, application, and security information. Transforming these logs into RDF triples enables their integration into knowledge graphs, improving interpretability, root-cause analysis, and cross-service reasoning beyond what raw logs allow. Large Language Models (LLMs) offer a promising approach to automate RDF knowledge graph generation; however, their effectiveness on complex cloud logs remains largely unexplored. In this paper, we evaluate multiple LLM architectures and prompting strategies for automated RDF extraction using a controlled framework with two pipelines for systematically processing semi-structured log data. The extraction pipeline integrates multiple LLMs to identify relevant entities and relationships, automatically generating subject-predicate-object triples. These outputs are evaluated using a dedicated validation pipeline with both syntactic and semantic metrics to assess accuracy, completeness, and quality. Due to the lack of public ground-truth datasets, we created a reference Log-to-KG dataset from OpenStack logs using manual annotation and ontology-driven methods, enabling objective baseline. Our analysis shows that Few-Shot learning is the most effective strategy, with Llama achieving a 99.35% F1 score and 100% valid RDF output while Qwen, NuExtract, and Gemma also perform well under Few-Shot prompting, with Chain-of-Thought approaches maintaining similar accuracy. One-Shot prompting offers a lighter but effective alternative, while Zero-Shot and advanced strategies such as Tree-of-Thought, Self-Critique, and Generate-Multiple perform substantially worse. These results highlight the importance of contextual examples and prompt design for accurate RDF extraction and reveal model-specific limitations across LLM architectures.
Abstract (translated)
云系统会产生包含关键基础设施、应用程序及安全信息的大规模异构日志数据。将这些日志转换为RDF三元组可使其融入知识图谱，从而在原始日志的基础上提升可解释性、根因分析能力以及跨服务推理能力。大语言模型（LLMs）为自动化生成RDF知识图谱提供了一种前景可期的方法；然而，其处理复杂云日志的有效性尚待深入探索。本文通过一个受控框架评估了多种LLM架构与提示策略，该框架包含两条流水线，用于系统化处理半结构化日志数据。提取流水线整合多个LLM以识别相关实体与关系，自动生成主谓宾三元组。这些输出通过专门的验证流水线进行评估，该流水线采用句法与语义双重指标，以衡量其准确性、完整性与质量。由于缺乏公开的基准真值数据集，我们基于OpenStack日志，通过人工标注与本体驱动方法构建了参考性的Log-to-KG数据集，从而确立了客观的基线。分析表明，小样本学习是最有效的策略，其中Llama实现了99.35%的F1分数与100%的有效RDF输出；而Qwen、NuExtract与Gemma在小样本提示下同样表现优异，思维链方法也保持了相近的准确率。单样本提示提供了一种更轻量但有效的替代方案，而零样本及树思维、自我批评、生成多项等高级策略则表现显著较差。这些结果凸显了上下文示例与提示设计对准确提取RDF的重要性，并揭示了不同LLM架构中模型特有的局限性。
URL
https://arxiv.org/abs/2603.29878
