Abstract
Large language models (LLMs) excel at natural language tasks but remain brittle in domains requiring precise logical and symbolic reasoning. Chaotic dynamical systems provide an especially demanding test because chaos is deterministic yet often misinterpreted as randomness or complexity. We introduce ChaosBench-Logic, a benchmark that evaluates LLM reasoning across 30 diverse dynamical systems using a unified first-order logic (FOL) ontology. Each system is annotated with truth assignments for 11 semantic predicates, and 621 questions are generated across seven reasoning categories, including multi-hop implications, cross-system analogies, counterfactual reasoning, bias probes, and multi-turn dialogues. We define metrics for logical accuracy, implication consistency, dialogue coherence, and contradiction, and we release an open-source evaluation pipeline. Initial experiments show that frontier LLMs such as GPT-4, Claude 3.5 Sonnet, Gemini 2.5 Flash, and the open-source LLaMA-3 70B achieve 91-94% per-item accuracy, yet still score 0% on compositional items and exhibit fragile global coherence. Dialogue-level accuracy ranges from 53.1% (GPT-4 CoT) to 75.5% (LLaMA-3 zero-shot). ChaosBench-Logic provides a rigorous testbed for diagnosing such failures and a foundation for developing neuro-symbolic approaches that improve scientific reasoning in LLMs.
Abstract (translated)
大型语言模型（LLMs）在处理自然语言任务方面表现出色，但在需要精确逻辑和符号推理的领域中仍显得脆弱。混沌动力系统提供了一个特别具有挑战性的测试环境，因为尽管混沌是确定性的，但常常被误认为是随机性或复杂性。我们引入了ChaosBench-Logic，这是一个基准测试工具，它使用统一的一阶逻辑（FOL）本体论，在30个不同的动态系统上评估LLM的推理能力，并涵盖了11种语义谓词的真实赋值以及七个推理类别的621个问题生成。这些推理类别包括多步推断、跨系统的类比推理、反事实推理、偏差探测和多轮对话。 我们定义了逻辑准确性、推论一致性、对话连贯性和矛盾的度量标准，并发布了开源评估管道。初步实验显示，前沿LLM如GPT-4、Claude 3.5 Sonnet、Gemini 2.5 Flash以及开源LLaMA-3 70B在单项准确率上达到了91%-94%，但在组合性项目中得分为0，并且整体连贯性表现脆弱。对话级别的准确性范围从GPT-4 CoT的53.1%到LLaMA-3零样本模型的75.5%。 ChaosBench-Logic为诊断此类失败提供了一个严格的测试平台，同时也为开发能够改善大型语言模型中科学推理的神经符号方法奠定了基础。
URL
https://arxiv.org/abs/2601.01982
