Abstract
We introduce SymbolicAI, a versatile and modular framework employing a logic-based approach to concept learning and flow management in generative processes. SymbolicAI enables the seamless integration of generative models with a diverse range of solvers by treating large language models (LLMs) as semantic parsers that execute tasks based on both natural and formal language instructions, thus bridging the gap between symbolic reasoning and generative AI. We leverage probabilistic programming principles to tackle complex tasks, and utilize differentiable and classical programming paradigms with their respective strengths. The framework introduces a set of polymorphic, compositional, and self-referential operations for data stream manipulation, aligning LLM outputs with user objectives. As a result, we can transition between the capabilities of various foundation models endowed with zero- and few-shot learning capabilities and specialized, fine-tuned models or solvers proficient in addressing specific problems. In turn, the framework facilitates the creation and evaluation of explainable computational graphs. We conclude by introducing a quality measure and its empirical score for evaluating these computational graphs, and propose a benchmark that compares various state-of-the-art LLMs across a set of complex workflows. We refer to the empirical score as the "Vector Embedding for Relational Trajectory Evaluation through Cross-similarity", or VERTEX score for short. The framework codebase and benchmark are linked below.
Abstract (translated)
我们引入了SymbolicAI，一个采用基于逻辑的方法来学习概念和流管理的通用和模块框架，用于生成过程。SymbolicAI通过将大型语言模型（LLMs）视为语义解析器，根据自然和形式化语言指令来执行任务，从而在符号推理和生成人工智能之间搭建起一座桥梁。我们利用概率编程原则来解决复杂任务，并利用具有各自优势的可导和经典编程范式。该框架引入了一组多态、可组合和自参考的操作来处理数据流，将LLM输出与用户目标对齐。因此，我们可以从一个具有零和零散学习能力的各种基础模型之间进行过渡，或者从一个或多个具有特定问题解决能力的专门模型或优化器之间进行转移。该框架还促进了可解释计算图的创建和评估。最后，我们引入了一个评估这些计算图的质量指标及其实验得分，并提出了一个基准，用于在复杂工作流中比较各种最先进的LLM。我们将实验得分称为“通过交叉相似性评估关系轨迹的向量嵌入”，或者简称为VERTEX评分。该框架的代码库和基准如下。
URL
https://arxiv.org/abs/2402.00854
