Abstract
The Tsetlin Machine (TM) is a propositional logic based model that uses conjunctive clauses to learn patterns from data. As with typical neural networks, the performance of a Tsetlin Machine is largely dependent on its parameter count, with a larger number of parameters producing higher accuracy but slower execution. Knowledge distillation in neural networks transfers information from an already-trained teacher model to a smaller student model to increase accuracy in the student without increasing execution time. We propose a novel approach to implementing knowledge distillation in Tsetlin Machines by utilizing the probability distributions of each output sample in the teacher to provide additional context to the student. Additionally, we propose a novel clause-transfer algorithm that weighs the importance of each clause in the teacher and initializes the student with only the most essential data. We find that our algorithm can significantly improve performance in the student model without negatively impacting latency in the tested domains of image recognition and text classification.
Abstract (translated)
Tsetlin 机（TM）是一种基于命题逻辑的模型，它使用合取子句从数据中学习模式。与典型的神经网络一样，Tsetlin 机器的性能在很大程度上取决于其参数数量：更多的参数可以提高准确率，但会导致执行速度变慢。知识蒸馏技术通常是在已训练好的教师模型和较小的学生模型之间转移信息，以此增加学生模型的准确性而不延长执行时间。 我们提出了一种新的方法，在 Tsetlin 机器中实施知识蒸馏，通过利用教师模型每个输出样本的概率分布向学生提供额外的信息上下文。此外，我们还提出了一个新颖的子句传输算法，该算法根据其在教师中的重要性对每个子句进行加权，并且只用最关键的数据初始化学生模型。 我们的实验结果表明，在图像识别和文本分类等测试领域中，本算法可以在不增加延迟的情况下显著提高学生模型的表现。
URL
https://arxiv.org/abs/2504.01798
