Abstract
Cross-attention transformers and other multimodal vision-language models excel at grounding and generation; however, their extensive, full-precision backbones make it challenging to deploy them on edge devices. Memory-augmented architectures enhance the utilization of past context; however, most works rarely pair them with aggressive edge-oriented quantization. We introduce BitMar, a quantized multimodal transformer that proposes an external human-like episodic memory for effective image-text generation on hardware with limited resources. BitMar utilizes 1.58-bit encoders, one for text (BitNet-style) and one for vision (DiNOv2-based), to create compact embeddings that are combined and used to query a fixed-size key-value episodic memory. During vector retrieval, the BitNet decoder applies per-layer conditioning, which increases the contextual relevance of generated content. The decoder also employs attention sinks with a sliding-window mechanism to process long or streaming inputs under tight memory budgets. The combination of per-layer conditioning and sliding-window attention achieves a strong quality-speed trade-off, delivering competitive captioning and multimodal understanding at low latency with a small model footprint. These characteristics make BitMar well-suited for edge deployment.
Abstract (translated)
跨模态视觉-语言模型（如带有交叉注意力的变压器）在图像文本生成和理解方面表现出色，但其庞大的全精度骨干网络使得它们难以部署到资源受限的边缘设备上。尽管内存增强架构提高了对过去上下文的利用效率，但大多数研究很少将这些架构与激进的面向边缘设备的量化技术相结合。 我们引入了BitMar——一种采用人类类比的外部分布式记忆系统的量化多模态变压器模型。这种设计使得在资源有限的硬件上有效进行图像-文本生成成为可能。BitMar采用了1.58位编码器，分别用于处理文本（基于BitNet风格）和视觉信息（基于DiNOv2），从而创建紧凑的嵌入式表示，并将其与固定大小的关键值外部记忆结合使用以查询相关信息。 在向量检索过程中，BitNet解码器应用逐层条件调节技术，这能显著提高生成内容的相关性。此外，解码器还利用带有滑动窗口机制的关注力汇节点来处理长输入或流式数据，在严格的内存预算限制下进行高效运算。这种结合了逐层条件调节和滑动窗口注意力的方法在保证高质量输出的同时实现了高效的计算速度。 这些特性使得BitMar非常适合边缘设备部署，能够在低延迟条件下实现竞争性的描述生成与多模态理解，并且模型尺寸较小。
URL
https://arxiv.org/abs/2510.10560
