Visual Text Matters: Improving Text-KVQA with Visual Text Entity Knowledge-aware Large Multimodal Assistant

Abhirama Subramanyam Penamakuri, Anand Mishra
Indian Institute of Technology Jodhpur
Paper arXiv Video Code

Abstract

We revisit knowledge-aware text-based visual question answering, also known as Text-KVQA, in the light of modern advancements in large multimodal models (LMMs).

We make the following contributions: (i) We propose VisTEL — a principled approach to perform visual text entity linking. The proposed VisTEL module harnesses a state-of-the-art visual text recognition engine and the power of a large multimodal model to jointly reason using textual and visual context obtained using surrounding cues in the image to link the visual text entity to the correct knowledge base entity. (ii) We present KaLMA — a knowledge-aware large multimodal assistant that augments an LMM with knowledge associated with visual text entity in the image to arrive at an accurate answer. Further, we provide a comprehensive experimental analysis and comparison of our approach with traditional visual question answering, pre-large multimodal models, and large multimodal models, as well as prior top-performing approaches. Averaging over three splits of Text-KVQA, our proposed approach surpasses the previous best approach by a substantial 23.3% on an absolute scale and establishes a new state of the art.

Model

VisTEL (left in the image below) is an LMM architecture that combines visual text from a recognition engine with visual context from ViT, and leverages an LLM (Llama 2) to link the visual text to its corresponding knowledge base entity. Further, KaLMA (right in the image) is also an LMM architecture (LLaVA) that utilizes the knowledge linked by VisTEL to accurately answer the given question.

The overview of our proposed framework.




BibTeX

@article{penamakuri2024vistel,
  author    = {Abhirama Subramanyam Penamakuri and Anand Mishra},
  title     = {Visual Text Matters: Improving Text-KVQA with Visual Text Entity Knowledge-aware Large Multimodal Assistant},
  journal   = {EMNLP},
  year      = {2024},
}