AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding

Authors

Ahmed Masry^1,2, Juan A. Rodriguez^1,3,4, Tianyu Zhang^1,3,5, Suyuchen Wang^1,3,5,
Chao Wang¹, Aarash Feizi^1,3,6, Akshay Kalkunte Suresh¹, Abhay Puri¹,
Xiangru Jian^1,7, Pierre-André Noël¹, Sathwik Tejaswi Madhusudhan¹,
Marco Pedersoli^1,4, Bang Liu^1,5,8, Nicolas Chapados¹, Yoshua Bengio^3,5,8,
Enamul Hoque², Christopher Pal^1,3,8,9, Issam H. Laradji^1,10, David Vazquez¹,
Perouz Taslakian^1,3,6, Spandana Gella¹, Sai Rajeswar^1,3,5

¹ ServiceNow, ² York University, ³ Mila – Quebec AI Institute
⁴ École de Technologie Supérieure, ⁵ Université de Montréal ⁶ McGill University,
⁷ University of Waterloo ⁸ CIFAR AI Chair, ⁹ Polytechnique Montréal ¹⁰ University of British Columbia

Abstract

Aligning visual features with language embeddings is a key challenge in vision-language models (VLMs). The performance of such models hinges on having a good connector that maps visual features generated by a vision encoder to a shared embedding space with the LLM while preserving semantic similarity. Existing connectors, such as multilayer perceptrons (MLPs), lack inductive bias to constrain visual features within the linguistic structure of the LLM's embedding space, making them data-hungry and prone to cross-modal misalignment. In this work, we propose a novel vision-text alignment method, AlignVLM, that maps visual features to a weighted average of LLM text embeddings. Our approach leverages the linguistic priors encoded by the LLM to ensure that visual features are mapped to regions of the space that the LLM can effectively interpret. AlignVLM is particularly effective for document understanding tasks, where visual and textual modalities are highly correlated. Our extensive experiments show that AlignVLM achieves state-of-the-art performance compared to prior alignment methods, with larger gains on document understanding tasks and under low-resource setups. We provide further analysis demonstrating its efficiency and robustness to noise.

Note: This repository contains the implementation of AlignVLM for the low-resource setup described in Section 5.2.2 of the paper. The corresponding experimental results are reported in Table 3.

Setup

1. Clone the repository

git clone https://github.com/ServiceNow/AlignVLM
cd AlignVLM

2. Create a Conda environment and install dependencies

conda create -n alignvlm python=3.10
conda activate alignvlm
pip install --upgrade pip
pip install -e ".[train]"

Training

Training consists of two stages: pretraining and fine-tuning.

Step 1: Prepare alignment connector initialization

Before training the Align connector, you may initialize its weights using LLM LM head. Edit the paths in the following script:

• prepare_align_weights.py

Run the script to generate the initialization weights.

Note: This script is designed for LLaMA 3 models, but can be adapted to other LLMs.
You may skip this step if you plan to train a standard MLP connector instead.

Step 2: Pretraining

Choose one of the following pretraining scripts:

AlignVLM connector:

• pretrain_llava_llama_align_layer_norm_siglip.sh

MLP connector:

• pretrain_llava_llama_mlp_siglip.sh

Edit the required paths and arguments as indicated in the script comments, then run:

bash scripts/train/pretrain_llava_llama_align_layer_norm_siglip.sh

Step 3: Fine-tuning

For fine-tuning, use one of the following scripts:

AlignVLM connector:

• finetune_llava_llama_align_layernorm_siglip.sh

MLP connector:

• pretrain_llava_llama_mlp_siglip.sh

After editing paths and arguments, launch fine-tuning:

bash scripts/train/finetune_llava_llama_align_layernorm_siglip.sh

Evaluation

To evaluate trained models:

1. Install the lmms-eval library.
2. Run one of the evaluation scripts:

AlignVLM:

• eval_align_layernorm_llava.sh

MLP:

• eval_mlp_llava.sh

Example:

bash scripts/eval/eval_align_layernorm_llava.sh

3. (Optional) To evaluate on the MM-Vet benchmark, set your OpenAI API key:

export OPENAI_API_KEY=your_api_key_here

Acknowledgements

We would like to thank the authors of LLaVA-NeXT which helped us in the development of the AlignVLM project.

Citation

If you use our resources, feel free to cite us:

@misc{masry2025alignvlmbridgingvisionlanguage,
      title={AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding}, 
      author={Ahmed Masry and Juan A. Rodriguez and Tianyu Zhang and Suyuchen Wang and Chao Wang and Aarash Feizi and Akshay Kalkunte Suresh and Abhay Puri and Xiangru Jian and Pierre-André Noël and Sathwik Tejaswi Madhusudhan and Marco Pedersoli and Bang Liu and Nicolas Chapados and Yoshua Bengio and Enamul Hoque and Christopher Pal and Issam H. Laradji and David Vazquez and Perouz Taslakian and Spandana Gella and Sai Rajeswar},
      year={2025},
      eprint={2502.01341},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2502.01341}, 
}