2026-06-29-QWenVL

Overall Architecture

The core goal of Qwen-VL is to extend a text-only LLM into a vision-language model so that it can understand both text and images.

The overall architecture can be divided into three parts:

Image
  ↓
Visual Encoder / ViT
  ↓
Position-aware Vision-Language Adapter
  ↓
Compressed visual tokens
  ↓
Qwen LLM
  ↓
Text / box / OCR / grounding output

Specifically:

• LLM: initialized from Qwen-7B;
• Visual Encoder: uses a ViT, which divides the image into patches with patch_size=14 and outputs image patch features;
• Visual Adapter: uses cross-attention to compress the long image patch feature sequence into a fixed-length visual token sequence. The final input to the LLM is not the raw image, but a continuous sequence of visual embeddings together with the text prompt.

---

Visual Encoder

The visual encoder in Qwen-VL is a ViT. It divides the image into patches, and each patch produces a feature vector.

For example, if the image resolution is 448×448 and the patch size is 14, then the patch grid is (32 \times 32 = 1024). Therefore, the ViT outputs about 1024 image patch tokens (X_{\text{img}} \in \mathbb{R}^{1024 \times d}).

These patch features contain information about local image regions, such as objects, text, textures, colors, and spatial layout.

However, directly feeding all 1024 visual tokens into the LLM would significantly increase the sequence length and computational cost. Therefore, Qwen-VL introduces a cross-attention vision-language adapter to compress the visual sequence.

---

Learnable Query Embeddings for Cross-Attention

The queries in Qwen-VL’s Visual Adapter are learnable query embeddings. These queries can be understood as a set of learnable information slots. Their role is to actively “read” useful information from the large number of image patch tokens output by the ViT.

Assume the ViT outputs (X_{\text{img}} \in \mathbb{R}^{N \times d}).

The Visual Adapter has 256 learnable queries: (Q_{\text{learned}} \in \mathbb{R}^{256 \times d}).

The cross-attention operation is roughly:

$$ \text{softmax} \left( \frac{QK^\top}{\sqrt{d}} \right)V $$

where:

• the queries come from the learnable embeddings;
• the keys and values come from the ViT image patch features;
• each learnable query attends to all image patches;
• each query outputs one compressed visual token.

In essence, these learnable queries compress the image into a fixed-length visual representation. After training, they may learn to capture different types of information, such as global scene information, object-level information, and text/OCR regions.

---

Output of the Cross-Attention Vision Adapter

The 256 continuous visual embeddings output by the Visual Adapter are inserted into the input sequence of the LLM.

Qwen-VL uses special tokens to mark the beginning and end of the image content:

<img> [visual token 1] [visual token 2] ... [visual token 256] </img>

Then the user text prompt is appended:

embedding"<img>" visual_embeddings embedding"</img>" text_embeddings..

Therefore, what the LLM sees is an interleaved multimodal sequence.

---

Multi-Task Pretraining

The training process of Qwen-VL consists of three stages:

Stage 1: Pretraining
Stage 2: Multi-task Pretraining
Stage 3: Supervised Fine-tuning

---

Stage 1: Image-Text Pretraining

The first stage mainly uses large-scale image-text pairs. In this stage, the LLM is frozen, and only the visual encoder and visual adapter are trained. The image resolution is 224×224. The training objective is standard text token cross-entropy, meaning that the model predicts the corresponding text given an image. The purpose of this stage is to let the visual encoder and adapter learn how to convert image information into visual embeddings that the LLM can understand.

---

Stage 2: Multi-Task Pretraining

The second stage introduces higher-quality and more fine-grained vision-language annotation data, while also increasing the input resolution. The image resolution is increased from 224×224 to 448×448. In this stage, the visual encoder, Visual Adapter, and LLM are all trained.

Qwen-VL jointly trains on seven types of tasks in this stage:

Captioning
VQA
Grounding
Referring Grounding
Grounded Captioning
OCR
Pure-text Autoregression

All of these tasks are unified as autoregressive text generation.

For example, a bounding box does not require an additional detection head. Instead, it is represented as text:

<box>(661,612),(833,812)</box>

The text span referred to by the box is marked with:

<ref>bees</ref>

Therefore, a grounded caption can be written as:

Beautiful shot of <ref>bees</ref><box>(661,612),(833,812)</box>

The key idea behind this design is that captioning, VQA, OCR, and grounding are all converted into the same next-token prediction problem. As a result, the model does not need a separate head for each task.

---

Stage 3: Supervised Fine-Tuning

The third stage is instruction fine-tuning, which produces Qwen-VL-Chat. In this stage, the visual encoder is frozen, and only the Visual Adapter and LLM are trained.

This stage improves the model’s:

instruction following
dialogue ability
multi-image understanding
localization ability
multilingual response ability

During training, multimodal dialogue data is mixed with pure-text dialogue data to prevent the model from losing its general language dialogue ability.

---

Comment

The main contribution of Qwen-VL is that it designs a visual encoder and a visual adapter that allow an LLM to receive both image and text information. Through pretraining and supervised fine-tuning, the model obtains multimodal reasoning ability. Its context length is 8192, and the model does not include additional training or architectural design specifically for extending the context window.