What is ColPali?
ColPali is a document retrieval model that embeds images of document pages with a vision language model, producing many vectors per page that are matched to query tokens by late interaction, with no OCR or text extraction step.
Checked
Most RAG pipelines start by turning PDFs into text, and that step quietly loses tables, charts, layout and anything scanned badly. ColPali skips it. It looks at each page as an image and produces embeddings for small patches of that image, so a query about a figure in a chart can match the chart itself.
ColPali was introduced by Manuel Faysse and colleagues in ColPali: Efficient Document Retrieval with Vision Language Models, first posted to arXiv on 2024-06-27. The same paper introduced ViDoRe, a benchmark for retrieving visually rich pages across domains and languages.
How ColPali works
A vision language model reads the page image and outputs one vector per image patch. The query is embedded into one vector per token. At search time, each query token finds its closest patch vector on the page, and those maximum similarities are summed into the page score. This is the late-interaction scoring that ColBERT introduced for text, applied to images.
The authors report that ColPali largely outperforms modern text-extraction pipelines on ViDoRe while being simpler and end-to-end trainable. The training and inference code for the ColVision family, which also includes ColQwen2 and ColSmol variants, is MIT-licensed on GitHub.
The storage cost
One vector per patch means many vectors per page rather than one. Qdrant's 2026-10-13 post on Perplexity's pplx-embed-v2-late model counts about 1,700 vectors of 128 dimensions per page. It reports a 21x storage reduction with 1-bit quantization while keeping 96.5% of float32 retrieval quality without rescoring.
That cost is why multi-vector support became a 2026 database feature. Milvus 3.0 (2026-07-16) added a StructList type for ColBERT and ColPali vectors. Weaviate 1.40 (2026-10-14) added multi-vector search on its disk-based HFresh index through MUVERA, which encodes a set of vectors into one fixed-length vector for candidate search and then scores the finalists exactly with MaxSim.
Where ColPali fails, and what it is confused with
Clean text gains little
For born-digital documents that extract cleanly, a text pipeline with hybrid search may match ColPali at a fraction of the storage.
Retrieval is only half
ColPali returns pages. The model that answers still has to read those pages as images, which costs more tokens than text.
ColPali versus ColBERT
ColBERT applies late interaction to text tokens. ColPali applies it to image patches, so it needs a vision model and handles layout that text never captured.
ColPali versus multimodal embeddings
Models such as Gemini Embedding 2 put a whole image or page into a single vector in a shared space with text. That is cheaper to store but coarser than per-patch matching.
How Motiontic's engineers apply it
Motiontic's engineers reach for page-image retrieval when the eval set shows text extraction losing answers: scanned forms, slide decks, engineering drawings, financial tables. It is tested against a text pipeline on the same questions, and quantization or a reranking-only role is used to keep storage inside budget.
Three ways to retrieve from visually rich documents
| OCR then text RAG | Single-vector multimodal | ColPali-style multi-vector | |
|---|---|---|---|
| What is embedded | Extracted text chunks | Whole image or page | Every image patch |
| Tables and charts | Often lost or garbled | Captured coarsely | Captured in detail |
| Vectors per page | A few chunks | One | Many, one per patch |
| Storage cost | Low | Lowest | Highest, cut by quantization |
| Needs OCR | Yes | No | No |
Common questions
What is ColPali used for?
Finding the right page in large sets of PDFs, scans or slides where tables, charts and layout carry the answer and text extraction loses them.
Does ColPali need OCR?
No. ColPali embeds the page image directly, which is the point of the method. The answering model may still read extracted text alongside the page image.
What is the difference between ColPali and ColBERT?
Both use late interaction, matching each query token to its closest document vector. ColBERT works on text tokens, while ColPali works on image patches from a vision language model.
Which vector databases support ColPali embeddings?
Engines with multi-vector support, including Qdrant, Milvus 3.0 and Weaviate 1.40. Vespa also supports late interaction natively.
Related
Late interaction (ColBERT)
Many vectors per passage, matched token by token: finer retrieval, more storage.
Multimodal RAG
RAG over scans, slides, charts, audio and video, not only extracted text.
Vector quantization
Fewer bits per dimension: smaller, cheaper indexes at a small cost in recall.
Reranker
A second pass that rereads the shortlist and puts the right passages first.
AI product development
New products and major AI features, built from scratch alongside the client's team.
Sources
- Faysse et al., ColPali: Efficient Document Retrieval with Vision Language Models (arXiv, 2024-06-27)
- illuin-tech/colpali on GitHub (ColVision models)
- Qdrant, pplx-embed-v2 late-interaction embeddings (2026-10-13)
- Zilliz, Milvus 3.0 lake-native vector database (2026-07-16)
- Weaviate, 1.40 release (2026-10-14)
- Google, Gemini Embedding 2 (2026-03-10)
Running agents that have to hold up?
Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.