85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is multimodal RAG?

Multimodal RAG is retrieval-augmented generation over content that is not only text, such as page images, charts, scans, audio or video, where both retrieval and the answering model handle those formats.

Checked

Much of what businesses know lives in formats a text pipeline flattens: scanned contracts, slide decks, engineering drawings, recorded calls, inspection photos. Converting all of it to text loses tables, layout and anything the converter misreads.

Multimodal RAG keeps more of the original. Since 2024 the pieces have matured on both sides: embedding models that place images, audio and video in the same space as text, retrievers that read page images directly, and answering models that read PDFs page by page as images.

Three architectures

  • Convert to text

    OCR, captions and transcripts feed an ordinary text RAG pipeline. Cheapest to store and easiest to search with BM25, but tables, charts and layout are often lost.

  • Shared multimodal embeddings

    One model embeds every format into one space. Google's Gemini Embedding 2 (2026-03-10) maps text, images, video, audio and PDFs into a single space, with limits of 6 images per request, 120 seconds of video and 6 PDF pages per input.

  • Page-image late interaction

    ColPali-style models embed each page image as many patch vectors. Qdrant's 2026-10-13 post counts about 1,700 vectors per page for Perplexity's pplx-embed-v2-late, which is why quantization matters here.

The answering side

Retrieval only finds the page or clip. The model then has to read it. Claude's PDF support, for example, converts each page into an image and supplies the extracted text alongside it. Anthropic's docs put text at roughly 1,500 to 3,000 tokens per page, plus image tokens for each page, with up to 600 pages per request, or 100 when the context window is under 1M tokens.

That cost shapes the design: retrieve precisely, then send only the pages that matter.

Where multimodal RAG fails, and what it is confused with

  • Token cost

    Image pages cost far more to read than text chunks. Sending ten full pages where two paragraphs would do inflates cost and latency.

  • Chunking audio and video

    Clips need boundaries too. Cut in the wrong place and a question about a moment in a call misses it, just as text chunking can split an answer.

  • Citations

    Pointing to a page is easy; pointing to the right cell of a table or second of a video is harder, and reviewers need that precision.

  • Versus a multimodal model

    A model that can read images is not multimodal RAG on its own. The retrieval layer also has to index and find non-text content.

How Motiontic's engineers apply it

Motiontic's engineers start by checking where a text pipeline actually loses answers on the client's documents. Only those document types move to image-based retrieval, and the answering model receives the few pages that retrieval ranks highest, which keeps per-question cost predictable.

Multimodal RAG architectures compared

Multimodal RAG architectures compared
Convert to textShared multimodal embeddingPage-image multi-vector
What is indexedOCR text, captions, transcriptsOne vector per image, clip or pageMany vectors per page
Keeps layout and chartsPoorlyCoarselyIn detail
Keyword searchYesNo, unless text is also indexedNo, unless text is also indexed
StorageLowestLowHighest
ExampleAny text RAG stackGemini Embedding 2ColPali, pplx-embed-v2-late

Common questions

What is multimodal RAG?

Retrieval-augmented generation where the knowledge base includes images, scans, charts, audio or video, and both the retriever and the answering model can work with those formats.

Do PDFs need OCR for RAG?

Not necessarily. Page-image retrievers such as ColPali search the page images directly, and models such as Claude read PDF pages as images plus extracted text.

Which embedding models handle images, audio and video?

Google's Gemini Embedding 2, released in preview on 2026-03-10, maps text, images, video, audio and PDFs into one embedding space. Page-image models such as ColPali focus on documents.

Is multimodal RAG more expensive than text RAG?

Usually. Image pages cost more tokens to read and multi-vector indexes cost more to store, so precise retrieval and quantization matter more than in text-only systems.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.