85 agentic tools audited for security incidents · Talk to the lab →
MotionticBook a call
Glossary

What is query rewriting?

Query rewriting is the step that turns a user's raw question into one or more search queries better suited to the retriever, by rephrasing, expanding, splitting or adding context from the conversation, before retrieval runs.

Checked

People ask questions conversationally: "what about the second one?", "is that still true after the update?". A retriever sees only the words in front of it. Without the earlier turns, those questions match nothing useful.

Query rewriting puts a model between the user and the index. It resolves references, removes filler, splits compound questions and sometimes writes a guess at the answer to search with. In agentic systems it stops being a single step and becomes part of the loop.

Common techniques

  • Condense and rephrase

    Rewrite a follow-up into a standalone query using the conversation. OpenAI's vector store search offers this as rewrite_query, turning "I'd like to know the height of the main office building." into "primary office building height".

  • Decompose

    Split a multi-part question into separate searches and retrieve for each, so one part does not crowd out the other.

  • HyDE

    Have a model write a hypothetical answer document and search with its embedding. Introduced by Luyu Gao, Xueguang Ma, Jimmy Lin and Jamie Callan on 2022-12-20; the encoder is meant to keep the relevance pattern and filter out invented details.

  • Trained rewriters

    Rewrite-Retrieve-Read (Ma et al., 2023-05-23) trains a small rewriter with reinforcement learning from the answering model's feedback, instead of prompting a large model each time.

Query rewriting inside agents

When retrieval is a tool the agent calls, the agent rewrites its own queries as it learns what is in the corpus. Anthropic's write-up of its multi-agent research system (2025-06-13) found agents defaulted to overly long, specific queries that returned few results. Anthropic's fix was to prompt them to start with short, broad queries, narrow down, and refine the next query after each result.

Where query rewriting fails

  • Intent drift

    A rewrite can quietly change the question, for example dropping a negation or a date range, and retrieval then answers a question nobody asked.

  • Invented specifics

    HyDE-style rewrites can add plausible but wrong names or codes. For exact-identifier lookups, the original query should still go to keyword search.

  • Latency and cost

    Each rewrite is an extra model call before retrieval starts. A small, fast model usually does the job.

  • Invisible failures

    If the rewritten query is not logged, retrieval bugs look like model bugs. OpenAI returns the rewritten query in a search_query field for this reason.

How Motiontic's engineers apply it

Motiontic's engineers add query rewriting when the eval set shows conversational follow-ups or compound questions failing retrieval, and log the original and rewritten queries in every trace. The original query is usually kept as one of the searches, so a bad rewrite cannot remove the plain-text match.

Query rewriting techniques compared

Query rewriting techniques compared
Condense and rephraseDecomposeHyDE
ChangesWording, referencesNumber of queriesSearches with a generated answer
Helps withFollow-up questionsMulti-part questionsShort or vague questions
Extra costOne model callOne call plus several searchesOne generation plus embedding
Main riskIntent driftLost connections between partsInvented details

Common questions

What is HyDE in RAG?

Hypothetical Document Embeddings: a model writes a fake answer to the question, and the system searches with that answer's embedding, which often sits closer to real answer passages than the short question does.

Does query rewriting help hybrid search?

Often. A rewrite can produce clean keywords for BM25 and a natural-language query for vectors. Keeping the original query in the mix protects exact terms.

Does OpenAI's file search rewrite queries?

Vector store search accepts rewrite_query=true, which rewrites the query automatically and returns the rewritten version in the search_query field.

What is the difference between query rewriting and agentic search?

Query rewriting is one transformation before retrieval. In agentic search the agent issues, reads and reformulates queries in a loop until it has enough evidence.

Running agents that have to hold up?

Describe the agent, the systems it touches and what it must never get wrong. The first call is with a lab engineer, who will say plainly whether it is ready for production, and whether Motiontic is the right team.