Skip to content
ENFRDescribe your project

RAG: letting AI answer from your documents without exposing what is sensitive

RAG (retrieval-augmented generation) has AI answer from your documents and cite its sources. How it works, how it differs from fine-tuning, examples, and the access rules that prevent leaks.

By Published on 6 October 2026

RAG, short for retrieval-augmented generation, has AI answer from your own documents. Before answering, the system searches your document base for the passages that deal with the question, then hands them to the language model, which writes its answer from them and says where it comes from. It is the usual way to connect an AI assistant or agent to a company’s knowledge without retraining a model.

The idea was described in 2020 by researchers from Facebook AI Research, University College London and New York University. It is simple; putting it into practice is less so: an answer is only as good as the passages found, and a poorly framed RAG can show someone a document they are not allowed to read.

How does RAG work?

RAG works in two stages.

  • Preparing the documents. They are split into passages of a few paragraphs. Each passage is converted into a series of numbers that represents its meaning (a vector, or embedding), then stored in a vector database with its origin, its date and its access rights. This is redone whenever a document changes.
  • Answering a question. The question is converted the same way. The database finds the passages whose meaning is close, even when they use different words. The model receives the question and those passages, with the instruction to answer from them and cite its sources.

A vector database is therefore an index by meaning rather than by words. Many systems combine both kinds of search, by meaning and by keyword, so they also find exact references: a contract number, a product name, an error code.

RAG or fine-tuning: which to choose?

Fine-tuning (retraining a model on your data) changes the model itself: it teaches it a style, a format, a vocabulary. RAG leaves the model alone: it supplies the relevant information at the time of the question. For company knowledge, the difference matters:

  • Information that changes. With RAG, you update the document and the answer follows, with no new training.
  • Answers you can check. RAG cites the passages it used; a retrained model cannot tell which document taught it what.
  • Access rights that hold. RAG can filter passages according to who is asking; whatever a model has learned is open to everyone who queries it.

Fine-tuning still has its place for a very specific output format or business vocabulary, and the two can be combined. In its November 2024 survey of 600 IT decision-makers at US companies, the investment firm Menlo Ventures found 51% adoption of RAG among the ways of designing a generative AI application, up from 31% a year earlier, and 9% of models in production fine-tuned.

Examples of RAG in business

  • An internal assistant: it answers staff questions from procedures, guides and documentation, and points to the original page.
  • Support: from the knowledge base and past tickets, an agent drafts a reply to a similar problem, which a person approves before it is sent.
  • Contracts and tenders: finding a clause, comparing terms, drafting a reply from the ones already written.
  • AI agents: RAG is often the tool through which an agent consults your documents before it acts. Our article on agentic AI explains what an agent does with that information.

Why a RAG system gives bad answers

When a RAG system gets it wrong, the cause usually lies before the model:

  • outdated or conflicting documents: two versions of a procedure, and the AI quotes the old one;
  • poor splitting: a table cut in half, an exception separated from its rule;
  • passages without context: with no date, author or scope, there is no way to prefer the right one;
  • a question the base does not cover: with no useful passage, the model fills in from its general memory unless told otherwise.

RAG reduces errors without removing them. In May 2024, Stanford researchers tested three RAG-based legal research tools on more than 200 questions: depending on the tool, 17% to 33% of answers contained false or misleading information, against 43% for GPT-4 queried without document search.

Hence three habits: sources kept up to date, each with an owner; the instruction to answer “I don’t know” when nothing is found; regular tests on a set of questions whose answers are known. This is the work of our offer Data ready for your AI agents.

The security risks of RAG

RAG opens your documents to an AI. Its risks are now well described, notably by OWASP, the foundation known for its lists of application security risks, in the 2026 edition of its Top 10 for large language model applications, published in August 2026.

  • Showing people what they are not allowed to see. If the database mixes documents from management, human resources and every department, a well-phrased question can bring up a confidential passage. OWASP asks for access rights to be applied at search time, passage by passage, rather than filtering the answer afterwards. In August 2025, ANSSI, the French cybersecurity agency, and its German counterpart BSI recommended checking the user’s role before data reaches the model.
  • Booby-trapped documents. A document can hide instructions that the model will read as orders: this is indirect prompt injection, described by researchers as early as February 2023 and at the top of the OWASP list. An instruction slipped into the database affects every conversation that retrieves it. In 2024, the PoisonedRAG study showed that by adding five malicious texts per targeted question to a database of millions of texts, attackers got the answer of their choice 90% of the time.
  • Vectors are not anonymous. The numbers stored in a vector database are not encryption: in 2023, researchers recovered 92% of short texts (32 tokens) word for word from their vectors. A vector database and its backups must therefore be protected like the original documents.

These risks overlap with those of agents, covered in our article on securing AI agents.

Personal data: what CNIL and ANSSI say

In July 2024, CNIL, the French data protection authority, pointed out that an organisation that connects generative AI to its own document base is responsible for that processing when the base holds personal data, and recommended involving the data protection officer and, where needed, carrying out an impact assessment. For personal data or sensitive documents, for example in a RAG system, it generally advises favouring an on-premises deployment. Our article on sovereign cloud covers hosting options in Europe.

ANSSI’s recommendations for generative AI systems (April 2024) point the same way: list every dataset used, the vector database included; build in the need-to-know principle from the design stage; log every call to the data; and regularly review the rights of AI tools over mailboxes and document spaces.

Setting up RAG, step by step

  • Start from real questions, the ones your teams ask, with the expected answer: they become your test set.
  • Pick a few reliable sources, each with an owner.
  • Carry over the access rights of the original documents and apply them at search time.
  • Require sourced answers, and “I don’t know” when nothing is found.
  • Log questions, retrieved passages and answers; measure quality before opening it to more people.

We do this work with your teams in the offer Data ready for your AI agents (see also Data and BI and Agentic AI under your control). To talk it through with an engineer: the free scoping day.

Frequently asked questions

What does RAG stand for?

Retrieval-augmented generation: the AI first searches your documents, then writes its answer from what it found.

Does RAG remove AI errors?

No. It reduces them when the documents are good and well retrieved, and it lets you check the source. An important answer should be checked against the passage cited.

Do you need a vector database?

Often, not always. For a few short documents, keyword search can be enough. Once the volume grows or questions are phrased freely, search by meaning becomes useful; many common databases can store vectors.

Are our documents used to train the model?

Not through RAG: passages are given to the model at question time, without changing it. What the model provider does with the exchanges depends on its contract: check its data retention and use terms, or use a model hosted in your own environment.

Sources

Facts and figures checked on 4 October 2026.