Getting an AI to answer from your internal documents: what it delivers, what it demands

How can an AI answer from our documents?

Two steps. First, your documents are indexed: split into passages, sorted by topic. For each question, the system selects the most relevant passages and hands them to the model, which writes from those excerpts — not from its general knowledge.

The consequences: your documents do not train the model, they are consulted at question time. Updating the knowledge means updating the document. Every answer cites where it comes from.

RAG is not an AI that knows things. It is an AI that reads fast and well, within a perimeter you control.

What does it change in practice?

The gain: no more hunting for information that already exists. Which procedure applies to this client? What does the framework agreement say about termination? The answer exists somewhere — a file server, a mailbox, a colleague’s memory.

A document assistant gives that scattered knowledge a single entry point: internal support (HR, IT, quality), pre-sales, onboarding new joiners.

One limit, upfront: the assistant retrieves written knowledge; it creates none. Written nowhere? It must not invent an answer — which is exactly what you want from it.

Why do our documents decide the outcome?

A RAG system answers from what it reads. Three contradictory versions of a procedure? It may cite the wrong one — confidently. Scans a machine cannot read? Invisible to it. No date, no status (draft, approved, obsolete)? It cannot arbitrate.

Hence the real starting point: a documentation clean-up. Identify the references, flag outdated versions, date everything, name an owner per domain. None of it is technical — and it is often what separates a reliable assistant from a confusion generator.

Good news: a sorted, dated base serves the company even without AI. Start with one domain you know you can keep current.

Could the AI expose confidential information?

The risk is real: index everything into one base anyone can query, and the assistant may serve anyone documents meant for a few — salary grids, HR files, confidential projects.

The remedy: at each question, enforce the same access rights as your existing tools. People only get what they may open. Design this in from day one; it retrofits poorly.

Then comes the legal frame. Personal data in the documents? The GDPR applies — France’s CNIL published recommendations on applying the GDPR to AI systems in July 2025. And since 2 August 2026, the EU AI Act requires that people be clearly informed when they interact with an AI system, unless that is obvious from the circumstances — an exception the Commission asks to read narrowly.

How do we know an answer is true?

An unverifiable answer is worthless. Three rules.

Cite sources. Every answer points to the documents it used — name, section, excerpt — one click away. Checking takes seconds.

Say “I don’t know”. When the base holds no answer, the system admits it instead of producing something plausible. That is a setting, not luck — and you test it before go-live.

Stay a guidance tool. For any decision with consequences — legal, financial, safety — the rule is simple: the AI points the way, the document is the authority. The assistant saves the searching, not the reading of the source. Write that down at launch.

Host in-house or in the cloud?

The real question: where are your documents allowed to go?

First route, online AI services: documents and questions transit through a provider’s infrastructure, often outside Europe. The fastest option — provided you examine the contractual guarantees: data location, no training on your data, reversibility.

Second route, open models on infrastructure you control — dedicated server, European cloud, on-premise machines: your documents never leave. More demanding to run, but now often within an SME’s reach for this use case, depending on your resources.

Weigh the sensitivity of the documents, your client commitments, your sector’s rules. For the most sensitive data, France offers a public benchmark: ANSSI’s SecNumCloud qualification, which designates trusted cloud providers. Sovereignty is not a dogma: a parameter, weighed corpus by corpus.

What does it cost day to day?

Two cost blocks — the second is the one people underestimate.

Building: connecting the sources, setting up indexing and access rights, tuning the assistant, testing it on real questions.

Running: every question consumes compute. On a usage-billed service, cost follows volume; on self-hosted infrastructure sized in advance, it is largely fixed. Add hosting and, above all, upkeep: documents change, go stale, new sources appear. Without someone maintaining the index and handling reported errors, the system degrades — and your teams’ trust with it.

Start small: a narrow perimeter, typical questions, a named owner. Watch real usage before extending. That is what separates an adopted tool from a forgotten demo.

When is it not worth it?

Sometimes the honest answer is: don’t.

Tiny corpus? A clear folder structure and a search engine will do.

Unreliable base, and nobody to fix it? AI would amplify the disorder. Clean up first.

You need to find a file, not get a written answer? Classic document management does it for far less effort.

The questions call for expert judgement? The assistant retrieves what is written; it does not arbitrate what is not.

The test: RAG creates value from knowledge that is written, scattered and alive. Not written? Write it first. Not scattered? Find something simpler. No longer current? Update it first.