How to build an internal RAG system for company knowledge

What is an internal RAG system?

Retrieval-augmented generation (RAG) pairs a search step with a language model. When someone asks a question, the system retrieves the most relevant passages from an index of your content, places them in the prompt, and instructs the model to answer from those passages and cite them. The model supplies the language; your data supplies the facts.

Internal RAG applies this to company knowledge: wikis, policies, contracts, support tickets, runbooks, code and records in databases. It runs either as a knowledge assistant employees query directly, or as retrieval behind a product feature or an agent. Compared with fine-tuning, RAG keeps knowledge current, since updating an answer means updating a document, and it makes every answer traceable to a source.

How should you ingest and chunk company documents?

Most RAG quality problems start in ingestion, not in the model. Each source needs a connector that extracts clean text, keeps structure such as headings and tables, and records metadata: source system, owner, last-modified date, document type and access controls.

• Chunk by structure first. Split on sections and headings, then by length, so a chunk holds one coherent idea rather than half of two.
• Keep context with each chunk. Prepend the document title and section path so a passage still makes sense when it is retrieved on its own.
• Use modest overlap between neighboring chunks so answers that span a boundary are not lost.
• Handle tables, code and tickets separately. A ticket thread or a function is often its own natural chunk.
• Re-index incrementally. Track changes per source and update or delete chunks when documents change, so stale content does not keep answering.

How does retrieval work, and how do you make it accurate?

Each chunk is converted to an embedding, a vector that captures its meaning, and stored in a vector index. At query time the question is embedded the same way and the nearest chunks are returned. Pure vector search misses exact terms such as product codes, error messages and names, so production systems usually run hybrid search that combines vector similarity with keyword scoring like BM25.

Metadata filters narrow results by team, product, region or date before ranking, and a reranking step reorders the top candidates with a more precise model. Retrieve more candidates than you need, rerank, then pass only the few strongest passages to the model. Too much context dilutes the answer as much as too little.

How do you handle permissions in a RAG system?

An internal assistant must never become a way around access controls. If a person cannot open a document in the source system, the assistant must not quote it. That rule has to be enforced in retrieval, because once a passage reaches the prompt the model can repeat it.

• Store access control lists or group memberships as metadata on every chunk during ingestion.
• Resolve the asking user's identity and groups on each request, and filter the search by them before ranking.
• Sync permission changes from the source systems as promptly as content changes.
• Log which chunks were retrieved for whom, so access can be audited later.

Do not rely on the prompt to hide restricted content. Instructions to the model are not a security boundary.

How do you keep answers grounded, cited and measurable?

Grounding starts with the prompt: instruct the model to answer only from the supplied passages, to cite each claim, and to say it does not know when the passages do not contain the answer. Render citations as links to the source document and section so people can verify. A clear refusal is better than a fluent guess.

Then measure it. Build an evaluation set of real questions with the documents that should answer them, and track retrieval metrics such as recall and precision at k separately from answer metrics such as faithfulness to sources and correctness. Run the set on every change to chunking, embeddings, prompts or models. In production, trace each request, collect user feedback, and review low-rated answers to find gaps in content or retrieval. Treat this like any other model lifecycle, with the versioning and monitoring described in MLOps from pilot to production.

How SigmaBit builds internal RAG systems

SigmaBit builds internal RAG systems over a company's own documents, tickets, code and databases as part of our AI automation work. That covers chunking and embeddings, vector search with metadata filters, permission-aware retrieval so people only see what they are allowed to, answers that cite their sources, and evaluation that measures retrieval quality rather than assuming it.

These systems run as internal knowledge assistants or as retrieval behind a product feature, and the same index can sit behind an MCP server so agents and assistants query it as a tool. We run them in production with full MLOps behind them: evaluation, monitoring, versioning and governance.