What Should Your Workplace AI Remember? RAG Permissions and Memory

If a workplace chatbot needs to give different answers to different departments, permissions must shape what it retrieves. Giving it a memory of someone’s preferences or previous conversations creates another boundary. Official company documents and personalized memory have different purposes, lifespans, and access rules.

This is the final post in the enterprise RAG series. Part 1 covered Markdown knowledge bases and tool selection, and Part 2 covered parsing and retrieval quality. Here, we bring together permissions, memory, security, and cost.

Enforce retrieval permissions on the server

Adding department or security-level metadata to a document does not finish the access-control work. The server should verify the signed-in user’s permissions and build a search condition that limits retrieval to the documents they can access. A department value supplied by the user or an instruction in a prompt should not serve as proof of permission.

A metadata filter in vector search is one way to narrow the search scope. Databases may apply filters at different stages, with different effects on search results. Test real combinations of users and documents to check that restricted documents never enter the candidate set and that filtering does not exclude too many relevant documents. Roles configured inside a product do not replace separate access controls at the infrastructure or API-provider level. In hybrid search, enforce the same permission conditions in both vector and lexical search. Do not send restricted documents to an external reranker or generation model and try to hide them only in the final response.

Permissions still matter after retrieval. If you reuse cached results, include the user or permission scope in the cache key. Apply the same rules to reranking, source display, conversation history, and exports. Check how long a document remains in search results or caches after its access is revoked, and define deletion and reindexing procedures. Test cross-department access, including cases where permission boundaries change.

Keep official knowledge separate from personal memory

A policy document contains information managed by the organization; a memory extracted from a conversation may capture a user’s preferences or ongoing context. Putting both kinds of information into one store without clear boundaries can make a system present someone’s personal statement as official guidance or expose one person’s memory to another. First decide what may be stored, who owns each memory, how long it is kept, and how it can be corrected or deleted.

Mem0 provides APIs for adding and retrieving facts to remember from conversations, with scopes such as user, agent, and app. Letta focuses on agent identity and memory blocks that persist across conversations. Zep is a service that provides agents with work context, while Graphiti, released by the Zep team, is a separate open-source project for building a knowledge graph with time-aware information. Distinguish the capabilities and operating scope of managed services from public libraries. These tools do not wrap the same storage design, so compare them based on the memory lifespan and update process you need.

Automatic extraction is convenient, but memories are not always accurate. A preference can be misunderstood and saved, or an outdated fact can remain after it changes. Set operating rules for whether users can inspect and correct their memories, how deletion requests reach indexes and backups, and how to prevent memories from mixing across users or sessions. A user_id groups memories; it does not prove access rights. The server still needs to verify sign-in and tenant membership and enforce the scope for storing and retrieving memories. You can begin by storing only essential memories and compare answers with and without them.

When to move from document links to GraphRAG

Wiki links between Markdown documents make it easier for people to explore relationships. An editor creates those links between documents. GraphRAG is a separate indexing and search approach: it extracts entities and relationships from documents, builds a graph, and can use community summaries when answering questions. Having document links does not provide the same functionality as GraphRAG.

If standard search is enough to find individual policies or product information, a graph may add little value. Try a small sample when you often need to follow relationships spread across documents or answer corpus-wide questions such as “What risk themes recur across these materials?” Building a graph requires additional indexing, model calls, and review; queries that summarize a whole collection can also require more computation. The model size and cost depend on the data and configuration.

Security controls from ingestion through tool use

RAG gives documents to a model, so a document may contain malicious instructions. Treat retrieved content as untrusted input. Telling the model to “ignore the document” is not enough to stop prompt injection. Track document sources and change history, keep retrieved context separate from user instructions, and have server policies validate the allowed scope and parameters of tool calls.

You can use tools such as NeMo Guardrails to apply policy checks to inputs, retrieved context, outputs, and tool use. Checking permissions and sensitive-data exposure before returning a response can help too. A tool such as Presidio provides components for detecting and masking personal information, but it does not replace security controls across the entire data flow. Check where information goes through retrieved documents, model providers, logs, and caches, and regularly test permission revocation and cross-tenant access. RAG does not remove risk; it distributes it across steps from ingestion to the answer.

Compare total cost under the same workload

Self-hosting gives you direct control over data handling paths and the choice of models and storage. It also means operating servers and accelerators, managing power, deployment and monitoring, incident response, and security updates. Commercial APIs have usage-based costs and provider settings and contract terms to consider, while they can take on some infrastructure work. It is not possible to generalize that either approach is always cheaper or always allowed under regulation.

To compare total cost of ownership, use the same time period, query volume, and quality goal to estimate API charges alongside self-hosted infrastructure, staffing, and availability costs. Review legal and contractual data-processing terms based on your organization’s jurisdiction, data types, and provider settings. First identify an approach that can meet your security requirements and be operated by your team, then weigh the cost estimate alongside pilot results.

Across all three posts, the common practice is to check the system against real documents, questions, and permission boundaries. RAG is less about choosing one model and more about designing a workflow that reads documents, finds evidence, delivers it according to permissions, and keeps it up to date in operation.