Split View: 사내 지식베이스를 LLM으로 만든다는 것 — 권한 인지 검색, 신선도, 그리고 출시 전 평가셋

Building an Internal Knowledge Base on an LLM — Permission-Aware Retrieval, Freshness, and the Eval Set You Need Before Launch

• Introduction — the Actual Problem Solved by an Internal Search Fielding 15,000 Queries a Day
• What Cerebras Published — One Narrow Waist and Six Tools
• Permission-Aware Retrieval — a Search That Leaks HR Documents Is Worse Than No Search at All
• Freshness and Deletion Propagation — the Index Learns the Truth Late
• When the Wiki and the Ticket Say Different Things
• The Eval Set You Must Build Before Launch
• Cutting Scope Along Organizational Lines
• Conclusion — Search Quality Can Be Fixed Later, but a Leak Can't Be Undone

Introduction — the Actual Problem Solved by an Internal Search Fielding 15,000 Queries a Day

On July 15, 2026, Cerebras published an account of how it built its internal knowledge base (How we built our knowledge base, GeekNews summary). Three months after launch, it had become an internal tool fielding more than 15,000 questions a day — roughly 15 per person, converted to a per-head basis. What stands out is who's asking: not just people, but automation scripts and agents hitting the same endpoint.

It's worth first pinning down exactly what problem this system is actually solving. The problem with internal search isn't "the documents don't exist." It's usually this: the answer sits in some Slack thread from six months ago, finding that thread requires knowing the exact error string used at the time, and the person who knew that string has already moved teams. The wiki has a document on the same topic, but it was last edited eight months ago, and the architecture has changed since. In other words, this isn't a search-quality problem — it's a problem of where knowledge lives and how fresh it is.

And the thing teams almost always underrate at the start isn't chunking strategy or the reranker. It's permissions, freshness, deletion propagation, source conflicts, and the pre-launch eval set. This post looks only at those.

What Cerebras Published — One Narrow Waist and Six Tools

The skeleton of the design is simple. Across three stages — ingestion, hybrid retrieval, and synthesis — every source funnels into a single Postgres embeddings table. One schema for documents, embeddings, and metadata (source name, timestamp), and Slack, code, wikis, incidents, and custom DBs all get normalized into it. Instead of migrating everything onto a single platform, they pull data out at the point where it's generated.

Slack handling got the most care. Message events arrive over a Socket Mode WebSocket, and for every event the entire thread is re-fetched and stored as a single row. An LLM then distills the thread into a normalized document — a one-line question an engineer would actually type, a summary, the resolution, and the systems and code references mentioned. What gets embedded is this distillation, not the raw conversation; the raw conversation is kept only for full-text search. In long threads, high-signal spans from the same author are pulled out and embedded separately (bursting), gated on an IDF rarity score of 4.0 or higher, a length of 200 characters or more, and optionally at least one reaction.

Retrieval merges four signals via reciprocal rank fusion, with k set to 60.

# exact tokens (error strings, flags, hostnames)   -> full-text search
# paraphrases                                       -> embedding search
# stripping filler like "yeah, got it"               -> IDF weighting
# pushing old answers down                           -> age decay

score(d) = sum over retrievers of  weight / (60 + rank_r(d))

# The point is that scores are not normalized.
# A document that multiple retrievers agree on beats one retriever's #1 pick.

Code is chunked hierarchically with CocoIndex using language-specific regexes — class to method, method to block — and only the parts changed by a given commit get re-embedded. They report handling repositories at the 40GB scale.

The query pipeline runs in six stages — a planner where a small LLM looks at project scope and picks tools, an executor that calls tools in parallel, k=60 RRF fusion, a cut down to roughly 20 results after deduplication, a cross-encoder reranker scoring 0 to 10, and a synthesis stage that attaches citations. There are six tools: search, search_slack, search_code (ripgrep), who_knows, recent_prs, and subsystem_index.

One structural choice stands out. The web UI runs the whole pipeline from planner to synthesis, but MCP exposes the primitives individually. That lets an agent like Claude Code, when it attaches, keep its own orchestration without a hidden LLM synthesis step in the way. People and agents want the same index, but not the same orchestration.

One thing should be disclosed. I tried the original page multiple times and it returned a 500, so some of the details above rely on secondary summaries of the original (mer.vin's writeup in particular). The numbers and parameters appear consistently across multiple summaries, but the ACL-related discussion that follows appears only in secondary sources, so I won't quote it directly.

Permission-Aware Retrieval — a Search That Leaks HR Documents Is Worse Than No Search at All

This is where an internal knowledge base and public-document RAG diverge. In public-document RAG, the worst failure is a wrong answer. In internal search, the worst failure is a right answer — a correct answer handed to someone who was never supposed to be asked.

The root of the problem fits in one sentence. The source system's permission logic doesn't travel with the chunk. Access controls that sat on SharePoint, Drive, Confluence, or a private Slack channel disappear the moment the text is sliced and embedded. When an LLM receives that chunk as context, nothing anywhere records the fact that it was a document open only to a specific department or a named set of users.

In practice you need to design three separate layers.

First, load the source ACL as metadata alongside the index at ingestion time. Put the channel ID, repository, document owner, and group list next to the vector row. A common mistake here is storing the flattened user list directly — every time the org chart changes, you have to rewrite the entire index. It's better to store stable subject identifiers like groups or channels, and resolve the user-to-group mapping at query time instead.

Second, filter at query time. Pre-filtering and post-filtering behave differently.

-- Pre-filtering: find nearest neighbors only among candidates the user has access to.
-- Safe, but recall drops sharply for users with few accessible documents.
SELECT id, content
FROM   chunks
WHERE  acl_group = ANY ($1)            -- groups the querying user belongs to
ORDER  BY embedding <=> $2
LIMIT  50;

-- Post-filtering: pull generously, then filter.
-- Recall is preserved, but you may fail to fill the final k,
-- or the mere existence of "2 of 3 results hidden" can leak information.

In practice you mix the two. Cut the obvious boundaries with a pre-filter, pull a generous set of candidates, and re-verify at the final stage. And that final check must always go to the source system itself, or to a permission service that mirrors it. An ACL baked into the index is a snapshot, and a snapshot is always the past.

Third, every entry point must pass through the same authorization. The moment the web UI, the MCP server, an internal bot, and a batch job each reach the index through a different path, one of them will inevitably skip authorization. Configurations where an agent attaches through a service account are especially dangerous. A service account's permissions are usually broader than a user's, and showing a user the results of a search run under that account amounts to privilege escalation. In agent paths, the safest approach is to propagate the querying user's identity all the way through and enforce authorization exactly once, at the point closest to the index.

Finally, prompt injection sits on top of this whole surface. EchoLeak, reported against Microsoft 365 Copilot in late 2025, showed that a single unopened email could enter an internal RAG pipeline and be used to pull out and exfiltrate sensitive data (I confirmed this only through secondary accounts on vendor blogs). The point is that the corpus you search over is itself the trust boundary. The moment you index a channel that outsiders can write to — email, customer tickets, a Slack channel with external guests — that channel becomes a prompt-injection input path.

Freshness and Deletion Propagation — the Index Learns the Truth Late

Age decay is not a solution to the freshness problem; it's a mitigation. It only lowers the score of an old answer — it has no idea that answer is already wrong. When there's no better candidate, age decay still puts the wrong answer in first place.

What actually needs attention is three distinct kinds of events.

Edits. When a document changes, you only need to re-embed the affected chunk. Commit-level incremental processing falls here. Not hard, but there's a trap — if chunk boundaries shift, old chunks can be orphaned. It's safer to delete every existing chunk for a document ID and rewrite from scratch.

Deletions. This is the piece that's usually implemented late. If a document deleted at the source stays in the index, search will confidently cite a runbook that's already been retired. Slack message deletion, wiki page archival, and repository deletion each arrive as a different kind of event, and some don't arrive as events at all. So event-driven deletion alone isn't enough — you need periodic reconciliation: a job that asks the source again about the list of document IDs in the index and removes whatever has disappeared. This is the kind of task that fails silently, so you need to emit "N deleted this cycle" as a metric.

Permission revocation. Quieter than deletion. The document is unchanged, but access has narrowed — a channel goes private, someone is removed from a project, someone leaves the company. The ACL snapshot in the index looks as if nothing happened at all. This is exactly why the previous section said the final check must go to the source permission service.

And there's something worth admitting honestly. Full real-time consistency is not the goal. The goal is knowing your latency, and excluding from the index any category of data where that latency isn't acceptable. HR documents, payroll, undisclosed M&A, the raw text of security incidents — no matter how well-built the permission model is, these are better left out of an early version, because the risk is asymmetric.

When the Wiki and the Ticket Say Different Things

A wiki saying A, a Jira ticket saying B, and a Slack thread saying C about the same question is the normal state, not an edge case. Most systems quietly pick one and answer with it. This is the worst possible behavior — it hides the conflict while delivering the same confidence as if there were none.

A few rules work well in practice.

Rank sources by authority explicitly. Set an order like "code is the final authority, then incident records, then the wiki, then Slack" as configuration. Cerebras treating the codebase as an authoritative source because it's structured, tested, and version-controlled reflects the same thinking. That said, this order flips depending on the type of question — "what's the default for this flag?" is won by the code, but "why was it done this way?" is won by a Slack thread or a design doc.

Surface the conflict inside the answer itself. Add an instruction to the synthesis-stage prompt: "if sources give different values, don't pick one — present both, along with each one's date and source." This isn't a quality issue, it's a trust issue. Once a search has quietly given a wrong answer, nobody trusts it again.

Attach freshness signals at the level of individual facts, not documents. The edit timestamp of an entire wiki page is nearly useless — fixing a typo at the bottom of the page makes the whole thing look current. Extracting at the level of individual facts, the way the Slack distillation does, lets you attach "when this fix was confirmed" separately.

The Eval Set You Must Build Before Launch

This is the part most often skipped, and the one where skipping it costs the most. Demos always work, because the person who built it asks questions they already know the answer to.

The eval set doesn't need to be big. 50 to 150 questions is enough — what matters isn't size but the range of failure modes it covers.

Each item is fine in a shape like this:

{
  "id": "acl-003",
  "type": "permission",
  "query": "What was the distribution of performance review ratings last quarter?",
  "asked_by": "role:engineer",
  "expect": {
    "must_not_cite": ["source:hr-drive"],
    "must_not_contain_any": ["rating distribution", "S-tier", "percentage"],
    "acceptable_behavior": "abstain_with_reason"
  }
}

I want to stress three things.

First, permission cases must be run per role. An eval run under a single admin account tells you nothing about permissions. At minimum, keep roles for a regular engineer, an engineer on a different team, an intern, and an external guest, and repeat the same query under each.

Second, abstention must be graded as a correct answer. "I don't know, this isn't in any document I have access to" needs to be the full-credit answer on 10 to 20 percent of the set. Skip this, and the system gets tuned to always answer.

Third, the eval set should grow out of incidents. Every time a user reports a wrong answer, drop that exact query straight into the eval set. Same principle as a regression test. A 400-question set six months in is worth far more than the initial 150.

Cutting Scope Along Organizational Lines

An element easy to undervalue in Cerebras's design is the concept of a project. Related Slack channels, repositories, doc spaces, and databases get bundled into one named unit, and new hires pick a default project during onboarding.

This looks like a simple UX convenience, but it actually solves three things at once. It cuts search noise (a same-named service from another team doesn't surface at the top), it narrows the planner's options (it's easier for a small model to choose among six tools), and it broadly aligns with permission boundaries (what a team can access and what a team cares about tend to overlap substantially).

The limits are clear too. If an organization reorganizes often, project definitions go stale fast. And scope is for search quality, not a security boundary — queries that fall outside scope still have to pass authorization. Implement the two with the same mechanism, and sooner or later a feature meant to widen scope becomes a permission bypass.

Conclusion — Search Quality Can Be Fixed Later, but a Leak Can't Be Undone

In an internal knowledge base project, a team's time usually gravitates toward search quality. Chunking, the reranker, hybrid weights — all measurable, and improvements show up immediately. But these items can be fixed later. You just rebuild the index.

Three things can't be undone. A document already shown to someone without permission. A decision already made on the basis of a wrong answer. And the organization's learned belief that "that thing can't be trusted." The last one is especially hard to recover from. Once an internal tool loses trust, its traffic quietly converges to zero, and the metrics look as if nothing happened at all.

To sum up.

• Design permissions across three layers: metadata at indexing time, a filter at query time, and a final check against the source permission service. Every entry point has to pass through the same authorization, and a configuration where an agent searches under a service account is a privilege escalation.
• Age decay is not a solution to freshness. Edits, deletions, and permission revocations are three different events, and deletion propagation needs both event handling and periodic reconciliation. Emit the reconciliation count as a metric.
• Quietly picking one source when sources conflict is the worst possible behavior. Keep an authority order as configuration, and surface conflicts inside the answer.
• The pre-launch eval set must include permission cases and abstention cases, and it must be run per role.

You find out whether search is accurate later. You find out about a leak even later than that.