Building a Knowledge Layer That Does Not Rot

We spent the last few months designing, testing and deploying an AI knowledge layer inside a live client operation. It is a proof of concept and it is still in beta. If it holds up through the gates we set for it, we build it into a proprietary system. Until then we run it on configured commercial software and we report what it does, including what it has not proven.

Two associates, two opposite problems

Our client is a SaaS company going through the growing pains of scaling. The product is deep, highly configurable and heavily integrated with other systems. One customer's configuration can look nothing like another's. Anyone who has worked inside a heavily customized Salesforce org knows the shape of this. The software is the same. What any given customer actually runs is not.

Two ScaleWise associates sit inside their departments, one in technical support and one in technical account management. They carry live queues and live accounts alongside the client's own staff.

When we asked each of them what slowed them down, we got two answers that turned out to be near opposites. That was the first useful finding, and it changed the design.

The support associate: the answer exists, finding it is the job

She diagnoses issues with customer integrations, which means she constantly needs to know how the product is supposed to behave in a specific configuration.

The answer almost always exists. This client employs a team dedicated to maintaining a public knowledge base, and that team has been productive. Hundreds of articles spanning every area of the platform. By the usual measure, a documentation success story.

Volume created a problem that volume cannot solve. Cataloguing that many articles produced a structure that even tenured agents struggle to navigate. A keyword search returns dozens of results with no reliable signal about which one holds the answer. She opens them one at a time and hopes the answer is in the first rather than the last.

Five to thirty minutes per non obvious question. Simple ones land near the low end. Questions that require reconciling several articles, some of them years old, run toward the high end.

We also got one thing wrong before we started. We assumed she needed a record of past solutions to recurring problems. She told us problems rarely repeat, and when they do it is usually a platform bug that leaves her queue entirely. What she needed was a fast, reliable answer to how the system is supposed to work. Two conversations changed the structure of the whole design.

The account manager: it was never written down

His problem was not findability. There was frequently nothing to find.

When a delivery team builds integrations for a customer and hands the account over, he inherits a configuration he did not design. Sometimes nothing was recorded. Sometimes it was recorded in an ad hoc format that takes as long to interpret as it would to ask someone. Sometimes it sits on the local drive of the person who built it. He learns the account by asking around, or by getting something wrong first.

He described a pattern worth sitting with. A delivery team builds four integrations, he helps build a fifth, and a year later only two are still in use. Nobody records that. The gap between what was built and what is actually used is invisible, and it is directly relevant to whether the customer renews. The person having the value conversation does not know what the customer values.

Why the distinction matters

Operations tend to sit in one of two states.

In the first, documentation never kept up with growth. Knowledge lives in the people who have been there longest, finding it costs more than asking a colleague, so everybody asks a colleague. That organization has an authorship problem and needs capture first.

In the second, documentation kept up and findability did not. The corpus outgrew its own organization. Retrieval time rises with corpus size, so the better the documentation effort performs, the worse the retrieval experience becomes. That organization does not need more documentation. It needs a better path into what already exists.

Most companies contain both at once in different departments. Ours did, in two adjacent roles. Working out which state a given team is in, before building anything, was the most consequential decision we made.

The part we decided not to build

The obvious move for the retrieval problem is to copy the client's documentation into a knowledge base and put a conversational interface on top of the copy.

We did not do that, and the reason is mechanical rather than philosophical. A copy starts drifting from the source the moment the source changes. Our client ships platform updates on roughly a monthly cycle. Configuration screens move, options get added, behaviors change. Within weeks the copy is partly wrong, and nothing in it signals which parts went stale. A knowledge base that is incomplete sends people elsewhere. One that is confidently wrong sends people to a customer with bad information.

So instead of copying, we pointed the system at the live documentation and let it read at the moment the question is asked. The client's documentation team already keeps that source current. That is their job, and they do it well. Pointing at their work means their maintenance becomes ours for free, and the single largest maintenance burden in the project stops existing because there is nothing to maintain.

This is not a universal rule. An operation whose procedures change once or twice a year faces a much weaker version of the problem, and mirroring there is defensible. The cost of mirroring scales with how fast the source changes. Measure that rate before you decide.

What the system does that a search box cannot is read all the candidates rather than the first few a search engine happened to rank. The associate gets an answer with the article it came from, so she reads one thing to verify instead of a dozen to find.

What we kept is small: a correction layer recording the specific places where the client's own documentation is wrong, outdated or silent. Those are checked first and override the source. A handful of entries, each one a genuine finding. It also accumulates into something the client does not otherwise have, which is a list of the weak points in their own documentation.

Capture, and why asking people to document things does not work

The account manager's problem could not be solved by better retrieval, because the information did not exist anywhere. Somebody had to write it down.

The usual answer is to ask people to log what they learn. That works for about three weeks. Then a busy week arrives, logging is the first thing dropped, and the record becomes a historical artifact. We have watched this happen enough times to treat it as a design constraint rather than a discipline problem.

The rule we applied is that capture has to sit inside work the associate is already doing.

The account manager already sits in meetings that produce transcripts. He pastes the transcript and the system proposes structured records, field by field, for review. He confirms or corrects. Review is a different order of effort than authoring, and it happens at the end of a meeting rather than in an evening he does not have.

The support associate already documents resolutions to close her work. We shaped that structure around how she thinks about a problem while solving it, rather than handing her a schema she would have to translate into.

We tested this path with a real handoff meeting transcript, and the useful part was where it pushed back. It refused to guess a missing meeting date. It flagged a contact name transcribed two different ways and asked which was right. It left a field blank even though a word in the transcript matched one of the options, correctly noting that a mention in passing is not a statement of fact. And it reported three categories of information our structure had nowhere to put, instead of forcing them into the nearest field. None of those three gaps had been anticipated. All three exist in the structure now.

The rule that keeps both sides honest

The system holds two sources. Live product documentation answers how the product is supposed to work. A structured record answers what is true about a specific customer.

Neither may be used to fill gaps in the other. If one customer runs an hourly sync, that says nothing about the product's defaults. If the documentation describes a default configuration, that does not mean any particular customer uses it.

This sounds obvious and it is the easiest rule in the system to break, because a blended answer reads as more complete and more confident than an honest one. In a product where every customer's configuration is different, it is also the rule that matters most.

What it costs

Two commercial subscriptions per associate. Thirty dollars a month on annual billing. No custom software, no infrastructure, no integration middleware.

That is the license cost, not the total cost. There was an initial design effort to understand how the associates actually work and write the operating rules, measured in days rather than weeks. There is a smaller ongoing effort to record corrections as they surface. Neither is zero, and a document that implied otherwise would be selling rather than reporting.

What we cannot claim

Three rounds of testing ran before deployment, using real work as input, with pass criteria set before anyone saw results. All three passed, including the questions we salted in that the documentation could not answer. That last part mattered more than the rest. A tool that invents one plausible answer, which an associate then repeats to a customer, teaches that associate to verify everything, and at that point the tool has no value at all.

But the time savings are reported by the associates, not measured. We have not run a controlled before and after. The sample is two people who have now developed habits around the system, and habits hide fragility. Whether capture survives months of ordinary pressure is unproven, and that is the failure mode that ends systems like this one.

We call it a beta because that is what it is.

The full document

The complete write-up covers the design decisions, the testing methodology and results, the full cost breakdown, and what transfers to operations outside software.