Turning a Ticket Queue into an FAQ Backlog: Group First, Compare Second

Your ticket queue already contains the FAQ you haven’t written. To get it out, group the questions before comparing them against what you have — and count not just how many customers asked, but how long they kept asking.

The backlog was written by the people who will read it.

Someone answers the same question for the fourth time in two weeks, notices, and writes an FAQ article if the week is quiet. That is how most knowledge bases grow. It works, and it has a predictable blind spot: it finds the questions that annoy whoever is on shift, and misses the ones spread thinly enough that nobody noticed — which are exactly the ones a written article serves best.

The information is already there. A question asked twenty times over two months is a specification for an article, written in the customers’ own words. Nobody reads it back, because reading it back by hand means going through two months of tickets while holding every existing article in your head.

The obvious way to automate that is to ask, for each ticket, whether an existing article already covers it. That produces one yes/no answer per ticket and not one of them can be acted on. Nobody writes an article because a single customer asked once. The criterion is frequency — and frequency is invisible from inside one ticket, because no ticket knows how many others said the same thing.

So invert the order. Group first, compare second.

The pipeline

Three stages, in increasing cost order:

Stage 1 is where the volume is, so it runs on the cheap model at low reasoning effort. Stages 2 and 3 are where the judgment is, so they run on the frontier model — and they stay cheap anyway, because there are so few calls.

That is the same shape as a staged alert funnel: the expensive call only runs where something has earned it. The difference is that the cheap stage here isn’t filtering anything out, it is making the rest comparable.

And what a run produces is a backlog, not a report — rows somebody works through, one at a time, in whatever surface they work in.

Stage 1: compress every ticket to one question

Two people asking the same thing write two very different tickets. One pastes a 20 KB log and a stack trace, one writes two lines from a phone, and both are asking why their data stopped showing up an hour ago.

Nothing can be grouped until that difference is gone. Each ticket becomes one sentence with the incidental detail stripped out.

The same call makes a second judgment, and it is the one worth spending time on: could a public article answer this at all?

• “Please export our audit logs for the last quarter” — no. It is a request for somebody to do something.
• “How do I export our audit logs for the last quarter?” — yes.

Those two sit next to each other in the queue and read almost identically. The line between them decides whether the analysis is about documentation or about workload.

Stage 2: group by need, not by wording

Grouping has to see every question at once. No single ticket knows how many others said the same thing, so this is one call over the whole window rather than a similarity score computed pairwise.

Only groups of two or more survive. A question asked once is not yet a gap.

The grouping is by underlying need, not by shared vocabulary. “How far back can I query my metrics?” and “Why has last March dropped off my dashboard?” share almost no words and want the same article; two tickets that both say “getting 429s” may want two. That judgment is where the value of the stage lives, and it is also where the pipeline’s worst outputs come from.

Stage 3: compare each topic against the whole FAQ

Every topic goes in with the entire knowledge base — all of it, in full, in one prompt.

That works only if the corpus fits in a context window, so measure it before you design around it. If it doesn’t fit, this step needs retrieval, and retrieval quality quietly becomes the thing that decides whether any of the analysis is any good.

Three verdicts come back:

• CREATE — nothing covers this, and it is worth writing.
• UPDATE — an article covers the subject but misses the point these customers got stuck on.
• NONE — either it is already answered, or it is not something an article can answer.

CREATE is the verdict you came for, and the one that costs most when it is wrong. It is only as sound as the grouping underneath it, and a bad one ends as a published page nobody needed.

UPDATE is the verdict that earns the pipeline. It finds gaps inside pages that look fine. There is an article about rate limits; it documents the account-wide ceiling accurately and says nothing about the per-method limits underneath it — and per-method is what every one of those customers hit.

NONE is the majority verdict, and that is the healthy shape. Most recurring questions in a mature FAQ are already documented.

Without caching, most of the bill is bytes you already sent

Most of what this pipeline sends is the same bytes over and over, and cached bytes cost a fraction of fresh ones. That is the biggest lever on what a run costs.

Stage 1 puts the same instructions in front of every ticket, thousands of times over — the instructions cache, and only the ticket is billed fresh. Stage 3 puts the entire FAQ in front of every topic, and the corpus dwarfs the topic, so after the first call almost the whole prompt is a cache read. Stage 2 gets nothing — it is a single call over the whole corpus, and there is no second call to read anything back.

Getting it is a matter of order. Caching matches on a prefix, so the reusable part has to come first and be byte-identical every time — instructions then ticket, corpus then topic. Reversed, every call writes an entry that nothing ever reads, and that costs more than not caching at all.

Frequency justifies the article; spread decides whether it stays true

Grouping gives every topic a count. Reading the count as the whole signal is the most expensive mistake available at this stage — the same mistake as reading alert volume without the time axis.

Take two topics from the same run. The first is customers reporting timeouts and gateway errors, all of them writing in on a Tuesday afternoon, none before or since. The second is customers asking which of two similarly-named plan limits applies to them — one or two a week, every week, for the whole window. The first has the larger count by some margin.

The first one is an incident, and nothing in the pipeline can see that. The tickets are real questions, they cluster tightly because everyone described the same error, and no article covers it — so it comes back CREATE, at the top of the list.

But an incident does not want an article. It was already handled where incidents get handled, and an FAQ documents how the product works, not what went wrong on one afternoon.

So give every topic a second number. Call it spread: how much of the analysis window its tickets span. Weight it at least as heavily as the count. A topic that trickles in across the whole window beats one that spiked on a Tuesday, every time. What is being decided is not what happened. It is what stays true in writing.

A recommendation is not an article

The output is a list of topics — a count, a spread, a verdict, and a paragraph of reasoning each — and the temptation is to let the same system write the pages. It will produce something that reads like a finished article: headings, numbered steps, the register your docs are written in. That is the problem, not the appeal.

As described so far, the analysis only ever sees published content. It does not know the internal process, the current path through the dashboard, or which of two workarounds support is recommending this month. Tool access changes that: an agent can go and read the internal wiki. What it does not change is whether what it read is still true.

Asked to write the page anyway, it will fill in the steps. The audit-log question from stage 1 comes back as Settings → Audit log → Export, whether or not that is where the button actually lives. Some of those steps are real and some are invented, and on the page they look exactly alike: same numbered list, same confident tone, nothing marking which is which. Neither the customer nor the reviewer skimming it before publishing can tell them apart. A step the system could not verify should come out as a gap for somebody to fill, not as a line the reader is expected to follow.

Hand over everything except the decision

Make the handoff wide and the authority narrow. Everything that makes the review possible should arrive in one place: the recommendation and its reasoning, the tickets in the cluster, links to the articles it was compared against, and the draft itself — unpublished. What must not arrive is a decision.

Nothing in the system publishes and nothing in it approves — the same line a self-improving triage agent has to hold, where the patches validate themselves and a human still cherry-picks which ones land. The last action is a person pressing publish.

That line stays put, and better context does not move it. Checking a recommendation means reading the cluster it came from — whether those tickets really are one need, and whether the proposed article would have answered them. That check exists only in the tickets, never in the output.

The page is also public: it goes out as your own answer, to everyone who searches that phrase, and stays wrong until somebody notices. The value of the analysis is that it says what to write. The value of the review is that a person decides whether it is true before customers do.

Two paths, one gate

The gate is on the action, not on who takes it. That is what lets the backlog be something you operate rather than a report you read. A person can work it in a web UI — read a recommendation, open the tickets underneath it, edit the draft. An agent can do the same through an API or an MCP server: read the recommendations, open a draft, revise one, discard a candidate that was never a gap.

Both paths are reversible and neither is customer-facing. Change the actor completely and the gate does not move. The boundary was never whether an agent is involved; it is which decision you moved.

The reviewer is the feedback loop

Most rows survive review as they stand. Now and then one doesn’t — a group that should have been two, a candidate that was never a gap — and the reviewer is the only person positioned to notice. What gets caught there is what makes the next run better.

A thumbs-down on a row is not enough to act on, though. The row is the output of three decisions and the mistake is rarely in the last one: a bad CREATE is nearly always a bad group, and a topic that never appeared is nearly always stage 1 discarding tickets it should have kept.

Feedback that names the stage is a change you can make: “these two are the same question, stage 2 split them”. Feedback that says “I wouldn’t write this one” is a complaint you can file.

Capture it where the review happens. The reviewer already has the cluster open and has already decided whether the page is worth writing — that is the only moment the evidence and the judgment sit together. A form somebody is meant to fill in later collects nothing.

What the feedback changes, and when

Where it goes depends on the kind of rule.

• Exact — into the filter, config rather than prompt, where it costs nothing to check and cannot drift. Tickets from this automation account are never real questions.
• Needs judgment — into the prompt. A question about how retention works is answerable; a request to restore one customer’s data is not.

Keep the exact ones out of the prompt, or it just grows — six months of feedback becomes forty ad-hoc clauses, a few of them contradicting each other, and nobody can say which one is doing the work.

None of this needs re-running on the spot. The reviewer has already dealt with this run by hand — the bad candidate discarded, the group merged, the draft edited. What the feedback changes is the next run.

And little is lost if the next run misses something too. A real gap is still being asked about next window, so a topic missed this time comes back with more behind it; a spike that was never a gap does not come back at all. The schedule is the correction, and feedback only makes it arrive sooner.

Where this doesn’t fit

Your knowledge base doesn’t fit in a context window. Stage 3 needs retrieval, and the design problem moves there.

Your ticket volume is low. Frequency stops meaning anything below some threshold — a topic with three members is noise, and the honest version of this tool is one person reading the queue on Friday afternoon.

Your queue is mostly action requests. If customers write in to have things done rather than to ask how things work, stage 1 discards most of the queue and the analysis has little left to group. That is a workload finding, not a documentation one, and it wants a different tool.

Closing

The questions were already there, in the customers’ own words. What was missing was anybody with the time to read a queue end to end.

The pipeline makes them comparable, counts them, and checks them against what is already written:

1. Compress — one sentence per ticket, and a judgment on whether an article could answer it at all.
2. Group — by need, two members minimum, one pass over the whole window.
3. Compare — each topic against the entire FAQ, for one of three verdicts.

Frequency says an article is justified. Spread says whether it will still be true. The verdict column says that most of the time, you have already written it.

Most of what comes back holds up when somebody checks it. What doesn’t, they correct — and the correction goes back into the filter or the prompt, so each run starts from a better place than the last.

The last step never moves. Somebody reads the draft and decides to publish — which is right, because what comes out at the end is a page customers will trust.