How an Agentic Workflow by DataGOL Helped an Enterprise SaaS Team

An enterprise SaaS company had already shipped the AI feature. Their assistant was live, serving end-users against internal knowledge bases, answering questions and closing support loops. By most definitions the roadmap item was done.

But the team responsible for its quality couldn't see how it was performing.

Was it helpful, or just answering? Which knowledge articles closed loops, and which sent users into dead ends? Why did some questions trigger follow-ups suggesting nothing had been resolved?

The signals existed. They were buried in product logs the VP of User Experience and the product team couldn't read without filing an engineering ticket.

This is the version of AI roadmap slippage that never shows up in a sprint report. The feature shipped. The demo worked. The team was flying blind, and every improvement meant pulling in engineers who had other things to build.

We see this more often than any other pattern in enterprise AI. The problem is rarely "we haven't built it yet." It is "we built it and now we can't manage it." The roadmap stalls at operationalization, not at delivery.

The problem: a live AI feature with no operational layer

Once the assistant was in production, the team hit a wall familiar to anyone who has shipped AI into a real product environment.

What they couldn't see

• Which questions were actually resolved, versus which triggered follow-ups signalling the user still had nothing

• Which knowledge base articles were working, and which led users nowhere

• Where the content gaps were that explained recurring poor responses

• How quality was trending across topic areas, and in which direction

The data was all there. It lived in raw technical logs that required engineering interpretation. The people who owned the customer experience had no direct access to it.

Every quality question meant a ticket. Every ticket meant engineering time. Every delay meant another week of running a product the team suspected wasn't good enough but couldn't prove.

What the first measurement pass found

We instrumented the assistant and traced every question through to its outcome. The shape of the problem was not what anyone expected.

Roughly four in ten questions returned nothing at all: no answer, no sources, an empty state. But when the system did produce an answer, that answer was grounded in real content almost every time, and judged good in the large majority of cases.

Answer quality was never the bottleneck. Answerability was.

The distinction carries real weight on a roadmap. One version of this problem costs a model migration. The other costs retrieval tuning and a content backlog. From the outside they look identical, and you cannot tell them apart without per-conversation measurement.

This connects to what we've written about in the 10 pillars of AI infrastructure. Shipping an AI feature is one milestone. Building the governed, observable layer that lets the business own and improve it is a different one, and most teams discover the gap only after the first milestone is behind them.

What DataGOL built

We didn't rebuild the assistant or touch the knowledge base architecture. We built the operational layer that should have existed from the start: governed, permissioned, and owned by the team accountable for the customer experience.

First, it found the instrumentation was broken

Before the layer could measure quality, it surfaced something more basic. Several signals the team assumed were being captured weren't.

The copy-answer event fired on a fraction of a percent of interactions, which means it was functionally not instrumented. Thumbs ratings appeared on around two percent of messages, and the structured reasons behind negative ratings were absent entirely. More than a third of messages carried no user identity, making per-user adoption analysis unreliable.

None of this was visible while the data sat in logs. It became obvious within days of putting the signals in front of the people who owned the outcome. An operational layer tells you how the AI is performing. It also tells you whether you are in a position to know.

An AI Usefulness Score on every conversation

The core addition was a per-conversation quality signal. An LLM judge evaluates the interaction against three inputs: the user's original question, the assistant's answer, and the user's follow-up question if there was one. The follow-up is the tell. A user who asks a clarifying question straight after a response is signalling that the first answer didn't land.

Running a judge across an entire production corpus is expensive, so the score is built in two layers. The judge scores a labelled set. A calibrated signal derived from answerability, grounding and user behaviour extends coverage to the rest, validated against the judged set, where it agrees on the useful, partial or not-useful classification roughly seven times in eight. Every conversation carries both a score and a confidence tier, so the team can weigh decisions by how much evidence sits behind each one.

The follow-up tell, measured

The follow-up is more than a tell. It is measurable, and the clock proves it.

We measured how long users waited before typing again, and the pattern split cleanly. After an answer that landed, the median pause was around 75 seconds, long enough to read it, use it, and come back. After an answer that missed, the median collapsed to 17 seconds. Roughly four in ten of those follow-ups arrived in under 15 seconds, too fast for anyone to have read the response.

Those are not follow-up questions. They are retries.

The practical consequence is that you can detect whether an answer was consumed without any click tracking, copy tracking or rating. The timing already sits in the logs, and almost nobody reads it. In this case that is why the broken copy instrumentation never blocked the analysis.

Embedded analytics inside the client's own product

We surfaced the results directly inside the client's UI, under their existing permissions model, so the product team could act without engineering involvement:

• Search behaviour: query volume, zero-result rates, and refinement patterns showing where users struggled

• Document access: most-viewed articles, search-to-open conversion, and content reached but not resolving

• Feedback lifecycle: how user feedback on answers was resolved over time

• Topic-cluster trends: which subject areas were improving, which were degrading, and how fast

Content gap reports that turned quality into a roadmap

This is the piece that changed how the team worked, and it produced the most uncomfortable finding in the engagement.

We grouped every failed answer by the question behind it and asked one diagnostic question: had this exact question ever been answered successfully?

Just over half of all failures were questions the system had answered correctly before, sometimes many times over. One routine procedural question came back answered on roughly half its attempts and empty on the rest, with the same wording against the same knowledge base.

This changes what the fix is. Those are not missing articles, because the article exists and is indexed. They are retrieval failures, where ranking, chunking and query handling drop content the system already holds. About a quarter of failures were genuine voids, with nothing in the corpus covering the question. The remainder were out of scope: users probing the assistant itself, asking general-knowledge questions, or typing test strings. No document will ever fix those.

Three buckets, three owners: engineering, the content team, and product. In a raw log file they look identical, and the default response is to commission new articles for problems new articles cannot solve.

Where users actually gave up

The most expensive moment in the product turned out to be the one nobody was watching: what happens immediately after a dead end.

When a user's opening question came back unanswered, fewer than three in ten ever asked anything else. The rest simply left. Of the small group who did persist and rephrase, sixty percent eventually got their answer, which tells you the content was reachable the whole time.

This inverts the priority. Fixing retrieval helps the minority who try again. Fixing the dead-end experience, whether that is a rephrase prompt, a suggested alternative, or anything other than a cheerful empty state, reaches the majority who currently disappear without a trace.

Silent abandonment was the single largest quality loss in the product, and it was invisible in the logs. The thing that defines it is the message that never got sent.

The after: AI quality as a managed product metric

Once quality was measurable weekly, the shape of the problem showed up immediately. The answer rate climbed substantially through the early observation window as coverage improved, then flattened.

The plateau mattered more than the climb. It marked the point where adding content stopped moving the number, which is what told the team the remaining problem was retrieval rather than coverage. Without a weekly trend, a plateau is indistinguishable from progress. You keep commissioning articles and wonder why the metric won't move.

A note on attribution: coverage was being expanded during the same period the layer went in, so the climb reflects several things happening at once. What the operational layer contributed with certainty was the ability to see the plateau and to know what it meant.

The pattern we keep seeing

This situation isn't unusual. Across the enterprise teams we work with, the most common version of a stalled AI roadmap is not a failed pilot. It is a shipped feature the business team can't manage, measure or improve without pulling engineering into every conversation.

What's missing is the operational layer: observability, governance, feedback loops and business-owned analytics that make an AI feature a real product rather than a technical artifact.

DataGOL is built for that gap, the layer between an AI feature and a production AI system the business can manage, improve and trust. It connects to existing systems through pre-built connectors, enforces enterprise-grade security and compliance at the infrastructure layer, and puts the operational controls in the hands of the teams accountable for the outcome.

Key takeaway: if your AI feature is live but your product team still can't answer "is it working?" without filing a ticket, the roadmap hasn't finished. Delivery was step one. The operational layer is step two, and most teams skip it.

We built that layer in weeks. The underlying stack stayed exactly as it was.

Talk to our team about auditing your AI delivery bottlenecks. No pitch, no deck. A focused conversation about where your AI roadmap is actually stuck and what it would take to unblock it.