A simulated 8-week support pilot focused on a practical scoping question: where should AI draft, where should software enforce a hard boundary, and when should the work move to a person? The exercise showed which discovery answers actually changed the design and which early metrics did not survive scrutiny.
How I worked with AI
I made the scope and boundary calls and decided which simulated findings were strong enough to change the design. Claude generated substantial first-draft material early in the exercise, including simulated stakeholder answers and target numbers. Meridian is a supporting exercise focused on the discovery, boundary, and measurement decisions I can defend. I specified the branching logic and rules in the flow below; Claude rendered the visual.
Agent flow
The AI sorts each ticket into a category, looks up the right information, drafts a reply, checks how confident it is, and hands off to a human if the conversation needs it. Money and security categories (refunds, credits, disputes, password resets, account changes, security actions) never reach drafting, regardless of confidence: that’s a hard rule, not a judgment call.
Ticket arrives → human handoff
AI / systemHumanCustomer
1
CustomerTicket arrives
2
AI / systemAI classifies categorysplits into two paths below, based on ticket category
Both lanes start from the same classification step above, and rejoin at step 3 below.
Path A · standard category
Retrieve knowledge (KB / tools)
Generate draft response
Assess confidence: confident → flag for standard review. Low confidence → flag + note why
Path B · high-risk: money / security
Skip drafting entirely. Route straight to a human; the AI only sorts it into a category, never drafts a reply
↳ both paths continue here
3
HumanHuman reviews / drafts response
4
HumanHuman decision → send to customerapprove, edit, or start over; nothing reaches the customer without a human touching it first
5
CustomerCustomer replies, if there’s a follow-upotherwise the ticket ends here, resolved
6
AI / systemEscalation checktriggers on any one of: 5+ exchanges · upset/repeating customer · high-risk category surfaces mid-thread · customer asks to escalate
7
HumanEscalate to human-only handlingshows the most recent AI draft plus the full thread for context; otherwise the ticket loops back to step 3 for the next reply
The 5+ exchange threshold isn’t arbitrary: a simulated 90-day dataset put average replies per ticket at 2.3, with about 12% of tickets running past 4 exchanges. The escalate-on-request signal was added after simulated rep discovery, since it doesn’t require inferring anything.
Discovery questions that changed the design
These questions most directly reshaped the pilot. I included the resolution for each one, not only the original question.
System data capabilityDoes the help desk tool actually distinguish a quick first-touch acknowledgment from a full ticket resolution, or is there a status field we could repurpose for this?
Resolution: the field exists but is inconsistently used, so instead of requiring reps to remember it, I used two hard system timestamps every ticket already has (first-reply-sent, resolved/closed) with no new tracking system or change in how reps work needed.
Did discovery cover rep-level habits?Does Meridian’s internal guidebook, or the product manager’s discovery work, already cover how reps actually work day to day, or just pilot scope and business goals?
Resolution: discovery covered scope and boundary rules, not rep-level instinct, which is better gathered directly than relayed secondhand. That led me to simulate a discovery pass with the 8 reps, which corrected an earlier assumption: only 2 of 8 reps do a first-touch acknowledgment, not the "mixed model" I’d been told, which changed the first-response target below.
Billing access approval scopeDoes giving the AI read-only access to billing/account-status data need the same approval process as the money/security action categories, even though it couldn’t act on that data, just see it?
Resolution: asking why reps look this up manually surfaced that the field already exists via an existing nightly sync, just configured to show on the Tier 2 view, not Tier 1. Turning it on for Tier 1 is a display-config fix, not a new integration or approval process, and solves the question it was asking about.
Draft scope for high-risk categoriesFor higher-risk categories (account access, money), does the AI still draft a response for human review, just flagged for extra scrutiny, or skip drafting entirely and escalate straight to a human?
Resolution: recommended skipping drafting entirely for these categories pending a legal answer, route straight to a human. If legal later clears drafting, these categories need to be added back into the quality target below, which currently skips them.
Knowledge base healthWhat’s the process for updating the knowledge base? About 20% of the roughly 180 articles are unreviewed in over 12 months, with conflicting or outdated guidance, which could realistically block the pilot if not addressed first.
Resolution: leadership held the 8-week timeline regardless, given renewal pressure, so instead of blocking on a cleanup, I built in a mitigation: track any pilot failure caused by stale or conflicting KB content as its own distinct root-cause category. Also seeded a future idea for a KB-maintenance agent, deliberately held back until pilot evidence actually shows it’s worth recommending, not pitched upfront.
Metrics I revised
The Success Measurement Plan separates signals so one strong number can’t hide a failure elsewhere. These are the two targets that changed once I reasoned through what the pilot’s own constraints actually allowed, plus the two I deliberately declined to pin down.
Handling time: hold roughly steady (~12.4 min per ticket), not an aggressive reduction target.
Mandatory human review of every AI draft caps how much time can actually be saved this phase; fact-checking an answer isn’t dramatically faster than answering it in some cases.
Changed my mind: I initially leaned toward proposing a meaningful handling-time reduction alongside the other targets. Working through what human review actually requires made me separate it from first-response time instead of treating both the same way.
First-response time: ~7 hrs blended, down from >8 hrs, revised down from an earlier ~6 hr estimate.
AI drafting a response the moment a ticket arrives removes the compose-from-scratch step before a human opens it, which should modestly speed things up even for tickets resolved in one reply.
Changed my mind again: the original ~6 hr target assumed a meaningful "mixed workflow" where some tickets got a distinct, extra-fast first-touch acknowledgment. Simulated rep discovery showed that’s rare, only 2 of 8 reps do it. I revised the target down to ~7 hrs: still a real improvement, but more modest than I’d first assumed.
Customer satisfaction score: monitor, no hard pilot target.
The link between faster response and customer satisfaction is an assumption, not a proven cause and effect. At this stage the more honest goal is proving AI accuracy and reliability; it’s a signal worth watching, not a metric to be pinned to yet.
Guardrails: zero-tolerance, single-incident trigger for money/security categories, not a rate.
Lower-stakes accuracy issues aren’t guardrail breaches, those are covered separately. Guardrails are reserved for the human-boundary categories, where even one real failure is too costly to wait on a rate threshold for.
Where I landed on this exercise: early on, I often reached for a plausible solution before I had dug far enough into the problem. Discovery for AI and agent workflows needs particular precision because the work often involves defining human judgment, not documenting a process that is already clear. A solo simulation also cannot replace being embedded with a company long enough to notice the questions worth asking. That remains a real limit of this exercise.