On this page
Quality assurance should reveal whether customers reached accurate, safe, accessible outcomes and where the service design failed. A rigid script score can reward performance theater while missing incorrect information, unresolved needs, or broken handoffs.
Define quality from the customer outcome
Start with issue resolution, factual accuracy, authorization, expectation setting, empathy without false promises, privacy, accessibility, escalation, documentation, and closure. Separate behavior controlled by the agent or system from policy, staffing, tool, knowledge, and routing defects that require operational owners.
Translate each dimension into evidence a reviewer can point to. “Clear expectations” might mean the agent stated what would happen next, who would act, and when the customer should expect an update. “Sounded confident” is a weaker criterion because confidence can accompany an incorrect promise.
Separate an action from its outcome. A replacement requested is not yet a replacement dispatched. A reviewer should assess whether the agent described the verified state accurately, even when fulfillment sits with another team.
The GOV.UK service standard on defining success emphasizes evidence that a service solves its intended problem. In a QA program, that means the rubric should remain connected to customer tasks rather than rewarding script compliance for its own sake.
Build a review system people can understand
- Published rubric with observable criteria and critical-failure rules
- Representative sampling across channels, times, outcomes, languages, and issue classes
- Reviewer access limits, retention basis, and handling of sensitive recordings
- Calibration examples and recurring agreement sessions
- Coaching process, appeal route, and improvement evidence
- Trend reporting that separates individual, team, policy, and system causes
Publish examples of pass, partial, fail, and not applicable where those categories are useful. Explain whether a missing required control overrides an otherwise high score. The rule should be known before the interaction is reviewed.
Choose the sampling method according to the question. A random or appropriately structured sample can help describe routine quality. A targeted sample of complaints or failed transfers can help investigate a specific defect. Do not present the latter as the average experience of all customers.
Record the population and selection period. If the team reviews only completed calls, abandoned calls and unresolved cases may be absent. If a channel or language is excluded because no qualified reviewer is available, state that coverage gap and arrange an appropriate review route.
Use the right review lens
| Lens | Question | Useful output |
|---|---|---|
| Outcome | Was the user’s job completed correctly? | Resolution and repeat-contact signal |
| Interaction | Was communication clear, respectful, and accessible? | Coaching and design feedback |
| Control | Were identity, consent, privacy, and escalation followed? | Risk correction and incident route |
Consider an illustrative refund interaction. The agent communicates respectfully and follows the visible script, but says the money will arrive tomorrow despite having evidence only that a request was submitted. The interaction lens may show good communication style while the outcome and accuracy lenses reveal a material problem.
The remedy depends on the cause. If the approved template makes the unsupported promise, the template owner must act. If the tool displays an ambiguous status, product or operations work may be needed. If the agent departed from clear instructions, targeted coaching is appropriate.
Keep these findings separate in the record. A single score can hide both the serious defect and the person who can fix it. A useful QA result identifies what happened, the applicable criterion, supporting evidence, and the responsible improvement owner.
Run a calibrated review cycle
- Select a documented sample method.
- Review independently against the rubric.
- Calibrate disagreements with evidence.
- Coach specific observable behavior.
- Route policy and system defects to their owners.
- Re-sample to verify improvement.
Have reviewers assess the same selected conversations independently before discussing them. Compare the evidence behind disagreements, not just the final totals. Zendesk's calibration documentation describes this shared-conversation approach to aligning evaluations.
If two reviewers give different results because the criterion is ambiguous, revise the criterion and its examples. If they apply the same rule to different facts, clarify which evidence is required. These are different calibration problems.
Keep a small set of reference cases that include ordinary success, a critical failure, an uncertain outcome, and a correct escalation. Replace examples when policies or tools change. A reference case tied to an obsolete process can teach consistent but incorrect scoring.
Give the reviewed employee a way to supply missing context or challenge a factual error. An appeal should examine evidence against the published rubric. It should not become an informal negotiation over whether the score feels fair.
Prevent distorted quality scores
- Reviewing only short or successful interactions
- Penalizing agents for required policy constraints
- Using customer sentiment as proof of factual accuracy
- Comparing AI and people without equivalent issue mix
Recording and monitoring require governance
Use qualified advice and current policy for notice, consent, access, retention, labor, and jurisdictional requirements. Collect no more interaction data than the program can protect and justify.
Compare similar work before comparing teams. A group handling complicated complaints may receive different customer feedback and longer conversations than a group answering straightforward status questions. Those differences should inform interpretation rather than disappear into a league table.
For AI-assisted interactions, identify what the agent or system could actually control. A suggested response may be incorrect, the handler may fail to check it, or the knowledge source may be out of date. The improvement could involve several owners.
Avoid using a small review count as a precise ranking. Show how many interactions were assessed and which types they represented. Use individual examples for coaching and suitably supported aggregate evidence for broader operational decisions.
Report distributions and causes
Show critical failures, outcome completion, first-contact resolution, repeat contact, escalation quality, accessibility, reviewer agreement, and recurring defect classes. Avoid hiding serious failures inside a high average. Publish owners and due dates for systemic corrective actions.
A practical report can show counts of critical failures, recurring accuracy defects, unresolved outcomes, and successful escalation, with the sample basis beside each. Include examples stripped of unnecessary personal information so operational owners can understand the issue.
Track corrective actions to completion. “Coach the team” is too broad unless the required behavior is named. “Revise the refund-status message and verify it in the next relevant sample” has an observable finish.
Check for unintended consequences after a change. A stricter closure rule might improve outcome verification while increasing old-ticket backlog. The program needs both the quality evidence and the operating context to decide whether the change is working.
Use sampled quality with operational measures
Review findings become more actionable when paired with demand, backlog age, response, resolution, reopen, and escalation patterns. Use the balanced support-metrics framework to avoid turning one quality score into the whole story.
Use QA as a source of questions as well as answers. If reviewers repeatedly find accurate responses but customers continue to return, inspect whether the service design forces unnecessary contact. The agent may be performing correctly inside a process that remains difficult for customers.
Schedule review of the rubric itself. Remove criteria that no longer distinguish useful behavior and add a criterion only when it represents a meaningful service requirement. The goal is a review program people can apply consistently and use to improve real work.
Continue with the next decision
Define what correct escalation looks like. QA needs observable ownership and closure criteria.
Use prelaunch tests before sampling live calls. Known failure cases should be tested deliberately, not discovered by customers.
Practical finish line
The program uses a transparent outcome rubric, representative samples, calibrated reviewers, proportionate data handling, coachable findings, appeals, and systemic corrective action.
Sources and further reading
Primary and contextual sources used to verify definitions or give readers a relevant next resource.
- Zendesk: Setting Up Calibration in Zendesk QA Calibration compares reviewers' assessments of the same conversations to improve consistency.
- GOV.UK: Define Success and Publish Performance Data Service teams should define success and use performance evidence to improve outcomes.