This article was based on the interview with Cyara CEO Sushil Kumar on finding the right balance of AI in your CX by Greg Kihlström, MarTech futurist and keynote speaker for The Agile Brand with Greg Kihlström podcast. Listen to the original episode here:
The directive from the C-suite is unambiguous: implement AI across the enterprise. For CMOs and enterprise marketing leaders, this is less a suggestion and more a mandate, a clear signal that the future of customer engagement is inextricably linked with intelligent automation. We see the promise everywhere—in the potential for hyper-personalized interactions, unprecedented operational efficiency, and the ability to serve customers at a scale previously unimaginable. The pressure is on not just to adopt AI, but to demonstrate its value, often through metrics like cost reduction and call deflection. It’s a compelling narrative, and one that rightly excites those of us who believe in the power of technology to forge better customer connections.
However, a more nuanced reality is emerging from the front lines of AI implementation. The very sophistication that makes modern agentic AI so powerful also introduces a new class of risks—subtle, significant, and often overlooked in the rush to innovate. As leaders, our role extends beyond green-lighting projects; it requires a deep, clear-eyed understanding of the potential brand damage that can occur when these complex systems inevitably falter. It’s not a matter of if, but when. The most successful deployments won’t be those that simply launch the technology, but those that plan with intellectual honesty for its imperfections. In a recent conversation, Sushil Kumar, CEO of the CX assurance platform Cyara, provided a masterclass in navigating this delicate balance, highlighting the critical mindset shift required to turn a potential liability into a genuine competitive advantage.
The 95% Seduction and the 5% Problem
One of the most alluring aspects of today’s agentic AI is its fluency. It sounds natural, understands sentiment, and can navigate conversations in a way that feels remarkably human. This capability gets us 95% of the way to a seamless experience, a massive leap from the rigid, scripted bots of the past. The danger, however, lies in becoming so captivated by this 95% that we neglect to build a robust strategy for the remaining 5%—the instances where the AI goes off-script, misunderstands intent, or worse, confidently hallucinates incorrect information. As Kumar points out, this isn’t a minor detail; it’s the central strategic challenge leaders must confront.
“The biggest risk that I think, if I have to summarize it, is knowing but not having a plan to deal with the reality that AI is amazing, but it’s not perfect. So it’ll get you 90, 95% there, but what’s your strategy to contain that 5%? And those 5% impacts could be devastating for a business, for your brand, for your trust. So if there is one thing that I always ask my customer executives is, ‘What is your plan for that 5% or 10% when AI doesn’t get it right?’”
This is a profound question that every marketing leader should be asking their teams. It reframes the AI initiative from a simple technology deployment to a comprehensive risk management exercise. The consequences of that 5% are not abstract. They manifest as frustrated customers who hang up after one or two failed attempts, as compliance breaches that can lead to massive fines, or as brand-damaging incidents like the airline bot that famously invented its own refund policy. The customer, quite rightly, doesn’t blame the algorithm; they blame the brand. Planning for that 5% means building in guardrails, establishing clear observability, and creating seamless escalation paths to human agents, ensuring that when the technology inevitably falls short, the customer and the brand are protected.
Rethinking Success: From Pass/Fail to a Multidimensional Rubric
For years, testing customer service technology was a relatively straightforward affair. In the world of scripted IVRs and rule-based chatbots, success could be measured in binary terms: did the user navigate the correct branch? Was the call successfully completed or deflected? It was a deterministic world where a pass/fail grade was sufficient. Agentic AI, by its very nature, shatters this paradigm. Its non-deterministic core—the very thing that gives it agency and conversational power—means we can no longer simply validate a predefined endpoint. A call can be “successfully” completed from a technical standpoint, yet still be a catastrophic failure for the business.
“The biggest lesson is that in this non-deterministic world, you have to rethink your testing and monitoring criteria as a multidimensional rubric where you look into a whole bunch of metrics in terms of, like I said, speech-to-text translation, in terms of latency, in terms of accuracy, but also the compliance angle. Did you comply? If you are in the healthcare sector, did you comply with it? The EU had just come with the EU AI Act, did you comply with the various provisions of that? And you have to demonstrate that.”
This calls for a fundamental shift in how we approach quality assurance. As leaders, we must champion a move away from simple completion rates toward a holistic “multidimensional rubric” of success. This scorecard should include technical performance metrics like latency and intent recognition accuracy, but it must also encompass critical business and brand health indicators. Was the information provided factually correct? Did the interaction remain free of bias? Did it adhere to all regulatory and compliance standards, such as protecting PII? Was the customer’s sentiment positive, neutral, or negative throughout the journey? Evaluating every interaction against this comprehensive scorecard is the only way to gain a true understanding of the AI’s performance and its impact on the customer relationship.
Moving Beyond Defensive Metrics
The pressure to prove ROI often leads organizations to fixate on efficiency metrics like call deflection and containment rates. While these are important for understanding cost savings, they are dangerously incomplete. A high containment rate might look great on a dashboard, but it tells you nothing about the quality of the interaction or the potential downstream consequences. Kumar’s insights serve as a critical reminder that an obsession with defensive metrics can blind us to significant risks. He illustrates this with a powerful example of a customer calling their health insurance provider.
“Here is a case where if the bot didn’t do that [prevent a PII breach], you will have a successful deflection rate, containment rate, even the customer would be happy, it’s just that you put your company out of compliance with potential bigger risks. So I would say a multi-dimensional rubric that looks against the compliance with the regulations, the policy, the biases, and the fact check… that’s what you have to look at.”
This is where marketing leadership is essential. We must advocate for a more sophisticated view of success that balances efficiency with efficacy, risk mitigation, and brand integrity. This means augmenting containment rates with measures of customer satisfaction (CSAT), sentiment analysis, fact-checking accuracy, and compliance adherence. It also requires creating a continuous feedback loop. Because agentic models are, as Kumar describes them, “living and breathing things,” their behavior can change from one call to the next. The insights gleaned from production—both successes and failures—must be used to continuously refine testing protocols and retrain the models. This closed-loop assurance process transforms quality control from a pre-launch event into an ongoing, dynamic discipline.
The Path Forward
The implementation of agentic AI is not a sprint; it is a secular transformation. As we look ahead, the capabilities of these systems will only grow, moving from routing and simple Q&A to performing complex, active operations like modifying accounts or rebooking multi-leg travel itineraries. The brands that win will not be those that simply adopt the technology the fastest, but those that adopt it the most wisely. This requires a mindset of confident humility—confidence in the transformative power of AI, coupled with the humility to acknowledge its limitations and plan for them rigorously.
Our role as leaders is to guide our organizations through this complexity. It means asking the tough questions about the “5% problem” before a project ever gets off the ground. It means championing a shift from simplistic, binary testing to a sophisticated, multidimensional evaluation of every customer interaction. And it means elevating the conversation beyond cost-saving metrics to a more holistic view of success that safeguards the brand and builds long-term customer trust. The journey is challenging, but for those of us genuinely excited about creating better customer experiences, it’s a challenge worth embracing. The future isn’t about choosing between humans and AI; it’s about orchestrating them in a way that is not only efficient, but also responsible, reliable, and deeply respectful of the customer.
