The rise of AI-powered shopping assistants, marketed as end-to-end “agentic commerce” systems, promises a seamless experience from discovery to checkout. However, a recent multi-platform evaluation reveals a significant gap between conversational fluency and the factual accuracy and completeness required for reliable commerce. The Smarter Sorting study, Product Truth in the Age of Agentic Commerce: A Multi-Platform Evaluation of AI Shopping Systems’ Accuracy, Completeness, and Regulatory Reliability (2025), exposes persistent challenges in delivering “product truth”—a SKU-level representation that is accurate, complete, and regulatory-aligned. For senior marketing and CX leaders, these findings underscore the critical need for robust data governance, infrastructure investment, and accountability frameworks to build truly trustworthy AI commerce solutions.
The Product Truth Deficit: Beyond Conversational Fluency
The study’s core concept, “product truth,” defines a SKU-level representation as factually accurate, complete on key attributes, and aligned with relevant regulatory regimes. This includes canonical identity, ingredient lists, hazard classifications, applicable regulations, merchant and channel information, and geo-specific availability, pricing, and delivery constraints. The evaluation of five leading AI shopping platforms (ChatGPT, Gemini, Perplexity, Claude, and Copilot) across 100 product scenarios and 2,500 interaction steps revealed a concerning reality: only 28.2% of steps achieved full success, with 36.5% partial success and 35.2% outright failure .
This performance deficit is particularly pronounced in critical areas. Attribute completeness scored a mean of 0.83 out of 2, dropping significantly in regulated categories such as batteries and aerosols (0.38 out of 2) compared to lower-regulation controls like apparel (1.18 out of 2) . Regulatory correctness also scored moderately at 0.91 out of 2, indicating that while systems surface some safety-relevant information, critical disclosures are often missed or replaced with vague language.
What this means: While AI shopping systems can provide seemingly helpful and fluent responses for initial product discovery, their reliability for critical transactional details and compliance is inconsistent. This creates a significant risk for enterprises, as customers may place undue trust in systems that generate confident but incorrect information, leading to downstream issues such as customer complaints, returns, or even safety hazards. For instance, an AI agent confidently recommending a cosmetic without accurate allergen information, or a cleaning product missing crucial hazard warnings, directly impacts brand reputation and regulatory adherence.
Systemic Failure Modes and Architectural Weaknesses
The study identifies systemic failure classes stemming not from mere model inaccuracies but from fundamental architectural and data pipeline weaknesses. These include:
- Semantic Drift: Models misidentify SKUs and persist with high confidence, even when presented with contradictory evidence. This is often rooted in embedding-space collapse, where subtle but commercially significant variant differences (e.g., specific shades or formulations) are not adequately distinguished .
- Attribute Hallucination: Systems fabricate plausible-sounding but incorrect product specifications, particularly ingredient lists or technical attributes that do not appear in authoritative sources. This arises from sparse or uneven pre-training exposure to specialized product terminology .
- Regulatory Vague-Fill: Generic or ungrounded safety claims (e.g., “safe when used as directed”) substitute for concrete regulatory flags. This occurs due to the absence of structured regulatory datasets and limited compliance information in pre-training corpora .
- Offer Fabrication: Invented prices, stock levels, or delivery estimates that never appeared in retailer feeds. This is a consequence of relying on outdated or inconsistently cached merchant data, and blending information without clear provenance .
These errors often compound. For example, a “variant confusion” in an initial step can propagate through the entire shopping task, leading to incorrect attributes, unavailable offers, and failed checkout attempts. The most problematic stage was “Availability and Localization,” which exhibited a 52.7% failure rate and only 5.6% success . This highlights that real-time inventory verification, accurate stock status, and reliable pricing remain fundamental weaknesses, often due to a lack of live integration with retailer systems.
Platform performance varies significantly, correlating strongly with investment in structured data integrations and checkout infrastructure. ChatGPT (43.4% success) and Copilot (37.8% success), which benefit from formal merchant partnerships and shopping-specific data pipelines, outperformed systems relying primarily on web crawling or lacking native transaction capabilities . Claude, for example, despite competitive attribute correctness, had near-zero checkout feasibility (0.02 out of 2), demonstrating that accurate product understanding does not automatically translate into commerce capability without the necessary transactional infrastructure .
What this means: Enterprises cannot rely solely on the “intelligence” of LLMs. The foundational data infrastructure—how product information is acquired, structured, and integrated—is paramount. Without robust, real-time connections to validated product catalogs, inventory management systems (IMS), and pricing engines, AI agents will continue to operate with unreliable information, rendering them unsuitable for high-stakes transactions, especially in regulated industries like healthcare or financial services where accuracy and compliance are non-negotiable.
Strategic Imperatives for Product-Truth-First Architectures
To bridge the gap between conversational competence and reliable agentic commerce, senior marketing and CX leaders must advocate for a shift towards “product-truth-first architectures.” This involves treating product truth as a core performance metric, on par with user satisfaction or conversion rates.
What to Do:
- Prioritize Structured Product Data: Invest in building and maintaining regulatory-grade, SKU-centric product intelligence. This includes canonical identity management, detailed attribute schemas (e.g., formulation, safety data sheets, hazard classifications), and clear provenance tracking for all data elements. Implement product information management (PIM) systems with robust validation workflows.
- Establish Real-time Integration: For transactional reliability, ensure AI systems are deeply integrated with enterprise resource planning (ERP), inventory management (IMS), and order management systems (OMS). Availability lookup should be live, not cached, to prevent offer fabrication (e.g., real-time API calls to inventory databases; SLAs for data freshness).
- Implement Strong Governance and Policy:
- Data Quality Thresholds: Define minimum data quality and accuracy thresholds for AI-mediated transactions, especially in safety-critical and regulated categories (e.g., pharmaceutical product attributes, financial service disclosures).
- Consent and Disclosure: Develop clear policies for how AI agents handle sensitive customer data and disclose the limitations of their product information (e.g., “This information is derived from public sources and should be verified on the product label”).
- Auditability: Ensure all AI agent outputs related to product truth are auditable, with clear logs of data sources, timestamps, and confidence scores. This supports tracing errors to their root cause (e.g., CRM ticketing systems flagging hallucinated attributes).
- Develop Robust Error Detection and Feedback Loops:
- Proactive Red-Teaming: Continuously test AI agents against ground-truth data, specifically targeting variant confusion, attribute completeness, and regulatory correctness in high-risk categories.
- Standardized Error Reporting: Implement APIs that allow customer service agents or even end-users to flag product-truth failures back to model providers and data stewards. For example, a “Report Data Error” button on an AI-generated product comparison.
- Post-Transaction Analytics: Integrate AI agent performance with post-purchase metrics such as return rates, complaint rates (e.g., “incorrect item received”), and customer satisfaction scores (CSAT/NPS). Use these signals to identify systemic data gaps or model biases (e.g., a high complaint rate for “variant_confusion” in a specific apparel line).
- Define Operating Models and Roles:
- Data Stewardship Roles: Designate clear ownership for product data quality, including content authors, regulatory compliance teams, and data engineers.
- AI Agent Oversight: Establish roles responsible for monitoring AI agent performance against product-truth metrics, defining guardrails (e.g., “do not confidently assert regulatory claims without verified source”), and escalation paths for critical failures.
- Cross-Functional Collaboration: Foster collaboration between marketing, CX, product, legal, and IT teams to ensure a holistic approach to product truth.
What to Avoid:
- Over-reliance on Web Scraping: Do not treat opportunistically scraped web data as an authoritative source for product truth, especially in regulated categories. Its volatility, personalization bias, and lack of a stable sampling frame make it inherently unreliable.
- Conflating Fluency with Accuracy: Do not evaluate AI shopping systems solely on conversational quality, perceived helpfulness, or click-through rates. Prioritize objective alignment with ground-truth data in high-stakes contexts.
- Ignoring Failure Accumulation: Do not dismiss early-stage errors as minor. Small inaccuracies in identity or attribute extraction will compound, leading to transactional failures and customer dissatisfaction.
- Fragmented Data Architectures: Avoid siloed product data sources or blending data without robust timestamping, source prioritization, and SKU-level identifiers. This creates semantic inconsistency and increases the risk of hallucination and offer fabrication.
- Premature “Agentic” Claims: Do not market AI systems as “shopping agents” or “commerce assistants” if they lack the underlying infrastructure for real-time transactional execution, such as native checkout integrations or verified availability.
Immediate Priorities (First 90 Days):
- Conduct an Internal Product Truth Audit: Identify critical product categories (especially regulated ones) and assess the accuracy, completeness, and regulatory alignment of their digital product data against ground-truth sources.
- Map Data Provenance for AI: Trace the sources of product information fed into existing or planned AI systems (e.g., PIM, ERP, merchant feeds, web scrapes). Identify high-risk data dependencies and areas lacking robust provenance.
- Define Core Product Truth Metrics: Establish measurable KPIs for identity accuracy, attribute completeness, regulatory correctness, and transactional reliability tailored to your enterprise’s product catalog. Set baseline targets (e.g., 95% identity accuracy for top 100 SKUs).
- Pilot Feedback Loop Mechanisms: Implement a simple feedback channel for customer service teams to report product-truth errors identified during AI interactions, logging failure modes (e.g., variant_confusion, missing_hazards) and their impact (e.g., time-to-resolution, FCR).
For enterprises, building trust in AI-mediated commerce hinges on an unwavering commitment to product truth. The current state suggests that significant investment in structured data, real-time integrations, and robust governance is not merely an IT challenge but a strategic imperative for CX and marketing leaders.
Conclusion
AI-powered shopping systems are poised to reshape e-commerce, but their full potential and trustworthiness remain constrained by a fundamental challenge: reliably delivering “product truth.” The evaluation demonstrates that while current systems show capabilities in surfacing product information and supporting basic discovery, they consistently falter on critical dimensions such as variant resolution, attribute completeness in regulated categories, and real-time availability verification. This product truth deficit stems from architectural limitations, fragmented data pipelines, and a lack of robust governance, leading to systemic failures that can erode customer trust and incur regulatory risks.
For senior marketing and CX leaders, the path forward requires a deliberate shift toward product-truth-first architectures. This means prioritizing investments in structured, verified product intelligence, establishing real-time integrations with core enterprise systems, and implementing comprehensive governance frameworks. By focusing on auditable data, clear accountability, and continuous feedback loops, enterprises can move beyond superficial conversational fluency to build truly reliable and trustworthy AI commerce agents. The question is no longer whether AI can assist with shopping, but whether it can be trusted to get the details right when the details matter most. The future of agentic commerce depends on this critical foundation.
Reference
