Visa Launches First-of-Its-Kind Agentic Commerce Benchmark; AI Models Fail Almost Half of Test Runs
Visa on November 5 in San Francisco launched the Visa Agentic Commerce Benchmark, a first-of-its-kind industry benchmark that puts leading AI models through realistic buying challenges — and lets people take the test, too. Across the first round, 22 AI models from 12 labs took on 15 shopping challenges, achieving an overall pass rate of about 57.2 percent — meaning almost half of the test runs failed.
Shopping is difficult for AI because consumers often do not clearly know what they want at the outset — they may know they want a TV, but not all the required specifications. Under the benchmark, models must identify a consumer's needs through conversation and make an informed purchase decision; if nothing fits the customer's requirements, the AI should opt not to buy, and it must uncover enough information to justify its decision.
The best-performing models passed 82.2 percent of attempts, while the worst passed 28.9 percent. The five models that uncovered the most hidden buying factors averaged a 70.2 percent pass rate, against less than half (44.4 percent) for the five that uncovered the least. The five models with the highest average cost per run passed 67.1 percent, compared with 50.2 percent for the five least expensive. Consistency is an issue: every model got three attempts at each scenario, and all three attempts passed in just 34.2 percent of cases. About 54 percent of failed runs still ended with the AI making a purchase that did not meet the customer's identified requirements, or without uncovering all of them.
"AI agents have the potential to power the next era of commerce, but trust is key to making it work," said Rubail Birwadker, Visa's Global Head of Growth. "Visa Agentic Commerce Benchmark is yet another way Visa is working to boost standards across the industry." Visa's Trust Index, released in September 2026, found that while 72 percent of U.S. consumers had used an AI assistant, only 23 percent trusted AI to handle payment transactions on their behalf.
Visa plans to add other shopping scenarios and product categories to the benchmark and extend testing to checkout, payment and order placement. The benchmark has been shared with leading model providers for feedback, and the public can attempt it on the Visa Payments Frontier page and watch replays of AI attempts.
(Source: Visa)
Source: Visa
Published by GSM · USA (original Chinese edition)
Photo: Visa 总部所在的旧金山 Mission Rock 办公楼群，Sarah Stierch 摄，Wikimedia Commons，CC0
