Market · AI in practice
Visa launches AI shopping agent benchmark
Visa launched the Agentic Commerce Benchmark on November 5, a standardized test of AI shopping.
What the sources report
Visa launched the Agentic Commerce Benchmark on November 5, a standardized test of AI shopping. In the first round, 22 models from 12 labs took on 15 challenges, such as choosing a TV, a laptop, or a personal-care appliance. The overall pass rate was 57.2%: the best models passed 82.2% of attempts, and the worst passed 28.9%.
Adriano's reading
What stood out to me was consistency. Each model got three attempts per scenario, and all three passed together only 34.2% of the time. In about 54% of failed runs, the model still recommended a purchase. The test covers discovering the customer’s requirements and making the correct buy or no-buy decision.
Visa plans to extend the evaluation to checkout, payment, and order placement. For teams building payments products and infrastructure, the consistency of these decisions deserves attention before agents gain more autonomy.