Price Lab 2.0 Benchmark: Sony WH-1000XM5 & Ninja AF101 (September 2026)

Overview

On September 13, 2026, we ran a head-to-head benchmark of AI Price Search against Bing Shopping and PriceGrabber using two products: a Sony WH-1000XM5 headphone (black, new condition) and a Ninja AF101 4-quart air fryer (new condition).

Bing Shopping finished first overall. AI Price Search finished second.

This is a Price Lab 2.0 benchmark report — transparent testing of our own engine against competitors. We publish what we got right, what we got wrong, and what we're doing about it. Price Lab exists to identify failures, improve the engine, and retest — not to cherry-pick wins.

Sony WH-1000XM5 Findings

Condition Eligibility

The query specified "new" condition, but AI Price Search surfaced refurbished and used listings. The condition filter did not fully exclude non-new offers, which means a shopper asking for a new product could be directed to a used or refurbished listing.

Brand vs. Model Parsing

The query "Sony WH-1000XM5" was parsed with "Sony" as the brand, but the model code "WH-1000XM5" was not always used to anchor exact-model matching. This led to broader results that included other Sony headphone models instead of locking onto the specific WH-1000XM5.

Color Matching

The query specified "black" but AI Price Search surfaced silver variants without clearly flagging the color mismatch. A shopper requesting black should not see silver listings presented as matching results.

Price Freshness

Sony's direct store showed a $299.99 sale price at the time of testing, but AI Price Search surfaced $399.99 — the regular price. This suggests the price data was not fresh enough to capture the active sale, meaning the shopper would miss a $100 discount.

Ninja AF101 Findings

Accessory and Model Rejection — Strong

AI Price Search did a strong job rejecting liners, accessories, and non-AF101 items that would have polluted the results. When the query asked for a "Ninja AF101 4-quart air fryer," APS correctly excluded:

• Air fryer liners and accessories
• Incorrect models (e.g., Ninja AF100, AF161)
• Wrong capacities (e.g., 6-quart, 5.5-quart variants)

This is a core strength of the identity verification engine — it prevents accessory poisoning and model confusion.

Retrieval Coverage Gap

However, APS missed at least one exact competing AF101 offer — specifically a Walmart listing for the Ninja AF101 4-quart air fryer that Bing Shopping surfaced. This is a coverage gap (a provider or retailer not being reached), not a filtering problem. The listing was correct and should have appeared.

What This Means

AI Price Search did not win this benchmark. Bing Shopping finished first overall, with AI Price Search second. The results expose real weaknesses in our engine:

1. Condition filtering — must enforce "new" when the shopper specifies it
2. Model-code anchoring — must use model codes like "WH-1000XM5" for exact matching
3. Color matching — must flag or filter color mismatches
4. Price freshness — must capture active sale prices, not just regular prices
5. Retrieval coverage — must reach all major retailers, including Walmart

We are documenting these issues for engine improvement. No fixes have been deployed yet. We will retest after improvements are made and publish the results transparently — whether we win or lose.