Price Lab 2.0 Benchmark: Sony WH-1000XM5 & Ninja AF101 (September 2026)

Price Lab 2.0 Benchmark Report

Test date: September 13, 2026

Bing finished 1st overall; AI Price Search 2nd

Bing Shopping finished first. AI Price Search finished second. Here's exactly what we got wrong — and what we're doing about it.

Overall Results

1st: Bing Shopping

2nd: AI Price Search

Products Tested

• Sony WH-1000XM5 Wireless Headphones — Sony WH-1000XM5 black new — Condition: new — Color: black (APS surfaced refurbished/used listings, silver color variants, and $399.99 regular price instead of Sony's $299.99 sale price.)
• Ninja AF101 4-Quart Air Fryer — Ninja AF101 4-quart air fryer new — Condition: new (APS strongly rejected accessories and wrong models/capacities, but missed a Walmart AF101 listing that Bing surfaced.)

Competitors Tested

AI Price Search, Bing Shopping, PriceGrabber

What AI Price Search Got Right

• Correctly identified the Sony WH-1000XM5 as the target product
• Rejected clearly wrong products (earbuds, other headphone models)
• Strong rejection of air fryer liners and accessories on the Ninja AF101 query
• Correctly rejected incorrect Ninja models (AF100, AF161) and wrong capacities (6-quart, 5.5-quart)

What AI Price Search Got Wrong

• Surfaced refurbished/used listings when the query specified 'new' condition (Sony)
• Model code 'WH-1000XM5' not always used to anchor exact-model matching (Sony)
• Surfaced silver variants without flagging the color mismatch when 'black' was requested (Sony)
• Showed $399.99 regular price instead of Sony's active $299.99 sale price — price freshness gap (Sony)
• Missed a Walmart Ninja AF101 listing that Bing Shopping surfaced — retrieval coverage gap (Ninja)

Results

#1 Bing Shopping — Finished first overall. Surfaced the Sony direct $299.99 sale price and the Walmart Ninja AF101 listing that APS missed.

#2 AI Price Search — Finished second overall. Strong accessory/model rejection on Ninja AF101, but condition filtering, color matching, model-code anchoring, price freshness, and retrieval coverage gaps on Sony WH-1000XM5.

Run This Search Yourself

Verify these results on AI Price Search:

• Sony WH-1000XM5 black new
• Ninja AF101 4-quart air fryer new

Overview

On September 13, 2026, we ran a head-to-head benchmark of AI Price Search against Bing Shopping and PriceGrabber using two products: a Sony WH-1000XM5 headphone (black, new condition) and a Ninja AF101 4-quart air fryer (new condition).

Bing Shopping finished first overall. AI Price Search finished second.

This is a Price Lab 2.0 benchmark report — transparent testing of our own engine against competitors. We publish what we got right, what we got wrong, and what we're doing about it. Price Lab exists to identify failures, improve the engine, and retest — not to cherry-pick wins.

Sony WH-1000XM5 Findings

Condition Eligibility

The query specified "new" condition, but AI Price Search surfaced refurbished and used listings. The condition filter did not fully exclude non-new offers, which means a shopper asking for a new product could be directed to a used or refurbished listing.

Brand vs. Model Parsing

The query "Sony WH-1000XM5" was parsed with "Sony" as the brand, but the model code "WH-1000XM5" was not always used to anchor exact-model matching. This led to broader results that included other Sony headphone models instead of locking onto the specific WH-1000XM5.

Color Matching

The query specified "black" but AI Price Search surfaced silver variants without clearly flagging the color mismatch. A shopper requesting black should not see silver listings presented as matching results.

Price Freshness

Sony's direct store showed a $299.99 sale price at the time of testing, but AI Price Search surfaced $399.99 — the regular price. This suggests the price data was not fresh enough to capture the active sale, meaning the shopper would miss a $100 discount.

Ninja AF101 Findings

Accessory and Model Rejection — Strong

AI Price Search did a strong job rejecting liners, accessories, and non-AF101 items that would have polluted the results. When the query asked for a "Ninja AF101 4-quart air fryer," APS correctly excluded:

• Air fryer liners and accessories
• Incorrect models (e.g., Ninja AF100, AF161)
• Wrong capacities (e.g., 6-quart, 5.5-quart variants)

This is a core strength of the identity verification engine — it prevents accessory poisoning and model confusion.

Retrieval Coverage Gap

However, APS missed at least one exact competing AF101 offer — specifically a Walmart listing for the Ninja AF101 4-quart air fryer that Bing Shopping surfaced. This is a coverage gap (a provider or retailer not being reached), not a filtering problem. The listing was correct and should have appeared.

What This Means

AI Price Search did not win this benchmark. Bing Shopping finished first overall, with AI Price Search second. The results expose real weaknesses in our engine:

1. Condition filtering — must enforce "new" when the shopper specifies it
2. Model-code anchoring — must use model codes like "WH-1000XM5" for exact matching
3. Color matching — must flag or filter color mismatches
4. Price freshness — must capture active sale prices, not just regular prices
5. Retrieval coverage — must reach all major retailers, including Walmart

We are documenting these issues for engine improvement. No fixes have been deployed yet. We will retest after improvements are made and publish the results transparently — whether we win or lose.

Retest Status

Pending — issues identified for engine improvement (condition filtering, model-code anchoring, color matching, price freshness, retrieval coverage). No fixes deployed yet. Will retest after improvements and publish results transparently.

Methodology

Data window: September 13, 2026. Analysis based on 2 searches run on aipricesearch.com. Prices are point-in-time snapshots and may have changed since capture. Free to cite with a link to this page.