Takes
Three AI advertising claims, zero holdouts: how to read platform numbers
Google, Meta and Magnite each published an AI advertising figure measured on their own product, and none can be checked from outside. Treat a claim like that as the good case in your forecast, and let a holdout on your own bookings decide whether it holds for you.
AI advertising claims arrive faster than anyone can test them, and three are in circulation this autumn. Google puts AI Max at 7% more conversions or conversion value. Meta puts Advantage+ placements at an average 11.7% improvement in cost per action. Magnite puts its first agentic campaign in Europe, the Middle East and Africa at roughly 70% less setup time. Three vendors, three numbers, and one thing in common: each was measured by the company selling the product, and none published a control group that anyone outside the company can inspect. That does not make them wrong. It makes them unverifiable, which for a budget decision is the same thing.
The three claims, and what each one left out
Google, AI Max, 7% more conversions or conversion value at a similar CPA or ROAS. The figure is Google's internal 2026 data, it excludes retail advertisers, and the comparison is narrower than the headline: the full AI Max feature set against search term matching alone, not AI Max against the setup it replaces (Google). The best-known counter-figure comes from smec (Smarter Ecommerce), a PPC software vendor, which found in November 2025 that opt-in AI Max traffic in more than 250 retail campaigns returned about 35% lower ROAS than the other match types in the same campaigns (PPC Land). The two measure different comparisons in different populations, so neither can be netted against the other. Neither says which population you are in.
Meta, Advantage+ placements, 11.7% lower CPA. Meta's placements page states that ad sets using Advantage+ placements saw an average CPA improvement of 11.7% (Meta). Separately, advertisers began reporting in late August that the manual placement control had disappeared from Ads Manager in some accounts, with no announcement from Meta (PPC Land). The page gives the average and nothing needed to weigh it: how many ad sets went into it, over which months, in which industries. A number with no denominator is a headline, not a measurement.
Magnite, agentic campaign in EMEA, about 70% less setup time. Magnite ran the campaign with Amnet France for an automotive manufacturer it did not name, and reported an approximate 70% reduction in campaign setup time, alongside a view-through rate given as 95 with no unit (GlobeNewswire). Time saved on setup is a real labour saving. What a reader would need to check it is missing from the release: setup time in hours before and after, a comparison campaign, the media budget, and how many campaigns the figure covers. For an advertiser the question is a different one: does inventory chosen by an agent return more than inventory chosen by a person, measured against a holdout? The release does not report that.
Six questions that separate AI advertising claims from findings
- Against what baseline? Manual setup by whom, with what experience, on what date? "Versus manual" is not a baseline; it is a category.
- Measured by whom? A platform measuring its own product has every incentive and every opportunity to choose the flattering window. That is not an accusation; it is a reason to ask.
- On whose conversions? Platform-reported conversions include sales that would have happened anyway. A business's bookings, invoices and calls do not.
- What sample, over what period? Ten accounts over two weeks and ten thousand over a year produce the same headline and opposite confidence.
- In which vertical? Google's own figure excludes retail. Ask what else it excludes.
- Was there a control? The only question that matters if the other five went well. No control, no causation, no matter how large the sample.
Vendors rarely answer more than two of these unprompted. The point of asking is not to catch them out; it is to place the claim correctly in your own forecast. A claim that answers all six is a finding you can plan around. A claim that answers two is the good case in the bad-case forecast, and the good case is never the plan.
How a small business runs the test itself
The platforms' own lift studies are better than nothing, but they are still run by the platform on the platform's conversions. The cheap, platform-neutral alternative is the holdout test: make the change in one comparable slice of the business and withhold it from another, run it for a full buying cycle, and count outcomes from your own records.
For a local service business the slices already exist. Two comparable districts. Two comparable services. A campaign that migrated to AI Max in the first week of September and one that migrated in the last. Four to eight weeks covers a buying cycle for most local services; shorter tests measure noise. Count bookings, quotes or invoices, and put each one in AARRR's one-metric-per-stage structure so "more conversions" cannot borrow success from "more leads." At the end, subtract the withheld slice's result from the exposed slice's. That difference, divided by the exposed slice's result, is the share of your own outcome the change caused. The same difference, divided by the conversions the platform reported for the exposed slice, is the share of the platform's claim that was real for you.
Why this matters more in an automated year
Meta aims to let advertisers fully automate ad creation by the end of 2026, according to a Wall Street Journal report from June 2025, and Mark Zuckerberg described the end point in an interview that month: a business states its objective, connects its bank account, and reads the results (Marketing Dive). Google moved campaign-level broad match and Automatically Created Assets into AI Max in September and moves Dynamic Search Ads in February 2027. As the platforms take over execution, the only decisions left to the advertiser are the offer, the margin arithmetic, and whether to believe the number on the dashboard. The first two are covered by the margin-back budget. The third is covered by refusing to let the seller be the only one holding the tape measure.
None of the three claims is likely to be a lie. Each is probably true of the accounts it was measured on. The question a sceptical owner should keep asking is the one no vendor volunteers: true of mine?
Frequently asked questions
- What is incrementality in advertising?
The share of outcomes that advertising actually caused, measured by comparing an exposed group with a comparable group that did not see the advertising. Platform attribution measures correlation; a holdout measures cause.
- Are Meta's and Google's own lift studies enough?
They are better than attribution alone, but they are designed and measured by the platform on platform-defined conversions. A holdout on the business's own records is the independent check, and for a local business it costs nothing but a few weeks of patience.
- How long should a holdout run?
One full buying cycle for the service, usually four to eight weeks for local services. A local business will rarely have enough bookings per group to pin the lift down to a percentage. It can still rule a claim out: if the two slices finish level over a full cycle, a large promised lift did not happen for you. If the result is too close to read, tighten the question or lengthen the test.
- What should I do with AI advertising claims I cannot verify?
Put the claim in the good-case column of the forecast, sign against the bad case, and run the holdout. If the claim was true for you, the holdout will show it, and you will have the only number that is.
Sources
Necati Atakan Alevli