
A marketing AI model can choose an audience, a message or a send time based on prior behavior. If it picks customers who were already likely to order, the campaign report can look excellent even when the intervention made little difference. A decision about the next campaign needs a credible estimate of that difference.
Define who was eligible before the campaign began

A system trained on campaign responders may learn that recent purchasers open email quickly. A useful test compares orders from equally eligible recent purchasers assigned either the email or a holdout. Fix the eligible population and assignment rule before delivery. Keep comparable treatment and holdout groups over the same dates, and count outcomes for everyone assigned, including people who never open the message. Otherwise purchasers who would have returned anyway can be mistaken for a campaign effect. Gordon and colleagues’ Facebook advertising experiments
Decide how a mid-campaign change will be evaluated

Record creative, offer, spend and delivery by version. When possible, reserve a stable comparison while a learning system reallocates other traffic. An attractive dashboard segment found after examining many segments deserves a fresh test. In paid search, eBay's experiments illustrate how customers who already navigate to a retailer can complicate attributed returns. The eBay paid-search field experiments
Use contribution and include costs borne by the seller

Estimate additional orders over the stated window, multiply by contribution per order after variable costs, and subtract campaign-specific costs once. Include unsubscribes and returns if they matter to the decision. Report uncertainty and the follow-up period. A positive early result says little about whether later full-price orders were simply pulled forward.
Try it with the evidence
Read a campaign's profit claim
Source notes. Fictional randomized test. Treatment and control each contain 2,000 customers. Treatment produces 130 orders and control 110. Contribution after discounts and variable fulfillment is $18 per order. The campaign itself costs $500. One order per customer at most.
Prompt. Compute absolute conversion lift and observed incremental contribution after campaign cost. Explain why crediting all 130 orders is misleading.
Check your reading
Conversion is 6.5% versus 5.5%, a one-percentage-point difference. The observed 20 additional orders contribute $360. After $500 in campaign cost the estimate is a $140 loss. The control indicates substantial purchasing without the campaign.
Illustrated field notes
Audience and creative choices
Find the audience definition and the evidence supporting the selected message.






