Expert Mode - Insights from marketing, AI, and CX pros

Expert Mode: Why Most Direct Mail Tests Fail Before Anything Prints with Evan Stewart from Direct One

This article is based on a Q&A with Direct One CEO Evan Stewart to questions and Greg Kihlström for The Agile Brand Guide’s Expert Mode series.

Most of us inherited direct mail. It was in the budget when we got here, and the reason it’s still there usually has less to do with performance than with inertia. Nobody has a strong enough number to cut it. Nobody has a strong enough number to defend it, either.

Evan Stewart, CEO of Direct One, a direct mail production company based in Winter Park, Florida, thinks that stalemate comes from judging a version of mail that doesn’t exist anymore. Then we measure it with tools that undercount it. He didn’t pitch mail as a fit for everyone, and he was blunter about where it fails than most people who sell it would be.

The Mailbox Is the Only Inventory Getting Less Crowded

Stewart’s argument starts narrow. Mail works, he says, when the household is the buying unit and the decision is significant: a mortgage, an insurance switch, a new roof. What makes it work there is that the sender controls delivery in a way no digital channel allows anymore.

You buy the address, you reach the address. In a decade where every digital channel has gotten less unique and predictable, mail got more valuable by standing still. … And the mailbox is the only inventory in advertising that has gotten less crowded. The average household gets a handful of pieces a day and a few hundred emails. Attention is priced by scarcity, and scarcity is moving in mail’s direction while it moves against everything else.

The more useful part of his answer is where he says mail doesn’t belong. He concedes it’s expensive per thousand and always will be, since you’re manufacturing and shipping something. If your average order value is under a hundred dollars, the math probably doesn’t work. And if you’re competing on price in a commodity category, he says you’re buying “an expensive coupon.” That gives us a filter to run before anyone opens a creative brief. Is the purchase considered, and is the household the one making it? Do we already have an addressable file? If any answer is no, mail isn’t the problem to solve this quarter. If they’re all yes and mail isn’t in the plan, ask whether that was a decision or a habit.

Creative Locks Late. The Audience Doesn’t.

Stewart first came to us with a pitch about production speed, tied to a new inkjet press Direct One installed this summer. Pressed on what speed changes mid-campaign, he made a distinction most of us have never been walked through: creative and the audience file lock at different times.

Creative can stay flexible surprisingly late in the process. On a roll-fed inkjet press running variable data, changing a headline, price, image, or QR code can be as simple as swapping the production file before the job goes to press. … If you change the audience after that process is complete, we have to redo that work and regenerate the materials that tell the Postal Service how the mail should be sorted and handled. That can add roughly a day and potentially change the drop date. Once the pieces are printed, sorted and prepared for mailing, you’re essentially at the point of no return.

He’s candid that the press isn’t the story. “I’m not going to sell you press as a strategy,” he said. What changed, in his telling, is that mail got wired into the CRM, and it runs in both directions. A quote request or an abandoned cart moves a record and fires a piece. A QR scan on that piece hands you “a named household that just raised its hand,” which can go straight into an email sequence or to a rep the same day. If your team treats mail as a creative deliverable with a print deadline, you’ll keep running into the audience lock. Treat the approved, suppressed file as the deliverable. Split budget into waves, as Stewart recommends, so the first drop tells you what to change in the second. Then ask about postage. USPS’s 2026 promotions offer 5% off qualifying mailings, plus 1% add-ons for Informed Delivery and sustainability that can stack to 7%. If your provider can’t say which ones you’re using, Stewart says “there’s probably money being left on the table.”

A Response Rate Without a Control Group Is a Vanity Metric

Measurement is where mail usually loses the budget argument, and Stewart thinks it loses for the wrong reason. QR codes and personalized URLs only capture responses that route through the piece. The household that sees the postcard and then searches the brand gets credited to search. Some segments also respond by phone or in person, so scan-only measurement understates mail most in the segments where it performs best.

A bad answer is a response rate with no control group behind it. Or site traffic that lifted during the mail window. Or a scan count. Or a vendor who ran the match-back on their own systems and handed you a deck. The worst one is “you can’t really measure mail, it’s a brand play,” which is the oldest dodge in my industry. Response rate without incremental lift is a vanity metric, and I sell mail for a living.

What he’d defend instead is a randomized holdout: carve 5 to 10% out of the mailable file at the household level, suppress it, then match the mailed and held-out households against your full transaction file over a window agreed before the drop. That’s how you get to incrementality rather than activity. Scans are a creative signal and a follow-up trigger. They aren’t the performance number.

Stewart also named the conflict himself. “I get paid on pieces produced,” he said. “You get paid on return and contribution margin.” They split when the easy move for the vendor is adding fifty thousand names to the file and the right move for the brand is mailing thirty thousand better ones. His answer is the holdout. His test for a vendor review: a mail partner who argues against a control group is “protecting their volume instead of your results.” Put the holdout in the contract. Get the mailed file back with every record and version coded, and run the match-back yourself.

Design the First Test So It Can Actually Answer the Question

Asked for a minimum viable test, Stewart gave actual numbers. Assuming a 1% response rate, he recommends 50,000 to 75,000 pieces with about 10% held out. His estimate is $30,000 to $50,000 all in, with postage as the biggest cost, and two to three weeks from final creative to the mailbox. Much below 25,000, he says, results get too noisy to read. They’re his estimates, not a benchmark, but they’re a usable starting point. He was more pointed about how first tests go wrong.

Somebody mails 10,000 pieces and calls it a day, then has a negative view of mail for the rest of their life. Or a company decides to skip the holdout audience, and nobody on the exec team believes the number that comes back. A test gets measured for two weeks and stopped before the response curve finishes, which happens a lot for purchases with a longtail. … Or the last-click hands the conversions to paid search or the test runs in a quarter where a price change or a competitor launch muddies everything.

Almost none of that is about mail. It’s test design, the kind of thing we’d catch right away in a paid media plan. Stewart’s prescription is to keep it narrow: one audience, one offer, two creative executions at most. Mail your own file first. Lapsed customers read cleaner than cold acquisition because you know what those households did before the piece arrived. Measure a full purchase cycle, sixty to ninety days for a considered purchase. Decide up front how conversions get credited, since last-click attribution will quietly hand the result to search. And do the file work he calls “unglamorous” before anyone debates the headline: standardize the addresses, run change-of-address processing, suppress current customers.

Stewart doesn’t argue that every brand should mail more. His narrower point is that most brands haven’t run a test capable of telling them whether they should. Before the next planning cycle, pull the last mail result your team cited and check whether a holdout sat behind it. If it didn’t, you don’t have a number yet. Get one at a volume that can actually be read, and hold your vendor to the return on the whole program rather than a single drop. Stewart asked to be held to exactly that.

The Agile Brand Guide®
Privacy Overview

This website uses cookies so that we can provide you with the best user experience possible. Cookie information is stored in your browser and performs functions such as recognising you when you return to our website and helping our team to understand which sections of the website you find most interesting and useful.