Why we can't run a clean A/B test
A proper A/B test shows an offer to one group and withholds it from a control group, then compares who buys more. Our app can't do that yet. Every offer goes to all users at the same time, so there is no control group to measure against.
Instead, we test in sequence. We run a standard offer, record how it performs, and treat that as the baseline. Then we change one thing, run it again, and compare it to the baseline. Periodically we re-run the standard offer to confirm the baseline itself hasn't shifted.
What would shift it? Mostly the Bitcoin price. When Bitcoin runs hot, demand for wallets rises no matter what our offer says. Re-running the standard offer tells us how much of a result we earned and how much the market simply handed us.
How big a discount we can afford
Part of every wallet's price is what it cost us to make it. A discount comes straight out of our profit, so the deeper the discount, the less we keep. Past a certain point, the offer makes us nothing at all.
JADE CORE · sells for $100
| Discount | They pay | We keep |
|---|
| None | $100 | $40 |
| 15% off | $85 | $25 |
| 25% off | $75 | $15 |
| 30% off | $70 | $10 |
| 40% off | $60 | $0 |
JADE PLUS · sells for $150
| Discount | They pay | We keep |
|---|
| None | $150 | $60 |
| 15% off | $127.50 | $37.50 |
| 25% off | $112.50 | $22.50 |
| 30% off | $105 | $15 |
| 40% off | $90 | $0 |
▸
At 40% off we keep nothing, so that's off the table, and we never go deeper than 30% off. We also don't judge an offer by its redemption rate, meaning how many people used the code. We judge it by profit per person reached: for everyone who saw the offer, how much did we actually keep? A deep discount can sell more wallets and still bring in less money.
What we change, and in what order
Each test changes one variable and holds everything else at the best setting found so far. Change two things at once and you can't tell which one moved the result.
1st
How big the discount is
This moves profit the most, and it's the easiest place to lose money. Because the effect is large, we can read it clearly even when few people buy. Test a moderate discount, then a deeper one, and rank them by profit per person reached.
2nd
How the deal is shaped
Spend the same margin, but structure it differently: a percent off, a flat dollar amount, free shipping, or a bundled accessory. How an offer is framed often shifts buying more than a few extra points of discount would. This gets the longest run in the program.
3rd
How the offer looks in the app
A small banner, a full-screen pop-up, or a feature built into the app. The pop-up draws the most attention but also irritates the most users. If it sells well yet drives people to delete the app, it loses. We only test this once the offer structure is locked.
Later
How long it runs and the countdown
Promo length and "ends soon" urgency. The effect is small, and it improves a lot once the app can send different timing to different users. Not worth a slot right now.
The plan, start to finish (about 3 months)
Each offer runs about seven to ten days, followed by a short gap so the pool of ready buyers can refill before the next one. That gives us seven or eight clean comparisons.
Before we start
Get set up
Give each version its own discount code so sales can be told apart. Agree on the standard baseline offer. Get one number from the app team: how many users each offer reaches.
Weeks 1-2
Run the baseline
A standard 15%-off banner. This tests nothing on its own. It sets the line everything else is measured against. Then a few quiet days with no offer to see what normal looks like.
Weeks 3-4
Test a bigger discount
Same banner, but 25% off instead of 15%. Only the discount changed. Did profit per person reached beat the baseline, after adjusting for the Bitcoin price?
Week 5
Run the baseline again
The standard 15% offer once more. This shows whether the baseline drifted. After this, we lock in the best discount size and stop changing it.
Weeks 6-9
Test the deal shape
Same margin spent, structured different ways: dollars off vs percent off, then the winner vs free shipping or a bundled accessory. This gets the longest run because it usually matters most.
Week 10
Run the baseline again
Drift check before we test how it looks. Lock in the winning deal shape.
Weeks 11-13
Test how it looks
The winning offer as a full-screen pop-up, then built into the app, vs the standard banner. Watch closely whether the pop-up drives people away. The output is one recommended setup we keep running.
▸
If too few people buy to separate two offers, widen the gap: test 15% against 30% instead of 25%. Going past 30% is only ever a one-time stress test, treated as a learning cost, not an offer we expect to profit on.
How we decide if an offer worked
Primary
Profit per person reached
Total profit from the sales, divided by the number of people who saw the offer. Highest number wins. This is the measure that decides it.
Also track
Who bought, and which model
Did the discount mostly shift buyers to the cheaper model? That can mean more sales but lower total profit. Hardware is a slow decision, so we wait two weeks before counting results.
Deal-breaker
Did it drive people away
If an offer, especially a pop-up, makes people delete the app, it can lose even when sales look strong. A lost user is worth more than a single sale, so this can overrule a sales win.
Outside factors
Bitcoin price and outside noise
Log the Bitcoin price and any major news while each offer runs. A jump in sales might be Bitcoin, not the offer. The repeated baseline runs are how we separate the two.
What could go wrong, and what we do about it
No control group
We can't fully separate the offer's effect from seasonality or the Bitcoin price. We handle it by re-running the baseline, logging what else was happening, and treating results as directional, not proof.
Running out of ready buyers
The first offer captures the people most ready to buy, so later offers reach a thinner pool. We handle it with equal recovery gaps, and the repeated baseline measures how much the pool thinned.
Training people to wait for deals
Frequent deep discounts teach people to never pay full price. We handle it by keeping the standing baseline modest at 15% and making deep discounts rare and event-like.
Not enough sales to see a difference
Few app users buy hardware, so small differences hide in the noise. We handle it by testing only high-impact variables and keeping the gap between versions wide.
Pop-ups driving people away
An intrusive format can cost more in lost users than it earns in sales. We handle it by letting the "did it drive people away" measure overrule a sales win.
Later, when the app can do more
Everything here is a workaround for one missing capability: the app can't yet show different things to different users. Once it can, three things improve.
One, we can finally hold back a control group that never sees an offer, which removes most of the guesswork around Bitcoin and seasonality. Two, we can test timing properly by sending different urgency windows to different groups. Three, we can show someone an offer a few days after they install, when interest is highest, instead of sending it to everyone at once.
The moment the app supports that, we rebuild this plan around it. What's here is the best we can do until then.
The whole thing in five lines
The problem
Everyone gets the same offer at once, so a clean A/B test with a control group isn't possible.
The fix
Test one change at a time against a standard offer we keep re-running as the baseline.
The limit
Never discount past 30%. Beyond that there's no margin left to give.
The order
Discount size first, then offer structure, then format. Timing is deferred.
The win
Highest profit per person reached. Driving users away can overrule a sales win.