Why we need this
60% organic
240 of 400 orders had zero paid touchpoints
FB over-claims 2.3×
Reports 63 purchases · Triple Whale attributes 28
Search = 100% brand
Every Google Search conversion is "blockstream jade" or similar
Platform-reported ROAS is unreliable. Facebook claims 63 purchases but Triple Whale can only attribute 28. Google Search converts exclusively on brand terms people would likely search organically. The only way to know what's incremental is to turn things off and measure what happens to total Shopify orders.
Current baseline (Mar 7 – Apr 6)
Every test measures against these numbers. The primary metric is Shopify orders per day — not platform-reported conversions, not ROAS, not clicks.
Shopify (source of truth)
Total orders
502
30 days
Orders per day
16.7
baseline run rate
Total revenue
$103,433
AOV
$201
New customer share
85%
427 of 502
Ad spend breakdown
Total ad spend
$19,576
Facebook Ads
$10,166
52% of spend
Google Ads
$9,410
48% of spend
Triple Whale attribution (last platform click)
Pure organic
240 orders
60% · no paid touch
Google: first + last touch
81 orders
20% · likely incremental
Google: closer only
54 orders
14% · discovered organically first
Facebook only
10 orders
2.5% · on $10K spend
FB + Google multi-touch
8 orders
2% · long organic journeys
Other paid
7 orders
1.5%
Post-purchase survey (full year, 716 responses)
Triple Whale's post-purchase survey asks buyers how they heard about us. 35% response rate across 2,022 survey views. This is the closest thing to ground truth on where demand originates.
Self-reported acquisition channel
YouTube
193 (27%)
$30,992 revenue · $161 AOV
None of the above
127 (18%)
$26,380 revenue · $208 AOV
Referred by a friend
124 (17%)
$20,959 revenue · $169 AOV
TikTok
61 (9%)
$15,401 revenue · $252 AOV
Email
52 (7%)
$16,179 revenue · $311 AOV
X
43 (6%)
$8,616 revenue · $200 AOV
Google
41 (6%)
$12,219 revenue · $298 AOV
YouTube + referrals = 44% of buyers. These are organic channels with zero ad spend. At a 35% response rate, the true number is likely ~550 YouTube-driven and ~350 referral-driven purchases for the year.
Facebook is not listed. Either too few buyers selected it to appear, or it wasn't offered as an option. Either way, customers aren't crediting Facebook with their purchase decision.
Selected free-text responses
| Order | Customer said | TW attributed to |
|---|---|---|
| #BS-35181 | Adam Livingston, a YouTuber | organic_and_social |
| #BS-35242 | Friend | google-ads |
| #BS-35335 | brother | google-ads |
| #BS-35227 | chatgpt | unknown |
| #BS-35237 | Discord | unknown |
| #BS-35141 | Unchained. | organic_and_social |
| #BS-35007 | Bitcoin Conference last year | organic_and_social |
| #BS-35060 | My coussin | Direct |
Note: several buyers who said "friend" or "brother" were attributed to google-ads by TW — they searched the brand name after hearing about it, clicked an ad, and Google claimed the conversion.
Test 1 result — Facebook budget reduction
Concluded Apr 21 after 21 days at the reduced budget (Mar 31 – Apr 19). Facebook ran at a ~$49/day floor the entire window while Shopify orders were tracked against the 16.0/day pre-cut baseline. The first six days looked alarming; the full read does not.
Final read — Apr 21, 2026
| Window | Days | Orders/day | FB spend/day | Δ vs baseline |
|---|---|---|---|---|
| Mar 14–30 · pre-cut baseline | 17 | 16.0 | $344 | — |
| Mar 31–Apr 5 · first six days | 6 | 11.7 | $76 | −27% (confounded) |
| Mar 31–Apr 19 · full post-cut | 20 | 15.4 | ~$49 | −3.8% |
| Apr 13–19 · latest 7d | 7 | 15.86 | $49 | −0.9% |
Sources: Triple Whale blended (orders), FB Ads API (spend). Post-cut daily order counts derived by subtracting brief's pre-cut weekly averages from 30d Triple Whale totals.
−0.9% orders/day
Latest 7d at $49/day FB · within the brief's ±5% natural variance band
Day-6 panic = trend tail
11.7/day dip was the pre-existing decline (18.3 → 14.9 → 13.3) completing, not FB causing harm
−$8.8K FB spend
21 days at $49/day vs $344/day = ~$6.2K/month savings continuing at the floor
Verdict: at the old $345/day budget, Meta spend was not incremental. Total Shopify orders recovered to baseline while Facebook ran at 14% of its prior budget. The pixel's 44 purchases on the $3.2K/month floor spend are being over-claimed — they sit inside the 15.86 orders/day total, not on top of it. The brief's hypothesis (FB does not drive incremental purchases at the tested spend level) is confirmed.
Decision: hold at $50/day floor, keep iterating. The test proved the $345/day budget was not pulling its weight. It did not prove Facebook can never drive incremental sales. We hold the floor as a live creative sandbox — testing new angles (Setup + price anchor, Trigger Event, Gift framings ported from Google) against the same incrementality bar. If a new creative moves Shopify orders above baseline while spend climbs, it earns scale. If it only moves pixel conversions, it doesn't. Declined FB-2026-04-20-01 ($50 → $100/day scale) pending that proof.
Field notes — the first six days
Preserved from the interim read (Apr 6). At day six the signal looked bad: orders had dropped 27%. Three weeks later, that drop is confirmed as the tail of a pre-existing decline, not a FB-caused wound. Keeping the original data here as a reminder to never read incrementality tests at six days.
Orders per day — before vs after FB reduction
| Period | Days | Orders/day | FB spend/day | Google spend/day |
|---|---|---|---|---|
| Mar 14–30 (before) | 17 | 16.0 | $344 | $231 |
| Mar 31–Apr 5 (after) | 6 | 11.7 | $76 | $290 |
Daily orders — Mar 14 to Apr 5
24
23
15
18
20
20
8
11
19
18
16
14
14
12
7
12
21
15
11
20
5
6
13
Mar 14
← FB cut Mar 31
Apr 5
−27% orders/day
16.0 → 11.7 · 6 days post-cut
p > .05
Not yet statistically significant · t = 1.66, need t > 2.57 at df = 5
Cohen's d = 0.82
Effect size is large but sample is too small to be conclusive
Confound: orders were already declining before the cut. Weekly average dropped from 18.3 (Mar 14–20) to 14.9 (Mar 21–27) to 13.3 (Mar 28–30) while FB was still at full budget. The post-cut 11.7 may be a continuation of that trend, not a result of the budget cut. Google spend also rose 25% after the cut ($231 → $290/day), absorbing some demand.
Lesson locked in. Six days of data was not enough to draw conclusions. The statistical framing at day six (p > .05, Cohen's d = 0.82) correctly flagged the read as premature — we held FB at $50/day through Apr 21, and the full window confirmed the underlying trend, not the cut, caused the dip. Future tests run at least 14 days before any directional read, 21 days before any action.
Three tests, run in sequence
Each test isolates one variable and measures Shopify orders per day against baseline. Test 1 concluded Apr 21 (FB budget cut on Mar 31, held at $50/day for 21 days). Tests 2 and 3 run on the original timeline.
TEST 1 — COMPLETE
Facebook budget reduction
HYPOTHESIS
Facebook is not driving incremental purchases. Triple Whale attributes only 10 Facebook-only orders on $10K spend. The post-purchase survey (716 responses) doesn't list Facebook as an acquisition channel. Cutting spend should not reduce total Shopify orders.
ACTION
Reduced FB budget from ~$345/day to ~$76/day on March 31. This gives us a clean before/after comparison while maintaining a small floor to preserve audience data. Holding at $50/day through at least April 21 — no changes to FB or Google until the read is complete.
MEASURE
Primary: Shopify orders per day vs 16.0 pre-cut average (Mar 14–30)
Secondary: New customer orders per day
Control: Google Ads spend and performance stay constant (currently $290/day)
Confound check: Compare order trend pre-cut (already declining 18.3 → 13.3) vs post-cut to isolate FB impact from underlying trend
Secondary: New customer orders per day
Control: Google Ads spend and performance stay constant (currently $290/day)
Confound check: Compare order trend pre-cut (already declining 18.3 → 13.3) vs post-cut to isolate FB impact from underlying trend
OUTCOMES
Orders stay flat (16+ /day): Facebook is not incremental. Reallocate $10K/month to Google or save it.
Orders drop 5–15%: Facebook drives some awareness. Consider reducing to $2K–3K/month floor.
Orders drop >15%: Facebook matters more than attribution shows. Re-enable immediately and investigate assisted paths.
Orders drop 5–15%: Facebook drives some awareness. Consider reducing to $2K–3K/month floor.
Orders drop >15%: Facebook matters more than attribution shows. Re-enable immediately and investigate assisted paths.
EXPECTED
Orders stay flat. FB touches 8.5% of orders in any attribution model, and most of those are long organic journeys where FB is one of many touches. At-risk revenue: $1,672 (FB-only orders) to $5,033 (generous multi-touch).
RESULT
Orders stayed flat. Latest 7d (Apr 13–19) at $49/day FB: 15.86 orders/day vs 16.0 baseline — within the ±5% natural-variance band. Full 20-day post-cut average: 15.4/day (−3.8%). Prediction held. Hypothesis confirmed at the tested spend level.
TEST 2
Pause Google Search campaign
HYPOTHESIS
The Search campaign cannibalizes organic brand clicks. 100% of Search conversions come from brand terms ("blockstream jade", "jade plus"). These searchers already know the product and would click the organic result if the ad weren't there.
ACTION
Pause the "Bitcoin Hardware Wallet | Search | USA" campaign only. PMax stays on. This isolates the brand-search variable without touching Shopping, Display, or YouTube.
MEASURE
Primary: Shopify orders per day from US customers vs baseline
Secondary: Organic click-through rate on brand terms (Google Search Console)
Secondary: Direct/organic traffic to store.blockstream.com (Triple Whale pixel)
Control: PMax campaigns unchanged, Facebook state matches Test 1 result
Secondary: Organic click-through rate on brand terms (Google Search Console)
Secondary: Direct/organic traffic to store.blockstream.com (Triple Whale pixel)
Control: PMax campaigns unchanged, Facebook state matches Test 1 result
OUTCOMES
US orders stay flat + organic CTR rises: Pure cannibalization. Kill the Search campaign permanently. Save $1,300/month.
US orders drop 5–10%: Some brand searches are competitive (competitors bidding on your terms). Re-enable but cap budget at brand-defense level ($500/month).
US orders drop >10%: Search ads are capturing intent that organic misses. Re-enable and test negative keywords on non-converting brand variants.
US orders drop 5–10%: Some brand searches are competitive (competitors bidding on your terms). Re-enable but cap budget at brand-defense level ($500/month).
US orders drop >10%: Search ads are capturing intent that organic misses. Re-enable and test negative keywords on non-converting brand variants.
EXPECTED
Orders stay flat. Search Console should show organic CTR rising as the ad no longer absorbs the top click. The 16.1 conversions Google claims will migrate to organic, not disappear.
TEST 3
PMax geo holdout
HYPOTHESIS
PMax is the most likely channel to be incremental (Shopping, Display, YouTube reach new audiences). But 54 of 147 Google-attributed orders started as organic visitors who later clicked a Google ad. A geo holdout reveals how much PMax contributes beyond organic discovery.
ACTION
Split US PMax into two geo targets:
Holdout region: Exclude 4–5 mid-volume states from US PMax (e.g., Pennsylvania, Ohio, North Carolina, Michigan, Georgia). These states should represent ~15% of US orders. PMax stops serving in these states entirely.
Control region: All other US states continue unchanged.
Match SA PMax and EU PMax by keeping them untouched — they serve as a parallel control.
Holdout region: Exclude 4–5 mid-volume states from US PMax (e.g., Pennsylvania, Ohio, North Carolina, Michigan, Georgia). These states should represent ~15% of US orders. PMax stops serving in these states entirely.
Control region: All other US states continue unchanged.
Match SA PMax and EU PMax by keeping them untouched — they serve as a parallel control.
MEASURE
Primary: Orders per capita in holdout states vs control states
Secondary: New customer acquisition rate in holdout vs control
Secondary: Revenue per state (normalize by population or historical order share)
Control: Non-US campaigns stay unchanged. Facebook matches Test 1 result.
Secondary: New customer acquisition rate in holdout vs control
Secondary: Revenue per state (normalize by population or historical order share)
Control: Non-US campaigns stay unchanged. Facebook matches Test 1 result.
OUTCOMES
Holdout matches control: PMax isn't incremental in the US. Consider US PMax reallocation to SA/EU where ROAS is higher.
Holdout drops 10–20%: PMax drives meaningful incremental volume. Current spend is justified. Optimize but don't cut.
Holdout drops >20%: PMax is a primary acquisition driver in the US. Test scaling US PMax budget.
Holdout drops 10–20%: PMax drives meaningful incremental volume. Current spend is justified. Optimize but don't cut.
Holdout drops >20%: PMax is a primary acquisition driver in the US. Test scaling US PMax budget.
EXPECTED
Holdout drops 10–15%. PMax Shopping and YouTube likely reach buyers who wouldn't find Blockstream organically. The 81 full-funnel Google orders (first + last touch) suggest real demand capture. But the magnitude is uncertain — this test pins it down.
Timeline
01
Test 1: Facebook budget cut DONE
FB budget reduced from ~$345/day to ~$76/day on March 31, then to ~$49/day from Apr 6 onward. Held through Apr 21.
02
Test 1: interim read DONE
14-day check showed orders stabilizing in the 14–16/day range, consistent with the pre-existing trend completing rather than a FB-caused drop.
03
Test 1: full read DONE
Latest 7d at 15.86 orders/day vs 16.0 baseline (−0.9%). Full 20d post-cut at 15.4/day (−3.8%). Within the ±5% natural-variance band. Verdict: FB at $345/day was not incremental. Decision: hold $50/day floor, decline scale recommendation, iterate creative.
04
FB creative sandbox at $50/day floor
Keep the Beginner CBO live at the floor as a testing ground. Ship new angles (Setup + price anchor, Trigger Event, Gift framings) into a separate $10–15/day ad set and hold the bar: any creative asking for scale must move Shopify orders above the 16.0 baseline, not just the pixel count. Re-evaluate at each Mon/Wed/Fri run.
05
Test 2: Google Search pause
Pause Search campaign only. Monitor US orders daily. Track Google Search Console organic CTR on brand terms. 7-day check on May 2. Full read on May 9.
06
Washout + Test 3: PMax geo holdout
3-day washout, then exclude holdout states from US PMax. Pull state-level Shopify baseline before starting. 7-day check on May 20. Full read on May 27.
07
Final report
Combine all three test results into a single incrementality model. Map true incremental value per channel. Set budgets based on proven lift, not platform-reported ROAS.
How to read the results
The core principle: if you pause a channel and total Shopify orders don't drop, that channel wasn't driving incremental sales. The nuance is in how much drop is meaningful.
Decision thresholds
| Shopify orders/day change | Interpretation | Action |
|---|---|---|
| ±5% or less | Within natural variance | Not incremental — cut or reallocate |
| −5% to −15% | Moderate lift | Partially incremental — reduce to floor, monitor |
| −15% or more | Significant lift | Incremental — restore and optimize |
What counts as natural variance
Your baseline is 16.7 orders/day. Daily orders naturally fluctuate — you had days as low as 4 and as high as 46 in the last 30 days. A single bad day means nothing.
The 5% threshold at 14 days: 16.7 × 14 = ~234 expected orders. A 5% drop means 222 orders. If you see 222+ orders in the test period, the paused channel was not incremental.
Weekly run rate is more stable: Compare week-over-week rather than day-over-day. Your weekly orders ranged from 98 to 134 in the baseline. A test week under 93 (30% below mean) is a clear signal.
The 5% threshold at 14 days: 16.7 × 14 = ~234 expected orders. A 5% drop means 222 orders. If you see 222+ orders in the test period, the paused channel was not incremental.
Weekly run rate is more stable: Compare week-over-week rather than day-over-day. Your weekly orders ranged from 98 to 134 in the baseline. A test week under 93 (30% below mean) is a clear signal.
The source of truth is always Shopify. During tests, ignore platform dashboards entirely. Facebook will report zero conversions (because it's off). Google may show inflated numbers (because it's absorbing some FB traffic). Neither tells you what you need to know. Only Shopify order count — pulled from Triple Whale or Shopify admin — reveals the truth.
Risks and mitigations
HIGH
PMax learning phase reset. Pausing and re-enabling PMax can trigger a new learning phase (1–2 weeks of unstable performance). Mitigation: Test 3 doesn't pause PMax — it uses geo exclusions, which don't reset the campaign's learning. Tests 1 and 2 don't touch PMax at all.
MED
Seasonal confound. If a Bitcoin price spike or Blockstream news event happens mid-test, it contaminates the results. Mitigation: Log BTC price daily. If BTC moves >15% during a test window, extend the test or discount that week. Flag any Blockstream press coverage.
MED
Cross-channel contamination. Pausing Facebook might cause Google to absorb some impression share (or vice versa). Mitigation: Track Google impression volume during Facebook pause. If Google impressions spike, that absorbed demand isn't incremental — it would have happened anyway.
LOW
Revenue loss during testing. Maximum at-risk revenue across all three tests: ~$8K (Facebook-only orders + Search conversions + holdout state orders). Over 6 weeks, you'll save ~$12K in paused ad spend. Net expected impact is positive even in worst case.
LOW
Competitor captures brand terms. When Search ads are paused, competitors bidding on "blockstream jade" will be the only ads showing. Mitigation: Check Search Console organic position — you should still rank #1 organically. If a competitor's ad CTR eats into organic, re-enable Search as brand defense only.
What we now know (and what's still to come)
Facebook
Confirmed: $10K/month of Meta spend was not driving incremental purchases. Orders held flat through 21 days at the $50/day floor. Holding the floor as a creative sandbox — new angles must prove incremental lift on Shopify orders to earn scale.
Search
Whether brand Search ads capture demand or just cannibalize organic clicks. Current evidence says cannibalization.
PMax
How much of PMax's 135 attributed orders are truly incremental vs organic visitors clicking an ad on their way to a purchase they'd already decided on.
Budget
A data-backed budget that allocates spend by proven incremental lift — not platform-reported ROAS.