MAPE Improvement Across All Rounds
→
→
Round 3
13.05%
hill_ec 0.5→0.3
→
Round 4
13.12%
per-channel adstock
Note: Round 4 PyMC ran as a fallback Bayesian LM (13.12% MAPE — consistent with Round 3's 13.05%). The full DelayedSaturatedMMM timed out without a C-compiler (gcc). The fallback does not compute per-channel contributions, so the 3-model CV% is inflated by PyMC zeros. The key Round 4 achievement is Ridge vs NNLS agreement on Meta Facebook (CV 71% → 7.9%), driven by the Notion knowledge layer.
Round Summary
Round 3
13.05%
Best test MAPE (PyMC)
3
Models running (Ridge + PyMC + NNLS)
0
Channels with ✅ High cross-model agreement
Global adstock decay 0.4
hill_ec = 0.3
lag = 1
Round 4
13.12%
Best test MAPE (PyMC fallback)
3
Models running (Ridge + PyMC fallback + NNLS)
1 ✅
Meta Facebook — first high-agreement channel
Per-channel adstock from Notion
hill_ec = 0.3
lag = 1
Knowledge layer active
Knowledge Layer — Per-Channel Adstock Decays
Key change in Round 4: Notion knowledge layer provided channel-specific adstock decay rates instead of a single global value (0.4). Search decays fast (0.2); video/display build slowly (0.6–0.7). This directly caused Meta Facebook to achieve high cross-model agreement for the first time.
Channel
R3 Decay
R4 Decay (Notion)
Rationale
Google Search
0.4
0.2
Intent-driven, fades in days
Google Video
0.4
0.7
Brand building, multi-week
Google Display
0.4
0.6
Awareness, medium carryover
Meta Facebook
0.4
0.5
Social, 2–3 week carryover
Meta Instagram
0.4
0.5
Social, similar to Facebook
Cross-Model Agreement: Round 3 vs Round 4
Meta Facebook achieved ✅ High agreement (CV=7.9%) for the first time — down from 71.2% in Round 3. This is the most defensible finding in the project: two independent models now agree on both the direction and magnitude of Meta Facebook's contribution.
| Channel |
R3 CV% |
R3 Agreement |
R4 CV% |
R4 Agreement |
Change |
| Meta Facebook |
71.2% |
❌ Low |
7.9% |
✅ High |
↑↑ Major improvement |
| Meta Instagram |
81.4% |
❌ Low |
34.6% |
⚠️ Medium |
↑ Improved |
| Google Shopping |
86.5% |
❌ Low |
42.2% |
⚠️ Medium |
↑ Improved |
| Google Search |
141.4% |
❌ Low |
100.0% |
❌ Low |
→ No change |
| Google Display |
141.4% |
❌ Low |
100.0% |
❌ Low |
→ No change |
| Google Video |
141.4% |
❌ Low |
100.0% |
❌ Low |
→ No change (PyMC needed) |
ROI Comparison — NNLS Model (consistent across rounds)
Meta Facebook revenue per unit spend · NNLS
Meta Instagram revenue per unit spend · NNLS
Why R4 ROI looks lower: R3 NNLS had no baseline floor — 100% of revenue was attributed to media, inflating ROI. R4 introduced a 33% organic baseline, so media ROI is more conservative and more realistic. The R4 numbers are more trustworthy, not worse. R3 Ridge ROI was zero due to a coefficient scaling bug fixed in R4 — Ridge is excluded from the R3 comparison for this reason.
Revenue Contribution — NNLS Model
Meta Facebook % of media-attributed revenue
Meta Instagram % of media-attributed revenue
Google Shopping % of media-attributed revenue
R3 NNLS had no organic baseline floor — all revenue was attributed to media (overattribution). R4 applies a 33% organic baseline, so media contribution % is lower but more structurally sound. Meta's relative ranking is unchanged across both rounds.
Key Takeaways
1. Knowledge layer works. Adding per-channel adstock decays from Notion reduced Meta Facebook's cross-model disagreement from CV=71.2% to CV=7.9% — achieving ✅ High agreement for the first time. The agent used domain knowledge it couldn't derive from data alone.
2. Meta Facebook is the most defensible finding. Both Ridge and NNLS agree on direction and magnitude (1.60× mean ROI, 39–42% revenue contribution). Directionally actionable within Meta portfolio.
3. Meta Instagram shows higher ROI efficiency (1.99×) but medium agreement (CV=34.6%). Warrants marginal budget shift toward Instagram within Meta, pending PyMC confirmation.
4. Google channels remain unreliable. Without PyMC's Bayesian priors, Google Search/Display/Video coefficients are noisy and sign-confused. Install pip install pymc-marketing and run Round 5 for three-model consensus.
5. Evaluation challenge confirmed. PyMC fallback MAPE held at 13.12% (vs 13.05% in R3) — MAPE is stable. But PyMC fallback returns zero channel contributions, inflating CV%. The real improvement is Ridge vs NNLS agreement on Meta Facebook (CV 71% → 7.9%). MAPE alone is a misleading scorecard for MMM attribution quality.