How I ran A/B tests at Indosat
We sent fewer than 50 campaigns a month to 100M+ subscribers. Every one was tested before it went to everyone. Many of them never did.
The problem
I owned pricing, bundles and personalization, so I could send an offer to a small group first.
The risk: an offer can push ARPU up on the dashboard and still lose money, because some people would have bought anyway, or just switched from a package they already had.
- Revenue holds against the holdout?
- Retention holds?
- Clears the payback threshold?
What I did
- 01
Start small
Send the offer to a small slice of one segment. Use what that slice actually did to estimate the full rollout.
- 02
Always keep a control group
About 10% of every campaign audience got nothing. We compared everyone else against them.
- 03
Count switching
Every business case had one clear line for switching: how many buyers just moved from a package they already had. That turned gross revenue into net revenue.
- 04
Check ARPU and usage together
If ARPU went up but data usage stayed flat, people were paying more for the same thing. That usually didn't last.
- 05
Check again after launch
After every release I compared the plan with the actual numbers and stopped launches that didn't reach the payback target.
- 06
Test the copy too
We tested ChatGPT-written copy for each customer type against our usual copy. Some tests we ran together with McKinsey.
What happened
Many campaigns never went to the full base, because the control group showed revenue and retention didn't hold, even when ARPU rose.
The most common reason a campaign failed wasn't switching. It was low demand: very few buyers, and flat clicks and views.
The campaigns that did scale raised ARPU by about 5% over time, and revenue per channel hit target.
- Test demand first. Most failed campaigns simply had too few buyers.
- Look at ARPU and usage side by side.
- Write the key assumption down where everyone can see it and question it.