BTL Acquisition Experiment Result (2019-12-18)
Participants: @yola, @Kenny G G G, @rama, @gifarie, @silvia, @fherline
Result: deck
Result Summary
- User segment with propensity score > 0.8 gave the best response (major conversion rate uplift), while for the segment with prop score between 0.5-0.8, only high CLV users showed major conversion uplift from the experiment.
- Apparently only coupon treatment that seemed to work to attract CA. Nonetheless, sending PN only works best to attract high CLV users with Prop score > 0.8 (top right box from the audience heatmap matrix).
- By targeting these 3 segments with the best response rate to the treatment (combining both coupon & PN only treatment, decided based on segment), we can bring up to IDR 160Mn GBV uplift, without major impact to CLV. If Xperience aims to improve CLV, then it is advised to reduce the offer.
Immediate actions
- Given PN only treatment with high CLV and high Prop score resulting in positive net-revenue, Xperience team can start excluding these segment from coupon campaigns as we can conclude this segment will buy the product anyway without coupon, thus we can optimize Xperience coupon cost.
- Dashboard to download audience for high CLV & high Prop score will be ready in January 2020 (timeline will be updated by Yoyo). In the meantime, should Xperience want to immediately exclude this audience in December 2019, can request to MarTics through JIRA.
Notes: MarTics & MarSci are planning to improve Propensity Model, it's advised for Xperience to re-run the experiment to validate the result once the model is ready.
2020 Xperience Experiment Planning
- Prioritized categories (sorted based on highest GBV contributor): Attraction, Tour, Beauty & Spa, Movies, Playground, Entertainment
- Xperience will need to find the balance between doing both acquisition campaign and retention campaign
- Specific for acquisition, we can do 2 scenarios:
- Acquisition from non-Xperience product to Xperience
- Acquisition from Xperience daily category <-> leisure category as we found that the overlap between this two are very small
- To make it efficient, PMM can directly submit the experiment proposal using this template and submit it to Offer Management. Once the proposal is ready, Offer Management will conduct session to finalize the proposal with Xperience PMM, Xperience DA, and MarTics
- BTL lifecycle intro: deck