We have all heard various recommendations on growth tactics: Mandatory registration, rich push notifications, frictionless user onboarding; the list goes on. Some of this advice is very convincing as they come in as learnings from successful growth projects.


Common Mobile Growth Tactics

Andy Carvell shared a case study from SoundCloud where he tested putting emojis into push notification, a heart 💗 at the beginning and the end of the message title. The open rate improved. The team went ahead and tested with two hearts 💗💗 and the open rate yet improved again.

You probably at some point saw one or more growth tactics like the below, most of which come with numbers:

  • Emojis can increase push notification open rate by 85%
  • Email open rate increase by 4x if a user's name is included 
  • In-app behavior based messages increase the open rate by 9x
  • Push notifications lift retention rate by 20%

 Next time when you encounter an exciting and seemingly convincing growth tactic, ask yourself the questions below to get a better understanding regarding the context of the data provided, and evaluate how applicable they are to your particular problems:

1. When was the data collected? Has the OS significantly changed in related features for these test results to stay relevant today?
2. How large is the sample size? Does the data cover one app, several apps by one company or many apps by many companies?
3. Is the data industry specific? For example, does it relate to e-commerce, game or utility apps?
4. Does the data cover multiple geo and/or language market?

Growth Tactics Don't Always Work, Now What?


Next time when we feel the urge to flood our push notification messages with emojis  😼🤣😘🎉(applause) in the hope of improving that depressing 2% push open rate, let's first take two actions:

1. Discover WHAT

I'm all for testing every sensible growth lever, but first and foremost we need to be crystal clear what we are really testing or other factors play a major role in the test result. 

 For example, a rich push was tested on both Android and iOS where users are required to press and slide down the notification in order to see the full content (e.g. an image, gif or CTA buttons) attached to the rich push. This test is highly dependent on users' understanding and adoption of a particular OS feature. If the open rate didn't improve, it may not be your image or CTA, as the users simply need more education on how to use the feature itself.


2. Find Out WHY

After we find out about what works and what doesn't, the next and more important step is to understand what drives the user behavior resulting in the data you see.

An app I've worked on tested emoji and saw absolutely no noticeable lift on the test cohort. User analysis revealed that the app's user base gears towards an older age group, and surfaced that some recipients found emojis overwhelming and made it harder for them to read the text of the actual message. 


Takeaway

The larger goal in testing is beyond individual wins. Rather, it is about getting better understanding of our own UNIQUE user base so that we can apply the learning systematically to future treatments. In the long term, we want to be able to design more meaningful test and treatments to address users' underlying needs.

So test away my friends, with your North Star in mind!
Growth best practice, AB Testing, Growth Case Study, Growth Hacking
Do we know what we're really looking for?


Read my related growth articles:

Test Improved Conversion over Control by 20%. Deploy? Think Twice.

A Newbie's Guide to AB Testing


Follow me on twitter (@qindizhang) or add me to your Google+ circle, my friend! Let me know in the comment what other topics you want to read about!