Saturday, March 18, 2017

Test Improved Conversion over Control by 20%. Deploy? Think Twice.

Common traps for AB testing and how to read split test result correctly. What to watch out for in split test

The Invisible Traps in AB Testing: A Conversion Optimization Case Study

As a conversion optimization hacker, you pride yourself in making data backed decisions for your business. You follow the AB testing guidelines to a tee. After dozens of tests, you've finally found your diamond in the rough - your new landing page is performing 15% better in converting visitors for registrations. You're so pumped, you're ready to go! 


At least not before you've checked all the data. 

Case study

Pinterest once ran a test on the visual design where they increased the size of the related pin to the original image that brought users to their landing page.
Conversion optimization, successful website design
It created more equality for the extended image roll and may get users into the mode of scrolling on more instantly. Brilliant idea! Data agreed too. They saw a lot of engagement and the new layout was 25% more effective at convincing a visitor to sign up. 

They didn't ship that page. 

Why? Because the new design broke SEO. Well, kinda. They saw a double digit drop in traffic coming through. The resizing of the images forced search engines to re-crawl tons of pages and the traffic through image search was negatively impacted. This was a highly trafficked landing page so the percentage drop in top of the funnel was considerable. 

Key Takeaway

It is a simple but often overlooked phenomenon in conversion optimization - when you're laser focused on improving one key metric, make sure you've buttoned up on the entire data flow when making your decision. Pride yourself on being the data detective - keep your ears open when your data is screaming a B-side story.

Follow me on twitter (@qindizhang) or add me to your Google+ circle, my friend! Let me know in the comment what other topics you want to read about!

Thursday, March 16, 2017

The Rise of Voice and Collective AI

Ben Evans is a partner at the renown venture capital firm Andreessen Horowitz (also called a16z). He generously shares his industry insights with the public and has hundreds of thousands of followers on Twitter.

In Ben's recent post Voice and the Uncanny Valley of AI, he started with

Voice is a Big Deal in tech this year. 

The Rise of Voice

Ben attributed the rise of voice to four causes: 
  1. The advancement in machine learning reduced the error rate in voice recognition and natural language processing.
  2. Better hardware, lower cost.
  3. If voice most existed as people's side projects in the past, the resource is abundant today. 
  4. The desktop-to-mobile platform evolution has shown, the more engaging UX is, the more control companies have which own these platforms - Apple and Google had more control over the mobile user base than Microsoft ever had with the desktop.

The twitter feed from Ben at the top of the post was mocking at AI not being able to connect dots in a human way. In fact, the ability of today's virtual assistants at handling context in a conversion is continuously increasing. You can ask a series of questions such as:
"What are the 5 star rating restaurants nearby?" 
"How far is the first one?" 
"How much  does it cost for Uber to take me there?"
And product such as Hound handles it like a champion. You don't even need to get out of the app to call for a Uber ride of your choice.

The Era of Sharing Economy, Knowledge & Intelligence

Collective AI came into view as one of the key contributors to the increased AI performance. Collective intelligence is shared or group intelligence that emerges from the collaboration, collective efforts, and competition of many individuals and appears in consensus decision making. What happens when we apply this to building AI?

SoundHound is the leading innovator in voice-enabled AI and conversational intelligence technologies. SoundHound's vision of Collective AI is to provide the tools and platform for developers to build more intelligent solutions easily and rapidly. 

The era of sharing economy, knowledge and intelligence brings a voice-enabled AI platform and its technology to the ISV community - something once only possessed and controlled by name brands such as Google, Apple, Microsoft and Amazon. What a gift!

The closing thought - what's the face of your personal AI? 

The Rise of Voice and Collective AI, Voice recognition, Natural Language Processing, Houndify: Collective AI from SoundHound Inc.
For the record, my personal AI definitely has the face of Wall-E. ;o)
Read my related machine learning articles:
Want More Funding? Machine Learning!
How to Build a Recommendation Engine without Machine Learning Experts?

Follow me on twitter (@qindizhang) or add me to your Google+ circle, my friend! Let me know in the comment what other topics you want to read about!

Sunday, March 12, 2017

Y Combinator is a Waste of Time - Unless You Do This

Tips & lessons shared by an ex-YC founder, for those of you applying or thinking of applying for YC S17

Steli Efti is the founder & CEO of,  a SaaS solution that helps people to close more deals.  As a YC W11 alumni, he is letting you in for a peek behind the curtain. In his own words.
Inside stories from YC alumni; how to get into Y combinator; Tips on YC application; How to prepare for YC application

Is Y Combinator worth it? It’s funny: You can ask two totally different YC alumni that question and get two totally different answers, even if they both attended the same batch.

How’s that possible? Why is it that? For some startups, Y Combinator makes a difference; while for others it makes all the difference? Trust me: It’s not coincidence, luck, or privilege.

I’d know: At, we owe a huge part of our success to Y Combinator. The lessons learned, resources gained, and relationships built during YC W11 were instrumental in both our initial traction and our continued growth.
So yeah, if you ask me, Y Combinator’s worth it—if you take the right approach. Because at the end of the day, thriving in the YC environment isn’t about having the best idea, the most disruptive product, or the smartest team. It’s all about having the right attitude.
More than any other factor, your mindset determines how much value you take away from the YC experience. If you’re willing to invest the (substantial) time and energy required for Y Combinator, you owe it to yourself to take an extra 10 minutes to read this article and make sure you’re setting yourself up for success.

The YC mentality: Academics  vs. Hustlers

I’ve been a part of multiple Y Combinator batches. Once as an attendee, other times as a speaker or advisor. And every time, I was fascinated by the stark divide between attitudes.
In every batch, there are two clearly-defined startup “camps”: The Academics and The Hustlers. Both sides find value in the experience, but only one truly thrives.

Meet the Academics

The Academics view Y Combinator as “startup school.” They show up, sit down, and follow the program. Their motto is, “If I just do what I’m told, I’ll ‘graduate’ successfully.”
If there’s a dinner Tuesday at 6:00 p.m., they show up Tuesday at 6:00 p.m. If there’s a speaker Friday at 3:00 p.m., they show up Friday at 3:00 p.m. They take notes, do their homework, and follow the schedule.
And this approach works. Y Combinator’s designed to help startups, so you’ll learn a lot even if all you do is follow the program. But just “following the program” is like graduating with a “C.” Sure, you passed, but you left the majority of your education on the table.

Meet the Hustlers

The Hustlers, on the other hand, don’t leave any source of potential value untapped. They recognize YC as the resource-rich environment it is and take full responsibility for their experience. Their motto is, “We’re here to make our business successful, and we’re going to leverage every possible resource to make that happen.”
Hustlers take, learn, leverage, and ask for as much as they can. If the doors of the YC offices are open, they’re there. They know the best opportunities and most valuable introductions are rarely a part of the official “program.” After all, that program is meant to be the launching pad, not the finish line.

Channeling your Inner Hustler

The biggest mistake I see YC attendees make is playing it safe. If you leave the program wishing you had spent more time with someone or gotten more help with something, that’s your fault, and no one else’s.
The YC leadership is there to help. They want you to succeed. They want you to ask questions. They want you to be proactive. But it isn’t their responsibility to make you do any of those things.
As a result, many YC attendees don’t ask questions. They aren’t proactive. And they don’t reach their full potential. But that’s good news for those that do.
Let me share three short stories to illustrate the importance of taking ownership of your YC experience; hopefully, they’ll help you make the best of yours.
“As it turns out, something opened up.”
As a relatively unknown founder, there’s practically nothing you can do to force a VC to care about your sense of urgency. I learned this lesson the hard way.
My team and I were trying to close a round of fundraising in the next two weeks, but the VC partners we needed to meet with weren’t available for at least another month.
That timeline didn’t work for us, so we sat down with Paul Graham and asked if he’d be willing to reach out to the firm on our behalf to tell them how amazing we are and urge them to prioritize our meeting.
When pg agreed, I was prepared. I pulled out my laptop, which already had a web browser open, and said, “Great! Can you do this right now?”
He did. Less than 12 hours later, we had a response from the firm, informing us they “suddenly” had an opening on their calendar in just two days.
“You’ve got better things to do, just sign it.”
Another time, I needed pg to sign a recommendation letter so I could get my visa. I’d already drafted the letter and all I needed from him was the signature. He agreed but, being the talented writer he is, wanted to re-write my draft to make it really good.
I needed that signature now, and I didn’t know how long it would take for pg (who is an incredibly busy guy) to get around to those revisions. So instead of waiting and hoping, I followed up. An hour later, I showed up at the Y Combinator office with letter and pen in hand. “Just sign it,” I said. “Just sign it. I know you’ve got better things to do.”
He did, revisions be damned, and I’m still here; so I must’ve done something right.
“These guys make shit happen.”
And yet another time, we were trying to recruit the unrecruitable by hiring Phil Freo. He wasn’t convinced, so we invited him to a YC dinner to have pg help us convince him.
He sat Phil down and said something like, “Look, these guys are weird, but they’re really effective. In fact, they’re one of the best startups in the batch, simply because they make shit happen. They’re aggressive. They’re shameless. They’re humble and open-minded, sure, but they’re intense. They don’t wait for things to happen. They make things happen.”
And to be honest? It didn’t work, at least, not at first. Although his initial answer was no, Phil ended up joining the team a few years later. I like to think pg's endorsement played a big part in that. (Thanks again, pg!)

You are your own biggest obstacle

I could share a thousand stories but, at the end of the day, it’s as simple as this: In Y Combinator (as in life), you’re going to get what you take. YC is a resource-rich environment filled with people who can and will help you; but no one there owes you anything, nor do they know what you need unless you tell them.
In YC, everything you need to succeed is right in front of you (or only an introduction away). The founders who get the most value out of this program are those who know this and leverage that knowledge constantly, sometimes even shamelessly.
So ask for an unreasonable amount of office hours. Ask for off-the-wall favors. Ask for more than you expect to get.
Because remember, you’re in an environment where people want you to succeed. You aren’t doing them (or yourself) any favors by being timid. So step up, be bold, be loud, and be proactive. Know what you want, and do whatever it takes to make that happen.
Do this consistently and I promise: You’ll crush it inside and out of Y Combinator.

Grow your startup with a free copy of From 0 to 1000 Customers & Beyond.

Follow me on twitter (@qindizhang) or add me to your Google+ circle, my friend! Let me know in the comment what other topics you want to read about!

Friday, March 10, 2017

Try Out Beet, Because it'll Make You Red Pee

The Red Pee Proposition

Most parents have had the challenge of convincing their kids to eat vegetables.

"Try some brocolie, honey. It's good for you." 
"Because it's healthy. It gives you lots of vitamins and fiber." 
" No, thanks." (Looking away)  

This conversation I had with my kindergartener yesterday wasn't so typical.

"What is this red food?"
" It's called beet and it makes you red pee tomorrow if you eat enough."
"How many is enough?" 
"20 little pieces, I guess." 
"One, two, three... I got twenty!" (Chewing) 

Get Inside of Your Audience's Mind

Much of a Product Manager’s responsibility is to juggle multiple streams of conversation and move them towards closure. As a PM, you're required to influence and sell on a daily basis, to different audiences. During a good portion of your day, you make data-driven decisions (hopefully) with your left brain. Is it good to extend that capacity and sell with evidence or on the technicality side of the story? 

It depends. 

To an engineering audience, it may very well be the best way to go about it.  At the meantime, you'd better be prepared to sharpen up your right brain in order to take on an audience made up of sales, design or anyone new to the technology or concept you introduce on. 

How to Talk Tech to Non-Technical Audience

Live like an adult and think like a child - If you want to convince a child, you first learn how a child thinks.

The focus of the left brain is verbal, processing information in an analytical and sequential way, looking first at the pieces then putting them together to get the whole. The right brain of the brain focuses on the visual, and processes information in an intuitive and simultaneous way, looking first at the whole picture then the details. Left brain thinking is verbal and analytical. The right brain is non-verbal and intuitive, using pictures rather than words. 

Left brained organization such as Amazon invented non-powerpoint presentation and evangelizes the beauty of 6 pagers. If you want to work at Amazon, be sure to brush up on AB testing, market sizing questions and learn how to write hypothetical product's press release and FAQ announcement. Amazon uses this "working backwards" approach because it forces the team to get the most difficult discussions out of the way early. They need to fully understand what the product's value proposition will be and how it will be pitched to customers. If the team can't come up with a compelling press release, the product probably isn't worth making. 

Case Study: Tell a Technical Story in the Language of your Audience

R2D3 is an experiment in expressing statistical thinking with interactive design. Using a data set about homes, they created a machine learning model to distinguish homes in New York from homes in San Francisco. This is the most powerful visual storytelling I've come across recently. How elegant!
Immerse yourself in the data story
machine learning 101, machine learning case study, R2D3 machine learning

This is part 2 of story telling series. Find part 1 there.

Follow me on twitter (@qindizhang) or add me to your Google+ circle, my friend! Let me know in the comment what other topics you want to read about!

Want More Funding? Machine Learning!

Crystal ball, what's on the investors' mind?

 posted on What VCs Talk About After Your Pitch yesterday and as suggested in the intriguing subject line, he has let all the entrepreneurs in on one of the most prized secrets in the startup world. 

All businesses chase after product market fit and in the same sense there's no intel more valuable than what investors are in fact looking for when evaluating an investment opportunity. In Parker's article, he pointed out two things to focus on not only in business plans but also in pitch decks:

  1. The problem the business is solving
  2. Building the moat

How to build a moat with machine learning

Machine learning can do the heavy lifting for you and turn your big data into a moat around your business castle. 

With machine learning, Netflix has come a long way from a DVD rental business to world's leading internet television network and from a content aggregator to an original content producer. Quite a moonshot accomplishment if you ask me! Youtube is catching up. With all the top influencers and the attached fan base in their pocket, they even have more interesting flavors added into the big data cookbook.

Machine learning can also turn the most bland business ideas into brilliant products that consumers can't live without. I've personally fallen in love with gmail's priority inbox feature. It takes into consideration several dimensions of data features:
  1. Social features to examine the relationship between the sender and receiver based on open rate
  2. Content & Thread feature to determine the recency and relevancy of email content to what receiver has acted on in the past
  3. Label feature indicates preference explicitly expressed by the user
Last but not least, machine learning isn't all science and no art. Gmail team skillfully balances user need on minimizing the false negative rate - users will likely have vastly different levels of emotional response to an urgent email ending up in the junk box vs. a spam message occasionally slips through the priority inbox.

So, ready to adopt machine learning yet, my entrepreneur friends? Or shall I rephrase, ready for a big fat paycheck from your next investor? When you're face to face with investors, discover how to effectively engage an audience and work a room.
Tips on funding pitch and investor pitch deck. how to build a moat around business

No Machine learning expert? No problem. Learn step by step on how to work around it.

Follow me on twitter (@qindizhang) or add me to your Google+ circle, my friend! Let me know in the comment what other topics you want to read about!

Thursday, March 9, 2017

Today Was A Complete Waste Of Makeup

An idea is a terrible thing to waste. As harsh as it may sound, the quality of the idea is not as important as the quality of the presentation of that idea.

Show, Don't Just Tell - The Power of Story Telling

A couple years ago I was at a workshop of James Whittaker where he was teaching stage presence and story telling at an internal Microsoft forum. James is a distinguished engineer, an accomplished public speaker and the author of popular article Why I left Google. I enjoyed every minute of his stories and believe that following his five laws of stage presence, I'll master that magical super power of story telling. 

5 Laws of stage presence by James Whittaker

  • Start Strong
  • Focus on the concept
  • Show, don't tell
  • Speak for the Audience
  • Stick the landing

So why it has anything to do with the waste of makeup? 

The 6th law

  • Tell a story by pumping up audience's imagination

James shared a interesting moment when he ran into a strange grumpy lady who apparently had a long daunting day. She said nothing more than 
"Today was a complete waste of makeup."
How to tell a story, six word story, story telling is a super power

She shared a story by not sharing one, but you, in the audience, are simply intrigued. Including James.

Six word story

Inspired by Hemingway’s famous six-word tale, “For sale: baby shoes, never worn,” the “six word story” has served as a prompt for decades, testing writers’ ability to create their own succinct masterpieces with all sorts of clever results, including the popular 2006 Six Word Memoirs project. Now, the well-worn idea has gotten new life on Tumblr and Reddit, where users are posting their own hyper-short creations online to show off their creativity and pithiness.

  • Birth certificate. Death certificate. Same Day.
  • We're lying in bed. She's lying.
  • "Total media blackout" agreed the President.
  • BREAKING: Simulated beings realize they're simulated.
  • "Wrong number", says a familiar voice.
  • Sorry soldier, shoes sold in pairs. 
  •  Brought rose home. Key didn't fit. 
  • "Male?""It's an old driver's license."  
  •  Introduced myself to mother again today.
  •  I'm done being your "sometimes". 
  • "Joining the President is his husband..." 
  •  I met my soulmate. She didn't. 
  •  Mom taught me how to shave. 
  •  Passengers, this isn't your captain speaking. 
  •  What's your return policy on rings?

Now it's my turn to share two of my 6 word stories today - Story telling is a super power. May the power be with you. 

Now investor pitching, a different story.

Follow me on twitter (@qindizhang) or add me to your Google+ circle, my friend! Let me know in the comment what other topics you want to read about!

Sunday, March 5, 2017

How to Build a Recommendation Engine without Machine Learning Experts?

Want to serve relevant content? YES! 
Have machine learning expert? NO.

Now what?

Before the machine learning era and personalized content, old timers pop the page with a list of most popular items of all time.
The common issues you'd notice with this approach:
  1. Content Gets Old - Users finding same content every time they sign onto your service.
  2. New Items Never Get Surfaced - Since the leader board may not normalize the popularity by time elapsed since publishing, great new contents have a hard time making it.
  3. You Don't Know How to Pick your Battle - If you decide to let the new content take over, your collapsed conversion may give you a heart attack, especially if you have a less active user base.

Build a simple and effective recommendation engine

How do you build a simple recommendation engine with the highest take rate and decent user experience? The answer lies in understanding your user behavior, in this particular case: Usage Pattern. 

Here is step by step to build a simple but high performing recommendation engine

  1. Show Only Convertible Items - Remove items from the recommendation list if the user has already taken it in the past, or if a user has clicked through very recently and abandoned the flow thereafter.
  2.  Segment your User Base - Analyze the distribution of user activity across a reasonable time period, say 4 weeks. Finding a cut off to group the users into high activity cohort and low activity cohort. For example people who are active once every other day will go to segment A and those who are active only once a week or longer go to segment B. There is no universal formula for that because it varies by the business you are in, use industry benchmark and your own data to make a decision.
  3. Balance your Recommendations - Test and observe how segment B respond to best-selling catalog items. Since they don't log in as often, you make sure they see the most bullet proof offerings when they do. On the other hand, offer newer popular contents to users who frequent your shop in general as they have likely consumed fair share of the most popular content.

The above three steps to build an effective recommendation engine is for a catalog of moderate number of genres and/or relatively homogeneous user base. For users that may range from young teenagers to business users that could have zero overlap in their consumption, the above approach is not the best - well I highly doubt you want to serve them with one product anyway.

Follow me on twitter (@qindizhang) or add me to your Google+ circle, my friend! Let me know in the comment what other topics you want to read about!

Saturday, March 4, 2017

How many nightmares have you ever had in your life?

Market Sizing Question & How to explain things to a kid

My 6-year-old woke up this morning and asked me a question: How many nightmares have you had in your whole life, Mommy? 
Interview questions. Consulting interview. Analyst interview.

Sounds familiar? If not, then how about estimate number of shampoos used in the US each year or how many gas stations there are in San Francisco Bay Area, or one of these. And isn't it more exciting now you have to answer that in a way a kindergartner can follow? Okay, let's leave MECE out for a second and see what else we can do here.

"We'll find that out together, shall we? First, let's agree on a couple small things. We'll count the nightmares by night. If I wake up in the mid of the night from a nightmare and fall back asleep and this nightmare continued, it count as one. Yes?"


"As long as I think it's a nightmare, even if it's not scary to you, it still counts. Yes?"


Great! Now let's chop up all my years into three blocks: 
1. When I was a kid, I was a happy kids just like you. I didn't have that many nightmares until I was a teenager. So until 12 let's say I had a nightmare only every 6 months, that makes it? Yes, Number A. 
2. As a teenager, I was scared of frogs and roller coasters, in addition to a few movies such as a scientist turned himself into a fly. So between 13 and 18, for every 20 dreams I had one nightmare. How do I calculate that? With some hustling we got to the Number B. 
3. In my adulthood I've learned how to rationalize scary scenes from ghost movies and cope with real life stress. I've been able to reduce on how often I have nightmares by 2/3. How much is that for the next 15 years? Number C.
OMG we have just one step left to do, adding them all up A+B+C. Ta-da - you got your answer!

Apparently I daydreamed about the second half of the conversation - Of course my kindergartner rambled away when I was just half way into teenager nightmare calculation, but I'm sure you could tell the steps I followed in solving a market sizing question. 
Step 1: Clarify the question and agree on assumptions.
Step 2: Break it down (ideally in a mutually exclusive and collectively exhaustive (MECE) manner).
Step 3: Solve each piece.
Step 4: Consolidate the result.

I hope you will never have any nightmare about interviews, or at least market sizing questions after reading this post. 

Follow me on twitter (@qindizhang) or add me to your Google+ circle, my friend! Let me know in the comment what other topics you want to read about!

Friday, March 3, 2017

3 Types of SQL Interview Questions You Must Ask to Spot Best Candidates

How to prepare for a SQL interview, as an interviewer?

SQL is relatively simple as a programming language. It is a must have for testing a candidate's technical ability when looking for analytics talent. I will share some effective questions for hiring managers to consider.
How to interview for data analyst

SQL interview question bank:On candidate's basic understanding on data structure and SQL fundamentals:

  1. Can you interchange where and having clause?
  2. What is a correlated sub-query?
On candidate's actual hands on SQL experience:

  1. How to perform unpivot in SQL?
  2. How large is the dataset you typically query against? What are some of the things you do to improve query efficiency?
On candidate's business exposure and understanding:

  1. What is a business case where you need to use full join?
  2. How do you handle missing days in a transaction report if dashboard users want this information captured and reflected?

For the technical interview, it's best to combine some actually code writing with conceptual questions  such as above. That way you as an interviewer can evaluate how familiar the candidate is with the basics, how comfortable she is at writing code, how much of her knowledge is from book vs. from practice. And furthermore, how well the analyst is able to grasp the underlying business requirement and deliver the analytics product that satisfies stakeholder's need.

Interviewers, I hope this gives you an easy start! 

Follow me on twitter (@qindizhang) or add me to your Google+ circle, my friend! Let me know in the comment what other topics you want me to cover!

Thursday, March 2, 2017

Why is Snapchat a camera company, even before they started selling Spectacles?

Snapchat IPO

Today, all eyes are on Snapchat. On its first day of trading, SNAP soared nearly 50%. And it's only the beginning.
Shall I buy Snapchat stock? Snapchat valuation and market capital.

3 Reasons Why Snapchat is a Camera Company

In its public filings, and on its corporate website, Snap refers to itself as “a camera company.” But what that actually means is open to some creative interpretation. Here's mine.

1. When it comes to consumer psychology, Snapchat is in a class of its own.

A camera product involves two parties of users, the one behind the lens and the one in front - or shall I add, traditionally speaking. Snapchat is the camera company that perfectly merged the two into one. Guess what, it happens to be (almost) everyone's most interested subject: I, me and myself. 

Instagram filters make me look as cute as the kitten in the Youtube video who just got another 10k views today. 
Said no one ever. You get the point.

2. Snapchat's monetization model seamlessly make ads engagement part of the product usage flow. 

The last company that did that? Google search engine. Machine learning has made personalized ads possible, but displayed ads still require users to disengage with whatever product they are using at the time in order to pay attention to the served ads, needless to say to engage with the ads they'll start a separate action flow which takes them away from their ongoing core product usage. Makes sense to me that Google has quietly invested in Snapchat not too long ago. 

3. Snapchat's product consumption is not dependent on network effect to be successful. 

Of course social features make a product more sticky, but the core value provided by snapchat is entertainment, to be consumed by one or more. It is the go-to-app of my 6-year-old's every time he manages to snatch my phone from me. He tries out each filter and make a video or two, and later on keep watching these videos for some more time. A valid use case.

Now get ready for the camera company that has 158 million daily active user, creating and exchanging 2.5 billion snaps per day, 10 billion daily video view. These numbers give me goosebumps.

Read another 2 minute case study on Pinterest.

Follow me on twitter (@qindizhang) or add me to your Google+ circle, my friend! Let me know in the comment what other topics you want to read about!

Wednesday, March 1, 2017

A Newbie's Guide to AB Testing

AB Testing 101

Don't set aside budget for Optimizely and Google Analytics before reading this simple and fun AB testing guide for newbies that ENTREPRENEUR put together to help get the basics out.

Key Takeaways

  • Be prepared to fail, and fail, and fail again before succeed. About 1 in 7 AB test has a winning one.
  • Try to test on one thing at a time. Be aware of different variables that may come into play. If you can't isolate individual drivers, try to be mindful about them when reading the results.
  • Make sure your test result is statistically significant. There are some online tools available to help you out.

More tips on split testing

Follow me on twitter (@qindizhang) or add me to your Google+ circle, my friend! Let me know in the comment what other topics you want to read about! 

What is AB testing and how to do a AB test? Fun facts about AB test. Ab testing for dummies.