Just had a good conversation about “lean analytics” with digital marketing guru Rahul Chhabria. He’s been working with some interesting startups on implementing data-driven decision making processes, and made an interesting point.
1. Nearly every startup uses an agile development process and has adopted Facebook’s “move fast and break things” mantra. This is good thing.
2. Startups increasingly make data-driven development and UI decisions. This is a good thing.
3. You need to have decent amount of data for analytics to be useful. In some environments, development cycles move faster than reasonable amounts of data accumulates, and design/engineering teams might be pressured to make decisions based on too little data.
It’s better to either let a test run for a little while – for a few dev cycles – in order to accumulate a decent amount of data, or if a decision needs to be made, to make it based on expert opinion (intuition?) than to make a decision that you think is data-driven but really isn’t.
Better yet, if you don’t have a strong userbase yet, resist the pressure to move fast and break things and focus resources on building more traffic and users.
What sort of numbers am I talking about? It really depends, but a good rule of thumb is that at a minimum don’t make a decision based on fewer than 100 datapoints. It’s better if you have 1,000. Under a hundred, and who knows? It could be a fluke. Don’t invest precious dev resources in making changes that may not be necessary, or worse, may be detrimental. Instead, invest those resources in things that drive more traffic to your site, so you can actually accumulate enough data to make a smart, data-driven decision.