Reflections on startup life: Week 65

Analysis Paralysis — or a little bit of stats can go a long way.

We didn’t “turn on” the viral thing after all — instead we just enabled the feature for everyone. Why? Well as we thought more about it, we realised that we still don’t really understand why people use Trunk.ly (so driving more people to the site still isn’t the right thing to do). It’s a strange thing to say in some respects, I mean, it’s an automated, searchable bookmarking site right? We’ve had some great growth and good press, so surely we know why people use it! Short answer — no.

It turns out that we are still trying to answer that question. Why is this important? The more you understand who and why your customers are, the better job you can do of presenting the offering in a way that they get it (critical in any business, but especially on the web where your home page has to sell for you). To help with this, we’ve been working through the product / market fit survey (http://startup-marketing.com/using-survey-io/) which is a joint partnership between Sean Ellis and KissMetrics.

The key question in this survey is simply “How disappointed would you be if (product x) no longer existed”. In the case of Trunk.ly, our first round of surveys (back mid Jan) showed 25% were “Very Disappointed (VD)”, while in the most recent round 36% would be VD. That’s cool — but the methodology says we need higher than 40% before we can begin to consider ourselves a “must have” product. How can we move the Somewhat Disappointed (SD) to VD?

Actually it’s more than just “get to the 40%” mark, more accurately this about customer discovery and understanding what’s the magic that that 40% see that other don’t. Working this out is much harder than it sounds. In practice however you’re looking for one of three things (from an email with Hiten Shah of KissMetrics):

1. Discovery: SD responders didn’t discover a specific feature that is buried that most VD responders did discover.

Next Step: Get more people to discover that specific feature faster or even rotate the whole product around that feature.

2. Discovery: VD responders describe the product in a different way than SD responders.

Next Step: Try testing the messaging on the website to tune towards what benefits the VD responders say they have received.

3. Discovery: SD responders have had “issues” with the product that prevented them from considering it a “must-have”.

Next Step: Try to solve those “issues”.

The first step was reasonably straight forward (although time consuming). We went through all the SD and VD responses and analysed all their comments, assigning them to categories in terms of what they liked and what they wanted to see.

Two differences became apparent. The SDs are more inclined to describe it as a service to “Store Links”. The VDs are more likely to talk about “Search”. Is this the magic difference — do VD responders “get” that it’s an online memory and a way to rediscover links, while SD responders see it more just as a backup service?

We’ve got a lot of stats in the backend of Trunk.ly — we measure most actions that people take to help us understand how they use the service. So we were able to take the individual SD and VD responders (or at least those who had provided email addresses) and match their actual use of Trunk.ly to their survey response. At first glance, we’d found it!

SD Responders

Searches, timelines, tags, clicks

(‘0.62’, ‘0.76’, ‘1.81’, ‘1.71’)

VD Responders

Searches, timelines, tags, clicks

(‘6.21’, ‘3.14’, ‘2.79’, ‘6.93’)

Clearly VD responders use features more on average. Yay — we just need to turn the messaging around on the front page to highlight the searching more, job done. That felt a bit too easy though, and didn’t match my instincts and the people I’d been talking with — also in the comments on the survey, the difference between the SD and VD mentioning search was there, but not this significant. After some discussion, we decided to break it down even further.

Instead of using an average, we started looking at the actual values. This was really useful — turns out that in both SD and VD categories there are people who don’t use those specific features (so lots of 0's), but that when they do, they use it a lot. Perhaps this isn’t the difference (in fact logically it isn’t — when a large percentage of VD responders have never even DONE a search, then search can’t be the key differentiator, although of course there may be more than one factor influencing users).

We stared at the numbers a bit, then decided to break them down by number of links. Low and behold what we were actually measuring became clear! The more links you store, the more searches you do (and interestingly the less tags you use). This was true regardless of if you were an SD or VD responder. VD responders do have more links, but it’s too much of a leap of logic to say “making people store more will make them happier!”.

Intuitively to me this makes sense — the more links you have, the harder it is to find old links again, so search increases in importance. SD responders with lots of links are discovering search, but it’s not search alone which will turn them into VDs yet.

The end result of this was some improved stats so we have great benchmarks for today. We also know a lot more about the relationship between links and feature usage. It’s also told us what the difference ISN’T — which means we can move on look elsewhere. As always we’ve got ideas and this week we are trying to explore that and see what happens with the measurement, we’ll keep iterating and trying and measure it again in around 3 weeks time.

Highlights

• Trying the Trunkler iphone app for the first time. Really excited to see this fairly substantial product being developed by a third-party (Dave Cheong).
• Angel pitch — first large scale public pitch (normally do small groups), so good to stand up in front of 40 or so people.
• Alex has moved from Shanghai and settled into Dunedin, New Zealand. This will mean improved bandwidth and communication (and less Chinese Great Firewall for him).

Lessons Learnt

• When reading statistics don’t leap to conclusions and be sure you know what you’re actually measuring (long term experience with survey in my previous corporate life was useful here — it often isn’t what you think it is).

Goal this week

• Focus on features, but specifically around trying to uncover and understand this VD / SD difference.