, , ,

Seeing Progress and Making Decisions Through Instrumentation

I was thrilled to see great progress with one of the startups I’m advising. In our meetings early on, there were questions from the entrepreneur and her team about what to do and where to go next. It was difficult as an outsider to give a good direction without specific industry experience. Instead, I recommended some best startup (business) practices including the importance of instrumenting their platform and using data to drive roadmap and business decisions.
In one specific example, the company wanted to immerse users in the experience of the platform. Their initial UI hid away menu options. However, they found users were not customizing their experiences very often. They finally knew, however, how long users stayed on the platform, what they did, etc. The team hypothesized that users would engage with the platform even more and for longer periods of time with more personalized experiences through customization.
So, the team started making cues to the menu options including explicit instruction during first-use onboarding. It worked. Engagement of the new menu increased as has customizations. Their goal of greater and longer app usage also increased.
Users are now seeing buying options more often. Their next goal is to drive increased sales.
The team is studying engagement data to help answer questions, create hypotheses, and make decisions to accomplish specific goals. It’s great to see progress.
P.S. You can check out related articles on instrumentation and metrics in “Don’t Know What Metrics to Track? Start Asking the Right Questions”, “Metrics vs. Instrumentation”, and “Metrics for the Early Stage Startups”.
/by
,

Don’t Know What Metrics to Track? Start Asking the Right Questions

(Scene from I, Robot. image source: https://i.imgflip.com/fwri0.jpg)
It’s often best to start with what goals you want to achieve when deciding what metrics to track and events to instrument. This, then, drives key questions. So welcome to today’s post – the prequel to Tuesday’s “Metrics vs. Instrumentation”.
Almost every facet of a business – technical, business, financial, operational, etc. – should have some strategy in place for improvement. As such, there is a need for an understanding of where the company sits today, in the past (if available), and the goals for the future. The natural thing to do, then, is to ask the right questions.
Here are some questions that can help determine the instrumentation and metrics needed:
  • At what rate are we losing existing customers (churn)? à can highlight product-market fit issues, education issues, misaligned expectations, complex UI/ UX, etc.
  • What is the lifetime value of a customer? à help determine the ROI of marketing strategies, especially against the cost of acquisition
  • What is our server uptime service level? à understand if service levels and how that may impact existing customers
  • How often are our users using Feature X? For how long? à help determine roadmap and UI/ UX issues, unnecessary feature sets, etc.
  • How are we acquiring website visitors, and at what rate are we able to convert these visitors to some call-to-action? à test the efficacy and messaging of marketing efforts
  • How often are users visiting our knowledge base? Is there a particular article that is visited most often? àbetter upfront knowledge sharing/ education, better UI/ UX opportunities

Goals and strategy of the company leads to questions on where the company sits today, and how to achieve tomorrow. Then, instrument as needed to capture status and improvement.

/by
, ,

Metrics vs. Instrumentation

I heard some confusion recently about: what exactly are metrics? What’s instrumentation? How do they differ?
Definitions from dictionary.com)
  • Metric: a standard for measuring or evaluating something, especially one that uses figures or statistics
  • Instrument (-ation): a means by which something is effected or done; a device for measuring the present value of a quantity under observation.
With yesterday’s 4th of July and 47th Annual Peachtree Road Race, I’ll use running as an example to illustrate the differences.
Important metrics for runners:
  • Time
  • Distance
  • Average speed
  • Heart rate
  • VO2 max (lung capacity)
Examples of how to instrument:
  • GPS (for distance)
  • Heart rate monitor
  • Stopwatch (time)
  • Metabolic cart (VO2)
As you can see, instrumentation (“instrument”) is the way to which results can be captured to provide insight — metrics. They are not the same, but rather, one feeds and enables (instrumentation) the other (metrics).

Instrument early on to answer questions and drive the startup forward — backed by data. Check out these 8 metrics for startups from proper instrumentation.
/by
, , , ,

8 Metrics from Proper Instrumentation of a Business and Its Product

I want to continue my post from Tuesday about the importance and value of instrumentation. Today, I want to share SaaS metrics that can be answered with proper instrumentation (operational and business).
  1. Cost of acquisition. This is cost to capture an unaffiliated buyer. Need to know the costs associated with closing this opportunity including marketing costs, engineering support, etc. For this metric, it’s important to track the flow and behaviors of a customer through websites, sales touches, etc.
  2. D1, D7, D14, D30 retention metrics. Here, D stands for “day” and the number refers to the number of days since a user first enters the system. This metric tracks the percentage of returning users to the service in D days –gauge “stickiness”.
  3. Open and Click-Through Rates of Emails. Many products these days have email engagement and nurture campaigns. Here, companies measure if users are opening these emails and, if applicable, are they clicking into a destination the company is looking for.
  4. Drop-off During Sign-up Process. Many products have multi-stage sign-ups which can deter and annoy users from completing sign-up. By measuring here, the company can quickly ascertain if the sign-up process needs to be simplified or be very valuable to motivate complete sign-up. If they never enter, they’ll never see the great product! (This, by the way, is why so many apps use Facebook, Twitter, Google login… plus, companies get personal data shared from those platforms.)
  5. In-App Engagement.This is a big bucket including what pages, tabs, profiles, features are viewed and used. You want to understand how users interact with the product – are they finding pages useful? Are features cumbersome?
  6. Customer Lifetime Value. Same concept of the revenue of a customer (or net profit) but extrapolated against the number of times a customer buys (subscription, multiple products, etc.).
  7. Churn.That is, what percentage of customers stop buying annually? Good annual churn for SaaS businesses according to Sixteen Ventures is 5-7%. High churn may point to poor value, mismanaged expectations, or an inherent problem in the product.
  8. Average Revenue per Customer. To be explicit, it’s total revenue divided by customer. In this case, revenue would naturally be weighted by where most of revenue comes from.

I get excited when a company properly instruments their products and services. It demonstrates tremendous maturity and understanding to recognize engagement data will drive confirmation (or rejection) of hypotheses, and thus, enables smarter business decisions.

What are some other metrics you find useful? How would you measure success in your company?
/by
, , ,

What Gets Measured Gets Improved: Tools for Instrumentation

One of the chapters of my book is all about measuring engagement and implementing analytics – chapter aptly dubbed, “What gets measured gets improved.”
The lesson is to “instrument” an application to gather data points of how users interact with the product. Quickly, you can assess if users are traversing all the steps of a Getting Started Wizard, getting stuck while building out a team, or exiting a page with above-average frequency.
We implemented Google Analytics at Body Boss which delivered anonymized data at an aggregate level – useful, but doesn’t give finer perspective into engagement. More powerful instrumentation would including capturing “events” to every actionable UI element (i.e. button), page, etc. For example, in Twitter, events would include when a user navigates to a user’s profile, scrolls down, Likes a tweet, and then searches for another user.
There are many fascinating tools for instrumentation on web pages, in-app uses, and beyond. Here are a few I’ve recently got to play with:
  • Fullstory– records behind the scenes how users interact with a site or app that can be replayed later. You can see where a user hovers his mouse, scrolls, stays on some block of text, etc.
  • Unbounce – taking A/B testing to multi-variate testing for landing pages. Quickly set up a landing page with multiple variants, and Unbounce automatically directs visitors and tracks conversions.
  • Pardot – Easily send automated mass messages personalized to recipients based on where users are in the sales funnel. Tracks users from first site visit and beyond for nurture campaigns
  • MixPanel/ Intercom – Very different in how each operate, but the feature I liked most was being able to trigger specific messages (more granular than Pardot) based on user interactions. High-degree of control by building out event-driven rules and trigger notifications.
  • Kevy – Marketing automation for ecommerce stores. Slick tool to understand consumer behavior and enables stores to better market to consumers by offering coupons, messages, and the like based on rules and triggers.

In gist, there are lots of tools available for instrumentation with overlapping features. It’s fun learning about these tools now, and dreaming of how great these would have been at Body Boss. Though, several tools didn’t exist three years ago… inherent problems don’t change, but solutions do.

What are your thoughts on instrumentation? What tools do you use?
/by