How to measure visitor engagement, redux

How to measure visitor engagement, redux

Back in December of last year when I first posted on measuring visitor engagement, I hardly imagined how much interest the topic would generate. Shortly after the first post, I commented thatmy definition of engagement was as follows:

Engagement is an estimate of the degree and depth of visitor interaction on the site against a clearly defined set of goals.

I then went and wrote over a dozen posts, publishing feedback from some incredibly bright peopleand demonstrating the utility of a well-defined measure for engagement. Since that time, however,some have questioned the value of such a metric and thusly prompted me to update and publish the following calculation for visitor engagement:

I presented this calculation to a completely full room last week at Emetrics but wanted to provide an update to all my patient readers who were not able to make the event. You can download my entire Emetrics on “Web Analytics 2.0″ which includes the slides on measuring visitor engagement from the White Papers and Presentations section of my site.

I very much believe that engagement is a metric, not an excuse, and that the metric described in this post provides a powerful measurement framework for sites looking for new ways to examine and evaluate visitor interaction. I know that for my own site, the use of simple measures like “bounce rate”, “conversion rate” and “average time spent” is simply insufficient for selling anything other thanmy books. But I’m now in the business of selling consulting, a complex and sometimes time-consuming sale, and so I’m always on the hunt for any web analytics measure that will give me an edge and help identify truly qualified opportunities.

I believe this metric is exactly that.

This post is an extension of the work I did in late 2006 and early 2007 and was written to clarify my position, update my thinking in the context of “Web Analytics 2.0″, and reiterate my desire to have an open and honest conversation with my peers and other interested parties regarding the measurement of visitor engagement. Web analytics is hard but not impossible; the same is true regarding the calculation and use of robust measures of visitor behavior.

I believe the visitor engagement measurement to be perhaps the most important of all “Web Analytics 2.0″ measurements. Given that this model fully supports both quantitative and qualitative data, and given that the model is build as much around the measurement of “events” as much as page views, sessions, and visitors, I (perhaps haughtily) believe this calculation to be prototypical of the types of measurements we will see as we continue to explore the boundaries of “Web Analytics 2.0″ (download my presentation from SEMphonic X Change).

The Web Analytics Demystified Visitor Engagement Calculation

The latest version of my visitor engagement metric, with notes about its calculation and use, are as follows. If you’re too busy to read this entire post but would like to learn more about this measure,please write me directly and we can set up a time to discuss it.

This is a model, not an absolute calculation for all sites. I agree with other analysts and bloggers who insightfully say that there is no single calculation of engagement useful for all sites, but I do believe my model is robust and useful with only slight modification across a wide range of sites. The modification comes in the thresholds for individual indices, the qualitative component, and the measured events (see below); otherwise I believe that any site capable of making this calculation can do so without having to rethink the entire model.

The calculation needs to be made over the lifetime of visitor sessions to the site and also accommodate different time spans. This means that to calculate “percent of sessions having more than 5 page views” you need to examine all of the visitor’s sessions during the time-frame under examination and determine which had more than five page views. If the calculation is unbounded by time, you would examine all of the visitor’s sessions in the available dataset; if the calculation was bounded by the last 90 days, you would only examine sessions during the past 90 days.

The individual session-based indices are defined as follows (and these are slightly updated from past posts on the subject):

• Click-Depth Index (Ci) is the percent of sessions having more than “n” page views divided by all sessions.
• Recency Index (Ri) is the percent of sessions having more than “n” page views that occurred in the past “n” weeks divided by all sessions. The Recency Index captures recent sessions that were also deep enough to be measured in the Click-Depth Index.
• Duration Index (Di) is the percent of sessions longer than “n” minutes divided by all sessions.
• Brand Index (Bi) is the percent of sessions that either begin directly (i.e., have no referring URL) or are initiated by an external search for a “branded” term divided by all sessions (see additional explanation below)
• Feedback Index (Fi) is the percent of sessions where the visitor gave direct feedback via a Voice of Customer technology like ForeSee Results or OpinionLab divided by all sessions (see additional explanation below)
• Interaction Index (Ii) is the percent of sessions where the visitor completed one of any specific, tracked events divided by all sessions (see additional explanation below)

In addition to the session-based indices, I have added two small, binary weighting factors based on visitor behavior:

• Loyalty Index (Li) is scored as “1″ if the visitor has come to the site more than “n” times during the time-frame under examination (and otherwise scored “0″)
• Subscription Index (Si) is scored as “1″ if the visitor is a known content subscriber (i.e., subscribed to my blog) during the time-frame under examination (and otherwise scored “0″)

You take the value of each of the component indices, sum them, and then divide by “8″ (the total number of indices in my model) to get a very clean value between “0″ and “1″ that is easily converted to a percentage. Given sufficient robust technology, you can then segment against the calculated value, build super-useful KPIs like “percent highly-engaged visitors” and add the engagement metric to the reports you’re already running.

The Visitor Engagement Calculation in Detail

The Click-Depth, Recency, and Duration indices are all pretty straight forward and are more-or-less the traditional indicators that most people (incorrectly) call “measures of engagement”. Each of these are very important to the overall calculation, but none of these alone are sufficiently robust to describe “engaged” visitors. I set the “n” values for my site’s calculation based on the average value for each and this seems to work pretty well (meaning my Ci looks for sessions more than “5 page views” in depth, my Ri looks for sessions more than “5 page views” that occurred in the “past three weeks” and my Di is looking for sessions longer than about “5 minutes” in length.)

Brand Index is a little more complicated. Here I have made a list of all the terms I believe to be “branded” for my site and business, terms like eric t. peterson, web analytics demystified, web site measurement hacks, web analytics wednesday, and the big book of key performance indicators. Whenever a session begins either with no referring domain or comes from a search engine with one of these terms attached, I count this as a “branded session” and score appropriately. While this index perhaps unfairly weights towards search engines, I firmly believe that if you’re starting your session with either my branded URL, my name, or the name of one of my books that you are already engaged.

Feedback Index is the sole qualitative input to this model but it can easily be expanded if necessary. Here I am simply scoring sessions based on whether visitors are providing qualitative feedback via the OpinionLab “O” present throughout my web site or writing me directly by clicking a “mailto:” link. I’m not looking at whether the feedback is positive or negative, only whether feedback was given, operating under the belief that anyone willing to provide direct feedback is engaged.

The Feedback Index could easily be expanded by scoring based on the answer to direct questions posed to the visitor, questions like “do you find the content on this site valuable?”, “do you plan on calling Web Analytics Demystified about consulting?” and “would you described yourself as engaged with this site?” Given a sufficiently robust mechanism for making the calculation, the Feedback Index can provide a tremendously powerful input to the visitor engagement model.

The Interaction Index captures sessions in which specific “engaged events” occur other than the site’s primary conversion event — events like downloading a white paper, providing an email address, requesting a presentation or PDF, commenting on a blog post, Digging a post, emailing content to a friend, printing a page, etc. The Interaction Index is designed to capture a small weighting from those measurable goals on your site you believe to be indicative of engagement.

The Interaction Index specifically does not examine commerce transactions and other conversion events of fundamental import to the site. While I have debated this in the past, here is the rationale for recommending the exclusion of primary conversion events:

1. These events already have their own key performance indicator: conversion. Given that conversion is likely already defined for most transactional sites and tracked in great detail, adding conversion to the visitor engagement calculation is superfluous in my opinion.
2. The visitor engagement metric is designed to provide information about the large number of visitors who do not convert. Given relatively low conversion rates online, having visitor engagement be decoupled from conversion provides a cleaner measure for use in exploring non-purchaser behavior, including looking for independent correlation between the two measures.
3. By excluding conversion, the two metrics can be used side-by-side to look for visitor behaviors may not be obvious otherwise. Given the lifetime of possible visitor behaviors, having a way to look for well-engaged visitors who have not completed a transaction online or have completed a transaction outside of the available data set provides a critical view not otherwise readily attained.

The Loyalty Index is a reflection of my belief that repeat visitation behavior is perhaps the best measure of engagement available. Based on the distribution of visitor loyalty data at Web Analytics Demystified, I score “1″ when visitors have come to the site more than five times in the past 12 months.

The Subscription Index is a reflection that truly engaged visitors are able to self-identify by subscribing to our blogs or newsletters; if you have taken the time to subscribe to one of the Web Analytics Demystified blogs I believe you to be engaged. If your site does not have some type of XML-based content subscription you can either drop this index or (perhaps better) look for an opportunity to develop a subscription service, thusly giving your visitors another good engagement point.

How Does This All Work in Practice?

Careful readers will likely have already figured out that as visitors come to your site over time, their cumulative “lifetime engagement score” changes as they satisfy the criteria of each individual index. So someone coming from a Google search for “web analytics demystified” who looks at 10 pages over the course of 7 minutes, downloads a white paper and then returns to my site the next day will have a higher visitor engagement value than someone coming from a blog post who looks at 2 pages and leaves 2 minutes later, never to return.

If you think about it for just a bit, and consider the components in the full calculation, the visitor engagement metric starts to make an awful lot of sense. Consider the following:

• A visitor can quickly move through a lot of pages, getting exactly what they need, and still be scored usefully through the Click-Depth Index
• A visitor can slowly and methodically read a few pages and be scored usefully through the Duration Index
• A visitor can come to the site frequently and do little more than read a single page of content and be usefully scored through the Recency and Loyalty Indices
• A visitor can come to the site once, subscribe to the blog, return later and download a presentation, and be usefully scored through the Subscription and Interaction Indices
• A visitor can come to the site, click on dozens of pages but fail to find what they are looking for, then tell me so using my feedback mechanisms and be usefully scored through the Click-Depth and Feedback Indices

The power of the metric is appreciated when you apply it to the commonly measured dimensions found in web analytics: referring domain/URL, search engine/phrase, campaign/placement/creative, content group and page, browser/operating system, etc. Suddenly instead of looking at simple measures, you’re examining the potential of visitors coming from or going to each element in the dimension. To see the metric in action, I encourage you to read my post on the gradual building of context, at least until I’m able to publish new screenshots later this week.

Some Parting Thoughts about Measuring Visitor Engagement

Some folks have complained that this metric is “not immediately useful”, that nobody will understand it, and that it is impossible to calculate. Perhaps, but I would argue that A) no metric is truly immediately useful and B) most people don’t understand web analytics because web analytics is hard. The assumption that a diverse organization is going to be more successful using “bounce rate” because it can be glibly explained by saying “your content sucks” is just wrong — all of this stuff needs to be explained regardless of the complexity of the metrics involved.

Regarding the metric being impossible to calculate, it fully depends on which application you’re using. If you’re trying to get by using free tools then yes, you’re out of luck. But if you’re using robust tools like the high-end offerings from Unica, IndexTools, Visual Sciences, and WebTrends then you should have little trouble using the metric I describe in this post.

I personally believe that Web Analytics 2.0 both requires and allows us to be more creative and thoughtful in our use of metrics. Why not use a robust indicator if one is warranted? Especially if you’re not selling anything online, or if you’re selling high-consideration items, my visitor engagement metric can be shown to be an extremely powerful measurement.

Given the assertion that some consultants are apparently charging $200,000 USD for complex “engagement index” work, and given that someone working for Google is in the process of trying to patent a much simpler version of this equation, I am happy to give my work away to the entire industry in an effort to promote the use of more meaningful metrics to be brought to bear on increasingly complex measurement problems.

What do you think? Did you see my Emetrics presentation and still have questions? Did you read every word of my series on engagement and still not believe me? Do you need to see engagement in action before you’re willing to say it’s not just an excuse? Or are you chomping at the bit to have a robust measure like this for use on your own site?

Especially on this subject I relish your feedback, either via comments or via email — your choice! I find the subject fascinating and welcome the opportunity to discuss it you, my (hopefully) engaged readers.

Post Date:

Monday, October 22nd, 2007 at 9:27 am

Categories:

Engagement, Key Performance Indicators, Presentations, The Engagement Project,Web Analytics 2.0

Subscribe:

RSS feed for comments on this post

Interact:

53 responses | Add to the conversation | Trackback URI

Source: http://blog.webanalyticsdemystified.com/weblog/2007/10/how-to-measure-visitor-engagement-redux.html