Category Archives: Predictive Analytics

Top takeaways from the Predictive Analytics Innovation Summit

Posted on by
Reply

I work in Digital Analytics. We are a relatively new field (or at least, a relatively new application of analytics) and one where I sometimes feel we don’t leverage enough of what’s been done before. Rather than re-invent the wheel, I headed off to the Predictive Analytics Innovation Summit in San Diego, to learn from a wider variety of industries, and see what they were doing with predictive analytics.

Here were a few of my key takeaways:

It’s not about the tools

I honestly thought this was just an affliction of the digital analytics industry, but it turns out, this is an issue in all area of analytics.

It’s not about the tools, it’s about two things –

1. The talent. Talent matters more than the tool. The “best tool” is one that you can hire great people to use. People are what will make analytics successful, not the tool.

2. Your needs. The “best tool” is the one that suits your business needs – not the most popular, the most expensive, or even the one with the most features. (If you don’t need those features, what’s the point?)

It is about strategy and culture …

Success with analytics isn’t driven by the tools you buy, the process you implement, or the technology you have – it’s about culture.

The creation and hoarding of data is not what will make you successful with analytics. What matters is the consumption of analytics by the organization, and allowing data to challenge beliefs and theories. It’s not about big data – it’s about using it to make big decisions. 

Case in point? Improving the efficiency of a predictive model is only worth a few percentage points in your model’s accuracy. But nailing your fundamental strategy? That’s where success comes from.

… and about communication …

 Analytics is also destined to fail if it’s not communicated well. This is often where hiring for analytics fails: too often, analysts are hired who can’t communicate with others in the organization. If anyone walks out of a meeting after an analytics presentation and doesn’t know the outcome or the next steps, that’s not their fault – it’s our fault.

Businesses today are in a difficult situation – we have too much data, yet too little knowledge. Analytics is critical to helping to understand what’s important, draw insight out of volumes of data, but without the right people and the right communication, you’ll never see that value.

Statisticians + Web Analysts = Awesomeness

Posted on by
2

One thing I have found can work very successfully is a hybrid team of web analysts and statisticians. When you combine the business and website knowledge that the analyst has with the “mad stats skills” that the statistician brings, you can create some truly powerful work.

There are a lot of different things that a web analytics team can leverage a statistician’s help for. This is by no means an exhaustive list, merely a place to get started.

1. Significance Testing

So you’ve run an A/B or Multivariate test. While your testing tool will likely also advise of you the statistical significance of your results, a statistician can dive deeper into this, and help you to measure significance outside of your tool. Perhaps you noticed shifts in site areas that weren’t one of your test success measures – a statistician can help you decide if these are merely interesting, or statistically significant.

Or perhaps you’ve tested in more of a time-series fashion. A statistician can try to tease out whether the change had an impact, or whether changes are due to seasonality. (This relates closely to the idea of an Impact Analysis.)

2. Impact Analysis

You make a site change, and you notice an increase in visits to a site area, or some key metric. You’re tempted to attribute this entire shift to the site change. (“Woo hoo! We’re up 5%!”) However, what about changes in marketing spend? Seasonality of your site traffic? Social initiatives? Are you taking those into account before reaching your conclusion?

A statistician’s analysis can attempt to tease out those additional variables to estimate the impact of the actual site change, vs. these confounding variables.

This same approach can be used to measure the impact of industry events or company changes (outside of the website) – anything, really. The benefit here is a better understand of the actual impact of events or initiatives, but a nice perk should be presenting your findings to the business and not having to freeze like a deer in headlights if someone says, “Yes but we spent another million dollars in paid search last week – did you factor that in?”

3. Standard reporting automation

Statisticians can use tools such as SAS to fetch data from FTP, combine and compute it, and deliver outputs to your system of choice (for example, Excel, if that’s somewhere you’re comfortable working.) This can allow you to schedule FTP delivery of SiteCatalyst reports, Discover reports, ad server reports (etc) – basically data from multiple sources – have SAS do the work of fetching multiple data sets, combine them and output to Excel.

That, however, doesn’t mean you need to deliver a huge scary data sheet to the business. On top of the data, you can build  a more user-friendly view (preferably formula-driven, so that you’re not manually updating!) in Excel to present the data.

This allows you to take a lot of the manual part (copy-paste, copy-paste) of standard reporting out the equation, and focus your time on explaining the shifts you might be seeing in the report. e.g. Perhaps traffic to a specific content area is down – start digging in. What traffic sources are driving it? Are there particular pages experiencing a more dramatic shift?

In addition, once the business sees the value of this work (the time it frees up for analysts to actually analyse!) it may actually help  argue for further automation and investment in further tools. So make sure you provide those insights, and use this work to prove why you shouldn’t spend your time copy-pasting.

4. Forecasting

Statisticians can build forecasting models to predict your site traffic, sales, ad impression volume – pretty much anything. You can go short-range, or long-range. Perhaps a simple “forecast through end of month” will suffice to start, or maybe you want to start forecasting three or six or twelve months in advance.

So why would you do this? Well, good analysts know that data needs context. That’s why we have KPIs, or compare month over month, year over year – to understand whether “2.6%” is “good”. Comparing to a forecast can be another way to get context for your data. If you’re diverging from your forecast, you can start digging in to see why. This divergence might be good – perhaps you saw a better than expected responses to your marketing initiatives. But on the flipside, you might also need to frantically search for why you’re suddenly down -10% compared to forecast …

Even a through-end-of-month forecast can be helpful here. An EOM forecast will tell you where you’ll likely end the month, based on current performance – even though you’re only on day 9. This will allow you to course correct throughout the month, rather than waiting till end of month to realise you didn’t match your forecast.

If your business sets site goals, forecasts can be the first step. First, forecast where your business will be for the next twelve months without any major initiatives. Simply assume the status quo. Then, look at the initiatives you want to add on top of that, and assess how much of an impact they may have. Forecast + specific initiatives = your goal. A statistician can also help you look back over time at previous initiatives and analyse their impact, to make sure that you’re not overstating how big an impact something new may have. (How many times have you heard “This is a game changer!” and found it barely moved the needle?)

There are still things you need to keep in mind when forecasting, but even starting small can bring value to your business.

Group Hug!

Still, analysts and statisticians may sometimes face some hurdles. Analysts need to learn the language of statisticians, and statisticians need to either learn the business, or be guided  by the analysts. A statistician exploring data with no understanding of the business, the website, or what any of it means normally doesn’t reveal great insights. On the flip side, the analyst really needs to start learning and at least dabbling in the world of statistics, and be able to translate complex concepts for the business users you support.

However, a cohesive team that learns to work together and leverage each other’s strengths can do amazing things.

Don’t have access to a statistician? Students often need real-life data for school projects. Consider seeking one out! (Who knows – you might find yourself a great future employee.)

A few thoughts on forecasting

Posted on by
2

It seems fitting that my first post should involve something that occupies a tremendous amount of importance (and potentially, debate) within an organisation (and certainly involves a lot of sleep loss for me personally!)

Forecasting.

If you are new, or fairly new, to forecasting on your website, I’ll share some hard-learned truths.

You’re always going to be wrong. Always. The very nature of a forecast means you will always be wrong … and that’s okay. (Don’t get me wrong. When it actually happens, it’s thoroughly depressing, and often has you chasing your tail to find out why, but it’s still okay.) Your aim is just to be wrong as little as you can – and to be able to identify why your resulting actuals are off from your forecast. I would argue that divergence from your forecast today will make tomorrow’s re-forecast better, but only if you can explain it and learn from it.

Your forecast is only as good as the information you have. As you look back over your site’s history, if there are events you can’t explain, or inputs into your traffic that you fail to identify, your forecast will be more off than you would like it to be.

Don’t have all the information you need? Get it. Gather anyone/everyone/any information or inputs you need. If the company’s eyes are on your forecast, their gaze will stray if you frequently prove too far off. If they’re not yet looking at your forecast, they won’t ever start if you can’t prove your history of accuracy.

There is a difference between a forecast and a goal. As you start forecasting site traffic, ad space, conversion rates, lead generation (etc), you’ll need to explain this. Many times. Your forecast will be based on your site’s history and its inherent trends. But as an analyst/statistician/forecasting guru, you alone can’t identify everything that will happen in the future. (And if you can, give me your number – I have a few questions I would love answered.)

This is where your business/development/product team come in. Your forecast can estimate where your site will be in the future, based on your current trajectory. But you need inputs from others to anticipate future planned growth, that’s not evident in the data.

Let’s say your forecast suggests your site will be up 5% year-over-year. If your executive team want your site to  be up 20%, you need your business/development/product team to either a) temper this expectation, if it’s not reasonable, or b) advise how they will achieve this. If your fine-tuned, well-informed forecast suggests you’ll be up 5% year-over-year, 20% is not a forecast, it’s a goal. You can’t reach 20% YOY without at least a basic idea of how you’ll get there, and any attempt to incorporate it into your forecast without a skeleton of a plan will reflect poorly on your forecast, rather than reflecting on the failed execution of the plan for growth.

The moral of the story? Forecasting can’t occur in a silo. Analysts, business and executives must all get their “feet wet” to produce something that all are comfortable with and can rely on.

Not forecasting (yet)? Even if it’s rudimentary (forecasting your basic web traffic metrics only, e.g. Visits, Unique Visitors, Page Views) get cracking. It’s wonderful to be able to analyse, segment and test your history, but your business and executive teams will really appreciate even a lightly dotted line of where they’re headed.