Expectations management in Data Science projects

I’m going to write a bit about expectations management in Data Science projects. Now this is something that the academic side of my personality finds a bit ‘business-speak’. However it matters in commercial environments (and probably in non-commercial environments like not-for-profits too).

It’s worth sharing a Dilbert comic. Just because it’s always relevant when discussing ‘business-speak’.

Copyright: Scott Adams and Dilbert

So the following are some tips I’ve learned. The reason I share these is that managing expectations is a difficult part of the Data Science process. Research and Development type work and even software work are hard, they’re hard to manage, we live in a world of rather inflated expectations – and a lot of commentary (which is often specious and rubbish) like ‘AI will steal our jobs’ doesn’t help matters. Nevertheless this is the world we live in. I’m not perfect at this, and this is a skill I’m still working on, but I wanted to write down some notions and frameworks I’ve thought about.

Firstly focus on simple stuff. Even delivering a data set or some kpis can be non trivial in an environment with poorly explored data. This also helps you grok the domain. A good strategy is to send updates to people – whether it is email updates or whatever. I also like to print out graphs and stick them on the wall, so people can see them.

Secondly deliver things iteratively as you go along. The political capital you lose from a 6 month R and D project can often be not worth it. Compare for example the insights you get from some SQL based product analytics – which could be say 20 insights in 6 months, and the insights from one recommendation engine – which may take over a quarter to see a ROI.

Thirdly deliver negative results too. “We discovered this feature was not strong enough” or there was no signal here. Jonathan Nolis writes a great blog post on this – it’s been my experience actually with some ambitious projects that sadly, it just won’t work. Experimentation involves accepting positive results AND negative results.

Sometimes we believe there is more signal in the data than there actually is! This is ok.

Fourthly – Build a very close working relationship with business partner, so you live in their shoes & make them part of the journey so even small decisions are joint. This came from my friend Padraic and it’s really important. Showing KPIs or analysis to a Business stakeholder – helps them understand the process, and really helps you as a Data Scientist learn what’s important and what each field REALLY means.

Let’s share a concrete example – in one project I did, it turned out that repayments weren’t all stored in the same column – so there was some funky SQL and business process understanding needed to really filter all the repayments. It was very easy to get a misleading revenue number – say revenue plus repayments, if you didn’t do that. I learned the hard way that showing the results quickly to a business stakeholder was important.

I’ve some other thoughts but those are my main ones for the moment.  It’s very easy to say YES to the exciting R and D project. However it’s not your job to say yes to that, it’s your job to deliver value to the business. Focus on that, not on $SHINY_TOOL

I’d like to thank conversations with James Stanier, Ollie Glass, Mick Cooney and Rob Horton that has shaped my thinking on this.