Data and creativity: A response to Timothy Egan

The New York Times columnist Timothy Egan wrote an odd and frankly troubling piece over the weekend: “Creativity vs. Quants.” In it, he argues that the move towards data journalism, and the general quantification of everything, is somehow opposed to creativity.

I must object. Data analysis and everything that goes into it can be highly creative. Like John Lennon — Egan’s leading example of a creative — I also get sudden and unexpected inspiration when I take a walk after having struggled with a data analysis problem for hours. I suspect many other data analysts do as well. Creativity is not limited to writers, musicians and satirists (as Egan implies).

Anyone who has taken an introductory econometrics class would immediately understand how much art there is to the alleged science of data analysis. At each step of the way there are creative choices to make: which data to choose, how to categorize it and arrange it, which questions to ask of the data and which methods of analysis to use. Sometimes the data you want are not available. Can you find ways to get new data? Data analysis is constantly evolving, led by some remarkably creative minds.

Because there are many ways to approach data analysis, there is often vigorous argument about how best to measure and estimate various outcomes. While election prediction has certainly become more scientific, there are a variety of different election models, each with their own creative choices. Some may fare better than others. And we can argue and debate different approaches. Prediction makes for good science, because it is falsifiable.

Certainly, there is plenty of bad, even sometimes dangerous, data work out there. For example, one can partly blame the financial crisis on poor modeling and analysis – some of which, by the way, was highly creative. And just as Orwell’s classic “Politics and the English Language” warned against the mindless use of familiar clichés and the abstruse complications of unnecessarily baroque phraseology, so data analysts can also mislead and confuse by using inappropriate or unnecessarily complicated methods — either because they don’t understand what they are doing or because they are more interested in showing off than communicating (and making it harder for others to check their analysis). Just as there are rules of good and bad writing, there are also rules of good and bad data analysis. And just as great writing requires some inspiration, so does great data analysis.

Certainly, there are limits to what data can do. As Einstein put it, “Not everything that can be counted counts, and not everything that counts can be counted.” But as we make more data available, we can reach better and more accurate conclusions. At Sunlight, we believe the more public data we have, the better our public sector will be. In part, this is because more data creates an opportunity for quants and hackers and other interested citizens to think creatively about both how to improve public service and hold public officials accountable.

And as for Egan’s claim that quants can’t do satire, he has obviously never read XKCD (as my friend Dan Hopkins pointed out).

None of this is to diminish what Timothy Egan does. He is generally an excellent writer, and I often enjoy his columns. But qualitative journalists and creative writers like Egan have far more to gain by working with data and data analysts than by dismissing them. There is no reason why science and the humanities have to exist as two separate cultures. They can inform and challenge each other to both be better.

At Sunlight, we view data collection, analysis and presentation as fundamentally creative work. We wish everyone shared that view.