A sea of data washes across our desks each day – alluring in its seeming precision, calling out for attention and action. We want these numbers. We need them. We are seduced. We take them and run.
At our peril.
The challenge: It’s not all just numbers. Methods, questions and analysis all matter; survey data collected and reported without regard to best practices simply are inferior to those created with optimal approaches. The question is what should inform our judgment, and how.
Before we give data our hearts, we need to apply our minds: Exactly how were these numbers produced? By what methods, with what theoretical and empirical foundations? Who produced them, when and why? What truly are they telling us – and what are their limitations?
Our awareness of these questions grew from a lifetime of exposure to media misreporting of survey data. But while the news media long have indulged themselves in the lazy luxury of being both data hungry and math phobic, other data users are equally ill-equipped to confront the challenges ahead. And given the proliferation of data sources, the necessity of separating the wheat from the chaff never has been greater.
We stepped into the fore years ago, creating an industry-leading standards and vetting operation at ABC News. We set guidelines for the news division’s reporting of survey research; examine data of interest to establish its validity and limitations; and then, if acceptable, evaluate its meaning in context.
We offer the same consulting services to a broader clientele today. In evaluating existing research, our questions are simple – although the answers often turn complex. Who was interviewed, how were they reached, from what sample source? What exactly were they asked? How were their answers recorded, validated, weighted and analyzed? Are the data valid and reliable? Is the approach fit for purpose? Are the conclusions supported by the results?
It’s an approach with value to all data consumers. Advantage comes not merely from possessing data, but equally as much from stepping out of the endless stream to assess, evaluate, and come to a better-informed understanding of just what these numbers mean.