A sea of data washes across our desks each day – alluring in its seeming precision, calling out for attention and action. We want these numbers. We need them. We are seduced. We take them and run.
At our peril.
The challenge: It’s not all just numbers. Methods, questions and analysis all matter; survey data collected and reported without regard to best practices simply are inferior to those created with optimal approaches. The question is what should inform our judgment, and how.
Before we give data our hearts, we need to apply our minds: Exactly how were these numbers produced? By what methods, with what theoretical and empirical foundations? Who produced them, when and why? What truly are they telling us – and what are their limitations?
At Langer Research Associates our awareness of these questions grew from a lifetime of exposure to media misreporting of survey data. But while the news media long have indulged themselves in the lazy luxury of being both data hungry and math phobic, other data users are equally ill-equipped to confront the challenges ahead. And given the proliferation of data sources, the necessity of separating the wheat from the chaff never has been greater.
We stepped into the fore years ago, creating an industry-leading standards and vetting operation at ABC News. We set guidelines for the news division’s reporting of survey research; examine data of interest to establish its validity and limitations; and then, if acceptable, evaluate its meaning in context.
We offer the same consulting services to a broader clientele today. In evaluating existing research, our questions are simple – although the answers often turn complex. Who was interviewed, how were they reached, from what sample source? What exactly were they asked? How were their answers recorded, validated, weighted and analyzed? Are the data valid and reliable? Is the approach fit for purpose? Are the conclusions supported by the results?
It’s an approach with value to all data consumers. Advantage comes not merely from possessing data, but equally as much from stepping out of the endless stream to assess, evaluate, and come to a better-informed understanding of just what these numbers mean.
Another of our consulting practices focuses on one of the most important and challenging decisions we make: hiring staff. We’ve honed our approach during the past 15 years by developing our Skills Drill, an in-depth assessment of candidates for survey research positions.
The timed drill draws on our experience in the skills required for success in our field. Subject areas, customized for the position offered, may include project design and management, sampling, statistical analysis, question writing and evaluation, questionnaire design, data assessment and analysis, report writing, PowerPoint presentation skills, editing, proofreading and related tasks. Knowledge, precision, creativity and critical thinking are evaluated, all from a survey research perspective.
We’ll consult with you to develop an appropriate drill, administer it to your selected candidates and grade their efforts. In addition to individual grade reports, we’ll deliver a narrative summary comparing candidates’ stengths and weaknesses and a spreadsheet for easy comparison of scores.
Our assessment is an invaluable tool at two stages of the hiring process – initially in identifying top-tier candidates for further consideration, and subsequently in selecting among finalists. Clients also have used our drills to assess skills among current staff members for promotion and training purposes.
Contact us for assistance with this crucial task.