What type of data are people really interested in? What type of data will people use to make changes in their programs?

Christina Christie set out to answer these questions in a study. She used simulated experiences with study participants to better understand what types of data most influenced them. Christie’s results, published in “Reported Influence of Evaluation Data on Decision Makers’ Actions: An Empirical Examination” (2007), used simulations with educational leaders to determine what types of evaluation data they use to make decisions.

To gather the data, Christie used simulations. She emailed electronic surveys to 297 alumni and current students in an educational leadership program. She also contacted 29 program directors from a county health department. The instrument consisted of scenarios that used three sources of evaluation information: large-scale evaluation study data, case study evaluation data, and anecdotes. The scenarios presented were based on a variety of programs such as pedestrian safety, substance abuse, and preschool programs.

After reading each scenario, the respondents were asked to choose which type of information would most likely influence their decision-making. The data were analyzed and the results were presented.

Results:

General large-scale and case study data were more influential to decision makers than anecdotal accounts.
Almost all respondents said that they were influenced by some type of evaluation data.
Christie suggested that evaluators specifically ask decision makers about their evaluation data needs prior to conducting the evaluation, so that the evaluator could provide information that will be most useful and influential.

Has your experiences with data used for decision making been similar to these findings? Are clients asking for large-scale and case study data?

DR. EVERETT’S USEFUL TIP: Ask the decision makers about their evaluation needs, but don’t forget that reports get shared. What may be influential for one group of people may not important to another. Knowing the other possible audiences for the report is important when planning an evaluation.