Week 11: Avoiding misuse: The evaluator’s perspective

Have you heard, “I only want the evaluation report to talk about what is good about our program…” or “doesn’t the fact that our participants improved in reading mean we caused that?” Oftentimes these questions are spawned out of a genuine belief that what a program is doing must in fact be having a positive impact on those it serves. Less often a program manager or other evaluation consumer purposefully coerces the words of the evaluation to make the conclusions sound different.

Last time I posted, I introduced a framework developed by Brad Cousins (2004) which outlined a way to understand and categorize different forms of use. Today, I want to talk briefly about how to prevent or address misuse when it does happen, from the perspective of an evaluator.

Previously, I talked briefly about an instance of misuse described to me by my professor when I first started my doctoral work in evaluation. A former client had taken a surgical knife to his words from an evaluation report he had produced for them and posted them on their website. It went something like this: “This curriculum can improve science and math scores, as proven by Dr. So-And-So in her recent report…” The reality was nothing even close to that. In fact, the evaluation report had not found conclusive evidence that the curriculum had any effect at all.

In this instance, there is not much to do on the part of the evaluator except to issue a statement. When an evaluator completes a report and submits it to their client, that client then owns the information. Hypothetically, the client can do what they please with it. What follow are a few tips I have picked up along the way about how to avoid this:

  1. Be clear and up front about the limitations of the evaluation and what types of conclusions can be made.
  2. Let the client know what they can say from the evaluation. By pre-empting a client’s inevitable desire to use the evaluation as proof of impact, you can deliver talking points around the evidence the evaluation has produced.
  3. Budget time for interpretation with your key stakeholders. By including money in the budget for time to review the report, and even facilitate discussion about the findings, you can be involved in the interpretation process and help prevent your stakeholders from reaching invalid or poorly supported conclusions.
  4. The evaluator’s involvement in the interpretation process can also help when clients are hoping to make decisions. As evaluators, we do not want our evaluation results to lead to unjustified firings or cuts to programs. Similarly, we don’t want our findings to be used to justify the continuation of a program which is not good. The evaluator’s involvement in this process helps prevent these types of situations.     

It is on us as evaluators to be clear about the strengths and weaknesses of the evaluations we do, and to work with our stakeholders to interpret these findings.

COREY’S USEFUL TIP: Budget and plan for time to work with your key stakeholders on data interpretation. This could be done with a draft report, which also helps you as the evaluator develop more valid and relevant conclusions and recommendations. You can also refer to steps 15 and 16 in Patton’s Utilization-Focused Evaluation Checklist available here http://www.wmich.edu/evalctr/wp-content/uploads/2010/05/UFE.pdf for guidance on how to get started thinking about the interpretation process

COREY’S USEFUL TIP: Budget and plan for time to work with your key stakeholders on data interpretation. This could be done with a draft report, which also helps you as the evaluator develop more valid and relevant conclusions and recommendations. You can also refer to steps 15 and 16 in Patton’s Utilization-Focused Evaluation Checklist available here http://www.wmich.edu/evalctr/wp-content/uploads/2010/05/UFE.pdf for guidance on how to get started thinking about the interpretation process

If you have any useful tips for addressing some of the issues I raised today, PLEASE share them in the comments section.