Week 51: Evaluator Qualifications and Use – The Doctoral Student

This week is the final post of a four part series where each of us took a turn to talk about our personal evaluation qualifications and how we believe those qualifications have helped us provide a focus on use for our clients. This is a particularly poignant time for me to be writing this post and contemplating the question of what makes me a qualified evaluator. In a little over two weeks I will be sitting for my comprehensive exams for my Ph.D., passage of which takes me to the last, and final, phase…the dissertation. As part of the preparation for these exams, one of the steps I must take is completing an evaluator self-assessment, whereby I must go through a list of competencies and provide evidence for why I have obtained each one. So, to say the least, it is a reflective time for me, and this post slots perfectly into that space.

Since I am last in this series all the good words/qualifications have been taken, and since I am a student and have been burying my nose in evaluation literature I have found a quote from Jane Davidson’s book Evaluation Methodology Basics: The Nuts and Bolts of Sound Evaluation that I want to use to frame my post. In this section she describes what it takes to become a good evaluator, writing that it: “involves developing the pattern spotting skills of a methodological and insightful detective, the critical thinking instincts of a top-notch political reporter, and the bedside manner and holistic perspective of an excellent doctor, among many other skills” (p. 28). 

When I read this, the skill set she mentions resonated with me, and so I figured it would be a way to frame my post for this series. I want to talk a bit about each one, and how it has helped me become a good evaluator, and one who is able to make evaluation useful.

The Detective. The other day I was talking to a peer of mine in our comps study group about that moment in an evaluation when everything sort of comes together, a lightbulb goes off, and I am able to put all the pieces of evidence into a cohesive thought or story. It’s like when a detective cracks a case or at the very least finds a lead. When I am able to speak holistically about a program in the context of the evaluation, it says to the client that I have an understanding of what is going on. As an evaluator you gain a unique perspective on a program. As an evaluator we are able to look at the program with some of the context, through the light of any of the evidence we have collected.

However, on the topic of evidence, the other part of being a detective that is pointed out in Jane Davidson’s quote is the methodological aspect. Good evidence is the foundation of any evaluation. As evaluators we are obliged to do our best to collect, analyze and use the best evidence we can. My path through my Ph.D. program had a heavy emphasis on methodology. Gathering and using good evidence gives evaluation consumers faith in the results, and, I believe, makes them more likely to use the evaluation.

The Reporter. Sometimes I think that in another life I would love to be a reporter. Collecting stories, breaking important news, shedding light on important issues. Some of the individuals I look up to and admire most in this world are reporters, often because of the questions they ask and the balance they attempt to bring to a story (see Diane Rehm). Good questions are a key part of a good evaluation. Questions have been a part of my life since I was a kid. Perhaps my Dad unknowingly was developing the evaluator within me when each day when I came home from school, he would challenge me, asking me what questions I asked that day. I have always been taught to ask questions, challenge existing knowledge, and think critically about what is put before me. I think this serves me well in evaluation.

I also think that being a reporter means maintaining a level of objectivity and distance from the object of evaluation so as to not have ones judgement clouded. To make evaluation useful, it must be credible. This is similar to what Kelley talked about in her post, but I think I am conceptualizing credibility in a different way. I think that if the evaluation is going to be used, the users have to believe in its {the evaluation’s} credibility. This demands a level of objectivity, and distance on the part of the evaluator.

The Doctor. I like the doctor metaphor a lot. Put simply, evaluation is about diagnosing a program and saying whether it is sick or healthy (good or bad). It is also about communicating that to the patient (the program stakeholders). So, when we communicate this is when the bedside manner comes into play. I think that having the ability to engage people, respect people, and empathize with people helps me to facilitate evaluation use. This is a part of my bedside manner that I think has served me well and will continue to do so in the future. I also think that part of being a doctor is communicating the diagnosis, then describing a treatment plan to address the disease. As an evaluator, it is important to make value claims about the program (e.g., good or bad, better or worse), but it is also important to help clients consider how to improve what they are doing. By putting forth the treatment plan, we also facilitate use.

COREY’S USEFUL TIP: Evaluators are superheroes, who need to be able to do many jobs. Become one today. Channel your inner detective, reporter and doctor all in one. With our powers combined….you know what comes next.