In the past week, I’ve gotten two interesting comments about evaluation my organization, iEval, has done, and I had to process both comments for a while before I settled on how I really feel about them.
Comment 1: Two minutes after receiving the emailed attachment of our evaluation report, the client called me and said, “It’s beautiful! It’s absolutely beautiful! I can’t wait to read it!”
My first reaction to that was thanking her for the positive feedback and asking her to let me know what she thinks after she actually reads the report. Then, after getting off the phone with her, I thought – Oh no! All the client cares about is that the report is visually appealing! The substance isn’t even going to matter! I ruminated on this for a while, wondering if I should call her back to guide her to some specific areas in the report to focus on when she’s reviewing it. I was worried she wouldn’t read for content, but instead focus only on parts of the report that were particularly pretty.
After thinking about it some more, I realize that her reaction was exactly what I wanted. She loved the look of the report – the combination of simplicity, visual stimulation, and usability that we strive for – and she was actually excited to read the report. I believe our report design will draw her in to thoughtfully read the report and hopefully use the findings in meaningful ways.
Since then, I haven’t yet gotten feedback on the quality of the report or questions about the findings, but I have gotten an email saying they definitely want to contract with us for another year. I’d say that’s a good indicator that they’ll be using the report findings, particularly because we’ll still be working with them!
Comment 2: We’re partnering with another firm in a large statewide early childhood evaluation project, and we were giving a presentation to the program directors across the state. One director came up to me, someone we worked with five years ago, and said something to the effect of “It’s been a long time since we worked with you, but we loved that process and are still using what we learned from it. I was excited when I saw iEval’s name as part of the evaluation team for the statewide evaluation.”
That comment made my heart swell! She found what we did with her (it was analyses of the communication patterns and relationships within the early childhood arena in their county using social network analysis) really useful on a long-term basis – how fantastic! Or was it? Wait, she’s still using that information and those findings five years later when it’s totally out of date? Then the doubts set in. Maybe we should have clarified a timeline for using the information and when it’s not particularly relevant anymore. Then I start going down a whole path of thinking what we might need to change in current and future reports to ensure that findings aren’t misused years after the reports are written (because it would be misuse if findings relevant to a specific point in time, context, staff grouping, or program are applied to a totally different setting, time, group, etc.).
But, similar to the first comment, after further digestion, I realized her comment was perfect. She didn’t say she was using the actual report, she said they were using what they learned from it. Ideal! When we worked with them years ago, there were immediate changes they made based on the evaluation report – they were extremely receptive to the data and excited to grow from the findings. Isn’t meaningful use of findings exactly what every evaluator dreams happens when they work with a client? It’s exactly what I want!
And, her good experience with that evaluation meant she wasn’t nervous about the evaluation plan we were laying out for the statewide project because she’d had a positive prior experience working with us. Double win!
DR. TACKETT’S USEFUL TIP: Evaluators need to be reflective in our practice, too. Don’t rush to change practices or processes based on immediate feedback from clients, take the time to think about what they said and what it really means. Then, if changes are still needed, make them.