We have been talking about using evaluation for program improvement. One of the frustrations and challenges evaluators can face with impacting program improvement is the reality that the lots of hard work needs to be done before one can share findings, and in-turn, impact programs. This lack of timeliness sometimes stymies enthusiasm for the evaluation process. The client waits for findings as the evaluator cleans, analyzes and finally reports on the data collected. This process can sometimes be long and tedious.
One technique that can be employed to keep a client involved and up to date on the key findings throughout the data collection and analysis phase is a ‘Top of the Mind’ memo. Essentially, it is a short (1-2 page) document that presents high-level themes within one week of key data collection points. For example, if researchers have been in the field collecting interview or focus group data, a ‘Top of the Mind’ memo would summarize the key ideas and issues that surfaced across groups, as well as some ideas that will be explored more deeply. Or, if a survey was conducted, some key descriptive statistics of some basic questions and/or survey responses that do not require a lot of cleaning can be reported in the aggregate.
These findings are broad and general themes and patterns that are clearly evident without a great deal of analysis, e.g., teachers are happy with the professional development they are receiving, but they do have some reservations about the upcoming changes; overall parents are happy with the program; crime was a concern that came up often when discussing neighborhood safety.
Why ‘Top of the Mind’ memos can be a good tool:
- It provides immediate feedback to a client while their attention is still focused on the project. Sometimes clients have a multitude of projects and programs they juggle. A ‘Top of the Mind’ memo provides feedback while the content, ideas, and program are still a fresh topic on their minds and/or agendas.
- It is a good way to garner feedback from a client about the direction or focus of analysis for longer and more formal reports. Which topics are of interest or surprise? Which themes would be of interest for deeper analysis?
- A ‘Top of the Mind’ memo helps frame and outline key findings for future and more in-depth reporting. You will always be happy you took the time to write down the key themes while they are fresh.
A ‘Top of the Mind’ memo may be a helpful tool for some clients in some situations. It should be noted somewhere in the memo that the findings are high-level and that deeper analysis will provide further and more valid insight. The memo can be a nice discussion tool that will help your clients feel engaged and involved in the evaluation process.
KELLEY'S USEFUL TIP: If you are planning to create a ‘Top of the Mind’ memo from focus groups or interviews, it is helpful to jot down one or two big themes from each group at the bottom of your notes. When you are done with all of the groups, looking across the themes you wrote down will help you develop your memo.