When you conduct program site visits or observations, do you always see the whole picture?

Here is an interesting exercise about paying attention, and, more importantly, looking for what you THINK should be there. Click on the link and try it out.

How did you do? Were you able to observe what was there, as well as what was not there?

How would you use this in thinking about evaluation and site visits?

It is apparent that not all evaluators see the whole picture when they conduct site visits and observations. The December 2015 of the American Journal for Evaluation has an article by Michael Quinn Patton called “Evaluation in the Field: The Need for Site Visit Standards.” It is an interesting read, especially for anyone who conducts site visits or is “site visited.’ He has many anecdotes of site visits gone wrong. Although they are all shocking in their unprofessionalism and lack of standards, one of the most jaw dropping stories relayed to Patton by an evaluation colleague about watching a graduate student conduct an observation:

“I remember the student sitting in one of the classrooms for quite a long time, more than a couple of hours. I circled back to this room and asked her how it was going. I was especially interested in what she was learning about the culture of the school, which was the focus of her observations. I asked, so what are you learning? She yawned, looked bored, and said, “Not much,” and added that she would probably leave soon. I noticed that she only had a half-page of notes. The next time I looked for her, she was gone. Imagine my surprise when I found out that she had written up her observations as a chapter in a book on school innovation. The methods description in that chapter did not correspond to the degree and nature of the field work I knew had actually occurred.” (p. 452)

How do we remedy this? Some of the standards Patton recommends to help remedy the site visitor only looking at certain things include (p. 458):

Preparation – Site visitors should know about the site they are visiting
Credible fieldwork – Not all of the activities/events of the site visit should only be led by the people at the site. The evaluator should be allowed to determine what activities are observed, arrange interviews, etc.
Neutrality – The evaluator should not have a preformed opinion on the program or activities.
Site review – The site should be able to see reports that are prepared based on the site visit and given an opportunity to change errors.

DR. EVERETT’S USEFUL TIP: Using Patton’s proposed site visit standards of preparation, credible fieldwork, neutrality, and site review can help ensure that just one side of the story is told from a site visits.