Let’s talk about pop culture this week. John Oliver, the host of the late night HBO talk show “Last Week Tonight with John Oliver” roasted the Miss American organization for their claim that they provide $45 million in scholarships annually to women. This 15 minute clip is a great spring board to talk about evaluation (warning: inappropriate language in the clip).
To start, he discussed the logic behind such a contest. If end long-term outcome is to provide scholarships to women, do the inputs and activities work to meet that goal?
What would a logic model of this program look like? Here’s my quick take on it:
However, I’m an outsider to the program. I’m guessing the program stakeholders would create a different logic model, reminding us it’s important to consider all stakeholders when creating a logic model.
John Oliver had a research question and collected data to answer his question. He wanted to know if the Miss America organization actually offers $45 million worth of scholarships for women each year. His staff combed through 990s, websites, and called state-level Miss America organizations to collect data.
What he found was that to reach that $45 million figure they claim, the Miss American organization counts all of the potential scholarships. For example, the Miss Alabama organization reported they provided $2,592,000 in scholarships to one school, Troy University. The show contacted Troy University and learned the organization got to that number by taking the amount of single scholarship ($54,000) and multiplying it by the amount of competitors who could have used it (48). $54,000 x 48 = $2,592,000. In actuality, zero people used the scholarship, therefore zero dollars were awarded. This, of course, brings up interesting questions related to reporting.
John Oliver also discovered a surprising finding in his research. Although the Miss America pageant does not give out $45 million dollars of scholarships each year, it is the largest single scholarship program for women. He said “even their (the Miss America organization’s) lowest number is more than other women-only scholarships that we could find. More than the Society of Women Engineers, more than the Patsy Mink Foundation and more than the Jeanette Rankin Women’s Scholarship Fund.” While he was speaking, the websites of the other three organizations popped up on the screen and he told the viewers you can donate to all of the other scholarship funds.
So, let’s review. John Oliver has questioned the logic model, asked a research question, and collected data. He analyzed and reported the data. Although most evaluators do this through a written report, John Oliver chose a different format. He presented his evaluation findings in the form of a 15 minute segment on his talk show. The findings have been shared over the internet to millions of people.
Now we get to evaluation use. How will the Miss America organization and the public use this evaluation data?
First off, the Miss America organization responded to the segment with a very general statement (which you can read here http://www.nj.com/entertainment/tv/index.ssf/2014/09/miss_america_john_oliver_scholarships_1.html)
How will the organization use those evaluation findings? Will the organization change their tagline about providing scholarships? Will they change the dollar amount they say they provide?
The other stakeholders for these findings are the viewers. John Oliver gave three examples of women-only scholarship funds, displayed their web addresses, and told the viewer they can donate to those funds. Did the results of evaluation change someone’s behavior? Will the viewer donate to the funds?
Although we may never know the impact of this segment on these organizations, it brings up interesting questions and shows us that evaluation is all around us.
*Thanks to Carolyn Jayne for the idea of looking at the John Oliver’s clip through an evaluation lens.
DR. EVERETT’S USEFUL TIP: Evaluation is all around us, we just have to look. We may not have our own late night cable show to share our evaluation results; but with the internet, we can easily share findings.
Note: Dr. Tackett is presenting on evaluation use at the European Evaluation Society conference this week. Stop in and say hi to her if you're there! You'll be hearing more about her experience in an upcoming post.