So you’re about to drop a few hundred dollars on new signs, or an interactive exhibit piece.Do you just put it out there, or are you conducting formative evaluation before putting all that money into the piece?If you’re not spending a few dollars for a foam core mock up, you may very well be wasting hundreds on your final piece.
Before you take that costly final step, try out your sign or interactive piece first.You will be surprised by what you can learn in just a few hours and with just a few dollars.When you create and test a mock up and then tweak it, you are conducting formative evaluation.
Two years ago I was working on a formative evaluation project with a professional design firm. The designers were present while we observed visitors interacting with several elements.One of the simplest elements contained three blocks of wood that demonstrated unique characteristics and three lift flap doors underneath.The wood blocks and lift flaps were approximately the same size.No one expected any difficulty with this simple interactive piece…a no brainer.
What we found was that visitors first tried to lift a block of wood, when it wouldn’t lift, they moved down to a lift flap and tried to lift it, but the lift flap hinges were concealed and tight, so when the flap didn’t lift immediately, the participants moved back to the blocks of wood and tried to lift them again.
The designers were awestruck.These were very familiar lift flaps, who wouldn’t know what to do?Of course, the visitors wouldn’t.These lift flaps would be very familiar to a designer that has installed them in many exhibits, but to a visitor who hasn’t seen this type of flap before, the wood blocks looked more like lift flaps they had seen in the past. Also the instruction ,“lift the flap” was just above the row of wood blocks.This final piece will probably cost the museum between $1,500 to $2,000 to have it professionally fabricated. In house it would cost a few hundred not including your staff’s time.What a waste of money if it had not been tested prior to completion. So by spending a little time and money to test a mock up, you save a lot of time and money and have an effective product.
Formative evaluation does not have to be complex. It can be as simple as putting out a mock up and watching to see how your visitors interact with it.Are they doing what you expected them to do?If not, what changes can you make to see if they interact properly?Make the changes and try it out again.You will also want to know what your visitors are learning from their experience…. so ask them.Do it by creating a few questions about the content of the exhibit piece.If you really want your findings to have some relevance, gather information on a random sample of visitors.That way, you are filtering out selection bias.
One of my favorite formative evaluation books is no longer in print, but it is still available online at resellers: This book: Try It: Improving exhibits through formative evaluation by Editors Samuel Taylor and Beverly Serrell is designed for the beginning exhibit evaluator.It includes case studies, methods and lots of encouragement.
What Not To WRITE 4
Recap: This is the 5th posting in this series What Not to Write.It is intended to help you create a comprehensive evaluation plan and provide a guide for what to write in your grant proposals.Last fall while giving this presentation at a conference, a woman stood up and introduced herself as the Associate Deputy Director of museum services for the Institute of Museum and Library Services (IMLS).To paraphrase her comment she said: You are exactly right what to include in the evaluation portion of IMLS grant proposals.You would be shocked by the percentage of grant proposals that we receive with the all encompassing, cover your bases sentence that you showed.In a later conversation that I had with her she said that although IMLS has offered an online course on evaluation planning for nearly 10 years, the majority of proposals that they receive still have a vague description instead of a real plan. So far we’ve discussed the critical elements 1 through 3.Rather than taking up valuable space, I’ll let you just peruse the previous postings for that information. Critical element #4:Sample size.This is pretty simple.Note that for the example they will be testing sample sizes of 60 for the first phase of both methods and 24 people for consecutive methods. example: The formative evaluation will be conducted in as many as three stages, each stage resulting in further refinement of the exhibit components.Both methods of the first stage will be conducted on a sample size of 60, and each consecutive stage conducted on asample size of 25.Sample members in the tracking and timing study will participate unknowingly; sample members ….. Determining sample size is controversial.The larger the sample, the more accurate it will be, but also more costly.My mentor, Beverly Serrell, believes that by the time you test 20 people you will see the trends.She usually chooses to test an additional 20 people to verify that the data is accurate.I tend to follow the same philosophy, but I am more comfortable with testing 40 to 60 people.It depends on the project.
(It's time to take a break from the What Not to WRITE series, but I'll pick it up where I left off soon. )
What Not To WRITE 3
Recap: So far in this series I have shown you what not to write and two critical elements that must be included in and evaluation plan and grant proposal.Those elements are the type of evaluation and the methods of evaluation that you will be using. The third critical element that you will need to include is what you want to learn from the evaluation. In the example plan you see that we want to measure the cognitive and affective outcomes and attraction power of the proposed exhibits. example: …two different methods of evaluation: 1) cued questionnaires that will test the cognitive and affective outcomes of the exhibits, and 2) timing and tracking which will test the attraction power of and interactions with exhibit components. The formative evaluation will be conducted in as many as three stages, each stage resulting in ….…..
If you’re doing front end evaluation, you probably want to know what your visitors want to learn, how they want to learn it, what they already know, how do they want to get that information…. etc.If you are doing formative evaluation you will want to know if interactive elements are working, are visitors reading or interacting as planned, what messages are they taking away from the experience…etc.For remedial evaluation you will be looking for obvious problems such as are interactive elements broken, is the lighting too poor for visitors to see the sign, is there a sign or element that is hidden or missed by your visitors….etc. And finally for summative you will be measuring exhibit effectiveness.What are the take away messages, what elements are getting visitor attention, what is being over looked.
What Not To WRITE 2
Recap: This series of blog postings is intended to help you create a comprehensive evaluation plan and provide a guide for what to write in your grant proposals.So far we’ve discussed the 1st critical element in the plan: the type of evaluation that you will be conducting.
The second critical element an evaluation plan must include is the methods that you intend to use to conduct the evaluation.In the example plan we see that cued questionnaires and timing / tracking are the two methods that will be used.
example:
The formative evaluation will be conducted on mock-ups of the proposed final exhibit components and incorporate two different methods of evaluation: 1) cued questionnaires that will test the cognitive and affective outcomes of the exhibits, and 2) timing and tracking which will test the attraction power of and interactions with exhibit components…….
There are many evaluation methods that can be used for collecting data.These are just a few: unobtrusive observation, questionnaires, one on one interviews, post it surveys, focus groups, concept mapping…. You have to pick the method that works best for your project.
What Not To WRITE 1
My previous post in this series was the first.For the next few posts I will be addressing seven critical elements of an effective evaluation plan.The purpose of this was originally intended to help grant writers successfully include an evaluation plan in the grant proposal, however, it is a simple exercise that anyone planning on conducting evaluation should do. The first sentence of your evaluation plan should tell the reader the stage or stages of evaluation that you plan to use.
example: Formative evaluation will be conducted by Monica Post an independent professional exhibit evaluator to ensure that the exhibits’ messages will be clearly understood by the audience.
If you are unfamiliar with the stages of evaluation then this should help: Front End Evaluation is conducted before the project gets underway.It helps youlearn what your audience knows, feels, wants to know, how they want to learn it… Formative Evaluation is conducted during the design phase when changes can be made.It is used to identify potential weaknesses and make improvements. Remedial Evaluation is conducted soon after the exhibit or program is open to the public.It is often conducted but overlooked.When you identify a problem that can be fixed,you have conducted remedial evaluation. Summative Evaluation is conducted after the exhibit or program is open to the public.Problems have been fixed and now you want to know how effective your product is.
Are you familiar with the What Not to Wear television show?When I was asked to create a presentation for grant writers on what to include about evaluation in their proposals, the words “what not to write” kept popping into my head.Grant providing organizations and donors are becoming more and more focused on outcomes based evaluation.They want to know that the money they are giving is being effectively used.They want measurable results.
This is what not to write:
We will be evaluating the program through exit interviews, on line surveys, questionnaires and focus groups.
In other words ….we don't really know what we're going to do yet so we are covering all the bases.
So now you might be asking: If that’s what not to write, then what should I write?You need to have a comprehensive evaluation plan before you write the grant proposal.You can’t just bluff through it and hope that the readers will buy it.Below is a good example of how a comprehensive evaluation plan can be written into a single (although long) paragraph.Use if for a guide, but don’t feel overwhelmed.Over the next few blogs I will break it down and you will see how easy it is to effectively write the evaluation portion of the grant proposal.
Formative evaluationwill be conducted by Monica Post an independent professional exhibit evaluator to ensure that the exhibits’ messages will be clearly understood by the audience.The formative evaluation studies will be conducted at both the Zoo and also at similar institutions that are more accessible to the designer and evaluator.This multiple-site evaluation system allows for a more extensive formative evaluation to occur and also acknowledges that the Zoo’s audiences are not always local and viewpoints from outside the region will help in clarifying the final product for all audiences.The formative evaluation will be conducted on mock-ups of the proposed final exhibit components and incorporate two different methods of evaluation: 1) cued questionnaires that will test the cognitive and affective outcomes of the exhibits, and 2) timing and tracking which will test the attraction power of and interactions with exhibit components. The formative evaluation will be conducted in as many as three stages, each stage resulting in further refinement of the exhibit components.Both methods of the first stage will be conducted on a sample size of 60, and each consecutive stage conducted on a sample size of 25.Sample members in the tracking and timing study will participate unknowingly; sample members in the cued questionnaire studies will do so knowingly. Eligible sample members will be randomly selected visitors over the age of 8, not including school and camp group participants.
The 10% Solution
Have you ever heard that we only use 10% of our brains?Does a 10% off sale get your attention?In an interview with Kurt Vonnagut, Susan Sonntag said that what she learned from the Holocaust is that 10 percent of the human population is cruel, and 10 percent is merciful.
If you read the article you might have been struck by the quote from Kirsten Ellenbogen with the Visitor Studies Association (VSA).Ms. Ellenbogen states that many museums dedicate 10% of their project budget to evaluation.As a member of VSA, I’ve heard this number and I agree with that.But I would also say that when you are putting together your budget for evaluation, you should contact a professional evaluator to help you with the plan and budget.
The total cost is all going to depend on the project itself.I’ve worked on multimillion dollar projects and I guarantee that we didn’t put 10% of the overall budget into evaluation.I’ve also worked on very small projects, and the cost of evaluation was easily more than 10%. So, it really depends on the project.There are ways to keep the cost down, but keep in mind that each cost saving measure can have pitfalls. Work with your evaluator to know what to expect and where you should put most of your money. Just don’t make the mistake of ball parking a budget number without consulting an expert.
Are You Watching?
I ran across an article yesterday that got my attention and got me thinking:The Museum is Watching You by Isaac Arnsdorf http://online.wsj.com/article/SB10001424052748704554104575435463594652730.html points out to the reader that museums are gathering data about the exhibits and exhibit elements that are catching our visitors’ attention, how long they are spending there and what information they are gleaning from their experience.Are you doing that?If you’re not, why?For some of us, formalizing our observations might feel like an invasion of privacy and from the title, it almost seems like that is what the author is trying to get at.But as you read it you see that the author is pointing out that museums are recognizing that visitors matter.That’s right, what they think, what they like, what they want to know, how they want to learn it, matters.We are past the days where staff determine the content and delivery methods for new exhibits.Visitor opinions, interests and prior knowledge matters, and this article it telling visitors just that. So do you care enough about what your visitors like, know, want to know and how they want to learn it, to study your visitors?I don’t mean just walk past and make a mental note of the comments,I mean really study them.Do you know how long your visitors are spending in front of your exhibits?Are they reading the signs?Do they learn something from their experience?Do you care?
