Case Studies:
Using an Audience Response System for Module Evaluation
Justin Savage, School of Psychology
Background
The School of Psychology is one of the largest departments of psychology in the United Kingdom. There are currently over 50 academic staff, 100+ research staff and a further 20+ Research Fellows (including Royal Society, BBSRC, ESRC, Leverhulme Trust and British Academy Research Fellows). The School also has over 120 full-time students studying for Doctorates, both in research and on our professional Doctorate programmes. Each year it admits around 200 students to the Undergraduate degree programmes in Psychology and Psychology with Professional Placement.
Every module is evaluated with a brief questionnaire. The standard evaluation comprises three questions per lecturer. In addition, an end of year questionnaire (which has traditionally comprised around fifty questions) has been given to each student.
The Challenge
Conducting and analysing the questionnaires was a slow, paper-heavy labour-intensive administrative process. It used to involve collecting A4 question sheets from the school office, handing them out to students who would fill them in during a lecture, gather them back in and return them to the office. The sheets would then be fed through a machine which would read the responses. It would take several days for administrative staff to feed all the sheets into the machine which read them, and some questions were tailored and needed to be hand transcribed.
It was several weeks before the analysis could be made available to module coordinators and lecturers. A faster and more flexible procedure was sought to provide feedback from the evaluation sooner with the scope for customising the information captured.
Intended Outcome(s)
Make analysis available quickly, reducing the administration time required to input and analyse the data
- Increase the flexibility for receiving customised feedback
- Gather student evaluation of modules with minimal impact on lecture time
- Maintain high response rates to the evaluation
- Reduce the amount of paper used
The Solution
Dr Justin Savage has been seconded, for one year initially, to look at the use of technology in education in psychology. To reduce the amount of time and effort involved in inputing and transcribing data, he decided to capture students’ feedback on their courses directly, and began looking for an appropriate electronic toolset.
One of the solutions considered was Questionmark Perception. This is a powerful tool for creating and deploying online assessments and its engine can also be used to capture survey responses. Questions are created and a survey is assembled which students can then access through a web-browser at and computer. Students can complete the survey at any time up to a preset closing date, at which point the software can generate a report analysing the responses.
However, unlike setting questionnaires in a lecture, where there is time specifically set aside for the activity and completion is supervised, it can be more difficult to ensure students engage. Having drawn on advice from other schools with experience in using online surveys, Justin was concerned the response rate might fall below the 60% required to get meaningful feedback. Some non-intrusive means of capturing responses electronically from a captive audience (in the lecture theatre) was required.
Audience response systems provide audience members (for example students in a lecture) with a device that allows them to respond to questions posed (for example by the lecturer). In its simplest form this may be two or more buttons which allow participants to choose from two or more pre-determined responses. Depending on the configuration of the system, responses may be anonymous or attributable.
TurningPoint comprises handheld wireless “clickers” and software installed on a PC which integrates with PowerPoint, and with which they communicated (by means of a small radio receiver plugged into the PC). This allows questions to be asked via PowerPoint slides, to which each participant can respond by pressing the buttons on his or her clicker. Whilst the limitation of this system is that it is impossible to enter “free text” responses (for example contributing comments to the evaluation) it is incredibly simple, and in combination with the range of types of slides that can be added to existing PowerPoint presentation it forms a very powerful tool for capturing a wide variety of responses (multiple choice, Likert scales, etc.)
In the School of Psychology, all modules are evaluated in the same week (this is around four modules per day). This required a set with enough clickers for the largest groups (around 180) to be made available for the whole week, and whilst many lecturers requested that the evaluation take place at the end of a session this was not possible in every case, since several modules have simultaneous teaching sessions.
Each evaluation took approximately 10 minutes. Initial concerns over the disruption that might be caused proved unfounded: Justin and his small technical team were able to go in, distribute the equipment and conduct the evaluation in around the same amount of time that distributing and completing the paper versions used to take. [JP1] However, unlike the paper versions, no further inputting or analysis was necessary. The results are captured and processed by the software making it possible to access the data immediately.
The Outcome for Students
The response to the system from the students was very positive. They liked the technology. However, for the end of year assessment, second year undergraduates had to answer fifty questions, and around half thought that this was too many (making the exercise “unbearably tedious”). In response to this, the end of year assessment for the first year undergraduates was reduced to twenty-five questions. Only four students (out of around 180) considered this too long. Students were asked whether they enjoyed using the clickers (responding by putting their hands up!) and they were unanimous in responding that they did. Next time this ad-hoc question will be added to the electronic questions J
Key points for successful innovation
- Be clear about your purpose in using an audience response system, and the type of responses you need to capture.
- Avoid over-using the clickers. Presenting students with a long series of questions in one go can be off-putting.
- If you intend to use the clickers regularly, consider issuing each student with their own individually coded clicker.
Conclusion
Capturing student evaluations directly using the ‘clickers’ took about the same time in lectures as asking students to fill in paper questionnaires. But rather than having to wait weeks for the completion of the administrative process, the analysis was available immediately.
Although capturing the data with clickers simply took the place of capturing it on paper in this case, the technology makes it possible to change the way feedback from students is gathered and used: for example, simply by inserting a slide into a presentation it would be possible to conduct an evaluation for a single lecture.
Students enjoyed using the clickers to respond to short sequences of questions, and having seen and experienced the technology several members of staff have expressed an interest in using clickers in a range of ways in their teaching.
As a result of our experience with the clickers loaned to us by BIOSI we have now purchased our own system of 200 handsets and plan to use them throughout the current academic year.
Further Reading
Evaluation of interactive teaching in bioscience degree courses, Dr Mike Mattey, University of Strathclyde (HE Academy subject centre report) http://www.bioscience.heacademy.ac.uk/resources/projects/mattey.aspx
Who wants to be… The use of a personal response system in statistics teaching, Dr Ernst Wit, University of Glasgow (HE Academy subject centre newsletter article) http://ltsn.mathstore.ac.uk/newsletter/may2003/pdf/whowants.pdf
The Technology: TurningPoint Interactive Response System
What Is It?
TurningPoint software integrates with PowerPoint (and other software) to make slides interactive. For example, you could pose a question in the title of a slide, set your bullet points as possible answers, ask all your students to select an answer, and display their responses as a bar chart graphic on the same slide.
How Does It Work?
The software is installed on a computer. Starting the software starts PowerPoint, but with additional buttons for adding interactive elements to the slides. A receiver is plugged into a USB port on the PC. This communicates with the “clickers”. The “clickers” are small battery powered handheld units that resemble a calculator without a screen. Each student (or group of students) is issued with a clicker. When a slide asks them to respond, they can press a number to indicate which response they would like to give.
The software can also integrate with other software (besides PowerPoint) and use other forms of response, for example more sophisticated clickers, web browsers or a mobile phone application.
Cost & support implications
The principle cost is the purchase of the clickers themselves, which can be bought in kits with the receiver to plug into the PC. The software for authoring slides is free and available on via the “network applications” on standard University PCs. The battery in the clickers typically lasts 6-12 months.
Additional Uses
- Wide range of pedagogical applications.
- You can set up anonymous voting, or identify students with particular clickers to track progress and make personal responses.
- Integration with Blackboard allows students to track their own responses.
- Researchers can quickly and efficiently getting instant response from experiment participants.
- The devices can also be used in meetings to gauge opinion and collectively set priorities.
