An adapted excerpt from Google Course Builder

 

 

Plan your course evaluation

You need to find out how effective your course is, so you'll need concrete information to guide future improvement of it. To do so, you need to collect and analyze data of various kinds.

Deciding early on what data you want to evaluate and how you will evaluate it makes it more likely that, as you continue to the develop the course, you will remember to collect the information necessary to do those evaluations. Do not wait until after the course has launched; it may well be too late by then.

We suggest you use three basic categories of data. Not all are relevant to all courses. Depending on your content and on the course objectives, you may use any, all of these categories of data.

Assessments, as discussed above, measure skill level and improvement in ability. You can collect this data by creating the pre-, mid-, and post-course assessments and comparing students' scores on them.

Reach and engagement data measures how many people you reached with the course and how engaged those people were. Common reach and engagement data includes: How many students registered? How many completed each unit and each lesson? What countries did the students come from? Was there a difference in course performance based on location? You can collect this data by using an analytics program to track visitors to your course.

Satisfaction data measures student attitudes towards the material and feelings of satisfaction. They answer questions such as how satisfied the students were with what they learned and whether they think they will be able to use the material. While not objective, satisfaction data is an important indicator of the success of your course. Even if a course was objectively effective, your students are not likely to recommend it to their friends if hated it. You can collect satisfaction data by sending a post-course survey to everyone who registered for your course. Consider sending different surveys to people who registered but didn't finish the course from those who did finish it.

[To these, I would add a fourth kind. I would gather feedback from tutors, who will identify patterns in what students easily grasp and what students students tend to struggle with. You should encourage your tutors to write down issues as they notice them, but most will probably prefer to be informally interviewed. This information will usually be very helpful, even though it is not completely objective. (R.W.)]

In a table array, it looks like this:

Category of data	What it measures	Examples	How it is collected
Assessment	Skill level and improvement in ability	What percentage of students scored at least 30 points higher on the post-course assessment than on the pre-test? What percentage of students missed questions on a particular topic?	Take snapshots of students' ability at various times. For example, administer pre-, mid-, and post-course assessments, then compare students' scores on them.
Reach and engagement	How many people you reached with the course and how engaged they were	How many students registered? How many completed each unit and each lesson? What countries did the students come from? What were the overall drop-off rates? When did students tend to drop off?	Use an analytics program.
Satisfaction	Student attitudes towards the material and feelings of satisfaction.	How satisfied were the students with what they learned? Do they think they will be able to use the material? Did they enjoy the course? Did they feel well supported by their peers? Did they feel well supported by the instructors?	Questionnaire Send a post-course survey to everyone who registered for your course. Consider sending different surveys to people who registered but didn't finish the course from those who did finish it. For more information, see Student satisfaction.

 

General tips

1. Remember to convey your privacy policy to students and let them know that they will be tracked on metrics you wish to measure.
2. If you're concerned with how well your material works for an international audience, figure out the completion ratio by country. Assume 80% of the people from your country completed the course, but only 5% of people from other countries. That might indicate that your material is less effective for people from other cultures.
3. During the course, collect data as you go, recording all of it in a centralized location.
4. Your time will be limited, so prioritize which data and analyses you need immediately, and which ones can wait.
5. Is a pre- and post-course assessment comparison needed?
6. Assuming you have a team of people working on your course, split the evaluation work among several people. If possible, have a different person be responsible for each of the categories above. Data collection and analysis takes time, and it helps to have the many team members think through what’s needed and how you analyze it.
7. If you have not done evaluations of this sort before, keep in mind that working with and analyzing data takes time and practice. The amount of data you collect may seem overwhelming and it will require time for analysis. If you know people who are comfortable in this area, reach out to them. In general, you’ll find that people are very eager to share their expertise.