On lecturer evaluation

Moving offices and attending meetings means that I am writing fewer posts than I would like to. I really wanted to do one about the stuff I discovered when pack up the old office - notes from my PhD, the handbook for new lecturers 1999 etc. But leaving nostalgia aside, once in a while I do management work that also makes for good blog content. Here's a re-post of the one I wrote for the School blog on lecturer evaluation:

In his book Blink, Malcolm Gladwell writes about people’s ability to “thin slice’ – to judge what is good or important from a narrow period of experience. He goes on to argue that having too much information can interfere with the accuracy of a judgement.

Putting all that to one side, the NWU-Puk has for some time been asking students what makes for a good lecturer? Formally these are known as lecturers’ evaluation and normally it takes place towards the end of a semester, ideally in all courses. We don’t ask students for their gut feel on who are the good lecturers, we ask them about the characteristics of good lecturers using a questionnaire. Is there a case for thin slicing, or does our analysis approach yield results?

Now, I have been to a faculty council meeting where the questionnaire has been pulled apart and people congratulated on their results in almost the same breath. Some Schools use the “old” ones, Institutional academic support services have developed one and then there is the “new” version. A number of Faculties use their own questionnaires because they are different.

Being Economists, our School decided to bring some rigour to the debate. In the first semester of 2012 we used the e-learning platform to administer the “new” questionnaire and with that data, we now have some clues as to what students think makes for a good lecturer.The questionnaire has 25 items that students score on a 4-point scale from strongly disagree to strongly agree. The items include things like: The lecturer plans thoroughly and prepares thoroughly for contact sessions. Or, the lecturer makes use of multimedia in support of learning. The lecturer may also explain the relevance of concepts and theories, or explain the relationship between theory and practice. So what is it that the high-scoring lecturers do? I have always argued that being well-prepared and enthusiastic is more than half the battle. To see what different constructs make up that elusive “good/great” lecturer we used factor analysis to analyse our data.

We have responses from seven undergraduate courses in Economics, International Trade and Risk Management, ranging from first years through to third years, and a total of 1962 observations. There are 12 lecturers involved.

The KMO measure of sampling adequacy was .975, which shows that the data are suitable for factor analysis and the principle component analysis extracted three factors, explaining 50% of the variation in the construct “good/great” lecturer. After a Varimax rotation the items are grouped as follows:

Component 1	Component 2	Component 3
The lecturer is on time for classes	The lecturer makes use of the study guide during contact sessions	The lecturer encourages students to attend learning support on campus when their performance is weak
The lecturer makes use of multimedia in support of learning / makes effective use of visual aids	The lecturer bases assessments on learning outcomes as stated in the study guide	The lecturer encourages students to work together during contact sessions
The lecturer plans thoroughly and prepares thoroughly for contact sessions	The lecturer explains how outcomes will be assessed	The lecturer encourages students to participate in the class discussions
The lecturer is friendly towards students	The lecturer presents study material in an organised manner as set out in the study guide	The lecturer encourages students to make a greater effort in their studies
The lecturer offers support and assistance when requested to do so	The lecturer states learning outcomes I have to master for every contact session	The lecturer presents contact sessions that are valuable learning opportunities for me
The lecturer promotes an atmosphere of mutual respect	The lecturer explains the relationship between study units	The lecturer explains the relationship between theory and practice
The lecturer uses a level of language that I can understand		The lecturer refers to relevant and recent developments in the subject
The lecturer assesses assignments and projects fairly and transparently
The lecturer gives feedback on tests and assignments within a reasonable time
The lecturer explains the relevance of concepts and theories

Component 1 seems like the baseline characteristics of a good lecturer – being friendly, accessible, fair, offering support, an all-round professional that can explain the work. It accounts for 19% of the variance in the data. Component 2 looks like the things that you have to learn to become a good lecturer. It is not only about knowing your field, you also need to know a bit about formulating outcomes, action verbs, putting it into a study guide and using it. This explains 15% of the variance. Component 3 looks like it is about the enthusiasm and adding value – engaging with the students. And it explains another 15% of the variance. The other 50% of the variance is not explained by these three constructs. What describes the other half of being a great lecturer? I would love to hear how students/ lecturers think we can measure that magic. Or should we just thin slice and ask the students to score the lecturer out of a 100?

Finally, I am glad to say that my colleagues in the School did really well in these evaluations and should take a bow.