INTRODUCTION WHAT IS EVALUATION? SOME COMMENTS ABOUT TERMINOLOGY

Before we move onto exploring the learner-centred framework in some detail, a few comments about terminology are important.

Distinction between Assessment and Evaluation

We need to be clear about the difference between assessment and evaluation. We are using evaluation in terms of looking at a broad range of evidence in order to gauge the effectiveness of a computer-facilitated learning project. Assessment is the process whereby teachers set specific tasks related to the learning outcomes which students undertake to do. Students all undertake formal and informal assessment tasks in the subjects they are studying and so we always have assessment data to use in evaluation. Their success in these tasks provides evidence of how effective their learning has been. But assessment results make up only one set of measures and these need to be considered alongside other pieces of evidence. While all evaluation plans should contain assessment data, that is just one aspect of evaluation.

Bear in mind also that there can be problems with using assessment data in evaluations. These need to be considered carefully. The Flashlight Evaluation Handbook(Ehrmann, 1999b) identifies a range of problems in basing evaluations solely on assessment measures, including problems deriving from the assessments themselves and from using them in before and after comparisons:

• Few academics have been trained in the design of effective assessments;
• It is difficult to know if another marker would come to the same conclusion about a student's work;
• Some academics grade 'on the curve' so that one year's results cannot be compared to another's;
• Educational innovations often result in changes in teaching and learning practices and objectives, so that assessment results cannot be reliably compared before and after the innovation.

Distinction between Evaluation and Research

There are similarities between evaluation and evaluation research. The distinction can become rather fine. We argue that evaluation is an integral part of effective teaching. However, evaluation can be formalised to an extent that it becomes research. Indeed, the focus of this project, where we are seeking explicit evaluation plans, is close to research.

The problem with the label 'research' is that a great deal of evaluation research is flawed. Reeves (1993) lists several of these flaws:

• vague specification of primary independent variables;
• lack of linkage to robust theory;
• inadequate literature review;
• inadequate treatment time (conclusions are sometimes made after students use materials for <30 minutes);
• measurement flaws (precise measurement of easy-to-measure variables; more complex variables, which might be the crucial ones, are ignored);
• inconsequential outcome measures;
• small sample sizes;
• obscure statistical analysis.

For research to be rigorous, you need to specify who the stakeholders are, describe the paradigm on which you are basing your evaluation, determine the questions you want to ask, and how you are going to answer them. You will also want to specify the characteristics of the sample of students who will provide evidence to answer your questions.

Your mentor will assist you with these issues. We are asking you to do research which is of value. We are asking you to take a limited task, spend a lot of time thinking about the exercise and try to design an evaluation plan which enables you to get some answers to the questions you are asking, and then have an interesting and useful time actually carrying out the evaluation.