INTRODUCTION WHAT IS EVALUATION? SOME COMMENTS ABOUT TERMINOLOGY

In this handbook we will try to assist readers to understand the nature of evaluation, its complexity, and some strategies which can be used in order to obtain answers to evaluation questions.

As human beings we naturally ask questions about how useful and how valuable our activities are. We can think of evaluation as a process of considerably sharpening this natural activity of checking on our ongoing work. A more formal definition is to think of evaluation as 'providing information to make decisions about the product or process'.

Evaluation is not equivalent to research, although it employs research techniques as a means of generating the necessary information, and uses similar criteria for the reliability and validity to judge the quality of the evidence. Also, evaluation tends to be broader than research, as it usually requires information about a range of situations, products and processes. However, the main difference between evaluation and research is that evaluation also involves making judgements about the value of what is being evaluated.

Evaluation in an educational setting is the process whereby we seek evidence that the learning experiences we have designed for students are effective. As we will discuss later, we evaluate educational activities for two overlapping reasons:

• to obtain information that can inform the ongoing design and development process (often referred to as formative evaluation);
• to decide whether an innovation is worth retaining (often referred to as summative evaluation). These forms of evaluation often meld together, and each can be difficult to undertake properly.

Indeed, the Flashlight Evaluation Handbook², (Ehrmann, 1999b) likens the process of evaluation to the use of a small torch to glimpse what sort of animal might be in front of you in a huge, dark cave. In this metaphor, the cave is the process being investigated, and the torch is the evaluation. The cave is large and complex, and the torch beam is narrow and weak; each evaluative question is equivalent to pointing the torch in a particular direction, to see what is there and what walks into the light.

Clarity is the key to successful evaluation because it determines what kind of a flashlight you use, who holds it and where it is pointed. It is particularly important to be clear about the following issues:

• What are the purposes of the evaluation? Who is the evaluation for? Who should participate and how?
• How can you unpack your own assumptions (about teaching and learning) so that you can check how these affect the evaluation?
• What specific outcomes are you aiming for? What audience/s do you wish the evaluation to reach and inform?

Let us consider each of these in turn, although they are really quite intertwined.

Who is the Evaluation for?

What do we mean by describing an educational experience as effective? From whose point of view is it effective?

The decision about effectiveness might be from several different angles. Evaluation, in general, is the process of finding out how effective or useful some activity is. Obviously the decision about how valuable something is depends on the perspectives and vested interests that various stakeholders have, and final decisions about effectiveness can vary quite markedly.

It is very important to ask who the evaluation is for. There are many stakeholders in the planning of university offerings and a variety of information may be sought. In educational innovations, there are several stakeholders. In Table 1.2 we have listed a range of possible stakeholders, and some of the interests they might have in an educational activity, whether this is an innovation in the curriculum or the continuation of existing practice. Each participant in an evaluation study in this project should scan Table 1.2 to see which stakeholders, other than teachers and students, need to be considered, and what implications that has for the information you will seek to gather.

Table 1.2. Description of various possible stakeholders in the types of evaluation studies addressed in the handbook.

Please note that this handbook places student learning at the centre of the evaluation enterprise and we will focus on discussing evaluation questions and strategies from that point of view.

How can you Unpack your own Assumptions about Teaching and Learning?

Evaluation of the educational impact of CFL is a complex field: different evaluators employ different paradigms and hence ask different questions when designing their evaluations. Whenever a measurement or observation is made, the situation being evaluated intrinsically alters (Keeves, 1988). This issue is rarely addressed in conventional evaluation. One must ask the extent to which the outcome of the evaluation was due to the evaluation design selected? In examining evaluation studies, there is a need to describe the context and clarify the educational rationale which has explicitly or implicitly been adopted by the evaluators.

Reeves, (1997) has mapped the dominant paradigms which are used in evaluation studies. He also describes models which researchers use within these paradigms. The paradigms are briefly summarised in Table 1.3, together with a commentary³. It is important that the members of each evaluation study spend time discussing their own paradigms, clarifying their own positions and explicitly looking at the assumptions underlying any models and associated methods they adopt. The role of the mentor is very important in this respect.

The Positivist-Quantitative Paradigm has been used in the majority of articles about evaluation of CFL projects. Surveys of articles in journals and conference proceedings were carried out by Reeves (1993), Alexander & Hedberg (1994) and Reeves (1995). They found that the majority of articles reported the use of experimental approaches, with control and treatment groups and quantitatively measured outcomes. Reeves (1993) and Alexander & Hedberg (1994) identified a range of serious methodological deficiencies in such studies and pointed towards the Constructivist-Interpretive-Qualitative Paradigm (Guba & Lincoln, 1989 and Patton, 1990) as being appropriate for evaluating the complexity of CFL materials.

Such evaluation is conducted in a naturalistic way (avoiding manipulation of the environment) with data produced largely through qualitative methods (sacrificing wide generalisability for richness and better understanding). While these characteristics can be contrasted to the experimental approach (manipulating the environment) with quantitative data collection (everything is measurable), current practice favours the Eclectic-Mixed Methods-Pragmatic Paradigm, involving a mixed approach to data production and analysis, with both qualitative and quantitative information obtained in the evaluation process.

The Constructivist-Interpretive-Qualitative approach requires a paradigm shift for many academics, whose fields of study are implicitly grounded in an objective, experimental view of the world.

Table 1.3. Brief summary of the dominant evaluation paradigms.

What Specific Outcomes are you Aiming for?

It is important to focus on what is achievable in a project of this short timeframe. Evaluation is an ongoing activity, and we hope that this period of focused evaluation activity will lead to new ways of approaching the design, development, implementation and reflection on all teaching and learning activities. But, in 2000, what do you hope to achieve?

To answer this question, you will need to devise an evaluation plan, considering the scope of the evaluation, the questions you want answered and how you want to report your results. The learner-centred framework for evaluation proposed here, (see §2) will provide guidance in structuring your evaluation plan.

The evaluation plan will vary depending on the nature of your project. Some possibilities might be:

• Evidence of how some specific strategies or materials work with a given group of students. This might well be a subset of a full subject, e.g. just how students use threaded discussions and chat sessions, or how students use a simulation exercise.
• Evidence of how some specific strategies or materials work across diverse groups of students, for example, the process of implementing offshore teaching using existing resources already developed. The use of similar resources with both full-time, young students and part-time, mature age students could also be the focus of the evaluation.
• Evidence of how some specific strategies or materials assist students learn specific concepts or procedures.

Consideration of the nature of your project will help you to determine the scope of your evaluation study. This may result in the creation of a relatively long list of initial questions. The process of planning an evaluation is about the refinement of this list of questions using criteria such as: what is of most interest, what is feasible (in terms of paradigm of inquiry and methodology), what is practically possible, etc.

The Flashlight Evaluation Handbook(Ehrmann, 1999b) emphasises the importance of carefully designing the questions you ask:

"The process of laying the foundations for asking a good question is one of the most important and time-consuming aspects of any evaluation. The more you learn about your own perspectives on education, and those of the stakeholders in the evaluation, the more such values you will consider."

Another relevant quote from Ehrmann (Ehrmann, 1999a) is:

"the quest for useful information about technology begins with an exacting search for the right questions".

Besides defining the scope of the evaluation study and the evaluation questions, you need to think about how you intend to report your evaluation. We are encouraging participants to undertake a process leading to the writing of a formal paper, but you may wish to choose other appropriate reporting mechanisms. For example:

• The involvement of some students in evaluation might become a negotiated assessment task.
• The strategic value of your work within your own university should be considered. Which university committee might this work be of relevance to?

³Readers sceptical about the different paradigms presented here are advised to read the original Reeves (1997) paper and refer to its extensive bibliography. Some of the issues presented are too complex for an introductory handbook.