A LEARNING-CENTRED FRAMEWORK FOR WHOLE PROJECT EVALUATION ANALYSIS AND DESIGN DEVELOPMENT IMPLEMENTATION INSTITUTIONALISATION

Perhaps the least obvious phase in which evaluation appears to be useful is the analysis and design phase, because this is usually seen as preparation for what is to be developed and evaluated and hence not in need of separate evaluation in its own right. Yet, as a recent meta-analysis of CFL projects has demonstrated (Alexander, 1999; Alexander & McKenzie, 1998), many relatively unconvincing projects lack a clear learning rationale, and result in products for which few if any benefits can be claimed. This problem has led Alexander to conclude (1999):

"A greater emphasis on formative evaluation at the design stage could potentially reduce the wasted time and cost involved in producing these projects which are never likely to realise the intention of improving learning." (p 181).

In other words, just as the development of the CFL should be enhanced through piloting and associated formative monitoring, so the conceptualisation of the whole project should be subject to the scrutiny of informed peers (focussing on plausibility informed by scholarship), in order that improvements can be made to the design before the project is commenced.

Unfortunately, there will be little opportunity for such a cycle of design improvement in the projects selected for ASCILITE evaluation because, to meet tight deadlines, participating projects must already be well under way. Even so, it is essential that a sound analysis and specification be included in the evaluation report because it provides the foundation for the remaining components of the evaluation.

Three questions need to be answered to the satisfaction of informed peers:

If these questions are answered satisfactorily, then the foundations for the remaining evaluation should be in place. That is, knowing what teaching/learning process is needed and how the CFL is supposed to provide it, it should be apparent what to look for as learning is underway, and what kinds of learning outcomes should be evident once the process is well advanced or completed.

Nevertheless, we realise that it is easier to ask such questions than to answer them. This is where the mentors have an important role to perform by probing participants' initial attempts to answer the questions and making suggestions for refinement. A scheme that might assist in sharpening some of the learning assumptions inherent in a project can be found in the article by Reeves and Laffey (Reeves & Laffey, 1999) and in the report by (Hargreaves, 1999). These papers provide bipolar learning and teaching dimensions upon which projects can be rated. It may be useful for the mentor and participants to rate the draft response (to the three questions above) on relevant dimensions, and explore the similarities and differences in their ratings as the basis for improving the draft. Interestingly, Hargreaves shows how it can be informative to compare staff and student ratings on these dimensions, after the project has been implemented.

Some approaches which may be used to resolve the questions raised in the Analysis and Design phase of the evaluation are shown in Fig. 2.2.

Table 2.2. Obtaining evidence for the Analysis and Design phase of the whole project evaluation framework.