A LEARNING-CENTRED FRAMEWORK FOR WHOLE PROJECT EVALUATION
ANALYSIS AND DESIGN
DEVELOPMENT
IMPLEMENTATION
INSTITUTIONALISATION


Questions concerned with the institutionalisation of CFL are rarely addressed in published evaluations, no doubt because they are difficult to produce evidence for, and because most projects are organisationally 'brittle'—there is little uptake outside of the host department and most projects don't survive the departure of the project champion (Alexander & McKenzie, 1998). Even so, it is hoped that some of the projects participating in the ASCILITE evaluation will be able to address some of the following questions, if only tentatively:

Impact evaluation:
What is the impact of CFL-enhanced learning on other aspects of the course? For example, have cognate units reported flow-on benefits (see (Gunn, 1999) for a positive instance, and (McNaught et al., 1999) for a negative case)? Is it possible to trace some improvements in 'generic' capabilities to the influence of the CFL? Have benefits been detected beyond the academy, for example in work placements or postgraduate employment? Are improvements in the grade distributions of the unit reflected in retention, progress and pass rates for the course? Has the project been reported in the scholarly literature and have there been any scholarly benefits (citations, uptake)?

Maintenance evaluation:
Are the educational benefits of the CFL (within and beyond the unit) sustainable given its maintenance and opportunity costs? For example, does the CFL require specialised computing resources that have limited utility outside the unit? Is the unit being subsidised by other units in the course (or could they also attract similar levels of support)? Are the peak loads on support staff interfering with the needs of other units in ways that cannot be offset? Have there been any flow-back benefits from uptake in other departments or institutions (enhancements, cost-recovery)?

For general advice on the conduct of cost benefit analyses, see:
http://www.icbl.hw.ac.uk/ltdi/cookbook/cost_effectiveness/index.html#endhead


Table 2.7. Methods suitable for obtaining evidence for summative evaluation of innovation appropriateness.

Method/Documentation and Purpose Further Information/Comment
Unit descriptions
To record changes in curriculum emphasis		 Before and after comparisons of syllabus structures and assessments.
Should be compared with students' perceptions of emphases because of potential 'hidden curriculum' effects.
Assessment records
To look for changes in the patterning of achievement across different areas of the curriculum		 It may be difficult to document changes in students' patterns of achievement if the assessments have been changed (from previous offerings of the unit) to optimise the fit with the CFL (see comments in relation to purpose built assessments in Table 2.6 above).
Student interviews
To obtain students' experiences of the curriculum, the emphases they adopted, and their reasons for doing so		 Individual:
http://www.icbl.hw.ac.uk/ltdi/cookbook/interviews/index.html#endhead
Group:
http://www.icbl.hw.ac.uk/ltdi/cookbook/focus_groups/index.html#endhead
Student questionnaires
To obtain evidence on, the emphases adopted by students		 General advice on questionnaire construction:
http://www.icbl.hw.ac.uk/ltdi/cookbook/questionnaires/index.html#endhead
Flashlight Current Student Inventory:
http://flashlightonline.wsu.edu/ (password available to project participants)
Peer and student ratings of pedagogical dimensions
To localise aspects of the CFL that may not be experienced as intended		 Refer to articles by Reeves & Laffey (1999) and (Hargreaves, 1999).
Staff allocation records
To note changes in patterns of staff support		 Before and after comparisons of staff deployment (quantum and pattern).