Section 2: Implementation

A LEARNING-CENTRED FRAMEWORK FOR WHOLE PROJECT EVALUATION
ANALYSIS AND DESIGN
DEVELOPMENT
IMPLEMENTATION
INSTITUTIONALISATION

The transition from development to implementation, although strongly drawn in Table 2.1, is often not so clear in practice, if only because committed academic teachers take every opportunity to fine-tune their teaching and resources with each new cohort of students. In other words, a summative evaluation of one offering of a unit often has formative consequences for the next offering. Nevertheless, there is a shift in emphasis once the CFL is fully incorporated into a subject unit and no longer has 'experimental' status. Students perceive it as one of many aspects of the unit and can react quite differently when their achievements are the focus of attention rather than the CFL itself. Also, in these days where unit outlines act as contracts between universities and students, there often is little opportunity to make major changes 'on the run' as is possible during pilot work. So, when the CFL is incorporated into the unit, it tends to be relatively stable for the duration of the unit except for bug fixes and other minor improvements (it's in this sense that the evaluation becomes 'summative' rather than 'formative' in emphasis). Three sets of questions now need to be answered:

Summative Evaluation of the Learning Process
Now that the CFL is part of the unit and not 'experimental', do students use it in the way intended? Does it encourage the desired cognitive learning process and is there evidence that the learning is occurring as the CFL is used? Are there unexpected cognitive benefits deriving from the ways in which the CFL is used (e.g. do students pose questions or connect ideas or create repetitive practice in ways that you didn't anticipate)? Are contextual processes much as you expected, or not (e.g. are students using the CFL in self-formed groups when it was conceived as a single-user system; are students using the software minimally or erratically when you thought it would be highly engaging)? What is the influence of the contextual processes on students' cognitive processes?

Summative Evaluation of the Learning Outcome
The questions under this heading shift from a focus on what is happening as the CFL is being used, to its impact on achievement towards the completion of the unit. For example, are improvements apparent in those assessment(s) that are directly based on the targeted learning? Are these improvements consistent with the learning objectives of the project, and can they be linked to the cognitive learning process fostered by the CFL? Have some unintended learning outcomes occurred, such as students forming self-study groups to bypass the poor quality of the CFL? Have other more 'remote' aspects of achievement benefited in understandable ways (e.g. curriculum flow-on or improvements in 'generic' capabilities)? Has there been a justifiable impact on pass rates or grade distributions in the unit? Are the benefits widespread or limited to some students?

Summative Evaluation of Innovation Appropriateness
Even if the CFL has the learning outcomes originally intended, other questions have to be answered before it can be claimed that the CFL is educationally appropriate in the context of its use. The following questions, although not exhaustive, give some sense of the additional matters to be considered:

Is the CFL integrated into the unit or does it function more as an adjunct, and what are the consequences? Do the learning benefits of the CFL outweigh its educational costs? For example, is the time allocated by students for use of the CFL appropriate given the other demands on their time? Is there evidence that students are trading-off the CFL area of the curriculum against other areas? If some aspect of the unit was displaced to make way for the CFL, is this omission appropriate given the objectives of the unit? Are there other educational cost benefits of the CFL? For example, has it allowed the reallocation of teaching time to other areas of the curriculum or enabled under-prepared students to be admitted to the unit? How have other aspects of the operation of the unit been affected by the presence of the CFL (e.g. impact on tutorials, seminars, exercises, group work, lecture attendance, etc)?

The methods relevant for summative evaluation of the learning process have been covered already in Tables 2.3 and 2.4, and are cross-referenced in Table 2.5. However, in the context of summative evaluation of the learning process, these caveats should be considered:

Usually students have no difficulty revealing how their knowledge and understanding are changing when the focus is on the CFL, as is the case when it is under development, but they often need reassurance about how the information will be used when, directly or indirectly, their use of the CFL is being assessed. This is a quite proper concern and is often best handled by ensuring that the staff undertaking the marking of the students' work are not involved in evaluation of the learning process.
It often is difficult to make use of all the detailed information that process evaluation generates, especially when the individual probing methods are used (e.g. think aloud, stimulated recall and teach-back). In these cases particularly, the sample should be limited to a small, workable number of students (between 6 and 12 is usually quite sufficient). To ensure that such a sample covers the range of student experience, it can be 'stratified' to include strugglers, those who are passing, and those who are doing well. If these distinctions cannot be made on formal grounds, then the students themselves can nominate which category they think they exemplify—most will be willing to do so provided the first caveat is satisfied.
Finally, it should be remembered that dilatory or inappropriate use of the CFL, and consequent inadequacy in the learning process, may have its origin beyond the CFL itself, and hence students should be asked at appropriate junctures why they are using the CFL as they are and how useful it is for their learning in the unit. For example, why would students diligently use the CFL if the targeted learning appears to be peripheral to the curriculum, or if the learning is not obviously being assessed even if it is 'core', or if they are learning quite well without it?

The methods relevant to summative evaluation of learning outcomes are detailed in Table 2.6, and the most useful documentation and methods for summative evaluation of innovation appropriateness are listed in Table 2.7.

Table 2.5. Methods suitable for obtaining evidence for summative evaluation of the learning process.

Method and Purpose	Further information
User tracking	See Table 2.3
Video + think aloud	See Table 2.4
Video + stimulated recall	See Table 2.4
Teach-back	See Table 2.4
Reflective journals	See Table 2.4

Table 2.6. Methods suitable for obtaining evidence for summative evaluation of the learning outcomes.

Method and Purpose	Further Information
Student confidence ratings To determine how confident students are with relevant areas of the unit	http://www.icbl.hw.ac.uk/ltdi/cookbook/confidence_logs/index.html#endhead
Concept maps To reveal how students interrelate and characterise key concepts	http://www.icbl.hw.ac.uk/ltdi/cookbook/concept_maps/index.html#endhead
Clinical interview To reveal how a student thinks about an idea or principle and/or how s/he reasons or solves problems	This method is often used in phenomenographic studies of students' conceptions of key ideas—see Lybeck, Marton, Stromdahl, & Tullberg (1988) for a detailed example. For general advice on interviewing: http://www.icbl.hw.ac.uk/ltdi/cookbook/interviews/index.html#endhead
Purpose-built assignments, exam questions To determine whether the CFL influences conventional learning outcomes	As noted in Section 1.x, standard assessments and grading procedures often are ill-suited to the evaluation of learning outcomes of new projects. Considerable care must be taken to ensure that the targeted learning is being tapped and graded appropriately—see McNaught, Whithear, & Browning (1999) and Reeves & Laffey (1999) for examples. Also see Biggs (1992) for an alternative way to grade.