A LEARNING-CENTRED FRAMEWORK FOR WHOLE PROJECT EVALUATION
ANALYSIS AND DESIGN
DEVELOPMENT
IMPLEMENTATION
INSTITUTIONALISATION


Consistent with the monitoring requirements of funding agencies like CAUT and CUTSD, the framework in Table 2.1 assumes that piloting will occur at well planned milestones during the development of the project, and that adjustments will be made to the CFL to optimise its use and functionality (i.e. it is assumed that the development process will be 'guided' towards its objectives, not launched like a 'ballistic' missile on the settings determined by the initial design). Two different sets of questions have to be addressed if formative monitoring is to be effective:

Formative Monitoring of the Learning Environment
Can students use the CFL easily (can they navigate, gain access to materials, perform the requisite tasks, etc) and do they find the CFL attractive, approachable and accessible? We assume this aspect of evaluation has occurred in most cases, and it will be useful to include a brief section in the final report summarising how the monitoring was undertaken and what was unearthed and changed as a consequence.

Formative Monitoring of the Learning Process
Does the manner in which students use the CFL encourage the desired learning process and is there evidence that the learning is occurring? This may not have been examined in all cases (e.g. in online discussions), so there may be scope for improvements using an action inquiry approach (see Section 3).

Note that 'learning process' refers primarily to the cognitive processes associated with learning (cognitive strategies, knowledge-building, higher-order thinking, problem-solving, reflection, etc.). However, it is often useful also to take note of the ways in which students make use of the CFL in the learning environment — the contextual aspects of the learning process (in pairs, sporadically, self-directed or with tutor support, etc.).

A common misconception has been to assume that monitoring based on the first set of questions (about the learning environment) will also address the second set (about the learning process), but this is not so (Alexander, 1999; Beattie, 1994; Gunn, 1999). Elegant software that has little impact on learning is as common as software that could enhance learning if only students would use it. Obviously, neither of these combinations is optimal. The aim of formative monitoring should be to ensure that the CFL has a sustainable and beneficial impact on learning, subject to verification once the CFL is fully operational.

Various methods are available to obtain evidence on the Development phase, i.e. evaluation of the learning environment and the learning process as the CFL is being developed. Some of the more useful methods are summarised in the Tables 2.3 and 2.4, and further information can be obtained from the web sites listed. Table 2.3 lists approaches which can be used to obtain evidence about formative monitoring of the learning environmentand the contextual learning processes. Methods relevant to the cognitive learning processare listed in Table 2.4.

Table 2.3. Methods of obtaining evidence relevant to the formative evaluation of both the learning environment and the contextual learning processes.

Method and Purpose Further Information
Interviews and questionnaires
To obtain student and peer comment on the attractiveness, usability and functionality of the CFL		 Interviews:
http://www.icbl.hw.ac.uk/ltdi/cookbook/interviews/index.html#endhead
Interface Questionnaire:
http://mime1.marc.gatech.edu/MM_Tools/UIRF.html
Resource Questionnaire:
http://www.icbl.hw.ac.uk/ltdi/cookbook/resource_questionnaires/index.html#endhead
Checklists:
http://www.icbl.hw.ac.uk/ltdi/cookbook/checklists/index.html#endhead
Focus groups4
To elicit a range of student reactions to the CFL and interpret and prioritise the difficulties, or to interpret questionnaire responses		 http://www.icbl.hw.ac.uk/ltdi/cookbook/focus_groups/index.html#endhead
Observation or video of students using the CFL
To obtain a detailed understanding of the ways students use the CFL and the problems they encounter		 Observation:
http://www.icbl.hw.ac.uk/ltdi/cookbook/supplemental_observation/index.html#endhead
http://mime1.marc.gatech.edu/MM_Tools/ARF.html
Video:
http://www.icbl.hw.ac.uk/ltdi/cookbook/split_screen_video/index.html#endhead
User tracking
To obtain a detailed understanding of problems that students experience in using the CFL, based on computer capture of the paths that students follow through the CFL. Requires specialised software		 http://www.icbl.hw.ac.uk/ltdi/cookbook/system_log_data/index.html#endhead


Table 2.4. Methods suitable for obtaining evidence relevant to the formative evaluation of the cognitive learning process.

Method and Purpose Further Information
Student ratings of learning confidence
To judge how confident students are with CFL content		 http://www.icbl.hw.ac.uk/ltdi/cookbook/confidence_logs/index.html#endhead
Video of think aloud
To record how students are thinking as they use the CFL		 Students are asked to verbalise what they are thinking as they use the CFL. Useful when online thinking is not too demanding, but verbalising can 'drop out' under heavy cognitive loads.
Video-stimulated recall
To reveal how students are thinking as they use the CFL		 Students are shown a video of themselves using the CFL and asked to say what they were thinking and why (best used with the split screen technique so that the CFL and student actions are both visible). Less prone to the 'drop out' problem, but reliant on the video to cue memories rather than confabulations.
Teach-back
To reveal how a student's understanding is linked to the CFL		 Students are asked to use the CFL to 'teach' the interviewer about the material, and in doing so to show how the CFL assisted their understanding. More useful in open-ended 'constructive' CFL than in highly structured practice environments, but can be used in the latter.
Discussion archive
To examine the nature of student discussion in 'chat' environments		 Analysis of the interchanges between students in real time and asynchronous discussions, examining the nature of the interaction process and the quality of what is said.
Reflective journals
To obtain students' interpretations of the process of understanding and learning		 Students are asked to explain in writing how the CFL may have assisted them to develop their understanding and learning of key ideas, with emphasis upon the understanding and learning processes. Requires careful structuring and exemplification if the journal is to move beyond a fairly low-level description of events and experiences.

4The term focus group is often used interchangeably. Technically, a focus group is a specialised form of a group interview.