Proposed Evaluation Strategy

Introduction and Scope of the Strategy

The document outlines a strategy for evaluating the CUTSD Project in terms of how well the project outcomes are achieved. The tangible outcomes of this project consist of four participant outcomes and two products as shown below.

At the conclusion of this project, participants will have:

gained transferable skills in conducting meaningful evaluation of CFL projects using high standards of practice;

conducted a well-grounded evaluation of a CFL development project from the viewpoint of student learning outcomes;

experienced the action inquiry process;

published a scholarly paper on the results of the evaluation study.

The project will also result in deliverable products:

a staff development guide which contains theoretical and practical aspects of evaluation in the form of a handbook; and process aspects of the evaluation derived from the action inquiry process;

a volume of scholarly papers which details the results of the evaluation studies carried out as part of this project.

The evaluation of the project will concentrate on these outcomes. However, there are also broader benefits to the whole higher education sector, including the following:

the project will improve perceptions of the worth of the scholarship of teaching and learning;

inter-institutional and inter-disciplinary collaboration and knowledge sharing will be fostered;

an internet community of learners can serve as a model for future projects; the project will address wide-spread deficiencies in evaluation of CFL projects;

the project potentially enables the achievement of improved student learning outcomes.

The evaluation study will not address these outcomes specifically although information collected may assist in judging the extent to which these outcomes are achieved.

Evaluation of Project Outcomes

Evaluation is one of the key aspects of the action inquiry approach, being used to inform further action at each stage of the cycle. The effectiveness of the project will be determined using the following methods at appropriate stages during the project:

To evaluate whether participants have gained transferable skills in evaluation of CFL projects, they will be asked to describe their practice at the beginning and end of the project. Analysis of the differences will indicate the extent to which participants have developed new skills.

The quality of participants' evaluation practices in respect to student learning outcomes will be gauged by analysis of reports from participants about the impact of CFL on student learning.

To evaluate the usefulness of the action research process, participants will keep a journal which records their actions and reflections on that action. Analysis of journals will indicate the effectiveness of the action research process. Mentors and participants will be required to submit regular progress reports in an attempt to ensure the success of each evaluation project.

Acceptance for publication of papers will provide evidence of the quality of participants' work.

The staff development guide will be refined by feedback from mentors, outcomes from the action inquiry process and input from staff development experts.

Summative evaluation of the project will involve interviews with all participants and mentors. Further feedback will be sought from a colleague of each participant to provide objective evidence of new ways of practising evaluation. Additionally, independent observers will be asked to provide written feedback on the participants' workshops.

The strategy for evaluating each of the main project outcomes is outlined below in a table detailing the process to be used to provide information on each of the project outcomes and the product(s) which will be produced. Participants and mentors should read this carefully as they have responsibilities to collect and provide information at various points in the project. Of particular importance are the following:

Monthly status reports from each participant/mentor group and submitted on the proforma to the participant discussion forum

A three part report from each participant tracking their approach to evaluating CFL projects, submitted to either the participant discussion forum or Chris Bailey, Murdoch University, chrisb@cleo.murdoch.edu.au

Feedback on the staff development handbook, submitted on the proforma to Chris Bailey, Murdoch University, chrisb@cleo.murdoch.edu.au

Journals reflecting on the action research process, where kept by participants

Draft of paper on the project submitted to a refereed journal.

Reminders of when reports are due will be posted to participants.

Project Outcome

Evaluation Process

Evaluation Product

Gained transferable skills in conducting meaningful evaluation of CFL projects using high standards of practice.

Participants will develop evaluation plans and the implementation of these plans will be monitored by monthly reports. The reports will be produced by the participants in collaboration with the mentor. A pro forma for the reports is given below.

Standardised reports for each project able to be analysed by project or by stage of project.

Conducted a well-grounded evaluation of a CFL development project from the viewpoint of student learning outcomes.

Participant reflection on process of developing and implementing the evaluation plan. Three part process:

At the start of the project, describe how they would have approached the evaluation process.

After development of plan, describe what if any changes occurred due to involvement on the project and influence of the mentor.

At the end of the project, describe how they would evaluate their next teaching innovation project.

Collect reports from colleagues/independent observers on the evaluation process.

A three part document describing the participant’s reflection on the evaluation process at three points in time. These will be comparable across projects to identify recurring and unique issues for improving the staff development guide.

Experienced the action inquiry process.

Negotiate with the mentor how to record their experiences, ie keep a journal during the project.

Journals from some of the participants.

Published a scholarly paper on the results of the evaluation study.

Papers will be developed by participants and initial draft discussed at workshop in December 2000.

Paper accepted for publication in a refereed journal. As this is likely to be after the project is completed, submission of the paper will be used as an interim measure.

Refinement of staff development guide.

Analyse effectiveness of the guide through feedback sheets, participant reports and journals, group discussion on guide at end of year workshops.

Agreed revisions of guide. Revised guide available on the web.

Project Outcome	Evaluation Process	Evaluation Product
Gained transferable skills in conducting meaningful evaluation of CFL projects using high standards of practice.	Participants will develop evaluation plans and the implementation of these plans will be monitored by monthly reports. The reports will be produced by the participants in collaboration with the mentor. A pro forma for the reports is given below.	Standardised reports for each project able to be analysed by project or by stage of project.
Conducted a well-grounded evaluation of a CFL development project from the viewpoint of student learning outcomes.	Participant reflection on process of developing and implementing the evaluation plan. Three part process: At the start of the project, describe how they would have approached the evaluation process. After development of plan, describe what if any changes occurred due to involvement on the project and influence of the mentor. At the end of the project, describe how they would evaluate their next teaching innovation project. Collect reports from colleagues/independent observers on the evaluation process.	A three part document describing the participant’s reflection on the evaluation process at three points in time. These will be comparable across projects to identify recurring and unique issues for improving the staff development guide.
Experienced the action inquiry process.	Negotiate with the mentor how to record their experiences, ie keep a journal during the project.	Journals from some of the participants.
Published a scholarly paper on the results of the evaluation study.	Papers will be developed by participants and initial draft discussed at workshop in December 2000.	Paper accepted for publication in a refereed journal. As this is likely to be after the project is completed, submission of the paper will be used as an interim measure.
Refinement of staff development guide.	Analyse effectiveness of the guide through feedback sheets, participant reports and journals, group discussion on guide at end of year workshops.	Agreed revisions of guide. Revised guide available on the web.