Graduate Skills Assessment
Report on the Exit 2000 and Entry 2001 tests

Christina Ballantyne
Teaching and Learning Centre
August 2001

• Introduction
• Murdoch Students and Programs
• Results
• Student Feedback
• Costs of Administering the GSA at Murdoch
• Conclusion
• References

Introduction

The Graduate Skills Assessment (GSA) is national test which is conducted by the Australian Council for Educational Research (ACER). The test was administered in the first instance to final year bachelor degree students in October/November 2000 - the Exit 2000 test - and to entering students in February 2001 - the Entry 2001 test. All Australian universities were invited to participate; nineteen institutions undertook the Exit 2000 test, twenty the Entry 2001 test. The rationale behind the entry and exit tests is that students who sit both tests will have a measure of the value-added to their skills by their university studies, and the institution will be able to identify the skills gained by their graduates and the programs of study in which those skills are being well developed. Graduating students will also have a measure of their skills benchmarked against national criteria to present as evidence to prospective employers. Murdoch University decided to take part in order to give its students the opportunity to have their skills assessed and to gain formal recognition of the results. Sitting the test is currently voluntary for students. Six thousand places for each test were funded nationwide by DETYA and all Murdoch students who applied to take the tests received a funded place The Teaching and Learning Centre is currently responsible for the administration of the test at Murdoch.

The four skills being assessed by the GSA are -

• Written Communication,
• Critical Thinking,
• Problem Solving, and
• Interpersonal Understanding.

These four skill areas were identified by DETYA as those most important to employers and most commonly listed by universities in their sets of graduate attributes. The Written Communication component is assessed by a one hour examination consisting of two tasks, a Reporting task and an Argument task. The other three skills are assessed by a two hour multiple choice examination. Sample questions are attached.

The tests are scored by ACER and the results returned to the universities. Results from the exit test were sent to students in March, those from the entry test in early June. Each student is provided with their individual result for each component. The results are presented graphically, showing the student's scores in relation to the middle 60% of students from a similar field of study, and from the total group of students tested. A sample copy of the test results for an individual student is attached.

Further information on the components and on the validity and reliability of GSA is provided in the ACER reports Graduate Skills Assessment, Summary Report GSA Exit 2000; and Graduate Skills Assessment, Summary Report GSA Entry 2001. http://www.acer.edu.au/unitest/index.html

Murdoch Students and Programs

Exit test 2000

At the suggestion of Careers Centre staff, it was decided to target those programs of study with the most students making requests for information on having skills assessed or recognised.

Several other programs were included to provide balance across disciplines. The programs targeted were -

• Environmental Science
• Biological Science
• Sociology
• Psychology
• Commerce
• Economics
• Tourism
• Communication Studies
• Engineering
• Veterinary Science

Multiple copies of the GSA brochure were distributed to chairs of the targeted programs and made available at the Careers Centre, Student Learning and Student Services offices and on the web. In addition all program chairs and divisional offices were given copies of the brochure and application form and asked to promote it to their students.

A somewhat disappointing total of 92 students sat the test in two sittings on the South Street campus and one sitting on the Rockingham campus. Table 1 shows the demographic make up of the group compared to the national group for the same test. For the Exit test, students who

were over 25 years of age, ie born 1975 or earlier, were considered as mature-age. This is in keeping with the guidelines used by the GCCA (Graduate Careers Council of Australia) when reporting employment data from the Graduate Destination Survey. Twenty-six percent of the Murdoch group were overseas students. There are no data available on the overseas/Australian ratio of the national group.

In an attempt to increase the participation of students, Divisions were consulted and it was decided to target students entering certain programs of study. The programs targeted were -

• Environmental Science
• Psychology
• Mass Communication
• Commerce
• Veterinary Science
• Biomedical Science
• All students commencing at Rockingham campus

Given the difficulty of communicating the information about the test to incoming students, each program determined its own strategies for enrolling students in the test. These included informing students of the test by letter, providing information at the pre-enrolment sessions and encouraging students to sign up for the test at enrolment. In conjunction with these measures to inform students enrolling in specific programs, all commencing students were mailed a flyer on the test with the 'Getting Started' booklet produced by the Office of Prospective Students.

As a result of this extra promotion a total of 173 students (nearly double the number who sat the Exit 2000 test) participated in the test which was conducted over four sittings. Three were held during Orientation Week, two on the South Street campus and one at Rockingham. Students from any program were invited to enrol for these sittings. The School of Environmental Science decided to hold a sitting for its own students; this was held during the first week of semester. Thirteen students took part in that test. Table 1 shows the demographic make up of the group compared to the national group for the same test. Only six per cent of the Entry 2001 students were overseas students. Overall, the two samples of Murdoch students who took the test were similar to the national samples.

Students were required to select, from a list provided by DETYA, which field of study their particular course was closest to, so that their results could be aggregated with those of other students in similar programs. This allowed students to gauge their performance against others studying in similar fields both at Murdoch and nationally. Table 2 shows how each of the Murdoch programs equate to the larger field of study areas.

¹Twenty-one students from the Biomedical Science program and nineteen students from Veterinary Biology sat the Entry 2001 test. Twelve Biomedical Science and six Veterinary Biology students assigned themselves to the General Science/Mathematics category. The remaining nine and thirteen, respectively, selected the Veterinary/Medicine/Dentistry category.

Results

The results for both the Exit 2000 and Entry 2001 tests are presented in this section. The number of students who took the tests is relatively small for any one university or field of study, therefore, extreme caution should be taken in interpreting these results as representing all students in a university or program, or for comparing results across programs, universities or cohorts. However, there are some clear trends in the results and the analysis below shows what is possible to extract from the tests in the future when more data will be available.

Table 3 provides the mean scores and standard deviations for the national and Murdoch groups for both tests. An examination of the scores shows that those taking the Exit 2000 test outperformed those students taking the Entry 2001 test on each of the skills. Care should be taken, however, in drawing any conclusions as the sample both nationally and for Murdoch, was small, largely self selected and involves different cohorts of students for the two tests. In the future, when the same cohort of students undertakes both tests, and hopefully larger samples are achieved, statistical tests could be run to ascertain whether the difference between Entry and Exit scores are statistically significant.

Even with the small samples for the current two tests, differences were found nationally in the performance of students from different fields of study. When looking at this 'value-added' between the Entry and Exit tests at institutional and national levels it is important to bear in mind the varying proportions of students in different fields of study. National totals given in Table 3 include fields of study which are not represented in the Murdoch cohort, eg Nursing. . Any comparison of Murdoch scores with national scores will be confounded by this varying mix of fields of study. Table 4 shows the proportion of Murdoch students and of the total group represented in each field of study for both tests. Murdoch is over-represented in General Science/Mathematics and Law/Legal Studies, but generally under-represented in Engineering/Architecture and absent in Nursing. Therefore the comparison of Murdoch's institutional results with the national averages in Table 3 will be affected by the different program mix of students who sat the tests. These comparisons should be treated with caution and may not represent the results which might be achieved if all students sat the tests. Given this caution, the Murdoch students who sat the Entry test performed as a group above the national average in all areas and those who sat the Exit test performed above the national average in the Report and Argument Tasks, but slightly below on the other three areas.

ACER also report that differences exist in performance on the basis of gender. For example, females generally outperformed males on the Interpersonal Understandings component, and males generally outperformed females on the Problem Solving component. There was no difference, however, in either of the Written Communication tasks or in Critical Thinking. Age also appears to be related to differences in scores in certain components, but further research is required in this area before any conclusions can be drawn. Murdoch student numbers were too small to make these comparisons, but there is no reason to suppose that the Murdoch cohort would be any different from the national group, particularly as the groups were similar demographically.

Possible uses of the data

While numbers are small for these particular tests, there are three possible useful ways of analysing the results in future given a larger and more representative sample.

1. A comparison of student scores across fields of study nationally, examining which fields of study have performed well in each skill area.



2. A comparison of student scores in each field of study at Murdoch University with those of students nationally in the same field of study, showing where Murdoch students are performing above and below the national average for their discipline.



3. Calculating the 'value-added' by a Murdoch University degree, ie comparing the scores of a cohort of students in a given field of study at entry and exit level.

Table 5 provides details of which fields of study performed well nationally in each skill area. Generally Veterinary/Medicine/Dentistry, Law/Legal Studies and Arts/Humanities students have performed well across all skills.

Table 6 provides the results for each of the skills on both tests for each of the eight fields of study which include Murdoch programs. Comparison of performance between sub-groups participating in the GSA needs to be done with caution because of the largely voluntary nature of student participation and the low degree of take-up both nationally and at Murdoch University. While Table 3 indicated that generally Exit 2000 students outperformed Entry 2001 students, when we examine individual fields of study in Table 6 this is not always the case. The different demographics and field of study mix of the two groups does not allow for valid comparisons on these tests. For future tests where the same cohort is being compared a measure of 'value-added' - either positive or negative - might be calculated.

3. Value-added

The numbers of students undertaking the GSA in 2000 and 2001 is too small to provide any valid measure of the value added by a university degree. This test, however, does have the potential with larger numbers and when a cohort takes both assessments to compare entry and exit. The changes between entry and exit scores seen in Table 6 are generally positive changes. Variation in the amount of change in different fields of study suggests that this is not simply maturation. A larger and more representative sample would allow better measurement of this 'value-added' in the various skill areas which are addressed by a university degree. Table 7 provides a model of how value added might be shown. The values in the cells show the extent to which the Exit 2000 scores were greater or smaller than the Entry 2001 scores for both Murdoch University students and the national cohort. Statistical tests could be run to ensure that there was a significant difference between the two tests and that the findings had not occurred by chance. This would represent the value added by a program if the students who took the Entry test and the Exit test were the same cohort. This is not the case for this analysis but will be in possible in future analyses.

As Murdoch students from two fields of study, Education/Social Work and Engineering/Architecture were represented in the Entry 2001 test only, scores from these fields of study have been omitted. Figures shown in bold indicate where Murdoch gains are greater than gains nationally. At Murdoch University the highest gains have been achieved by Arts/Humanities students in the Argument task, by Veterinary students in the Report task and by Computer Science/IT students in Problem Solving.

Student Feedback

At the end of each test, students at Murdoch were asked to complete a feedback sheet asking why they had decided to participate, what they thought they might gain and for their overall impressions. The results show that students who completed the Entry 2001 test were sitting the test to get an estimation of their own generic skills and ascertain where they needed to improve. To a lesser extent the results of the test were seen as something which would be useful to include in future employment applications. Exit 2000 students gave very similar responses. They felt that test would provide them with an accurate assessment of their skills and with a report that may be useful to include in a resume. Few students commented on the test itself. Of those who did most felt that the multiple choice section was too long.

Costs of Administering the GSA at Murdoch

The ACER, who designed, printed and marked the GSA, estimated that the cost to them of this was $22.00 per student. DETYA provided funding for 6000 student places across the first two tests. For the Exit 2000 and Entry 2001 tests, all Murdoch student places were funded by DETYA. There are also administrative costs at university level which include overall supervision by the chief supervisor (ie the organisation and administration of the tests, preparing information for program chairs and students, mailing results to students, preparing reports and liaising with ACER), supervision for individual tests, catering for students in the break between the two parts of the test, promotion, stationery, and postage. Costs for the Exit 2000 and the Entry 2001 tests combined are outlined below. Apart from the Exit 2000 test on the Rockingham campus where only one supervisor was required, each test sitting required an average of two supervisors for five hours. The Teaching and Learning Centre covered the administrative costs of the first two tests.

Murdoch University has approximately 2500 students commencing in undergraduate programs each year, and approximately 2000 graduands. To obtain a sample size which would provide the university and Schools of study with information on the value-added by a Murdoch degree we would hope for around 15 to 20 percent of the university's commencing and graduating students, with a minimum of 30 students from each School, giving a target figure of approximately 800 students across the two tests each year. An increase in numbers of this nature would not necessarily incur more supervision costs as duties of the chief supervisor would remain unchanged and the same number of test sittings would be adequate with some extra supervisors required. The resulting rise in the costs to cover 800 students sitting the test each year would increase the total to approximately $8,500, thus resulting in a per student cost of $10.63.

For the upcoming GSA Exit 2001, DETYA are asking the universities to contribute $10 for each student who sits the test. This would increase the overall costs to Murdoch University to $16,500 or $20.63 per student for 800 students. The TLC budget does not currently provide for costs of this magnitude. It is suggested that if divisions feel this type of assessment is sufficiently beneficial to their students as well as an indicator of the value added by their programs, they should be willing to fund $20.00 per student from divisional budgets.

Conclusion

The GSA has the potential to provide the university with valuable information about the skills with which its programs are equipping graduates. However to do this students need to be encouraged to take the test, particularly at entry level. Programs whose students complete both entry and exit tests will gain the most benefit from the assessments as the differences in mean scores between the two tests for the same cohort of students will give the most valid measure of the value added by their program. Student dropout will always mean a smaller pool will be available to take the exit test, so efforts are needed to promote the test to ensure an adequate number of entering students take it. As shown above, a greater number of students will not only be able to provide more useful information but will also be more cost effective for the university.

The overall takeup rate of Murdoch students for both tests was somewhat disappointing, however it is probably to be expected given that it is a new initiative, and is consistent with anecdotal evidence from other universities. The GSA is continuing with the Exit 2001 test which will be conducted in September/October. If Murdoch University is to continue to participate then there needs to be greater support from the Central Administration and Divisions to ensure a wider distribution to students, of the information regarding the test and its benefits to encourage greater participation.

References

Australian Council for Educational Research, Graduate Skills Assessment, Summary Report GSA Exit 2000; http://www.acer.edu.au/unitest/index.html, February 2001

Australian Council for Educational Research, Graduate Skills Assessment, Summary Report GSA Entry 2001, http://www.acer.edu.au/unitest/index.html, June 2001