Multiple Choice Tests: Test scoring and analysis
Christina Ballantyne
The example following
shows the analysis of a test with forty items which was taken by two hundred
and nine students. Student names and numbers have been changed to ensure confidentiality.
The basic
output produced and interpretation of results is as follows.
Students' Total Scores
The total scores for all students taking the test are available
in two forms ie by student number only or by student name and
number (in alphabetical order).
| Student Number | Total Score
|
| 421733 | 012
|
| 500750 | 033
|
| 501365 | 026
|
| etc | |
| Student Number | Name
| Initial | Total Score
|
| 500750 | CUUKU | T
| 033 |
| 421733 | GKIFFIN | T
| 012 |
| 501365 | INDUKKUN | F
| 026 |
| etc. | | |
|
Top of page
Group Statistics
The table below shows the mean scores and the standard deviation
(spread of scores about the mean) for the whole class and for
the lower, middle and upper groups of students in the class. This
division of students in to three groups according to ability on
the test is an important first step in item analysis and will
be explained later:
| ************** Group Statistics **************
|
| Group | Number of people
| Mean score | Std. Deviation
|
| All | 209
| 22.92 | 05.27
|
| Lower | 65
| 16.69 | 02.69
|
| Middle | 77
| 23.09 | 01.45
|
| Upper | 67
| 28.78 | 02.28
|
Top of page
The Distribution of Students'
Total Scores
The print out includes a numerical and graphical representation
of the distribution of students' scores.
Score Frequencies of Students
| Score | 0
| 1 |
2 | 3
| 4 |
5 | 6
| 7 |
8 | 9
|
| No. of students | 0
| 0 |
0 | 0
| 0 |
0 | 0
| 0 |
0 | 1
|
| Cumulative Frequency |
0 | 0
| 0 |
0 | 0
| 0 |
0 | 0
| 0 |
0 |
| | |
| | | |
| | | |
| Score | 10
| 11 |
12 | 13
| 14 |
15 | 16
| 17 |
18 | 19
|
| No. of students | 1
| 2 |
3 | 3
| 1 |
7 | 6
| 11 |
11 | 11
|
| Cumulative Frequency |
2 | 4
| 7 |
10 | 11
| 18 |
24 | 35
| 46 |
57 |
| | |
| | | |
| | | |
| Score | 20
| 21 |
22 | 23
| 24 |
25 | 26
| 27 |
28 | 29
|
| No. of students | 8
| 14 |
17 | 13
| 14 |
19 | 14
| 7 |
14 | 10
|
| Cumulative Frequency |
65 | 79
| 96 |
109 | 123
| 142 |
156 | 163
| 177 |
187 |
| | |
| | | |
| | | |
| Score | 30
| 31 |
32 | 33
| 34 |
35 |
| | | |
| No. of students | 6
| 8 |
3 | 2
| 2 |
1 |
| | | |
| Cumulative Frequency |
193 | 201
| 204 |
206 | 208
| 209 |
| | |
|
Cronbach Alpha coefficient 0.7121
1Coefficient alpha is a general formula assessing the
internal consistency of the items in a test. In the case of dichotomous
items it specialises to what is known as the Kuder-Richardson
Formula 21 and is the average correlation among all possible values
of a test. This value, being effectively a correlation, ranges
between 0 and 1.
This value can be affected by the real distribution of abilities
- if everyone is very similar then it will have a low value. A
value of 0.712 indicates reasonably good internal consistency
and separation (discrimination) among persons.
Total Score Frequencies
Mean score of all students = 22.9
Issues to consider when interpreting the distribution of students'
total scores:
Distribution
- Is this the distribution you expected?
- Was the test easier, more difficult than you anticipated?
- How does the mean score of this year's class compare to scores
from previous classes?
- Should the scores be moderated or scaled in any way?
- If the distribution is bimodal, what characterises the two
groups?
- Is there a ceiling effect- that is, are all scores close to
the top?
- Is there a floor effect- that is, are all scores close to
the lowest possible?
Spread of Scores
- Is the spread of scores large?
- Are there students who are scoring low marks compared to the
majority of the students?
- Can you determine why they are not doing as well as most other
students?
- Can you provide any extra assistance? Is there a group of
students who are well ahead of the other students?
- Is it possible to extend these students?
| The test shown has an even spread of scores about the mean. There is a group of 11 students who have done less well than most of the group (ie scored 14 or less out of 40) and 5 or so students who are at the top of the class. If this test was designed to discriminate between the students it has succeeded. If, on the other hand, the test was designed to determine mastery of a topic or subject area, there are a significant number of students who require more preparation.
|
Top of page
Analysis of the Items on the Test
This print out shows:
Difficulty Rank and Item Number
Items are ordered from least to most difficult. Item 9 is the
easiest item on the test and item 15 is the most difficult.
Number of Categories (No. of Cat)
Multiple choice tests have two categories of response ie correct
or incorrect.
Item Score All
This column shows the number of students in the class who were
correct on each of the items.
Item Proportion
| all: | the proportion of all students who were correct on each item.
|
| lower: | the proportion of students in the bottom third of the scores who were correct on each item.
|
| middle: | the proportion of students in the middle third of the scores who were correct on each item.
|
| upper: | the proportion of students in the upper third of the scores who were correct on each item.
|
************** Item analysis **************
------------------------------------------------------------
Difficulty Item Item Proportion
| No. of Score ________________________
Rank Item Cat. All all lower middle upper DI
------------------------------------------------------------
01 09 2 187 0.89 0.72 0.96 0.99 0.26
02 39 2 173 0.83 0.75 0.83 0.90 0.14
03 40 2 170 0.81 0.75 0.86 0.82 0.07
04 07 2 165 0.79 0.66 0.84 0.85 0.19
05 27 2 165 0.79 0.62 0.81 0.94 0.32
06 08 2 163 0.78 0.63 0.83 0.87 0.23
07 06 2 160 0.77 0.62 0.78 0.90 0.28
08 21 2 152 0.73 0.55 0.74 0.88 0.33
09 28 2 152 0.73 0.52 0.77 0.88 0.36
10 14 2 149 0.71 0.45 0.77 0.91 0.46
11 12 2 144 0.69 0.45 0.73 0.88 0.43
12 04 2 142 0.68 0.55 0.66 0.82 0.27
13 22 2 142 0.68 0.55 0.65 0.84 0.28
14 19 2 140 0.67 0.57 0.57 0.88 0.31
15 10 2 139 0.67 0.37 0.70 0.91 0.54
16 17 2 138 0.66 0.46 0.68 0.84 0.37
17 25 2 138 0.66 0.58 0.60 0.81 0.22
18 13 2 137 0.66 0.46 0.66 0.84 0.37
19 30 2 126 0.60 0.38 0.61 0.81 0.42
20 01 2 123 0.59 0.37 0.61 0.78 0.41
21 24 2 123 0.59 0.45 0.55 0.78 0.33
22 23 2 120 0.57 0.32 0.57 0.82 0.50
23 18 2 119 0.57 0.37 0.57 0.76 0.39
24 05 2 115 0.55 0.29 0.61 0.73 0.44
25 31 2 113 0.54 0.32 0.58 0.70 0.38
26 34 2 112 0.54 0.40 0.58 0.61 0.21
27 36 2 111 0.53 0.32 0.55 0.72 0.39
28 20 2 106 0.51 0.32 0.49 0.70 0.38
29 03 2 099 0.47 0.17 0.43 0.82 0.65
30 32 2 093 0.44 0.32 0.38 0.64 0.32
31 02 2 092 0.44 0.20 0.47 0.64 0.44
32 29 2 086 0.41 0.38 0.38 0.48 0.09
33 26 2 079 0.38 0.17 0.44 0.51 0.34
34 37 2 077 0.37 0.35 0.35 0.40 0.05
35 33 2 076 0.36 0.22 0.36 0.51 0.29
36 35 2 072 0.34 0.28 0.22 0.55 0.28
37 11 2 070 0.33 0.14 0.35 0.51 0.37
38 38 2 058 0.28 0.22 0.22 0.40 0.19
39 16 2 034 0.16 0.25 0.19 0.04 -0.20 *
40 15 2 031 0.15 0.17 0.14 0.13 -0.03 *
------------------------------------------------------------
NB: * -- Item with negative Discrimination Index (DI).
Issues to consider in relation to the item analysis:
- Are the easiest items in your test, ie those with the lowest
difficulty ranking, the first items in the test?
- If the more difficult items occur at the start of the test,
students can become upset because they feel, early on, that they
can not succeed.
| The first item on the test (item 1) has a difficulty ranking of 20, ie it is the 20th most difficult item on the test, the second item a ranking of 31 and the most difficult item (ie difficulty ranking 40) is item 15. The easiest items on the test are items 9, 39 and 40. Clearly the items need to be rearranged so the easiest items are towards the front of the test. If the items in the test are arranged in sections, according to how the unit content has been covered, then items can still be ranked according to difficulty within each section.
|
- Are there any items which students have found to be more difficult
than you expected, ie they have a high difficulty rank with few
students gaining the correct answer?
- Are they genuinely difficult items in terms of understanding
unit concepts, or are they badly written or obscure items?
- Did the students have sufficient time to complete all of the
items on the test?
DI or Discrimination Index
This is calculated by subtracting the proportion of students correct
in the lower group from the proportion correct in the upper group.
It is assumed that persons in the top third on total scores should
have a greater proportion with the item correct than the lower
third.
This calculation of the index is an approximation of a correlation
between the scores on an item and the total score. Therefore,
the DI is a measure of how successfully an item discriminates
between students of different abilities on the test as a whole.
Any item which did not discriminate between the lower and upper
group of students would have a DI=0. An item where the lower group
performed better than the upper group would have a negative DI.
In general, DI's above +0.30 indicate an item which is working
well, but 0.20 is not bad.
The discrimination index is affected by the difficulty of an item,
because by definition, if an item is very easy everyone tends
to get it right and it does not discriminate. Likewise, if it
is very difficult everyone tends to get it wrong. Such items can
be important to have in a test because they help define the range
of difficulty of concepts assessed. Items should not be discarded
just because they do not discriminate.
- Are there any items with a negative discrimination index (DI)?
That is, items where students in the lower third of the group
did better that students in the upper third of the group?
- Was this a deceptively easy item?
- Was the correct answer key used?
| Items 15 and 16 have negative DIs. In the case of item 16, for example, 25% of the weakest students on the test were correct and only 4% of the best students were correct. An analysis of the alternatives will provide further information.
|
- Are there any items that do not discriminate between the students
ie. where the DI is 0.0 or very close to 0.0?
- Are these items which are either very hard or very easy and
therefore where you could expect a DI of 0?
| Items 15, 29, 37 and 40 have DIs below 0. 10. Item 40 is an easy item and therefore possibly not meant to discriminate between students (but note as an easy item it should not be the last item in the test). Items 15, 29 and 37 are difficult items and it may be expected that students from the top group are as likely as students from the lower group to be incorrect.
|
Overall, the items on the test discriminate as expected ie students
who score well on a particular item tend to score well on all
items in the test and students who score poorly on a specific
item tend to score poorly across all the items.
Appendix A shows the item analysis of
this test after the unit coordinator decided to remove items 15
and 16 . This analysis shows the items are discriminating well
but some re-ordering from least to most difficult items is still
required.
Top of page
Overall Structure of the Test
In the 1950s, Louis Guttman contributed a great deal to the structure
of tests. He enunciated the following key requirements that a
set of items should meet before the scores on the items could
be added meaningfully to give a single score.
If student A scores better that student B on the test, then student
A should have scored all the items correct that student B has,
plus some other items that are more difficult.
The perfect pattern of the Guttman Scale for 6 items scored 0
for incorrect and 1 for correct is shown below. Note that the
items are ordered according to the number of students who answered
the item correctly ie from easiest to the most difficult item
and the students are ordered according to their total scores ie
the least successful to the most successful student.
Guttman Pattern
The item analysis described above, and all item analyses, are
implicitly based on this structure, although it is seldom made
explicit. Making it explicit can be helpful in an overview of
the results.
This pattern will only be approximated in test results. However,
it can be expected that there will be predominantly 0's in the
upper triangle of the table and 1's in the lower triangle. If
this structure is not approximated it may be that something has
gone wrong in the test, either in the construction of the items,
the preparation of the students or the administration of the test.
In this case the items should not be added together to give a
total score for each student.
If an approximation to the Guttman structure is not achieved it
could be because the students are close in ability and the items
are of a similar difficulty. Therefore, the lack of a Guttman
structure is not always a cause for concern.
A printout showing student scores in order of total scores appears
below.
Student score in order of total scores
(this printout has been edited to allow ease of reading)
Person Total Responses
ID Name Score (items in difficulty order)
------------------------------------------------------------------------
541527PUKUIKI............ 009 0100000000011001000010001101000001000000
543574WUNG..............R 010 1000010110100000000001000100010000110000
545750TIYRUK............K 011 1011010100010000100000100000000100000011
513501WUU...............K 011 0000001110000101110110000000000000000011
421733GKIFFIN...........T 012 0110001100001000110000101100000100010000
511653HUNDKICKK.........B 012 1000101000010010011010010010010000001000
511553HUDGU.............J 012 0110100100101100000011101000000001000000
511518HU.KHIU.WU......... 013 1010110001010001000000100101110000100000
511622KHIY..............K 013 1111101101101100000010000000000100000000
511866TIN...............I 013 0111011110001100000100000100000000011000
512030BIKNUK............D 014 1100100101001111010010000000000100100010
511311INDUKKUN..........M 015 0111011010011100000010100110000100000010
510236BUNNUTT...........R 015 1111111000110000010100001111000000000000
523751CHIN..............C 018 1111000110010111001101001010011000010000
521522CUURKUN...........G 018 1011111000011100111000110101000101000000
522122DUVINUY...........J 018 1111011110000011110100010000010010110000
521301RIVINGKTUNU.......N 018 1111111011000011100010010001000001100100
520765RITTRU............K 018 1110011001111010101100000100011100100000
521186MINNING............ 018 1101111100011100101100000001100100010010
520672MCMURRUN..........M 018 1011111010101110001000100001100000010110
521002PUKUZ.............. 018 1110001100001110101111000001100001011000
541454PUKTUK............. 018 0011110001111001101010001010000001000111
520554TIUKNUY...........M 018 1111110101010100010011111100000100000000
544137BIKKIKI............ 019 1111110111110101100110101000000001000000
591888BURT..............M 019 1101001111000001101110100011001101000010
591015BUNI..............K 019 1111111000010101011010011100001001000100
553080HU................C 021 1101011111111011011010011011000010000000
551715KUKUH.............R 021 1110110111110101010101110001111000000000
553135KUKUMU............M 021 1011101111101011101100010110110000000100
550552NUKKIK............J 021 1111111111101011111000100100010000000001
553881TIN...............U 021 1111011110111100110101000011000001100001
551255THUMPKUN..........K 021 1111110100101110111101000111100000000100
551665VU................I 021 1111011101010011111001111000000001100001
551621IINKRIU............ 022 1111111111010101011000000100100110101100
541752BIKKUTT...........U 022 1111111100101011110011011100100010010000
543535CHINDKI...........J 022 1010010110100111001110010111101100011100
531034CRUIKY............J 022 1111011111001110000101001110001010010110
570727CUNTUNCIN.........K 022 0111111111100011001110000110001011101000
570527DIVIUK............K 022 1111111001001111110010011110000110100000
570307GIBBK.............J 022 0111111101111110100110010100001101001000
571850GUUDWIN...........C 022 1001111001011100101110100110011000111010
570836KHIW..............I 022 1100001011110100011111011011101100011000
530733RITKUVIC........... 022 1111101111101111101000010010010001011000
571610RUWKU.Y...........T 022 1011110101011101001011110000011110100001
570558CIMMIKINU.........K 023 1111111110111001110010011100111000000100
531613FUNCK..KITKIUN..... 023 1111111011101110101100001110010111000000
570566MUY...............C 023 1001111111111001110011111010000010011000
572828MUN................ 023 1001100011111001001111110101011101011000
571867UKURRIVIN.........N 023 1110011111111000011100111001101011001000
533706TIN...............W 023 1111111111111001111000011011110000000000
573883TIY...............K 023 1111101101111101110010111010101010000000
512717TIUDUMINN.......... 023 1111111011011111111110010001000001100000
571630THUKNTUN..........V 023 1111111110110011011001010110001000100011
583750HIRIM.............K 025 1111010111110011010111111110100010100010
580810HIKKURD...........P 025 1111111111001111111110001101100001100000
583801KUIH..............I 025 1110110101111101111111110101001001001000
583102RUUW..............M 025 1111111111100111011101011110011100000000
583785RIM...............Y 025 1111111111111110111101110100010100000000
583728MDKKIRRUH.........I 025 1111110100110111101101011010101111000010
533631MUNUNU............C 025 1101111111110111011110111101101000000000
583210NG................C 025 1111110111111100110001111101100011100000
581585PUKTUK............D 025 1111111111110100110111111000111001000000
541510KICHMUND..........K 025 1111111011010101111111011101000010000101
521157THIM..............K 025 1011011101110111011101111011101010001000
503805WUNG..............K 025 1111111101011011011010111010000110010101
503707WU................. 025 1110111011010111111111111100100110000000
501365INDUKKUN..........F 026 1101111111011110110111010111110000001100
531778DIWKUN............R 026 1111111111111101111110000010101101000100
521558FUKNINDU..........M 026 1111100010110111110001110010111111110100
533701HIN...............K 026 1110111111101111011111100101101010010000
503533HU................C 026 1001011111111011111110111011110110000000
503225KHUU..............K 026 1111111111111011001001111100011101010000
503530KUNG..............W 026 1110100111111111011101111110100000110001
503536RIM...............K 026 1110111111111110100111110011101010001000
563067KIKTIINDI.KUPIIT..M 028 1110101111111111111101110011111000011000
533783TING..............Y 028 1111100111111111111101110111100011010000
533636TUU...............C 028 1111111111101111101111101001100111100000
506667WIRKINKUN.........M 028 1111111111111110111011011101110000110000
560511CUNNINGHIM........J 029 1111111111110111111110111001101110001000
560667DUTI..............M 029 1011111111101011111111111010110100101100
56668?UR.KIDGU..........J 029 1101111011100101111111111111110100101100
521662FIRKURR...........K 029 1111111110111111110111111100110010100100
560556RUU................ 029 1111111111111111110110111101010100110000
560565MCGKITH...........C 029 1111111011011111111010111111111010100000
560766MUNUN.............J 029 1111111101111100111011001111110101001101
563081NG.PUI.RING........ 029 1110111111111111111101110011101100011000
513668UUY...............R 029 1110111111111111011101111011101110001000
560505VIN.KIUKKUN.......T 029 1111111101111111010100111101111110101000
530366BICK............... 030 1111111011111111110111111110110001010100
563672CHING.............T 031 1111111111010110011111111011101110111100
563673GUH...............G 031 1111111111111111011111110110101011110000
560738HINKITTY..........C 031 1111111111101111110111011110110101101100
561310RIM................ 031 1111111111111110111111111111100010110000
563853KUCKIRINGIM........ 031 1101111111111111111101100111111101011000
563568TUUH..............C 031 1101111111111011111111111011101001111000
565785WUNG............... 031 1111111111111101111111111100111010100010
533102CHU...............C 032 1111111111111111111111110011101111100000
530618DURINKCHUK........M 032 1011101101101111111011011111101111111101
561887TUNNUCHY..........M 032 1110110111111111111101101011111011101101
500750CUUKU.............T 033 1111111111111111101111111101011101101100
563806TIN...............U 033 1111111111111111111111111111110110001000
565807IKHRUY............. 034 1111111111111111111011001011111111111100
560662MCDUNIRD..........I 034 1101111111111111111111111101110101111100
560525MCGKITH...........K 035 1111111101111111111111111111111011111000
------------------------------------------------------------------------
| A printout of the Guttman pattern for your individual test is available on request.
|
Issues to consider in relation to the overall structure of the test
- Does the structure adhere to the Guttman Pattern?
| It is not easy to discern the pattern of responses when large numbers of students are involved. However, the test above has mostly 0's in the top triangle and 1's in the bottom triangle.
|
- Is there any missing data?
- Were some students unable to complete the test?
- Unless the test is a 'speed test' all students should be able
to attempt all of the items.
| Missing data is shown as a space (where a student has not answered the question) or an asterisk (where two responses to one question have been given). There is no missing data on the test shown.
|
- Does the pattern of responses for individual students warrant
further investigation?
- Are some students responding in a way that does not fit the
expected pattern?
That is, they got easy items wrong and difficult items right.
- Would it be useful to interview a selection of these students?
| For example, the student with the highest score on the test (35 marks) has answered one of the easiest items incorrectly and two of the weakest students overall (11 marks each) have answered two of the most difficult questions correctly. Were slips made? Were correct answers guessed? Are able students not learning the basic material and answering a selection of easy and difficult items correctly but not performing well on all items?
|
Top of page
Alternative (or Distractor) Analysis
This analysis provides the opportunity to study the responses
students make to each alternative on an item. The efficiency of
alternatives can be judged by inspecting the tables below. These
tables show the number and proportion of students in the lower,
middle and upper group who selected the correct answer as well
as the number of students choosing each alternative.
Alternative Analysis for Item 9
| Alternative Analysis for item 9 DI = 0.26
|
| Answer key: C
|
| Observations
|
| Alt | Lower
| Middle | Upper
| All |
| A | 8
| 2 | 1
| 11 |
| B | 8
| 1 | 0
| 9 |
| C | 47
| 74 | 66
| 187 |
| D | 2
| 0 | 0
| 2 |
| Total | 65
| 77 | 67
| 209 |
| Proportions
|
| Alt | Lower
| Middle | Upper
| All |
| A | 0.12
| 0.03 | 0.01
| 0.05 |
| B | 0.12
| 0.01 | 0.00
| 0.04 |
| C | 0.72
| 0.96 | 0.99
| 0.89 |
| D | 0.03
| 0.00 | 0.00
| 0.01 |
| Mean Score | 16.69
| 23.09 | 28.78
| 22.92 |
This is the easiest item in the test; 89 per cent of the students
have passed this item. Alternative D does not appear to be serving
any function as only two students have selected it. Alternative
C is clearly discriminating in the right direction but with most
students making the right choice. This item could be the first
on the test.
Alternative Analysis for Item 37
| Alternative Analysis for item 37 DI=0.05
|
| Answer key B
|
| Observations
|
| Alt | Lower
| Middle | Upper
| All |
| A | 41
| 40 | 25
| 106 |
| B | 10
| 6 | 3
| 19 |
| C | 5
| 3 | 5
| 13 |
| D | 9
| 27 | 34
| 70 |
| Total | 65
| 77 | 67
| 209 |
| Proportions
|
| Alt | Lower
| Middle | Upper
| All |
| A | 0.54
| 0.57 | 0.55
| 0.56 |
| B | 0.35
| 0.35 | 0.40
| 0.37 |
| C | 0.09
| 0.05 | 0.03
| 0.06 |
| D | 0.02
| 0.03 | 0.01
| 0.02 |
| Mean Score | 16.69
| 23.09 | 28.78
| 22.92 |
Item 37 is a difficult item, ranking 34th in the test. Alternative
A is a possible alternative answer and should be examined to ensure
that it is not, in fact, a correct answer, especially as a number
of students in the upper group have selected it. Alternatives
C and D are weak and need to be revised. Very few students are
'distracted' by these alternatives.
Alternative Analysis for Item 11
| Alternative Analysis for item 11 DI=0.37
|
| Answer key: D
|
| Observations
|
| Alt | Lower
| Middle | Upper
| All |
| A | 41
| 40 | 25
| 106 |
| B | 10
| 6 | 3
| 19 |
| C | 5
| 3 | 5
| 13 |
| D | 9
| 27 | 34
| 70 |
| Total | 65
| 77 | 67
| 209 |
| Proportions
|
| Alt | Lower
| Middle | Upper
| All |
| A | 0.63
| 0.52 | 0.37
| 0.51 |
| B | 0.15
| 0.08 | 0.04
| 0.09 |
| C | 0.08
| 0.04 | 0.07
| 0.06 |
| D | 0.14
| 0.35 | 0.51
| 0.33 |
| Mean Score | 16.69
| 23.09 | 28.78
| 22.92 |
Item 11 is also a difficult item, ranking 37th in the test. The
correct answer, D, discriminates well. Alternative A has been
chosen by a large percentage of students in the lower group and
this may identify a common misunderstanding. B and C have been
chosen by few students and need some revison. This item should
be placed later in the test.
Alternative Analysis for Item 16
| Alternative Analysis for item 16 DI=-0.20
|
| Answer key: A
|
| Observations
|
| Alt | Lower
| Middle | Upper
| All |
| A | 16
| 15 | 3
| 34 |
| B | 16
| 21 | 10
| 47 |
| C | 32
| 41 | 54
| 127 |
| D | 1
| 0 | 0
| 1 |
| Total | 65
| 77 | 67
| 209 |
| Proportions
|
| Alt | Lower
| Middle | Upper
| All |
| A | 0.25
| 0.19 | 0.04
| 0.16 |
| B | 0.25
| 0.27 | 0.15
| 0.22 |
| C | 0.49
| 0.53 | 0.81
| 0.61 |
| D | 0.02
| 0.00 | 0.00
| 0.00 |
| Mean Score | 16.69
| 23.09 | 28.78
| 22.92 |
Item 16 is one of the most difficult items in the test and it
discriminates negatively. Few students have selected the correct
answer, A, and of those who have, the majority are in the lower
third of the scores. The majority of students have chosen alternative
C. This item needs to be examined to ensure it is not ambiguous,
or, in fact, that C is not the correct answer. Only one student
has selected alternative D. Item 16 needs to be revised or discarded.
Summary
One of the principal advantages of having MCQ tests scored and
analysed by computer is the feedback available on how well the
test has performed.
Information is provided on:
- questions which are too easy or too difficult.
- questions where alternatives are not performing their proper
functions.
- common misconceptions students have (as indicated by alternatives
chosen).
- any students who are performing poorly and who require further
preparation.
- any outstanding students who could be extended.
Reading List
Andrich, D. (1994) Educational Measurement. Unit Materials:
Module 6. Murdoch University.
Ebel, R. L. (1965) Measuring Educational Achievement. Prentice-Hall
Inc: New Jersey.
Newble, D. and Cannon, R. (1989) A Handbook for Teachers in
Universities and Colleges. Kogan Page: London.
Top of page
Back to General Information on Multiple Choice Tests
Contact IRES (ires@central.murdoch.edu.au)
if you wish to organise the marking of a multiple choice test.
Appendix A
Item analysis of test after substituting items for items 15
and 16.
************** Item analysis **************
-----------------------------------------------------------
Difficulty Item Item Proportion
| No. of Score ________________________
Rank Item Cat. All all lower middle upper DI
-----------------------------------------------------------
01 25 2 281 0.91 0.84 0.95 0.94 0.11
02 10 2 260 0.84 0.64 0.89 0.99 0.35
03 01 2 258 0.84 0.68 0.87 0.95 0.27
04 30 2 258 0.84 0.85 0.84 0.83 -0.02*
05 33 2 253 0.82 0.67 0.87 0.91 0.24
06 28 2 251 0.81 0.68 0.84 0.92 0.24
07 07 2 244 0.79 0.62 0.84 0.91 0.29
08 09 2 241 0.78 0.59 0.84 0.90 0.30
09 08 2 236 0.77 0.52 0.81 0.97 0.45
10 22 2 233 0.76 0.56 0.75 0.97 0.41
11 31 2 231 0.75 0.52 0.78 0.95 0.44
12 29 2 227 0.74 0.46 0.78 0.95 0.49
13 14 2 222 0.72 0.55 0.71 0.92 0.37
14 15 2 222 0.72 0.48 0.75 0.92 0.44
15 20 2 218 0.71 0.53 0.73 0.86 0.33
16 13 2 216 0.70 0.44 0.74 0.92 0.48
17 05 2 204 0.66 0.49 0.65 0.86 0.37
18 24 2 204 0.66 0.49 0.62 0.91 0.41
19 11 2 203 0.66 0.42 0.72 0.83 0.41
20 18 2 203 0.66 0.51 0.64 0.85 0.35
21 23 2 200 0.65 0.49 0.62 0.85 0.36
22 02 2 191 0.62 0.56 0.67 0.61 0.05
23 04 2 181 0.59 0.37 0.58 0.83 0.45
24 19 2 181 0.59 0.38 0.58 0.80 0.42
25 32 2 178 0.58 0.32 0.61 0.80 0.49
26 21 2 177 0.57 0.33 0.54 0.89 0.56
27 17 2 173 0.56 0.22 0.60 0.86 0.64
28 35 2 169 0.55 0.47 0.55 0.63 0.16
29 06 2 165 0.54 0.32 0.49 0.83 0.51
30 03 2 158 0.51 0.30 0.51 0.75 0.45
31 37 2 155 0.50 0.36 0.45 0.72 0.36
32 40 2 151 0.49 0.19 0.51 0.78 0.59
33 39 2 133 0.43 0.16 0.44 0.70 0.54
34 26 2 126 0.41 0.24 0.42 0.57 0.33
35 34 2 116 0.38 0.25 0.38 0.51 0.25
36 36 2 114 0.37 0.25 0.36 0.51 0.25
37 12 2 100 0.32 0.18 0.28 0.55 0.38
38 27 2 093 0.30 0.12 0.27 0.54 0.42
39 16 2 084 0.27 0.14 0.21 0.51 0.36
40 38 2 077 0.25 0.10 0.17 0.53 0.43
-----------------------------------------------------------
NB: * -- Item with negative Discrimination Index (DI).
Top of page
|
