PRAGMATIC PROCESSES OF EVALUATION RESEARCH QUESTIONS THE RESEARCH METHODOLOGY 5

There has been an argument among researchers as to whether the paradigm of inquiry determines the methodology employed. Some, such as Guba and Lincoln (1988), contend that research and evaluation methodologies are determined by the paradigm of inquiry that is adopted, while others, such as Patton (1988), argue that, the paradigm of inquiry adopted does not necessitate a single, inflexible methodological position. Patton's point of view is supported by others (Shulman, 1988; Salomon, 1991), and is reflected in the eclectic, mixed method approach adopted here, which enables the most appropriate evaluative approach to be used for each situation.

A useful approach is to develop an evaluation matrix (Reeves, 1999) to help you consider the most appropriate and feasible data production method for each of the questions identified in your evaluation plan. The set of research questions is tabulated against a selection of possible analysis tools, with the questions on the vertical side of the matrix, and a list of feasible tools on the horizontal side of the matrix. As you consider each question carefully you can choose the most appropriate data production method.

However, decisions have to be made about whether to obtain quantitative or qualitative data to provide evidence for each question. Without going into the issue in detail, it is necessary to consider a major difference between qualitative and quantitative data that helps to determine which kind we need. Because we create data instead of 'collecting' it, the data we create is a function of how we set out to create them, that is, the method. One way into this essential idea is the understanding that what makes quantitative data look so different from qualitative data is simply a matter of selection. Just as a map of scale 1:1 has its limitations, we would not wish to include everything about everything in an evaluation, even if we could. Therefore, we have to include some things and exclude others. The fundamental quantitative/qualitative choice in data creation is determined by the way in which we select the number of different components of how many different cases to observe. In evaluation, the different 'components' of a program tend to be the people, things, events, circumstances (context) and relations. The amount of data we need to produce for each of these is determined by our questions. In a crude, bipolar form, the options are:

• getting many examples of a few (of 'the same') components from many cases
  (ie. if one wanted to get every example of a phenomenon across a whole population over an extended period of time, one would produce the ultimate single dependent-variable statistical study);
• getting few examples of many ('different') components from one case
  (ie. if one were to try to get data on every aspect of every component in and about a situation, one would produce the ultimate holistic case study. This would have to be qualitative, as statistics don't work that well on cells where N=1, and the sheer number of words necessary would confine it to a single case).

Being more realistic, an example tending towards the first pole might be recording the number of words posted by all tutors to all students during all the online teaching across the whole of a university system. A study tending towards the other pole would be an analysis of the content, reading and cognitive levels, style and tone, pedagogic processes, typographical errors, logical sequencing, phatic interaction, etc. of all words posted by a particular tutor to a particular student during an online university unit. What data we choose to get on which components of a situation is, of course, our construction, because our choice will be determined by what it is we want to know.

Further information will be added to this section based on mentee needs and mentor feedback.