Developing interactive multimedia courseware: Evaluating instructors' goals and learners' characteristics

Introduction

This illustrative project was designed for first year students taking introductory courses within the School of Electrical and Computer Engineering. These students have widely varying backgrounds and include Western Australian school leavers, overseas students from a range of countries and mature students who already have quite extensive relevant industrial experience. They thus represent a broad spread of familiarity with the oscilloscope and would be expected to vary greatly in their knowledge and skills concerning the instrument. An important aim of the multimedia project development group was to go beyond the capacities normally addressed by existing instructional resources in this area and instead provide a solid foundation for an informed problem solving approach to the use of the oscilloscope. This ambitious goal requires an endpoint quite different from that often anticipated for instruction since it requires the development of high level cognitive skills that allow the student to deal effectively with novel situations. Notwithstanding evidence to show that meaningful problem oriented approaches to learning are more likely than fact oriented approaches to lead to application of knowledge to solve problems, a low level emphasis on factual information remains typical of the majority of instructional resources (Adams, Kasserman, Yarwood, Perfetto, Bransford & Franks, 1988). The oscilloscope project sought to use multimedia to provide rich learning experiences that would stimulate more meaningful, problem oriented approaches involving higher levels of cognitive processing.

The focus in area of multimedia is often with the technology rather than on the outcomes it has for human learning. However, for the technology to fulfil its potential as an instructional tool, it is fundamental that it is used effectively since the technology of itself will not compensate for poorly designed instruction. Because the surrogate laboratory development went beyond the bounds of conventional instructional projects, it was considered important that the process of evaluating the project should run parallel to the development rather than simply measuring the end results. For this reason, the evaluation includes both formative and summative components. This paper deals with the formative evaluation that has been used to provide a information base to the developers about relevant aspects of the instructional context (including both instructors and learners).

Background to the research

Although the approaches used in this evaluation bear a superficial resemblance to traditional types of front end analysis (eg., Dick & Carey, 1990), they are in fact driven by a fundamentally different set of concerns. Whereas this conventional type of analysis is typically concerned with the structure of the content itself, the approach used in the evaluation reported here was founded on a cognitive rather than a behavioural approach (Case & Bereiter, 1984). What was of particular interest was the cognitive structures of the lecturers and students with respect to oscilloscope information and how these mental representations differed in terms of their contents and organisation. Previous research in a variety of subject domains has shown that those who are relatively naive in the area differ from those with expertise in the field in terms of both qualitative and quantitative dimensions of their mental structures. These differences have been widely reported in the expert-novice comparison literature (Chi, Glaser & Farr, 1988).

In the current context, attainment of the goal of establishing problem solving skills from the oscilloscope instruction would seem to rely on the capacity of the learner to build an appropriate mental representation of the oscilloscope as a dynamic device. This would allow the learner to model its operation mentally (Gentner & Stevens, 1983) and thus engage in the type of problem solving processes that were the desired outcome. For this reason, one of the important considerations in the formative evaluation was to explore aspects of the mental models that both lecturers and students had for the oscilloscope and provide this information to the course developers to guide their design processes. At one level, problem solving with this instrument involves processing a complex and abstract display (see Larkin, 1989). Both static elements (such as scales and labels) and dynamic elements (such as the knobs and the screen) comprise this display which in itself presents a considerable processing challenge to the user in terms of the volume and complexity of the visual information involved.

However at another more subtle level, effective problem solving also requires an understanding of how invisible events within the oscilloscope should be associated with external changes that take place in the display. Hence to obtain a proper characterisation of the task facing the designers of the project, detailed information is needed about the mental representation of this more abstract level of function. The evaluation used a variety of complementary techniques designed to collect such information and this combination of techniques helped to build up a picture of differences in the way students and lecturers perceived the oscilloscope.

Four components of the formative evaluation process are addressed in this paper: structured interview, content analysis, survey and simulation. Selected results from the first three of these evaluation components are presented here that illustrate (a) the fundamentally different perspectives of staff and students concerning various aspects of the oscilloscope and (b) the wide range of students' entry level knowledge and skills that had to be addressed in the instructional design and its implementation. The paper concludes with a consideration of preliminary work on the fourth component.

Structured interviews

Content analysis

Analysis of both types of manuals was also used as a basis for developing a range of techniques for characterising both the content to be addressed by the proposed instructional materials and the entry knowledge of the student population for whom the instruction was intended. The goal of these techniques was to establish the students' existing knowledge structures with respect to oscilloscopes and the knowledge structures that would be best suited to their future requirements in their chosen course. Information about the differences between these knowledge structures is central to developing sharply focussed instruction.

Survey instruments

1. Familiarity of relevant terms and concepts.
2. Knowledge of external parts of an oscilloscope and their functions.
3. Understanding an oscilloscope's internal processes.
4. Knowledge of operating procedures.

Results

In terms of existing knowledge of the oscilloscope, students in the Instruction-Operation groups would be expected to know most about how an oscilloscope works. Analysis with respect to key terms and ideas of this groups' responses to an open ended question about how an oscilloscope works showed that the most frequent responses concern aspects with manifestations that are external to the electrical processes which occur within the instrument (eg., Display, Screen). Although the next most frequent cluster of responses was concerned with internal or less immediately apparent aspects of operation, these tend to be very generalised (eg., Deflection, Electron Beam). The more detailed aspects of the fundamental concepts involved occur with a low frequency (eg. Trigger, Timebase). These findings suggest that the knowledge base of even the most experienced group emphasises surface level characteristics of the oscilloscope rather than the way it responds internally to electrical signals.

The mean values of the Instruction-Operation group's self ratings of their understandings of 43 terms and phrases associated with the oscilloscope showed that while understanding of some terms and phrases such as Input and Voltage were generally rated as 'good' to 'very good', results for others such as Double Channel and Dual Trace were much more mixed. In addition, there were still others such as Detent and Graticule where the largest response fell into the 'very poor' category. These results indicated that even among the students who were most familiar with the oscilloscope (ie., who had previously learnt about the oscilloscope and also had experience in using it), there was a considerable variation in understanding of basic oscilloscope related content. This indicated that across the whole group of students who would comprise the ultimate clients for the surrogate oscilloscope instruction, a very wide range of entry level knowledge and skills would have to be assumed. Clearly to deal with this amount of variation would constitute a major challenge for the instructional design process, requiring a highly level of flexibility to be built into the instructional materials.

More detailed consideration of the terms and phrases that were not generally well known showed that although some of them were little more than simple vocabulary items (such as Graticule), others referred to sophisticated concepts that are fundamental to understanding how the oscilloscope functions as an analytical tool. For example, the notion of a Dual Trace facility in an oscilloscope is fundamental to investigations that compare the input and output characteristics of components or circuits. Such comparisons form a major part of the work that students were required to do in their university electronics courses. An effective mental representation of the internal processes that occur within the oscilloscope in dual trace mode is central to interpreting the display produced when using this facility. It is not sufficient that students are merely familiar with this term; they must also be able to conceptualise the processes implied by the operation of the instrument in this mode.

Figure 1: Percentage of respondents mentioning basic
operations required to make oscilloscope function

Figure 1 compares the percentage of respondents in the staff group and the three categories of student who mentioned various basic operations needed to make an oscilloscope function correctly. It can be seen that adjustments referred to by a comparatively high percentage of the staff were often mentioned quite infrequently by the students, even those who reported having used the oscilloscope before (there is an interesting difference in the percentage of staff and students who mentioned that it was necessary to turn the power on!). These results suggest that even at the very basic level of setting up before any measurements are attempted, students' ideas of what is necessary to operate an oscilloscope accord poorly with those of their lecturers. It seemed that it was not simply the more sophisticated aspects of the oscilloscope that had to be addressed in the instructional materials, but also a variety of basic operations necessary before the more complex tasks could be carried out properly. This supported the view that the scope of the materials to be developed would need to be wide ranging.

A major outcome of the analysis of the jobs that respondents said could be done by the oscilloscope is shown in Table 1 (see appendix). This table provides a conceptual summary based upon four main categories of uses for the oscilloscope. The first three categories concern generic functions that can be applied across a range of situations (displaying, measuring, and comparing). These categories involve progressively more sophisticated use of the oscilloscope. For example, the measurement of a single frequency is less complex than the comparison of two frequencies. The fourth category covers various specific applications of these functions (such as testing other types of equipment or monitoring real world events). Within each of these categories, the entities that were of interest varied. For example, waveforms appears in all four categories because respondents mentioned the use of the oscilloscope to display waveforms, to measure waveforms, to compare waveforms and to perform more specific tasks such as testing and monitoring. However, particular types of electronic equipment (both domestic and more specialised) appeared only in the fourth category.

This categorisation was used as the basis for characterising how the various types of respondents saw the use of the oscilloscope. There were clear quantitative differences in the relative frequency with which many of the items from the table were mentioned by staff and students. For example, the measurement of waveforms and the display of sounds was mentioned only by student respondents while the comparison of frequencies and the measurement of phase were mentioned only by the staff. In addition, staff were much more inclined than the students to mention the measurement of certain entities (such as current and pulses) and the comparison of others (such as phase and amplitude). In contrast the students' responses emphasised the display itself and the context in which the oscilloscope was used (such as in monitoring heart beats or testing radio equipment) rather than the abstract operational aspects. The jobs many of the students tended to mention were consistent with the way this instrument is portrayed in the popular media. Since the staff were asked to list the jobs that students should know an oscilloscope could be used for, these results suggest that there may be important qualitative differences between the way staff and students mentally represent the oscilloscope as well as the clear quantitative differences in the knowledge base that would be expected.

Simulation

Conclusion

References

Bonner, J. (1988). Implications of cognitive theory for instructional design: Revisited. Educational Communications and Technology Journal, 36, 3-14.

Case, R. & Bereiter, C. (1984). From behaviourism to cognitive behaviourism to cognitive development: Steps in the evolution of instructional design. Instructional Science, 13, 141-158.

Chi, M. T. H., Glaser, R. & Farr, M. J. (1988). The nature of expertise. Hillsdale, NJ: Erlbaum.

Elen, J. (1991, August). Psychological bases of instructional design models. Paper presented at the Fourth European Conference for Research on Learning and Instruction, Turku, Finland.

Gentner, D. & Stevens, A. L. (1983). Mental models. Hillsdale, NJ: Erlbaum.

Kozma, R. B. (1991). Learning with media. Review of Educational Research, 61, 179-211.

Larkin, J. B. (1989). Display based problem solving. In D. Klahr & K. Kotovsky (Eds.), Complex Information Processing (pp. 319-341). Hillsdale, NJ: Erlbaum.

Perkins, D. N. & Salomon, G. (1989). Are cognitive skills context bound? Educational Researcher, 18(1), 16-25.

Winn, W. (1990). Some implications of cognitive theory for instructional design. Instructional Science, 19, 53-69.

Appendix: Jobs done by oscilloscope

Please cite as: Lowe, R. and Williamson, J. (1992). Developing interactive multimedia courseware: Evaluating instructors' goals and learners' characteristics. In Promaco Conventions (Ed.), Proceedings of the International Interactive Multimedia Symposium, 247-255. Perth, Western Australia, 27-31 January. Promaco Conventions. http://www.aset.org.au/confs/iims/1992/lowe.html