IIMS 92 contents
[ IIMS 92 contents ]

Developing interactive multimedia courseware: Evaluating instructors' goals and learners' characteristics

Richard Lowe and John Williamson
Faculty of Education
Curtin University


Introduction

This paper arose from a development project that aimed to investigate the potential effectiveness of multimedia systems as a means of providing instruction in the use of complex laboratory equipment. It reports work in progress on the evaluation of the Surrogate Laboratory Project developed by the Image Technology Research Group of the School of Electrical and Computer Engineering at Curtin University of Technology. This project is designed to explore the opportunities that multimedia systems provide for overcoming the current limitations on teaching the use of high technology measuring and diagnostic instruments to large numbers of students with varied backgrounds. The intention of the pilot development that is dealt with in this paper is to provide a highly realistic multimedia replica of an oscilloscope which also has the operational characteristics found on the actual instrument. This combination of features gives the instructional system the capacity to go far beyond what can provided by conventional simulations and thus allows the student to participate in a well rounded and convincing laboratory experience (hence the concept of a surrogate laboratory).

This illustrative project was designed for first year students taking introductory courses within the School of Electrical and Computer Engineering. These students have widely varying backgrounds and include Western Australian school leavers, overseas students from a range of countries and mature students who already have quite extensive relevant industrial experience. They thus represent a broad spread of familiarity with the oscilloscope and would be expected to vary greatly in their knowledge and skills concerning the instrument. An important aim of the multimedia project development group was to go beyond the capacities normally addressed by existing instructional resources in this area and instead provide a solid foundation for an informed problem solving approach to the use of the oscilloscope. This ambitious goal requires an endpoint quite different from that often anticipated for instruction since it requires the development of high level cognitive skills that allow the student to deal effectively with novel situations. Notwithstanding evidence to show that meaningful problem oriented approaches to learning are more likely than fact oriented approaches to lead to application of knowledge to solve problems, a low level emphasis on factual information remains typical of the majority of instructional resources (Adams, Kasserman, Yarwood, Perfetto, Bransford & Franks, 1988). The oscilloscope project sought to use multimedia to provide rich learning experiences that would stimulate more meaningful, problem oriented approaches involving higher levels of cognitive processing.

The focus in area of multimedia is often with the technology rather than on the outcomes it has for human learning. However, for the technology to fulfil its potential as an instructional tool, it is fundamental that it is used effectively since the technology of itself will not compensate for poorly designed instruction. Because the surrogate laboratory development went beyond the bounds of conventional instructional projects, it was considered important that the process of evaluating the project should run parallel to the development rather than simply measuring the end results. For this reason, the evaluation includes both formative and summative components. This paper deals with the formative evaluation that has been used to provide a information base to the developers about relevant aspects of the instructional context (including both instructors and learners).

Background to the research

The purpose of the formative evaluation was to provide initial guidelines and ongoing support for the development team in their approach to the instructional design of the package. In line with important recent shifts in the approach to instructional design (Bonner, 1988; Elen, 1991; Winn, 1990), the evaluation was undertaken from a cognitive perspective that emphasised the learners' internal mental processing rather than the structure of the content area or the characteristics of the delivery platform. Nevertheless, both of these aspects were seen as an important background to the evaluation because of the specialised nature of the subject domain (Perkins & Salomon, 1989) and the instructional media concerned (see Kozma, 1991). The intention was to find ways in which the power and flexibility of the technology supporting the surrogate laboratory approach could be used to best effect within the specified instructional context.

Although the approaches used in this evaluation bear a superficial resemblance to traditional types of front end analysis (eg., Dick & Carey, 1990), they are in fact driven by a fundamentally different set of concerns. Whereas this conventional type of analysis is typically concerned with the structure of the content itself, the approach used in the evaluation reported here was founded on a cognitive rather than a behavioural approach (Case & Bereiter, 1984). What was of particular interest was the cognitive structures of the lecturers and students with respect to oscilloscope information and how these mental representations differed in terms of their contents and organisation. Previous research in a variety of subject domains has shown that those who are relatively naive in the area differ from those with expertise in the field in terms of both qualitative and quantitative dimensions of their mental structures. These differences have been widely reported in the expert-novice comparison literature (Chi, Glaser & Farr, 1988).

In the current context, attainment of the goal of establishing problem solving skills from the oscilloscope instruction would seem to rely on the capacity of the learner to build an appropriate mental representation of the oscilloscope as a dynamic device. This would allow the learner to model its operation mentally (Gentner & Stevens, 1983) and thus engage in the type of problem solving processes that were the desired outcome. For this reason, one of the important considerations in the formative evaluation was to explore aspects of the mental models that both lecturers and students had for the oscilloscope and provide this information to the course developers to guide their design processes. At one level, problem solving with this instrument involves processing a complex and abstract display (see Larkin, 1989). Both static elements (such as scales and labels) and dynamic elements (such as the knobs and the screen) comprise this display which in itself presents a considerable processing challenge to the user in terms of the volume and complexity of the visual information involved.

However at another more subtle level, effective problem solving also requires an understanding of how invisible events within the oscilloscope should be associated with external changes that take place in the display. Hence to obtain a proper characterisation of the task facing the designers of the project, detailed information is needed about the mental representation of this more abstract level of function. The evaluation used a variety of complementary techniques designed to collect such information and this combination of techniques helped to build up a picture of differences in the way students and lecturers perceived the oscilloscope.

Four components of the formative evaluation process are addressed in this paper: structured interview, content analysis, survey and simulation. Selected results from the first three of these evaluation components are presented here that illustrate (a) the fundamentally different perspectives of staff and students concerning various aspects of the oscilloscope and (b) the wide range of students' entry level knowledge and skills that had to be addressed in the instructional design and its implementation. The paper concludes with a consideration of preliminary work on the fourth component.

Structured interviews

These were carried out with selected staff members to determine what they saw as the ultimate goal of instruction in the use of the oscilloscope. In general, the staff wanted the students to be able to use this instrument independently as an analytical tool for problem solving within the domain of electronic engineering. Clearly this involves a high level of performance that goes well beyond the routine aspects of operating this equipment. Students would have to be able to handle novel situations that could not be addressed by the simple application of low level procedures. Rather, a sophisticated understanding of both the oscilloscope itself and the way it responded to various types of input signals would be required. Students would need to have a coherent mental model of the situation that integrated the internal functioning of the oscilloscope, the general characteristics of the system providing the input to the oscilloscope and the significance of the display as an index of the interaction between that input and the oscilloscope's internal functioning.

Content analysis

Both the manufacturer's manual for the oscilloscope model used in the course and the student laboratory manuals were analysed to characterise their content matter. As well as describing the construction of the oscilloscope and the layout of its parts, the manufacturer's manual also covered the basic operation of its controls. However, it assumes that the reader already understands the functions of these controls and thus provides neither information about the circumstances under which the controls would be used nor any guidance about the significance of the instrument's output. The student materials contained introductory exercises dealing with basic operations of the oscilloscope as well as a series of structured investigations involving a variety of electronic circuits. The purpose of these investigations was to introduce students to the characteristics of a number of key electronic components and arrangements that are constituents of more complex functional devices. A proper understanding of the way an oscilloscope's visual output reflects these characteristics requires an appreciation of the way it behaves internally when connected to the circuit being investigated. Analysis of the manufacturer's manual and the student materials showed that there were critical gaps in the total set of information provided in terms of its potential to cover this type of understanding.

Analysis of both types of manuals was also used as a basis for developing a range of techniques for characterising both the content to be addressed by the proposed instructional materials and the entry knowledge of the student population for whom the instruction was intended. The goal of these techniques was to establish the students' existing knowledge structures with respect to oscilloscopes and the knowledge structures that would be best suited to their future requirements in their chosen course. Information about the differences between these knowledge structures is central to developing sharply focussed instruction.

Survey instruments

Surveys were administered to 84 beginning students of Electronic Engineering in an introductory lecture. In addition to details of the students' backgrounds with respect to the oscilloscope, four main categories of oscilloscope related information were collected:
  1. Familiarity of relevant terms and concepts.
  2. Knowledge of external parts of an oscilloscope and their functions.
  3. Understanding an oscilloscope's internal processes.
  4. Knowledge of operating procedures.
A survey dealing with similar subject matter was also administered to 10 Electrical and Computer Engineering lecturing staff with expertise in use of the oscilloscope. These staff were involved in teaching subjects with a practical component requiring the use of this instrument. The survey questions for the staff were phrased differently from those in the student survey. For example, while students were asked to rate their own understanding of a list of oscilloscope terms and phrases, staff were asked to rate the importance of students understanding these items. Similarly, the students were asked to list jobs that an oscilloscope could be used for but the lecturers were asked to list what jobs the students should know that an oscilloscope can be used for as a result of their introductory course.

Results

The students were broadly classified according to their level of experience with oscilloscopes. The Instruction-Operation group (n = 40) had both learnt about the oscilloscope and had carried out practical activities which involved operating the instrument. The Instruction Only group (n = 24) reported having learnt about the theory of the oscilloscope but had not used the instrument. The Non Instruction group (n = 20) had neither learnt about the oscilloscope nor had any practical experience with the instrument. Thus there was a considerable range of experience amongst the target audience for the proposed instructional materials.

In terms of existing knowledge of the oscilloscope, students in the Instruction-Operation groups would be expected to know most about how an oscilloscope works. Analysis with respect to key terms and ideas of this groups' responses to an open ended question about how an oscilloscope works showed that the most frequent responses concern aspects with manifestations that are external to the electrical processes which occur within the instrument (eg., Display, Screen). Although the next most frequent cluster of responses was concerned with internal or less immediately apparent aspects of operation, these tend to be very generalised (eg., Deflection, Electron Beam). The more detailed aspects of the fundamental concepts involved occur with a low frequency (eg. Trigger, Timebase). These findings suggest that the knowledge base of even the most experienced group emphasises surface level characteristics of the oscilloscope rather than the way it responds internally to electrical signals.

The mean values of the Instruction-Operation group's self ratings of their understandings of 43 terms and phrases associated with the oscilloscope showed that while understanding of some terms and phrases such as Input and Voltage were generally rated as 'good' to 'very good', results for others such as Double Channel and Dual Trace were much more mixed. In addition, there were still others such as Detent and Graticule where the largest response fell into the 'very poor' category. These results indicated that even among the students who were most familiar with the oscilloscope (ie., who had previously learnt about the oscilloscope and also had experience in using it), there was a considerable variation in understanding of basic oscilloscope related content. This indicated that across the whole group of students who would comprise the ultimate clients for the surrogate oscilloscope instruction, a very wide range of entry level knowledge and skills would have to be assumed. Clearly to deal with this amount of variation would constitute a major challenge for the instructional design process, requiring a highly level of flexibility to be built into the instructional materials.

More detailed consideration of the terms and phrases that were not generally well known showed that although some of them were little more than simple vocabulary items (such as Graticule), others referred to sophisticated concepts that are fundamental to understanding how the oscilloscope functions as an analytical tool. For example, the notion of a Dual Trace facility in an oscilloscope is fundamental to investigations that compare the input and output characteristics of components or circuits. Such comparisons form a major part of the work that students were required to do in their university electronics courses. An effective mental representation of the internal processes that occur within the oscilloscope in dual trace mode is central to interpreting the display produced when using this facility. It is not sufficient that students are merely familiar with this term; they must also be able to conceptualise the processes implied by the operation of the instrument in this mode.

Figure 1
Figure 1: Percentage of respondents mentioning basic
operations required to make oscilloscope function

Figure 1 compares the percentage of respondents in the staff group and the three categories of student who mentioned various basic operations needed to make an oscilloscope function correctly. It can be seen that adjustments referred to by a comparatively high percentage of the staff were often mentioned quite infrequently by the students, even those who reported having used the oscilloscope before (there is an interesting difference in the percentage of staff and students who mentioned that it was necessary to turn the power on!). These results suggest that even at the very basic level of setting up before any measurements are attempted, students' ideas of what is necessary to operate an oscilloscope accord poorly with those of their lecturers. It seemed that it was not simply the more sophisticated aspects of the oscilloscope that had to be addressed in the instructional materials, but also a variety of basic operations necessary before the more complex tasks could be carried out properly. This supported the view that the scope of the materials to be developed would need to be wide ranging.

A major outcome of the analysis of the jobs that respondents said could be done by the oscilloscope is shown in Table 1 (see appendix). This table provides a conceptual summary based upon four main categories of uses for the oscilloscope. The first three categories concern generic functions that can be applied across a range of situations (displaying, measuring, and comparing). These categories involve progressively more sophisticated use of the oscilloscope. For example, the measurement of a single frequency is less complex than the comparison of two frequencies. The fourth category covers various specific applications of these functions (such as testing other types of equipment or monitoring real world events). Within each of these categories, the entities that were of interest varied. For example, waveforms appears in all four categories because respondents mentioned the use of the oscilloscope to display waveforms, to measure waveforms, to compare waveforms and to perform more specific tasks such as testing and monitoring. However, particular types of electronic equipment (both domestic and more specialised) appeared only in the fourth category.

This categorisation was used as the basis for characterising how the various types of respondents saw the use of the oscilloscope. There were clear quantitative differences in the relative frequency with which many of the items from the table were mentioned by staff and students. For example, the measurement of waveforms and the display of sounds was mentioned only by student respondents while the comparison of frequencies and the measurement of phase were mentioned only by the staff. In addition, staff were much more inclined than the students to mention the measurement of certain entities (such as current and pulses) and the comparison of others (such as phase and amplitude). In contrast the students' responses emphasised the display itself and the context in which the oscilloscope was used (such as in monitoring heart beats or testing radio equipment) rather than the abstract operational aspects. The jobs many of the students tended to mention were consistent with the way this instrument is portrayed in the popular media. Since the staff were asked to list the jobs that students should know an oscilloscope could be used for, these results suggest that there may be important qualitative differences between the way staff and students mentally represent the oscilloscope as well as the clear quantitative differences in the knowledge base that would be expected.

Simulation

Some preliminary work has been done toward developing a simulation that will help evaluate the effectiveness of the instructional segments as they are developed. This involved videotaping students as they worked on one of the early laboratory exercises requiring the use the oscilloscope to determine the operating characteristics of a particular circuit. Analysis of the video material will be used to devise an activity to place students in a simple problem solving situation that can be used to gather think aloud data about how they use mental models of the situation to perform the given task. The intention will be to characterise the initial and developing mental models of students as they work through the instructional modules and so provide feedback to the designers about the effectiveness of the approaches being used. In a project such as this where little specific guidance is available regarding how best to shape the instructional treatments, intensive monitoring of students' learning in progress is essential.

Conclusion

An important finding of this part of the formative evaluation has been the evidence of subtle qualitative as well as quantitative differences in how the instructors and students perceived oscilloscopes. Of most interest to the current project are the qualitative differences which may reflect differences in the way the two groups mentally model the oscilloscope as a scientific instrument. For example, it appears that the instructors are more concerned with the comparison functions of the oscilloscope that allow it to show the effect that a particular assembly of electronic components can have on an input. By comparing the characteristics of the input and output signals, the informed observer can investigate the characteristics of that assembly. However, it appears that for the students, the focus is upon a single display or measurement. They appear to lack an appreciation of how the instrument is used as investigation tool and focus upon its simpler and more explicit operational aspects. This difference between instructors and students sets a complex challenge to the designers of the multimedia project. It means that the instructional task is not simply one of giving students more information than they previously had about oscilloscopes (although this is clearly necessary with many of the students because of the diversity in their background knowledge). Rather, it involves changing the fundamental way that the oscilloscope is perceived in the context of the electronics laboratory. Meeting this challenge will therefore have at least as much to do with the effectiveness of the instructional design underlying this project as it does with the powerful presentation and interaction facilities that multimedia technology can provide.

References

Adams, L., Kasserman, L, Yarwood, A., Perfetto, G., Bransford, J & Franks, J. (1988). The effects of fact versus problem oriented acquisition. Memory and Cognition, 16, 167-175.

Bonner, J. (1988). Implications of cognitive theory for instructional design: Revisited. Educational Communications and Technology Journal, 36, 3-14.

Case, R. & Bereiter, C. (1984). From behaviourism to cognitive behaviourism to cognitive development: Steps in the evolution of instructional design. Instructional Science, 13, 141-158.

Chi, M. T. H., Glaser, R. & Farr, M. J. (1988). The nature of expertise. Hillsdale, NJ: Erlbaum.

Elen, J. (1991, August). Psychological bases of instructional design models. Paper presented at the Fourth European Conference for Research on Learning and Instruction, Turku, Finland.

Gentner, D. & Stevens, A. L. (1983). Mental models. Hillsdale, NJ: Erlbaum.

Kozma, R. B. (1991). Learning with media. Review of Educational Research, 61, 179-211.

Larkin, J. B. (1989). Display based problem solving. In D. Klahr & K. Kotovsky (Eds.), Complex Information Processing (pp. 319-341). Hillsdale, NJ: Erlbaum.

Perkins, D. N. & Salomon, G. (1989). Are cognitive skills context bound? Educational Researcher, 18(1), 16-25.

Winn, W. (1990). Some implications of cognitive theory for instructional design. Instructional Science, 19, 53-69.

Appendix: Jobs done by oscilloscope

Display Measure Compare Test, check, repair, monitor
Waveforms
AC/DC components
addition and subtraction of
Waveforms WaveformsWaveforms
Sound
sound wave
voice
audio



VoltageVoltage
DC level
direct
peak values

Voltage
rails
DC levels

Current
indirectly
using test resistor
using current probes



Period (time)


Frequency
AC
of signal
absolute
triggered sweep method
using X-Y method
Frequency
relative
of separate sources using Lissajous patterns
comparison


Phase
triggered sweep method
Lissajous pattern
Phase
between voltage and current
relationships
comparisons - Lissajous patterns
difference between signals
phase shifts by comparison of traces




TV set and radio equipment

Pulses
lengths
rise and fall times
of duty cycle



Amplitude
of signal
absolute
of AC and DC
Amplitude
relative

Distortion
signal quality
Distortion
harmonic
non linear


Electric circuits
output from


Electric circuits


Gains/attenuation
voltage or current comparisons
compare input and output


Outputs




Other equipment
ECG
heart beat monitor
brain scan
spectrum analyser

Please cite as: Lowe, R. and Williamson, J. (1992). Developing interactive multimedia courseware: Evaluating instructors' goals and learners' characteristics. In Promaco Conventions (Ed.), Proceedings of the International Interactive Multimedia Symposium, 247-255. Perth, Western Australia, 27-31 January. Promaco Conventions. http://www.aset.org.au/confs/iims/1992/lowe.html


[ IIMS 92 contents ] [ IIMS Main ] [ ASET home ]
This URL: http://www.aset.org.au/confs/iims/1992/lowe.html
© 1992 Promaco Conventions. Reproduced by permission. Last revision: 4 Apr 2004. Editor: Roger Atkinson
Previous URL 9 Apr 2000 to 30 Sep 2002: http://cleo.murdoch.edu.au/gen/aset/confs/iims/92/lowe.html