IIMS 94 contents
[ IIMS 94 contents ]

Measuring learning outcomes using IMM systems

Ron Oliver
Edith Cowan University, Western Australia

Interactive multimedia systems

The term, multimedia, is broadly used in the context of computers, education and training. It is not a new term and there are many references to teaching with it in the educational literature spanning the past few decades (Brown, Lewis & Haclerod, 1973). Multimedia describes the simultaneous use of different media forms in instruction. Recent advancements m technology have seen the emergence of interactive multimedia (IMM), where multiple media forms are applied in instructional settings using computer based technologies (Copeland, 1991). IMM applications currently attract considerable attention and popularity although in most instances, the applications are really only extensions of the computer based learning programs developed over the past two decades. It is the strong visual capabilities of the of multimedia learning materials cause thein to stand apart from conventional computer based materials. In many other respects, there is little difference between them.

Today we see three main applications of IMM in presentation, instructional and information systems. In term of education and training, the most widely used applications are the instructional and information systems. Typically the multimedia base forms an envelope of resources that can be accessed by the user in varying ways. Most materials that are loosely classified as multimedia are better described as hypermedia. Locatis, Charuhas and Banvard (1990) describe hypermedia as 'an organisation in which knowledge is broken into small units (nodes) that are interconnected and among which a user can navigate freely'. An organisation scheme is used to define the links and nodes. Whereas in conventional CBL, materials, knowledge was organised in a hierarchical sense, it would appear that the preferred mode of knowledge organisation in hypermedia, is referential, Figure 1 (Locatis et al, 1990).

Figure 1: Hypermedia links

Figure 1: Hypermedia links

Multimedia or hypermedia

It is the referential accessing available with hypermedia systems that appear to offer the most educational value to the users, although this form of navigation and interaction is not dependent on multimedia nor the technology it embraces.

So when most people talk about multimedia, they are in fact talking about hypermedia and much of the advantage that is ascribed to the system stems not from the technology but from the instructional approach that it embraces. The question of whether the medium has any influence on learning is still the subject of debate. Clark (1983) commenced this debate with the observation that media are the 'vehicles' that deliver instruction and as such need not necessarily influence the instruction. Although some scholars do not totally agree with this statement, (eg, Kozma, 1991; Petkovich & Tennyson, 1986), the confusion of many people today in not being able to distinguish the vehicle from the object being delivered, provides tacit support for Clark's arguments.

There have been many occasions where technology has found its way into education and training for reasons other than need. In such instances the educational rationale for implementing the technology is often the least important factor in the uptake. We refer to such activities as being technology led and there are many instances where technology finds its way into schools through reasons other than instructional advantage and opportunity. Dunnett (1990) also refers to this as the media trap. As a tertiary educator, it has been a frequent experience for me to become aware of instructional applications that are technology led. In the past, it was not uncommon to be invited to speak to the staff at schools to tell them how they might effectively use the expensive computer system they had just purchased. In a recent evaluation, I was asked to consider the utility of CD-I as a technology suited to providing instruction to distance education students. In the project a CD-I device had been purchased and placed in a school in order that its utility might be judged. Software and instructional applications were not considered important issues. In such instances, one is always left wondering how it is that people can fail so often to dissociate the vehicle from the message that is being delivered.

The problem with technology led applications in education and training is that they frequently fail to live up to the expectations placed on them. There are many examples in the past of important and innovative technologies being used in education and training for which research had clearly validated and demonstrated the potential of significant learning advantage, and which have not produced the expected fruit, television and radio being two obvious examples (Cuban, 1986). As we look at the proliferation of IMM into education and training today, and we compare current practises with those of the past, the prospect of IMM becoming just another piece of technology is quite possible. Many current applications of IMM are technology driven. There is an urgent need to study the pedagogical principles upon which IMM applications are developed and the environments in which they are being used (Reeves, 1993).

Computer based learning

There has been considerable research conducted in the past two decades into the effectiveness of computer based learning materials. The outcome of this research has been to establish that CBL offers significant advantage over conventional teaching, however, has no more advantage than other innovations and activities (Hattie, 1992). The message that must come from these forms of research, is not to expect any marvels or wonders from IMM developments. Applications of the new technologies must be guided by considerations not of increased learning but more of increased opportunities for learning. Many developers and publishers of IMM seem totally unaware of the previous two decades of research and development in computer based learning. Multimedia seems to have assumed a place of high standing among the educational community and much of this high standing is based more on perceptions of quality and effectiveness rather than concrete information drawn from experience and practice.

Interactive multimedia applications in education and training

IMM looks good. As with most applications of the technology, there seems to be some form of force that draws users and students towards computer technology and as the technology improves, this attractive force seems to increase. In fact there have been a number of research activities that have sought to probe reasons behind the attracting power of IMM materials. When I am moving in multimedia circles, it is not uncommon to be told about a great new multimedia product that has just come onto the market. Astute questioning will often reveal that the description of a new product frequently becomes a great new product in reference to multimedia publications. Every new product seems to be great. Why is this so? I suspect it is because IMM looks great and people appreciate the significant amounts of work and effort that are required for publication. Whether the product will serve its purpose or will provide significant advantage and learning opportunity is never considered. It may well be that many products justifiably deserve the description of being great but I think that we should reserve this judgement and make it in the light of more measured and meaningful indicators.

The application of new technologies in schools has frequently been hailed as the means by which meaningful and effective information skills teaching and learning can finally be achieved (eg, Sheingold, 1986; Downes, 1989; Shaw, 1991; Megarry, 1991; Riding & Chambers, 1992; Russell & Russell, 1993). But studies have revealed that the new technologies despite their facility to provide increased avenues of access to information, have not fully lived up to expectations. Despite its strong potential, hypermedia and interactive multimedia (IMM) also have inherent problems in use and application as an information source in educational settings (Ford & Ford, 1992). Collis (1991) describes the more serious problems as:

In describing children's use of electronic data sources, Fasick (1992) observed that much research activity has focused on investigating the knowledge required by the user in adapting and learning to use the new system. In many instances this has required extensive instruction and skills development in system usage before the information required has oven been considered.

Many potential users of hypermedia have been found to lack the cognitive skills, the motivation and attitude to learning required to take full advantage of the medium (Heller, 1990; Trumbull, Gay and Mazur, 1992). Although many publishers build design metaphors into their interface and navigational structures, naive users are often unaware of this encompassing and potentially useful structure (Gay & Mazur, 1989). Trumbull et al. (1992) found that novices typically tend to browse when seeking information and this was a very inefficient form of navigation when compared to indexing and using online guidance.

Navigation strategies are very difficult to plan when broad usage is required of a program. Keyword searching is problematic for novices who need to loam and apply, what is ostensibly, a new language. Similarly even alphabetic lists have been found to be difficult for novice users especially when indices use bands such as D-J and S-V (Fasick, 1992). Studies of the use of hypertext by novices have revealed that non-skilled users in these environments view more screens than skilled users and do so in a non-sequential and inefficient mode. Levels of learning and understanding are significantly less among novices than trained users. Although users prefer the control they are given over the media to other forms of media, they do not choose the best paths in information seeking activities (McGrath, 1992). Heller (1990) compares researching with hypermedia with discovery learning and suggests that many users may not have the intellectual capacity to be able to actively ignore non-essential information as they browse a hypermedia system. Several authors have suggested that different learning styles will influence the way in which hypermedia systems are used (Marchioni, 1989; Heller, 1990; Riding & Chambers, 1992).

Information seeking with hypermedia

In a study that sought to examine learning outcomes achieved through use of a hypermedia system, Frau, Midoro & Pedemonte (1992) found that their subjects often made ineffective use of the hypermedia in inquiry activities. They found the students chose to look mainly for facts in their interrogation of the system in constructing a mental image of the topic under investigation. The students were guided in their investigation by a task that directed their inquiry. It was found that the conventional method of inquiry practised by the students was an obstacle to exploiting the full potential of the system. The students' exploration was judged to be sequential and systematic (as if reading from a textbook) rather than thematic and hierarchical. the more appropriate forms when accessing hypermedia information. Frau et al. (1992, p.50) concluded that 'optimising the learning environment by the provision of hypermedia does not by itself produce a significant improvement in learning'.

My own research (Perzylo & Oliver, 1992) has also raised questions about the utility and efficacy of hypermedia applications as information tools. The study found that naive users of a hypermedia systems made very little use of the alternative forms of media available to them in a hypermedia system. While students gathered significant and useful amounts of information from the hypermedia source for their projects, little of the gained information was from sources other than the text. The students were observed to access the video and sound and graphical images while using the system to gather information for a project but only the text was used by the students as an information source for their project submissions.

Possible explanations for this may come from research into learning with media that has considered students' perceptions of the cognitive difficulty associated with media usage. Salomon (1984) found that younger students' perceptions of the mental effort associated with a learning activity often influenced the learning that was achieved. When a medium was judged to be difficult to learn from, students appeared to work harder in order to effect the learning. A recent study by Saga (1990) investigated this phenomenon with multimedia and older students and observed a reverse effect. Saga found that older students who judged a medium to be difficult to learn from, were generally accurate in their perceptions and were not able to learn as efficiently from those media. On the other hand, the students in Saga's study made some interesting comments about hypermedia and there was an impression among many of them that the hypermedia was distracting and playful and gave information too easily. Cennamo, Savenye and Smith (1991) also describe a study in which the users' demonstrated varying preconceptions of ease of use of various media. A correlation was observed between the level of mental exertion and scores in tests of recall and inference. In this study as in others, the media included those with visual and pictorial imagery, forms frequently encountered in hypermedia applications.

Possible reasons for the children's preference for text based materials can be found in research into teaching and learning. The registration of pictorial information in memory is achieved by processes which differ substantially from those employed in the registration and storage of verbal and written material (Cohen, 1973). If students have not developed the processes required for accurate and efficient processing of pictorial information, they will be less likely to demonstrate a preference for these tasks over others with which they are more familiar, for example text processing.

Multimedia and hypermedia is a medium that can be judged by naive users, to be one that requires less effort to find relevant information than conventional materials (Perzylo & Oliver, 1992). Interviews with the novice users in this study indicated a general perception that the hypermedia system was easy to get information from. Observations of students activity confirmed this as very little mental effort appeared to be expended by students when viewing graphics, full motion video and attending to sound tracks. There was no note taking and often distracted behaviour. On the other hand, significant effort was observed in their reading and attention to the text and hypertext screens. In line with the research of Salomon (1984), Saga (1990) and Cennamo, Savenye and Smith (1991), it should not be surprising to find that students gained little useable information from these 'easier' sources.

Despite these problems, however, comparison of learning and information retrieval from conventional and IMM sources do show some gains and advantages for the latter (eg, Saga, 1990; Riding & Chambers, 1992). It is likely that with appropriate training and instruction, students can be taught to become better users of the hypermedia environments. Furthermore as research informs us more about the appropriate designs of these products, the more successful can learner interactions be expected to become (Barker & Manji, 1991).

Learning outcomes from IMM applications

There has been considerable research conducted into the psychology of learning. From this, a number of distinct epistemologies have emerged that attempt to describe the process by which learning occurs. In the past, instructors have been guided by an objectivist epistemology that embraces the notion that learning occurs through the acquisition of knowledge through the senses. Instruction based on this premise sees knowledge broken into small chunks and specific goals and objectives established through which learning and instruction will proceed (Rieber, 1992). Direct instruction is then used to address each of the intended learning outcomes. The process treats learners as empty vessels to be filled with learning (Reeves, 1993). An associated aspect of this form of learning is the apparent lack of attention given to the learner in the design of the teaching/learning program. The instruction focuses directly on the content and material to be learned. This information transmission model is still widely practised in schools today (De Corte, 1990) and an identifying feature of a large amount of IMM materials.

An alternative epistemology to objectivism is based on the notion that learners are not passive recipients of information but actively construct their knowledge and skills through interaction with the environment and through reorganisation of their own mental structures (De Corte, 1990). This model of learning actively takes into account individual differences in student readiness for learning, the role of prior knowledge and the need for students to be active participants rather than passive recipients in the learning process. The constructivist epistemology has frequently been asserted as an optimal form of instruction for many learning environments and in particular well suited to CBL. Constructivist psychology and pedagogy sees instruction that enables each learner to individually acquire and construct knowledge through a process that builds on existing knowledge to establish cognitive and cognitive structures. Rather than being viewed as an empty vessel, the learner's mind is viewed as a vessel with existing structures and schema into which new knowledge must be built and incorporated. The knowledge acquired by the learner is actively constructed and built through relevant and meaningful activity.

If it is the goal of IMM materials to develop the knowledge of the learner then it is very important for the designers of IMM to consider the appropriateness of the epistemology being used and its capacity to influence the learners' mental states. A review of much of the IMM material that is currently available and being produced, quickly reveals that the instructivist approach is alive and well. If in practice, IMM materials produced with thew methodologies do not return the anticipated results, it is likely that IMM will be blamed rather than the instructional approach. This is very much the argument that Clark (1983) was trying to make when he stated that 'media are mere vehicles that deliver instruction but do not influence student achievement any more than the truck that delivers our groceries causes changes in our nutrition'.

For the instructional applications of IMM applications in education and training to be effective teachers, they must deliver instruction in a form that enables understanding as well as knowledge and skills to be developed. They must deliver transferable skills, knowledge and understandings. Furthermore, the instruction should avoid the empty vessel mentality and actively seek to build new knowledge bases that are framed and structured in light of learner's existing mental states.

To achieve this, IMM instructional materials need to fully make use of the resource envelope that is provided by the hypermedia application. Hypermedia has the capacity to organise knowledge by mirroring the structure of human thinking. One of the most remarkable aspects of human thinking, for example, is ability to be reminded, to branch from one thought to a related yet wholly distinct experience. Interactive multimedia recreates this web of relations with the concept of hypermedia (Amthor, 1992).

Measuring learning outcomes

The true test of the achievement of learning and knowledge gained through interactions with IMM materials is not the level of success of the user in objective testing contained within the instruction modules. The real test must come from assessments of the capacity of the user to transfer the knowledge gained in the IMM environment to the world in which the knowledge, skills and understanding are to be used. This form of assessment becomes even more important in IMM environments that are based on instructivist epistemologies because the learned knowledge can be quite distinct from the applications in which it is to be used. In such IMM applications, learners progress at a rate that depends on the level of mastery of the learning materials. In-built testing procedure that accompany the instructional materials and are used to assess the level of competency tend not to consider more than the knowledge itself. Learners are often tested on low level knowledge, in settings that parallel the instructional materials and in a time span that doesn't necessitate a full understanding of the content. Furthermore the learning is measured in the context of the computer program with little reference to transferable skills or knowledge.

It has been the premise of this paper that such instructivist strategies are less likely to achieve the level of learning and understanding achievable by constructivist approaches. Coupled with an inadequate assessment schedule, the actual learning outcomes achieved from interacting with the materials may never really be known. The forms of strategies that are required to assess the real effectiveness of instruction must encompass tasks across a broad spectrum. These tasks need to be able to measure understanding, rather than memory of facts. They must be able to give some measure of the transferability of the knowledge to the intended domain and must also provide evidence that the knowledge and skills are not only short term gains.

The power and potential of IMM seems to stand this technology in a world of its own in terms of its capacity in education and training. It is important to remember from past experiences that many potentially useful forms of technology can fail to deliver when the development of the technology is not matched by the development and application of the appropriate instructional components. Educational research has made us aware of the forms of instruction and activity that are best suited to knowledge acquisition. It is important to ensure that these principles are used in IMM if this technology is to provide the impact that it promises.

References

Amthor, G. (1992). Interactive multimedia in education. Ultimedia Digest, 1, 27-30.

Barker, P. & Manji, K. (1991). Designing electronic books. Educational and Training Technology International, 28(4), 273-280.

Brown, L., Lewis, R. & Haclerod, F. (1973). AV instruction: Technology, media and method (4th edn). New York: McGraw Hill.

Cennamo, K., Savenye, W. & Smith, P. (1991). Mental effort and video based learning: The relationship of pre-conceptions and the effects of interactive and covert practice. Education Training Research and Development, 39(1), 1042-1629.

Clark, R. (1983). Reconsidering research on learning from media. Review of Educational Research, 53, 445-459.

Cohen, G. (1973). How are pictures registered in memory? Quarterly Journal of Experimental Psychology, 25, 557-564.

Collis, B. (1991). The evaluation of electronic books. Educational and Training Technology International, 28(4), 355-363.

Copeland, P. (1991). The multimedia mix. Educational and Training Technology International, 28(2), 154-163.

Cuban, L. (1986). Teachers and machines: The classroom use of technology since 1920. New York. New York: Teachers College Press.

De Corte, E. (1990). Learning with new information technologies in schools: Perspectives from the psychology of learning and instruction. Journal of Computer Assisted Learning, 6, 69-87.

Downes, T. (1989). Integrating information skills and technologies into the curriculum. Scan, 8(4), 310.

Dunnett, C. (1990). Open access - open media. Educational Media International, 27(4), 198-207.

Fasick, A. M. (1992). What research tells us about children's use of information media. Canadian Library Journal, February, 5 1-54.

Ford, N. & Ford, R. (1992). Learning strategies in an 'ideal' computer based learning environment. British Journal of Educational Technology, 23(2), 195-211.

Frau, E., Midoro, V. & Pedemonte, G. (1992). Do hypermedia systems really enhance learning? A case study on earthquake education. Educational and Training Technology International, 29(1), 42-51.

Gay, G. & Mazur, J. (1989). Conceptualising a hypermedia design for language learning. Journal of Research on Computing in Education, 21(2), 119-126.

Hattie, J. (1992). Measuring the effects of schooling. Australian Journal of Education, 36(1), 5-13.

Heller, R. (1990). The role of hypermedia in education: A look at the research issues. Journal of Research on Computing in Education, 22(4), 431-441.

Kozma, R. (1991). Learning with media. Review of Educational Research, 61(2), 179-211.

Locatis, C., Charuhas, L. & Banvard, R. (1990). Hyper video. Journal of Education Training Research and Development, 38(2), 41-49.

Marchioni, G. (1989). Information-seeking strategies of novices using a full-text electronic encyclopedia. Journal of the American Society for Information Science, 40(1), 54-66.

McGrath, D. (1992). Hypertext, CAI, paper or program control: Do learners benefit from choices? Journal of Research on Computing in Education, 24(4), 513-532.

Megarry, J. (1991). Europe in the Round: Principles and practice of screen design. Educational and Training Technology International, 28(4), 306-315.

Perzylo, L. & Oliver, R. (1992). An investigation of children's use of a multimedia CD-ROM product for information retrieval. Microcomputers for Information Management, 9(4), 225-240.

Petkovich, M. & Tennyson, R. (1984). Clark's "Learning from media": A critique. Educational Communication and Technology Journal, 32(4), 233-241.

Reeves, T. (1993). Interactive learning systems as mind tools. In P. Newhouse (Ed), Educational Computing Association of Western Australia Annual Conference, 2. Mandurah: ECAWA.

Riding, R. & Chambers, P. (1992). CD-ROM versus textbook: A comparison of the use of two learning media by higher education students. Educational and Technology Training International, 29(4), 342-349.

Rieber, L. (1992). Computer-based microworlds: A bridge between constructivism and direct instruction. Educational Technology Research and Development, 40(1), 93-106.

Russell, N. & Russell, A. L. (1993). Interactive multimedia in libraries. Access, 7(1), 26-29.

Saga, H. (1992). Are we ready enough to learn from interactive multimedia? Educational Media International, 29(3), 181-188.

Salomon, G. (1984). Television is 'easy' and print is 'tough': The differential investment of mental effort in learning as a function of perceptions and attributions. Journal of Educational Psychology, 76, 647-658.

Shaw, S. (1991). The learning disc: Learning to read using interactive CD-ROM. Educational and Training Technology International, 28(4), 316-320.

Sheingold, K. (1987). Keeping children's knowledge alive through inquiry. School Library Media Quarterly, Winter, 80- 85.

Trumbull, D., Gay, G. & Mazur, J. (1992). Students' actual and perceived use of navigational and guidance tools in a hypermedia program. Journal of Research on Computing in Education, 24(3), 315-328.

Author: Ron Oliver, Senior Lecturer, Library and Information Science, Edith Cowan University, Bradford St, Mt Lawley WA 6050. Tel: +6109 370 6372 Fax: +6109 370 2910 Email: R.Oliver@cowan.edu.au

Please cite as: Oliver, R. (1994). Measuring learning outcomes using IMM systems. In C. McBeath and R. Atkinson (Eds), Proceedings of the Second International Interactive Multimedia Symposium, 377-382. Perth, Western Australia, 23-28 January. Promaco Conventions. http://www.aset.org.au/confs/iims/1994/np/oliver.html

[ IIMS 94 contents ] [ IIMS Main ] [ ASET home ]
This URL: http://www.aset.org.au/confs/iims/1994/np/oliver.html
© 1994 Promaco Conventions. Reproduced by permission. Last revision: 12 Feb 2004. Editor: Roger Atkinson
Previous URL 29 July 2000 to 30 Sep 2002: http://cleo.murdoch.edu.au/gen/aset/confs/iims/94/np/oliver.html