International
Journal of
Educational
Technology

Evaluation of a multimedia package on pedagogical design and display of visuals

Madhumita Bhattacharya, National Institute of of Multimedia Education;
Kanji Akahori, Tokyo Institute of Technology;
K. L. Kumar, University of Botswana

The quality of instructional visuals produced for a medium is the single most important parameter which affects the quality of learning. The paper address two issues: first, principles of design and display visuals and second, evaluation through collaborative protocol analyses. A multimedia package on design and display of visuals, or the DDV package, developed by the authors is presented in some detail. In doing so, some aspects of the multimedia presentation have been retained to highlight the general and media-specific principles with several examples. The DDV package is now available for instructional designers, software developers and others interested in the design and development of multimedia packages.

Evaluation of the DDV package has been made by employing the collaborative protocol analysis technique. A sample of subjects was selected from end-users in different fields. Working in pairs, they were asked to draw some pre study visuals, study the DDV package collaboratively and then redraw the same as post study visuals by employing the practical knowledge gained by them from the package. Analysis of responses resulted in an effectiveness index, which was observed to be lower than that determined through the administration of questionnaires as well as deductions from routine pretest and post-test comparison. It is concluded that the collaborative protocol analysis results in a realistic and reliable effectiveness of such software.

Introduction

Recent research on design and presentation of information through different media reveal that the application of the basic tenets of educational psychology, learner modeling and above all, common sense can enhance the effectiveness of instruction (Kumar, 1998). It is also observed that general principles for good software design are common for all audiovisuals and the same are valid for different modes of teaching-learning, e.g., classroom teaching, distance communication and self learning. Some principles are, however, media-specific and hence apply specifically to computers, video, or overhead projection. The purpose of the DDV multimedia package on design and display of visuals is to present the general and the media-specific principles for the benefit of instructional designers, software developers and others interested in the development of multimedia packages. Design and display of visuals in the package are also in conformity with the principles being advocated in order to illustrate the same by role-playing the principles. Some excerpts are being presented from the multimedia package in order to save rewriting on the one hand and to provide glimpses of the visuals on the other. Most screens are in hypertext and in Windows format. Examples are given alongside, wherever necessary. It may be mentioned that all the visuals have not been reproduced, but the main points have been brought out. The ‘thinking aloud’ protocols have been video recorded when the subjects were interacting collaboratively while going through the package. It was considered worthwhile by the authors to conduct this experiment for the package on DDV since they have not come across any study based on collaborative protocol analysis especially for evaluation of computer software.

Specifications of the Package

The package has been developed by employing an authoring system AUTHORITY and the programming language C on an IBM compatible PC with DOS operating system.

Scope of the Package

The package commences with introductory visuals and light music in order to draw the attention of the learner. The package also commences with a brief introduction and a statement of purpose, together with music to help orient the learners. A visual may consist of anything that may appear on the screen to the learner. It may be in print or on a monitor. A visual may, therefore, consist of graphics, pictures, video or animation. It is also emphasized that visuals need to supplement the verbal information in order to motivate the learner and improve his or her learning effectiveness.

The learner is given the option to proceed in any order to repeat a subtopic or to quit as shown in Figure 1. A built-in counseling system ensures that a learner may not quit without learning.

Figure 1 (D)

Pre and Post Tests

A learner is prompted to find out how much he/she already knows about the subject through a pretest consisting of five multiple-choice items, an example of which is shown in Figure 2. The learner may either respond to or skip an item. The responses may be entered either by clicking the mouse or by keying in the option.

Figure 2 (D)

A subsequent screen shows the percentage score of the learner and an appropriate comment. If the score is less than 40%, the learner is told that he or she would learn a great deal by going through the package. If the learner scores 100%, he/she is given the option to quit; if the score is between 40% and 100%, his/her knowledge is appreciated but he/she is urged to proceed with the package.

A similar set of five multiple-choice items is offered as a post-test at the end of the package. There, a learner is allowed to attempt repeatedly until he or she arrives at the correct response. He or she is however penalized in term of his or her score, which is reduced by 50% for successive attempts. Finally, the post-test score, or gains, are displayed and the learner is guided to seek an exit or go back and forth in order to attain mastery in the subject. Pre-test and post-test have been included in the package in order to make it an effective self learning package. According to Bruner (1972), it is only by performance that we recognize the acquisition of knowledge. Immediate feedback to the learner sustains interest.

Design Features

In view of the fact that most of us are already familiar with visuals in print, the same is taken as a reference medium. For example, every visual should have a cue on top left corner as shown in Figure 3.

Figure 3 (D)

Whenever a statement from a contemporary researcher is quoted, an icon appearing at the bottom may be activated.

Windows appear one by one or, as one on top of the other depending upon the intended depth of study. Display keys at the bottom right corner have the same function throughout the package. It is demonstrated in the package, following the concept of ‘blocking’ (Steinburg, 1991) that separate areas of the screen and colours are used for instruction and for directions.

Attributes of Visuals

A visual consists of elements such as words, lines, figures and blanks. Careful arrangement of its elements can maximize the impact of a visual. This is achieved by considering the attributes of a visual, i.e., unity, colour, shape, balance etc.

Colour adds a new dimension to a visual. According to Dwyer (1978), use of colour in a visual contributes to improvement in the achievement of specific objectives. Examples have been given for pie-graphs, as well as cool and warm colours. Research on colour-coding reveals that colour should be used selectively and meaningfully. Colours which produce less strain and fatigue should be preferred. An example of balance of a visual is shown in Figure 4.

Figure 4 (D)

The advantage of interactive learning is emphasized by means of an interactive sequence on a light-water nuclear reactor. Learners are required to pick up labels and drag them to the corresponding components. They are encouraged to succeed by trying again after any failures.

The question-answer technique is perhaps the most pragmatic of all methods for interactivity in computer assisted learning. It has, therefore, been used extensively in the package in order to bring home the point.

Composition of a visual is as important as its balance. The rule of the thirds for composition is illustrated in the package with examples. It is shown that an increase in the degree of 'approximation to reality' does not necessarily improve learning; a detailed and complicated visual may interfere with learning.

Learner Modeling of Visuals

The fact that visuals are meant for learners makes a case for designing visuals by learner modeling, i.e., by taking learner characteristics and skills into account as shown in Figure 5. The use of hypertext supports the concept of holistic learning, while being able to access different sources of information, facts, figures, etc. Degree of visual literacy of a learner should also be taken into account in designing visuals. Information density of a visual on a screen or on a monitor should be kept low. Two or more visuals may be displayed alongside as split-screen images, and are particularly useful for comparative and contrastive studies. It is necessary to display the text and stationary images before displaying animation.

Figure 5 (D)

Display of a visual can be effective if it highlights the salient points. For example, in two or more consecutive displays of similar information, only the alterations need to be highlighted.

It is necessary to keep some principles of psychology of retention in mind as shown in Figures 6 and 7.

Figure 6 (D)

Figure 7 (D)

Techniques of Highlighting

The information density could be minimised using rules e.g. visuals are read from left to right, and from top to bottom.

It is essential to highlight a word, phrase or a component of a visual at times by employing one of the methods shown in Figure 8. Different techniques of highlighting are available with different software. One must consider the relative impact and pedagogical value of alternative techniques before employing them. For example, change of colour of some components of a visual may be more effective than writing them in bold, particularly on a monitor.

Figure 8 (D)

A multimedia package should also permit annotation to be displayed online to a visual, wherever possible. It may be in the form of marking, crossing out, pointing, writing, or highlighting in one of the different ways shown in Figure 8. Two identical sets of annotation tools could be provided; one for the instructor only and the other for learners.

Media Specific Consideration

Audiovisuals are classified as non-projected, optically projected and electronically displayed. The same visual may, therefore, not appear well for all media. The same colours neither appear nor appeal equally on all media. Besides, the aspect ratio, i.e., height-to-width ratio is different for different media as shown in Figure 9.

Figure 9 (D)

Although many might believe that special effects are beneficial to the learning process, research has shown otherwise (Travers, 1978).

Computers are ideally suited to display visuals on monitors, particularly by employing Windows and zoom effects. Meaningful computer animation may be used effectively either to demonstrate the dynamics of a phenomenon or to represent the invisible system functions and their behaviour (Park, 1994).

At the end of the package, the learners are given a post-test and they are made to seek the correct answers through successive attempts.

Formative Evaluation of the Package

Although the package has been developed after a comprehensive literature survey, it was considered necessary to evaluate the same by a sample of users before finalisation. It was evaluated by the participants of six faculty development courses (Madhumita, 1994). An appraisal form was designed in order to collect their feedback. The form along with the feedback received from 100 respondents is shown in Table 1.

Table 1

Feedback on appraisal forms (D)

Some aspects such as presentation appeal, explanation of elements of a visual, explanation on colour, highlighting and building up overall presentation were found to be acceptable. However, there was scope for improvements in explanation on composition, animation, as well as the use of learning psychology in design, display, and features of visuals in general. Feedback was used to improve the package.

Procedure for the Effectiveness

The DDV package has since been made available as a learning resource to users across the globe. It was, therefore considered appropriate to examine its effectiveness as a learning resource from the point of view of users. Sample subjects with different fields of specialization were chosen for the experiment. The fields included Computer Science, Architecture, Cognitive Science, Electrical Engineering, Chemistry, Linguistics and Management Science. This provided the opportunity to gather data from people having wide differences in the knowledge they possess and the knowledge they evoke and discuss in a particular situation. Two subjects were chosen for each of the three experiments. The subjects were instructed beforehand to come prepared with some topic for which they would like to prepare some visuals. The subjects were given 15 to 30 minutes for pre and post viewing drawing sessions, i.e., before and after going through the package on DDV, respectively.

Video recording of the collaborative learning events was done by the use of two video cameras from two different angles. One camera was used to capture the visual on the computer monitor and the other was used to record the reactions of the subjects. It was so planned in order to record and study the reactions of the subjects for corresponding visuals. Subjects were asked to interact as freely as possible while going through the package. They were also briefed on the procedure of ‘loud thinking’ together with a short practice session before starting the actual evaluation session.

Collaborative Protocols

The basic assumption of this study is that learning consists of many different paradigms and, through the following experiment, it was intended to evaluate the effectiveness of the package in at least three paradigms; cognition, behaviour modification and a part of interpersonal communication. We have attempted to find out the effect of the DDV package on the cognitive structure of the learners. In doing so, we tried to identify the cognitive changes in the learners through the analysis and interpretation of collaborative protocols. We hypothesized that the cognitive changes would effect the behaviour, i.e., the skill or performance of the learners, which can be evaluated by identifying the changes in the 'post study visuals' as compared to the 'pre study visuals'. Evaluation of post visuals and interview of subjects revealed the effect of collaborative learning as well as established our assumption as tabulated in Table 2. Collaborative protocol sample analysis may be used for to the analysis of response of the target population. It also brings out the effectiveness of the package in terms of learning by the subjects.

Table 2

A typical example of dialogue between the subjects while responding to a pretest question. (D)

For example, after going through the package, the subject BA used scratching as a highlighter in his post study drawing as shown in Table 3.

Table 3

A typical example of dialogue between the subjects while discussing about schematization. (D)

We have found that the maximum conversation was held between the subjects on topics where one of the members of a pair had more knowledge about the content of the DDV package than the other. In this way, they helped each other to understand the contents. Less conversation was observed for the displays where the content of the package was clear to both of them.

Comparison of Pre Study and Post Study Visuals

In this experiment, the subjects designed some visuals before and after going through the DDV package. The subjects designed the visuals on paper or on transparencies at will. The result shows that the skill of designing visuals improved noticeably after learning from the DDV package. Also, the subjects became more knowledgeable about different aspect of design and display of visuals as shown in Table 4.

Table 4

Difference between Style of Construction of Pre and Post Visuals (D)

* Number of subjects out of 6.

A graduate student of Architecture drew the two visuals shown in Figs. 10 and 11. She was already aware of a number of features of good visuals. Even then, we have found that she has learnt a great deal from the DDV package. The post study visual is more organized with better highlighting techniques such as bullets, framing, colour, etc. Also, she has made the statements in the question form to make her presentation more interactive. The difference between the pre study and post study visuals revealed the effectiveness of the package as a learning resource.

Figure 10 (D)

Figure 11 (D)

The subjects were interviewed at the end of post study drawing session. They were asked to explain the changes they have made between the pre study and post study drawings. Their explanations are recorded in Table 5. Their improvements in the post study visuals show that subjects have learned from the DDV package. The subjects also expressed that collaborative learning benefited them.

Table 5

Excerpts from the interview session (D)

Categorization of Protocol Interpretation

The following is a categorised summary of responses obtained during the process of study of effectiveness of the package:

     Well appreciated visuals: The example of a well-organized visual, different techniques of highlighting, concept of visual literacy level of a learner and the technique of interactivity.

     Well accepted new concepts and terms: Foreground and background colour of a visual, the rule of thirds, the split screen technique, the meaning of different types of lines, one idea per visual, unity and balance of a visual.

     Thought provoking statements: Visuals control learning process, tendency of scanning a display, effect of sound and special effects on learning.

     Terms need more explanation: Learner modeling, analytical strategy, and the concept of example and non-example.

     Confusing term: The magic number '7+2' (whether this is the number perceived at one time or remembered?).

     Examples required for concept clarification: the concept of hue, colour coding and line spacing.

Although the effectiveness study on the basis of general responses by the participants as reported in Table 1 and that on the basis of protocol analysis reported in Table 4 are different in nature, one is tempted to compare the two outcomes. Firstly, the average rating in Table 1 is on the higher side of 4/5 whereas the number-count in post study drawing sessions adds to 54/84 on fourteen positive attributes. Collaborative protocol effectiveness is more reliable because it is on the basis of actual acquisition of knowledge and not merely on the basis of opinions. A participants' overrating may be partly due to the novelty value of the package or amicable environment. The collaborative protocol analysis is indeed based upon outcome of the different sections of the package and provides a true index of effectiveness of the package.

Discussion of Results

The DDV package on the design and display of visuals has now been available to all those interested in learning to design better visuals. While developing the package, different aspects of design of effective visuals were incorporated and different theories of learning and information processing were taken into consideration. The package has been revised on the basis of feedback received from the participants, in different training courses. Finally we have conducted an experiment to find out the effectiveness of the package as a learning resource. Since it is difficult for adults to think aloud, collaborative protocol recording proved to be more successful (Madhumita & Akahori, 1998a). In collaborative learning environment, the subjects could converse easily and depicted their thoughts through discussion. The pre-study and post-study visual drawing sessions provided an environment approximating reality because the subjects used their previous knowledge in drawing pre-study visuals and employed the newly acquired knowledge in drawing post-study visuals. On the basis of these results, the collaborative protocol analysis technique is recommended to be a reliable method for qualitative evaluation of educational software (Madhumita & Akahori, 1998b.)

Conclusions

The package provides a pedagogical perspective derived from contemporary research on the design and display of visuals for multimedia. An attempt is made to present some general principles and not to state dictums. This is to encourage creativity and innovation in design and display of visuals. The package goes on to provide some media-specific tips. Throughout the presentation, references and examples have been cited alongside the proposed guidelines. Feedback of faculty members engaged in developing CAI and multimedia software showed that they found the package informative, exciting and useful in their software development.

Results of protocol analysis revealed that the present package is more effective as a presentation package and as a collaborative learning resource rather than for individual learning. This is because some of the terms need more explanation and some concepts require more examples. In the collaborative learning environment, the subjects could clarify their doubts by asking each other. Finally, the value of collaborative protocol analysis to evaluate such packages has been established.

References

Bruner, J. S. (1972). The Relevance of Education. Harmonds worth: Penguin.

Dwyer, F. M. (1978). Strategies for improving visual learning. A review of 650 articles. Learning Services.International Journal of Instructional media 10, 303-328.

Gropper, G. L. (1988). How text display adds value to text content. Educational Technology, 23-35.

Hannafin, M. J. (1985). Keeping interactive video in perspective. Educational media and technology. Yearbook (Vol. 11), pp. 13-25, Englewood, CO: Libraries Unlimited.

Haber, R. N., & Wilkinson, L. (1982). Perceptual components of computer display. IEEE, 23-35.

Heinich, R., Molenda, M., & Russell, J. (1985). Instructional media and the new technology, New York, NY: John Wiley and Sons.

Knirk, F. G., & Gustafson. K. L. (1993). Instructional design. Instructional technology (p. 122). New York, NY: Holt, Rinehart & Winston.

Kumar, K. L. (1998). Educational Technology. New Delhi: New Age Publishers.

Madhumita, B., & Akahori, K. (1998a). A comparative study of single and paired thinking aloud protocol analysis, Joint Conference on Educational Technology Proceedings, 741-742.

Madhumita, B., & Akahori, K. (1998b). An evaluation method for audiovisual software based on protocol analysis; Media and education, 1(1), 19-26.

Madhumita, B., (1994). Effectiveness of faculty development programme for science education. Unpublished doctoral dissertation, University of Delhi.

Miller, G. A. (1968). The magical number seven, plus or minus two: some limits on our capacity for processing information. In Haber, R. N. (Ed.) Contemporary theory and research in visual perception. New York, NY: Holt, Rinehart & Winston.

Park, O. C. (1994). Dynamic visual displays in media-based instruction. Educational technology (4), 21-25.

Raymond, J. (1990-1991) LeProf software. Quebec, Canada.

Romizowski, A. J. (1993). Development of interactive multimedia courseware and networks. Instructional design, development and evaluation. Syracuse, NY: Syracuse University.

Steinberg, E. R. (1984). Teaching computers to teach. Hillsdale, NJ: Lawrence Erlbaum Associates.

Suppes, P., Walker & Hess (1984). Instructional software. Belmont, CA: Wadsworth.

Travers, R. M. (1978). An introduction to educational research. New York: Macmillan.

Triesman, A. M. (1974). Selective attention in man. In Fried, P. A. (Ed) Reading in principle and practice. Lexington, MA: Health.

Visual literacy (1985). First Dimension. Journal of visual verbal language (5), 7-15.

IJET Homepage | Article Submissions | Editors | Issues

Copyright © 1999. All rights reserved.
Last Updated on 11 July 1999. Archived 5 May 2007.
For additional information, contact IJET@lists.ed.uiuc.edu