IIMS 1994: Alexander and Frampton - technology and thinking

Technology and thinking: A qualitative study with interactive multimedia products

Malcolm Alexander and David Frampton
Griffith University, Queensland

Research concerns

Interactive multimedia, like many earlier technologies with some propensity for use in learning and instruction, has become the focus of a range of theoretical and practical issues in the field of education. What we see currently in interactive multimedia is a mixture of active proselytisation, utopian futurism, creative experimentation, the practical implementation of novel systems, and a reflective stance which suspects that, to a large extent, many earlier, contentious research issues surrounding instructional technologies are being revisited and re-rehearsed. An important factor is that any such revisiting takes place in the context of a new combination of political, economic and academic circumstances. If they do nothing else, such circumstances tend to influence strongly the current terms of reference.

Motivation for the research described below contains several of those elements, three of which can be identified as the basis of the program. The first is a concern with what are termed 'higher order' cognitive skills. Hedberg (1993) notes how a new emphasis on the need to promote higher order thinking skills has been increasingly apparent in curriculum debates; Jonassen and Wang (1993), reporting on experiments with college students, observe that the latter were 'not accustomed to higher order learning outcomes from instruction'. However, the competencies agenda has also seen a tendency for academic opinion in higher education to assume that, whatever higher order cognitive activity may be, there is a strong likelihood that more of it occurs in higher education learning than elsewhere.

The second element is that of claims and assumptions, which have grown steadily in number, that interactive multimedia is particularly apt for stimulating the incidence of such higher order cognitive activity in learning (Dede, Fontana and White, 1993; Cognition and Technology Group at Vanderbilt, 1990). We have some reservations about the validity of this construct in the research we are conducting, but we acknowledge the pragmatic value of referring to it in relation to current concerns.

Adding to the above reservation that such utopian 'aptness' may not be readily realisable, we had as our third motivation to link such concerns with observations of the cognitive skills which are manifested when hypermedia is used in the humanities, where its propensity for promoting learning in richly linked and non-rule based domains has received a good deal of attention (Spoehr and Shapiro, 1991; Jonassen, Ambruso & Olesen, 1992).

These concerns are reflected in the key issues for multimedia research discussed by Ferretti (1991): 'What kinds of knowledge and skills are best taught with multimedia ... and how does such knowledge/skills affect performance in multimedia environments?', 'What are the characteristics of multimedia that contribute important academic and motivational outcomes?', and 'What skills acquired by students while using multimedia will generalise to other settings?'

Research objectives

In response to the above issues, our objectives for the program we describe below can be formulated as follows:

To develop observation methods which allow us to track the incidence of students' cognitive skills in interactive multimedia environments, and particularly to identify those which may be characterisable as 'higher order' cognitive skills;
To begin from a sceptical position. Thus, we are reluctant to attribute any incidence of higher order cognitive skills to the multimedia environment itself, unless there is strong evidence for such an interpretation.

Development of methodology: Cognitive categories

An approach structured around these objectives presupposes that one can identify the higher order cognitive processes to which we refer. There is, however, a lack of agreement among researchers concerning appropriate categories of cognitive processing. We reviewed categories which researchers had used in similar types of user observations (Atkins and Blissett, 1992; Spoehr and Shapiro, 1991; Anderson and O'Hagan, 1989; Clements and Nastasi, 1988; Garhart and Hannafin, 1986), some which have been used in other situations (Duschl and Gitomer, 1991; Jonassen and Grabinger, 1989), and Ericsson and Simon's (1985) theoretical basis for assigning verbal reports of cognitive activity.

We also revisited Bloom's (1956) taxonomy of educational objectives which, for all its comprehensiveness, is not couched in terms which can be used for en route analysis of user interactions with computer programs. As discussed below, it was decided on completion of the first set of user observations to attempt to code video recordings and transcripts using categorisations by Cradler (1985), Gubbins (1985) and Rowland (1992).

Development of methodology: Media effects

A second methodological concern is to track the incidence of cognitive processes which can be said to occur because a learner is placed in an interactive multimedia environment, ie. to demonstrate a causal relationship. There are strong and weak hypotheses concerning possible cognitive outcomes in interactive multimedia environments which are relevant to this study:

No media effect
Those which claim that learning (type, quantity, quality, etc.) is significantly technology independent and much more influenced by other factors
Unique media effect
Those which claim that the exploitation of interactive multimedia technology as an internalised cognitive tool, rather than as a neutral carrier of knowledge as facts, is likely to produce learning outcomes which differ qualitatively from those attainable under other conditions

The first position is substantially that promoted by Clark (1983), who challenged researchers to demonstrate clearly the pedagogical contribution of technological media to learning. The second is discussed by The Cognition and Technology Group at Vanderbilt (1990), Hasselbring, Goin and Bransford (1991), Pea (1985, 1987) and Kommers, Jonassen and Mayes (1992).

The prime objectives of this project have led us to adopt a broadly phenomenological approach (see, for example, Goodrum and Knuth, 1991). The costs associated with the selected research strategy do not permit controlled comparisons to be made between learning environments. Moreover, a phenomenological approach is typically sceptical about assuming that the complexity of learning situations can be accounted for through appeal to a limited set of supposedly known variables in the manner of a scientific experiment (Guba and Lincoln, 1982).

Development of methodology: Trials and observations

The multimedia product

The product selected for the first set of user observations was Flashback! (c1992), an interactive multimedia product in CD-ROM format on Australian history. Flashback! has been developed on the basis of the NSW secondary school history curriculum. It contains a significant bank of textual materials, but its primary aim is to give access to a large amount of visual, audio and movie materials. In terms of our research concerns, it claims to operate in a diverse, richly linked and non-ruled based domain (history), and it claims to allow the full range of learning and teaching activities within the curriculum (eg. critical thinking, assessment of primary sources, hypothesis building).

Our initial familiarisation with the product determined one of the parameters of the design of the first set of observations: users' activities had to be task oriented during the observations. This was partly a practical issue of time. However, we also took account of reservations in the empirical research literature concerning unstructured hypermedia use, where behaviour may become 'entropic, goal-lost, impulsive, and distracted by the browseables offered by the system' (Hasselbring, Goin and Bransford, 1991).

The task

Following the style of period based research instanced in the Flashback! manual (p.76), students were referred at the observation sessions to the written task: 'Find out all the information you can about Australia at the beginning and during World War Two'. Given that this would be the subjects' first encounter with CD-ROM the task was kept simple and the period was one which is clearly delineated in the application.

Paired users

We accepted the advice offered by Anderson and O'Hagan (1989) that the verbal protocols obtainable from pairs of users would be more spontaneous than those which could be prompted from an individual thinking aloud. As these researchers note, think aloud protocols are extrinsic to the task and slightly unnatural.

Consequently, volunteer pairs of students were sought largely from the foundation year in the Faculty of Humanities at Griffith University, complemented by a small number of third year students. All had some familiarity with the subject matter of Flashback!, but it was not a criterion for the observation studies that test populations should be homogeneous.

Computer familiarity

It was thought desirable to have some measure of the computer self efficacy and confidence of the students. Those who volunteered for this first set of observations had had little experience of working with computers. A form of Computer Attitude Measure (CAM) devised by Kay (1993) was adapted for the purpose. One of its useful features was a semantic differential scale of users' anticipated degree of comfort, pleasure and satisfaction in working with a computer.

It was decided that the same attitude instrument would be used as post-observation measure, but that the semantic differential scale would in this case refer to 'working with Flashback!' rather than to 'working with computers' as it had done in the pre-test. It was thought that this might reveal shifts of feeling and attitude between the mediating technology and the knowledge representations which it embodied.

Interface evaluation

It had been determined from the outset to use an evolving research design, rather than to adhere to a preconceived theoretical framework. Consequently, following the observations, but prior to a detailed analysis of them, aspects of users' interactions with the interface were reviewed to judge whether further categories of activity should be used in the analysis.

Relevant category formers for this purpose were (a) time taken from start to first content related search activity, (b) whether there was 'closure' (ie. a pair of users deciding that they had completed the task), (c) rate of asking for reassurance about navigation decisions, (d) how often main menu was revisited, (e) rate of intervention of the research assistant (eg. to help subjects emerge from closed loops), (f) number of main sub-menus of the database searched.

Observation set-up

The equipment configuration for observations was adapted from an experimental set-up suggested by Perlman (1992). One camera views the users in semi-profile and medium close-up. The second camera views the computer screen. Both cameras are routed through a simple video mixer which allows the computer screen image to be inserted in the corner of the picture. After conducting the pre-test, the researcher informs the students of the task, shows them how to operate the mouse, then sits behind the students and intervenes as little as possible.

Observation procedure

For the first set of observations, video recordings were obtained of fifteen pairs of students who had chosen their own pairings. They were invited to talk freely about their en route findings and reactions, so that although the task was, in relative terms, not demanding, no impediment was placed in the way of the students' using higher order cognitive processes

Quantitative data

Initial quantitative data resulting from the above trials have been reported elsewhere (Alexander and Frampton, 1993).

Coding of observational material and protocol analysis

Transcriptions of verbalisation and navigation

The videotapes of user interactions with the application allow for a close analysis of the cognitive processes subjects use to move around the application and engage in various activities. We began by making a transcript of the verbalisations of the subjects laid alongside the navigational decisions they made to move around the hypertext of the application. Combined with the reviewing of the videotape which allowed us to see subjects' expressions and non-verbal gestures (hand and head movements, etc.) this provides an extremely rich record of the subjects' use of the application. The encoding problem is, however, to infer what cognitive processes are taking place. This level of encoding is crucial to the whole phenomenological approach but needs to be applied systematically and consistently. Protocol analysts as developed by Ericsson and Simon (1984) is the foremost description of this process, and we found that the specific procedures developed by Rowland gave the best guidance to this analysis.

Episodes and segments

Rowland's methods based on the Ericsson and Simon framework were used for a study of instructional designers working out the basic design of a training manual. The basic strategy Rowland used was to encode each speech segment by identifying the cognitive activity (eg. READ, INFER, GENERATE) and the object of the task to which it was being applied (eg. READ about machines, EVALUATE information etc.) These segments were then grouped as episodes. The episodes could then be categorised with reference to the types and patterns of segments within them and mapped to trace the pattern of activity over the course of a session (Rowland, 1992: 68-71). Rowland was able to then identify the main phases of each session and to compare the typical session patterns of novice and expert designers.

The objects of cognitive and navigational activity

The key to this process is, therefore, the accurate and valid encoding of segments, a task which is much harder once several coders are being used. The first task we faced was that of generating a listing of the objects in our application which we could associate with any activity. This required us to devise a hypermap of the application which identified each screen (or Macromind Director sequence) according to its place in the navigation structure and content categories. Thus we could code each segment; eg. READ first screen of Australia at the outbreak of each conflict: Overview: ASSESS APPROPRIATENESS of Bad Times for Many. (In future we will be able to time code each of these activities as well.) The second task we faced was that of extending the list of cognitive activities provided by Rowland in order to bring in activities characteristic of the decision making task we had set our subjects as opposed to the problem solving task he had set for his.

Relating observed activity to navigation, subject and content

The next step in the coding process was to arrange the segments into episodes. The technology made this step rather easy since each point of decision making is clearly marked by the choice of a menu item and the movement to another screen or to a sequence. The inferences which had to made about the episodes related to the learning that was occurring. Was it learning about the structure of the interface or navigational system? Was it learning about the arrangement and categorisation of the content, or learning about items of subject matter and their relation to the task at hand? At this point we found that we had to make use of the more elaborated lists of cognitive activities provided by Cradler (1985) and Gubbins (1985). These episode classifications largely pertain to the activities which Gubbins identifies as basic processes.

Tracking the pattern of episodes

The final mapping of each session involved another level of synthesis in that we had to move from considering the pattern of segments in episodes to now considering the pattern of episodes across a whole 30-40 minute session. At this level the pattern of subjects' activity largely reflected the task we had given them. That task involved a simple retrieval of information so that, once they had mastered the navigation system and the arrangement and categorisation of material, they simply reviewed the information and decided on its relevance. We expect the major cognitive processes here (at the level of the episode) to be those identified by Gubbins as related to the complex task of decision making (ie. Classifications and Relationships). On the other hand, we expect to find that the first phase of each session, where the subjects are still finding their way around the application, has more cognitive activity characteristic of problem solving.

Further steps in the research

In this final section, we point to some of the factors and questions which will influence further steps in the research. We commented earlier on the difficulty of finding an appropriate account of 'higher order' cognitive skills which can be applied in the interactive multimedia environment. It was in the iterative design of the research, after the first set of user trials with Flashback!, that we entertained particular misgivings about the lack of clarity in this area. Concepts of 'higher order' cognitive skills have both a history and contingent social and cultural determinations. Their nature is influenced by a society's evaluation at any particular time of what count, for example, as remunerable cognitive skills.

Any attempt to use the concept in relation to learning with technologies in a particular set of social circumstances must therefore defer to some extent to current moves towards greater explicitness concerning the desired outcomes of education and training. However, the value placed on particular kinds of cognitive skills, as expressive of 'intelligence', seems unlikely to remain constant. Does this mean, perhaps, that the pedagogical value of interactive multimedia in higher education will increase with emergent social interpretations of intelligence which take greater account of its alternative manifestations (Gardner, 1983)?

Again, the intense interest in theories of representation - in cognitive science, psychology, anthropological biology, linguistics, philosophy and aesthetics - gives evidence of convergence towards the mapping out of a domain of pre-conceptual experience which may specifically determine the categories of reasoning and logical cognitive activity which are most highly prized as 'higher order' thinking. Is interactive multimedia somehow particularly consistent with this 'new order'? Does its ostensibly close mapping with the bodily basis of rationality (Johnson, 1987), with the 'ecological information processing' proposed by Lewis (1992), or with the 'cognitive trails' in Cussins' (1992) theory, make it the learning medium par excellence of the 21st century?

One cannot ignore either endeavours to redefine some of the cognitive aims of education or the influence of these latter research directions. We cannot yet be certain that the categories of cognitive skills we are using and developing take adequate account of the role of the pre-conceptual and 'ecological' skills of human learners. We may well suspect that current trends in information technology, and the ever increasing use of graphical interfaces and the intuitive spatial manipulation of information, will accelerate the acceptance of alternative expressions of cognitive competence and 'intelligence'.

It is in such circumstances that interactive multimedia environments for learning will not be judged simply for their potential to stimulate the forms of propositional manipulation which are best accounted for in some existing schemas of cognitive processes. We believe that the frameworks for observation and analysis which we are developing will need to accommodate broader and more subtle expressions of cognitive activity.

References

Alexander A. & Frampton D. (1993). Student interaction with multimedia: Cognitive processing and user interface evaluation. In R. Iannella (Ed.), QCH'93 Symposium Proceedings. Gold Coast, Australia: Bond University.

Anderson, A. and O'Hagan, F. (1989). Dyadic interactions at the microcomputer interface: A case study in computer assisted learning. Journal of Computer-Assisted Learning, 5, 114-124.

Atkins, M. & Blissett, G. (1992). Interactive video and cognitive problem solving skills. Educational Technology, 32(1), 44-50.

Clark, R. (1983). Reconsidering research on learning from media. Review of Educational Research, 53(4), 445-459.

Clements, D. & Nastasi, B. (1988). Social and cognitive interactions in educational computing environments. American Educational Research Journal, 25, 87-106.

Cognition and Technology Group at Vanderbilt (1990). Anchored instruction and its relationship to situated cognition. Educational Researcher, 19(6), 2-10.

Cradler, J. (1985). CompuTHINK and criteria for selecting software. In A. Costa, Developing minds: A resource for teaching thinking. Alexandria, VA: Association for Supervision and Curriculum Development.

Cussins, A. (1992). Content, embodiment and objectivity: The theory of cognitive trails. Mind, 101(October), 651-688.

Dede C., Fontana L. & White C. (1993). Multimedia: A gateway to higher order thinking. In N. Estes & M. Thomas (Eds.), Rethinking the roles of technology in education: The 10th International Conference on Technology and Education, Vol. 1, 376-378. Cambridge MA: MIT Press.

Ericsson, K. & Simon, H. (1984). Protocol analysis: Verbal reports as data. Cambridge MA: MIT Press.

Ferretti, R. P. (1991). Interactive multimedia research questions: Results from the Delphi study. In Wilson, J. & McBride, K. (Eds), Multimedia technology seminar: Proceedings (Washington DC, May 2021), 64-79. ERIC document ED 338 008.

Gardner, H. (1983). Frames of mind: The theory of multiple intelligences. New York: Basic Books.

Goodrum, D. & Knuth, R. (1991). Supporting learning with process tools: Theory and design issues. Proceedings of Selected Research Presentations at the Annual Convention of the Association for Educational Communications and Technology. ERIC Document ED 334 984.

Guba, E. & Lincoln, Y. (1982). Epistemological and methodological bases of naturalistic enquiry. Educational Communication and Technology Journal, 30, 233-252.

Gubbins (1985). Matrix of cognitive processes. In A. Costa, Developing minds: A resource for teaching thinking. Alexandria, VA: Association for Supervision and Curriculum Development.

Hedberg, J. (1993). Expanding roles for interactive multimedia. Technology and Higher Education Conference Proceedings. Sydney: AIC Conferences.

Johnson, M. (1987). The body in the mind. The bodily basis of meaning, imagination and reason. Chicago: University of Chicago Press.

Jonassen, D., Ambruso, D. & Olesen, J. (1992). Designing a hypertext on transfusion medicine using Cognitive Flexibility Theory. Journal of Educational Multimedia and Hypermedia, 1(3), 309-322.

Jonassen, D. & Grabinger, R. (1989). Problems and issues in designing hypertext/hypermedia for learning. In D. Jonassen & H. Mandl (Eds.), Designing hypermedia for learning. Berlin: Springer-Verlag.

Jonassen, D. & Wang, S. (1993). Acquiring structural knowledge from semantically structured hypertext. Journal of Computer-Based Instruction, 20(1), 1-8.

Kay, R. (1993). An exploration of theoretical and practical foundations for assessing attitudes towards computers: The Computer Attitude Measure (CAM). In N. Estes & M. Thomas (Eds.), Rethinking the roles of technology in education, Vol. 1: The Tenth International Conference on Technology and Education, 302-304. Cambridge MA: MIT Press.

Kommers, P., Jonassen, D. & Mayes, J. (Eds) (1992). Cognitive tools for learning. Berlin: Springer-Verlag.

Lewis, M. (1992). Situated visualisation: Building interfaces from the mind up. Multimedia Review, 2(4)/3(1), 23-40.

New South Wales Board of Studies (c1992). Flashback! User manual. Sydney: New South Wales Board of Studies.

Pea, R. D. & Sheingold, K. (Eds.) (1987). Mirrors of Minds: Patterns of experience in educational computing. Norwood NJ: Ablex.

Pea, Roy (1985). Beyond amplification: Using the computer to reorganise mental functioning. Educational Psychologist, 20(4).

Perlman, G. (1992, November). Practical user interface evaluation. Workshop presentation to OZCHI '92 Conference. Gold Coast: Bond University.

Rowland, G. (1992). What do instructional designers actually do? An initial investigation of expert practice. Performance Improvement Quarterly, 5(2), 65-86.

Authors: Dr Malcolm Alexander, Faculty of Humanities, Griffith University, Nathan, Queensland 4111, Australia. Tel: 07 875 7169 Fax: 07 875 7730. Email: M.Alexander@hum.gu.edu.au
David Frampton, Director, Educational Technology Services, Griffith University, Nathan, Queensland 4111, Australia. Tel: 07 875 7142 Fax: 07 875 7845. Email: D.Frampton@ins.gu.edu.au
Please cite as: Alexander, M. and Frampton, D. (1994). Technology and thinking: A qualitative study with interactive multimedia products. In C. McBeath and R. Atkinson (Eds), Proceedings of the Second International Interactive Multimedia Symposium, 4-9. Perth, Western Australia, 23-28 January. Promaco Conventions. http://www.aset.org.au/confs/iims/1994/alexander.html

[ IIMS 94 contents ] [ IIMS Main ] [ ASET home ]
This URL: http://www.aset.org.au/confs/iims/1994/alexander.html
© 1994 Promaco Conventions. Reproduced by permission. Last revision: 19 Jan 2004. Editor: Roger Atkinson
Previous URL 22 Apr 2000 to 30 Sep 2002: http://cleo.murdoch.edu.au/gen/aset/confs/iims/94/alexander.html