Full Paper

Lynn Wilss

l.wilss@qut.edu.au

Division of Information Services

Queensland University of Technology

This paper details results of an evaluation of Computer Assisted Learning (CAL) at the Queensland University of Technology. Qualitative methodology was employed to investigate factors that constituted cognitive, relevance, and accessibility dimensions of learning with CAL. In addition, processes of developing and integrating CAL programs within courses were examined. CAL programs considered were from the following schools: Mathematics, Optometry, Humanities, and Nursing. Data were gathered from three groups: lecturers, students, and instructional designers. Analysis of the results indicated the CAL programs fostered student engagement and learning that matched educational objectives set by lecturers; CAL facilitated a constructive approach to learning for students; and CAL initiated deeper learning for some students. It also became apparent that multimedia features assisted comprehension and understanding and students were afforded opportunity to practise flexible learning strategies. Perceived limitations of CAL included lack of relevance of CAL to the rest of the course, in some instances limited access to computers, the lengthy process of development, and relationship issues between members of development teams. Findings relating to cognitive and relevance dimensions of CAL as well as developmental procedures are reported in this paper. Some recommendations regarding development and integration are suggested.

Computer assisted learning (CAL) is becoming increasingly prominent in education and as a result is changing the face of teaching and learning. While the driving force behind such innovations is multifaceted, the desire to increase teaching efficiency (Davies & Crowther, 1995) and quality (Coopers & Lybrand et al., 1996) have generally been regarded as the major contributing factors. This paper details some results of a comprehensive evaluation undertaken to determine the efficacy of CAL at QUT. Specific objectives of the evaluation were: cognitive - to determine how CAL accommodated students learning needs; relevance - to determine studentsí perceptions of CAL as part of their course of study; accessibility - to determine how CAL was integrated into courses; developmental procedures - to determine the reasons for adoption of CAL and subsequent processes of development. Findings relating to cognitive, relevance and integration, and developmental processes are reported in this paper.

Higher education institutions are making greater use of CAL. This had led to the establishment of ëcentresí in some institutions where customized software is developed. Jacobs (1996) proposed conditions for the establishment of such centres stating they should be:

• staffed by experts who can advise departments and facilitate software development;
• supported by technicians;
• equipped with multimedia;
• backed by sufficient financial resources to ensure continuous upgrades.

While these conditions may facilitate development of CAL programs, the actual process of development is complex and time consuming (Reid, 1994). Production of effective educational CAL programs entails incorporating elements that will ensure student engagement. This means addressing specific design considerations such as:

• Incorporating an interface that will facilitate student movement within the program. Park and Hannafin (1993) suggested the use of familiar visual ëmetaphorsí as they believe this will lessen cognitive processing demands associated with use of new software.
• Deciding on the multimedia features that will be suitable to the nature of the content. Some believe the multimedia features such as graphics including illustrations, animations, digital video (de Laurentiis, 1993), or simulations (Somekh, 1996) enhance the processes of learning as they help students to clarify topics and allow for the development of constructive learning interactions.
• Determining the type of interactivity (Pea & Gomez, 1992) and feedback (DeLaurentiis 1993) that should occur.

Taking into consideration the factors mentioned above, it becomes apparent that development of educationally sound IMM is a complex process. Some (Reid, 1994) consider the process of developing CAL to be fraught with difficulties. Factors that may contribute to developmental difficulties include: lack of educationally sound examples on which to base development, departure of staff involved in development, the lengthy time of development and the high costs involved. Darby (1992) suggested developing software that can be used over a number of institutions as part of a ìcollaborative consortiaî (p. 194) to alleviate the problem of high costs. Allen et al. (1996) consider the working relationship between developers and subject matter experts to be complex and a potential source of difficulty. If advantages of learning with CAL are to be realized, integration must be planned carefully.

Successful development and integration of CAL within courses in higher education institutions is dependant, to some extent, on the culture of the institution. Darby (1992) maintained that major changes to the organization of higher education would have to occur for CAL to be used on a wider basis. He suggested making more funds available for courseware development and that lecturers accept a changing role to become more involved in interacting with students as managers of learning. Reid (1994) reported that CAL developed by isolated enthusiasts without recognition or support from peers or the institution can result in the demise of a CAL project if the enthusiast left. To avoid this he advised that development become a part of ìthe academic life of the institutionî (p. 8). Integration must also ensure that students make links between the different types of learning they are experiencing. As Wills and McNaught (1993) contend ìclear information needs to be given about the objectives of any course in which CBL is being usedî (p. 699). The influences of CAL on teaching and learning are explored in the following section.

Ultimately instruction is conducted so that students will learn. Some caution that use of IMM does not necessarily mean learning will follow (Bruder, 1991; Allen & Booth, 1996). De Laurentiis (1993) believes that for learning to occur the surface structure of a curriculum, such as the activities and graphics, should match the deep structural issues of the knowledge domain. He maintained IMM has the potential to do this as it incorporates every educational medium except human contact and suggested considering the following points when developing IMM to ensure it is educationally sound: it should be adaptable to studentsí current knowledge and skills, IMM should utilize different media depending on the content and educational objectives, and it should include feedback that is immediate and serves to focus learning activities.

In addition, it is essential that CAL programs are supported by a clear model of learning and teaching (Coopers & Lybrand et al., 1996). For example contemporary theories of learning place emphasis on gaining knowledge through appropriation of information. This is central to a constructivist tradition of learning that emphasizes the role of the learner in knowledge creation (Biggs, 1991). Implications for teaching involve encouraging students to plan and define their goals for learning based on their existing knowledge; allowing students opportunity to examine their processes of thinking and problem solving, that is to become metacognitive processors of information; and providing learning experiences that engage the learner. Hedberg et al. (1994) proposed a method for developing IMM, based on the principles of constructivism, that incorporated three phases: 1) describe the project space, based on the content, and give consideration to structure of the material and what the target audience understands about the material; 2) review the description, make links through an appropriate instructional presentation strategy, and identify metaphors that will facilitate information structure; 3) link the design ideas into an interactive structure.

In summary, productive learning is dependent upon considerations such as content, delivery, the needs of the learner, and specific aspects of the learning environment. Evaluation of courses that incorporate technological innovations helps to illuminate factors that may contribute to educationally sound practices and productive learning outcomes. This evaluation set out to do that as indicated in the objectives detailed above. Methods employed and subsequent findings are presented below.

This evaluation was summative as it set out to determine if CAL programs used across four schools at Queensland University of Technology had achieved intended instructional and integrative objectives. Qualitative methodology was employed. Data were collected from students, lecturers, and instructional designers (IDs).

The sample comprised three groups: four lecturers, one involved in development and integration of each CAL program; four instructional designers (Ids), one from each program; students who used the programs. The student profile is detailed in Table 1. In summary students were undergraduates with three groups of students in the first year of their course and optometry students in their final year (4^th year). The majority of students were aged in their early twenties with ages ranging from 17 years to 36 years. Ages of students who worked through ëClick On To Mathsí were not available (see Data Collection for ëClick On To Mathsí below).

The CAL programs and educational objectives for each program were:

ëClick On To Mathsí - Mathematics, was developed to provide students with a standard level of understanding of trigonometry and complex numbers;

ëEye Managerí - Optometry, was developed to provide individualised instruction in crucial areas of clinical decision making and patient management;

ëProblem Based Learningí (PBL)- Nursing, was developed to allow students to develop and practise decision making skills and to allow students to develop and integrate knowledge;

ëLanguage Masterí - Japanese, was developed to allow students to practise aural comprehension, conversation, and reading skills in beginning Japanese.

Each program contained elements of multimedia and offered interactive learning. Data were gathered after students had used the program during semester 1, 1997; data for ëClick On To Mathsí was also based on student use during semester 1, 1996. Student use for ëPBLí - Nursing and ëEye Managerí comprised one session lasting approximately 45 to 60 minutes. ëLanguage Masterí - Japanese was timetabled for a one hour session each week for weeks 4 to 12 of the semester. Students worked individually, in pairs, or in small groups. ëClick On To Mathsí was used during a tutorial session by a group of students in semester 1of 1996. During 1997 use was voluntary, as explained below.

Some methods of data collection spanned all respondent groups while other methods were specific to a particular group. The methods used for each group are as follows:

• Lecturers were interviewed, asked to maintain a ëlogí detailing events that related to the integration and student use of the CAL program, and were asked for copies of documentation pertaining to development and integration.
• Students were asked to complete a questionnaire relating to their learning with CAL, some were observed using the program, and focus group discussions were conducted with students who used ëEye Managerí, ëPBLí - Nursing, and ëLanguage Masterí - Japanese. Students from the focus group for ëLanguage Masterí - Japanese were also asked to maintain a ëlogí that related to their learning with CAL.
• Instructional Designers were interviewed and also asked for documentation pertaining to the program they had developed.

The lecturer who had instigated the development of ëClick On To Mathsí retired at the end of 1996, consequently use by students during the semester this evaluation occurred was voluntary. Only 11 of 500 students used the program and 3 students completed the questionnaire. Students had been surveyed after use of the program during a tutorial in Semester 1, 1996. These surveys were made available as a source of data for the current evaluation.

Qualitative data were analysed, in line with Richardís and Richardís (1994) belief that qualitative research involves ìrecognition of categories in the data, generation of ideas about them, and exploration of meaningsî (p. 446). The analysis, carried out using the computer program (NUD.IST) version 3.04, revealed categories which are delineated below along with interview extracts that illustrate each category.

Lecturers were asked for the educational objectives of their specific program; IDs were asked what they perceived students would learn as a result of using the program; students were asked what they had learnt. The main response categories ranged from learning nothing, to subject specific learning, and finally to learning content and skills. A response was categorised as subject specific if it did not relate generally to learning from the CAL programs but concerned specific aspects of a particular program. A sub-category of skills was defined as working through a process. Examples of responses for each CAL program are presented below.

Most responses for learning from Click On To Maths related to learning content. This was evidenced in the lecturerís response and two student responses. One student felt they had learnt nothing, another studentís learning was subject specific, and the ID saw learning as skills based.

There was consensus among the three groups about learning from Eye Manager with the lecturer, ID, and 16 students stating that learning involved gaining the skills to work through a process. This process involved gaining skills in clinical decision making and management of patients. Other student responses included learning nothing (3) and subject specific learning (6) which indicated that students had gained a greater awareness of their diagnosis and management capabilities.

The majority of responses about learning with PBL indicated that the skills of working through a process to solve a problem were gained. This was evidenced in the Lecturer, ID, and eight student responses. One student felt they learnt nothing.

The lecturer stated that use of Language Master - Japanese would improve studentsí listening and reading skills. While nine students concurred with this, the ID and majority of studentsí responses (16) were subject specific and indicated cultural aspects of Japan would be or had been learnt. Four students felt they learnt content such as meaning of words and grammar.

Each group was asked to identify features of the program that would or did facilitate comprehension and understanding. Four main categories of responses became apparent: the interface; in-built features - that is specific elements of the program that were incorporated to promote student engagement; the interactivity; student strategies. In-built features included pictures, the HELP option, feedback, and the logical sequence of content. Students also reported using other strategies, that is student strategies, to assist their comprehension and understanding, these were asking someone and self-initiated strategies such as trial and error. (Examples of responses will be included at the conference presentation).

Factors pertaining to relevance and integration of CAL emerged from responses about the limitations of CAL. Some felt the content was either too directed in approach while others felt that it was too limited. Socially, CAL was regarded as being addictive as well as not allowing for social interaction as the human element was missing. Integrative limitations included technical problems and limited access. Some students expressed uncertainty as to the relevance of the program in relation to the rest of the course as well as specific program objectives. (Examples of responses will be included at the conference presentation).

Lecturers and IDs were asked about positive experiences as well as difficulties faced during development of the CAL programs. Positive aspects comprised two main categories: personal and technology related and derived mainly from lecturersí responses. The personal category was defined further as gaining satisfaction from the process of development and enjoying the peer collaboration. Two IDs responded that they had gained nothing or could not think of anything that was positive about the process of development. Technology related positive aspects included gaining a greater understanding of the process of development, being able to incorporate deeper issues, and developing leading technology. Negative responses comprised money issues in that developing CAL was an expensive process, time in that development was a lengthy process, procedural difficulties, staff problems as staff were working on other projects or ID staff left during development, and technology related matters.

Another negative aspect of development concerned the working relationship between lecturers and IDs. Negative categories constituted disagreements and feeling a lack of support, lecturers and ID teams having different ideas about development, and a lack of communication between the groups. (Examples of responses will be included at the conference presentation).

Student responses to questions about learning with CAL revealed that each of the programs facilitated learning according to the lecturersí objectives. Students also reported that learning with CAL had allowed them to gain a broader knowledge of the subject content. This implies that students used knowledge they already had as a basis for further learning and indicates that the CAL programs fostered a constructive approach to learning (Biggs, 1991). Features of the programs that lecturers and IDs stated were included to foster comprehension and understanding were identified by students as doing just that. In particular multimedia elements like graphics and the notepad were reported as assisting comprehension and understanding. As de Laurentiis (1993) stated multimedia features, such as graphics, enhance the processes of learning as they help students to clarify topics and allow for development of constructive learning interactions. Students also used their own strategies, such as asking someone, thus reinforcing the flexibility of learning with CAL. While the CAL programs achieved educational objectives, some limitations became apparent. The most commonly reported limitations from students were that the programs lacked relevance to the rest of the course and there was uncertainty about objectives for learning with CAL .

The majority of positive comments about development of CAL were reported by lecturers. Most stated they were satisfied with development and pleased with the completed program. Allen and Booth (1996) noted that difficulties of development can relate to relationship issues. Results of this evaluation indicated that this was the case particularly for IDs. In fact one ID experienced great relief when the project finished. Other negative aspects were similar to those identified in the literature and included high cost, the lengthy time taken (Reid, 1994), insufficient staff working on projects, and communication difficulties (Phillips, 1996). Integration was also problematic for one program as the lecturer involved left and there was no support within the school to use the program.

The results of this evaluation indicated that the CAL programs fostered student engagement and learning and allowed students to develop a constructive approach to their learning. The multimedia features assisted comprehension and understanding of information and in addition to this afforded students the opportunity to practise flexible learning strategies. The need to make closer links between CAL and the rest of a course, including objectives for learning with CAL became evident. It also became apparent that the process of development is complex and can be fraught with difficulties that include both technological considerations as well as relationship issues. Additionally development and integration should involve support from within a school. Some recommendations are suggested:

• development processes need to be defined clearly;
• there is a need for a ëLecturer Awareness Programí relating to development of CAL;
• a workable ëstaff to development projectí ratio needs to be considered and deployed;
• development of CAL should involve a ëwhole schoolí commitment rather than single lecturer focus;
• integration procedures should be determined early in development.

(c) Lynn Wilss

The author(s) assign to ASCILITE and educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author(s) also grant a non-exclusive licence to ASCILITE to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the ASCILITE 97 conference papers, and for the documents to be published on mirrors on the World Wide Web. Any other usage is prohibited without the express permission of the authors.