Methodological Considerations for the Evaluation of Computer Assisted Learning in Higher Education

Lynn Wilss

Queensland University of Technology, Brisbane

Objectives/Outcomes:

What's happening when students use CAL?
How does it fit in with the rest of the course?
How does CAL assist their understanding?
What is the lecturer's role?
How was the program developed?
Will it be on the exam?
But are they learning anything?
How do I find out the answers?

Introduction

If you have participated in the process of development and integration of Computer Assisted Learning (CAL) you may know the answers to the above questions. If you are considering embarking on this process, then you need to know the answers. During 1997 the Queensland University of Technology (QUT) conducted an evaluation to determine the answers to these questions and others. This evaluation was part of the Commonwealth "Quality" grants to evaluate trends in the development and integration of educational technologies.

Specifically, the evaluation was conducted to ascertain how CAL programs developed by Teaching and Learning Support Services, a Division of Information Services at QUT, were integrated within courses and if educational objectives were being achieved. When conducting such an evaluation, decisions must be made with regard to the most suitable methodology to be employed, what groups of people should be involved, what aspects of CAL development and integration need to be investigated, and how data that is gathered should be analysed. This workshop aims to address these methodological considerations within the context of the CAL evaluation conducted at QUT and in doing so will present findings that relate to:

student learning with CAL
strategies students use when interacting with a CAL program
the gains and pitfalls of CAL development
the interplay of relationships between those involved in the developmental process

The Workshop

Qualitative Data Collection and Analysis

As the evaluation of CAL at QUT constituted a qualitative design, the workshop will focus on qualitative data analysis using the computer program "Non-numerical Unstructured Data Indexing Searching and Theory-building" (NUD*IST) (Q.S.R. NUD*IST Power Version 3.04, 1994). The following factors were also integral to the process of the evaluation and as such will be addressed briefly:

Qualitative Methodology: An Overview of Theory and Practice,
Needs Analysis - Determining the Groups Involved
Setting the Evaluation Objectives,
Data Collection Techniques

Qualitative Methodology: Theory and Practice

Salomon distinguishes the qualitative research paradigm as assuming that elements are "interdependent, inseparable, and even define each other in a transactional manner so that a change in one changes everything else". Qualitative research procedures must therefore define clearly the phenomenon under study and the natural setting of the investigation. During the workshop these factors will be discussed along with terminology specific to qualitative research.

Needs Analysis

Determining the Groups Involved and Setting the Evaluation Objectives Methods employed to determine the target groups of the QUT evaluation into CAL will be presented along with factors that contributed to formulation of evaluation objectives.

Data Collection Techniques

Qualitative research employs multiple methods of data collection. The methods used in the QUT CAL evaluation will be presented.

Qualitative Analysis - NUD*IST

As data gathered was non-numerical and unstructured, the computer program NUD*IST was used to facilitate its organisation and subsequent analysis. During the workshop the structure of a NUD*IST project will be presented. This will incorporate the development of a branching system and accompanying nodes, exploring data documents and text units, and categorisation and indexation of qualitative data.

You will be given the opportunity to practice qualitative analysis techniques. This will be done within the context of issues that emerged about the "Development of CAL at QUT". The following data exemplifies some of the findings that will be explored in relation to positive and negative aspects of the developmental process:

Technology Related: Ability to incorporate deeper issues, Developed a greater understanding of the process of development, Developed leading technology
Time Factors: Delays, Became a lifestyle, Lengthy Process
Relationships: Collaboration - learning from others, Communication
difficulties between the groups involved, Need for consistent team effort

Intended Level of Presentation:

Overview