Full Paper

Mary T. Rice

mrice@deakin.edu.au

Deakin Centre for Academic Development

Deakin University

In recent years, rapidly changing economic, sociological and technological imperatives have presented enormous challenges for universities world wide and resulted in unprecedented scrutiny of the work of academics. In Australian higher education, amalgamations and restructuring came hand in hand with more stringent government funding and closer accountability. These changes also resulted in mass tertiary education which brought with it increased class sizes and more diverse groups of students with varying levels of knowledge. Higher education now draws students from all backgrounds and all age groups. They want to study at different levels for different purposes and often face financial pressures when doing so. They expect institutions to respond to their needs in flexible ways.

Ehrmann (1995) contends that most universities are now facing the "triple challenge" of improving outcomes and extending access for an ever broadening range of students while at the same time controlling costs. He further suggests that extensive use of technologies will be needed in order to meet the challenges of providing new pedagogical opportunities and greater flexibility. Norris and Dolence (1996, p.12) argue further for a transformation in higher education "from Industrial Age educational institutions to Information Age learning enterprises" where the nature of teaching and learning and the nature of IT systems and infrastructure will be fundamentally changed.

Since its establishment in 1977, Deakin University has been a dual mode institution and has built its reputation on development and delivery of distance education. High quality, print-based materials, with some audio and video resources have been the mainstay of Deakin's programs. The huge production process established to develop and deliver off-campus materials epitomised the 'industrial age' referred to earlier. Amalgamations and restructures have given rise to a multi-campus university whose mission is to provide flexible, innovative, technologically-mediated programs to service the needs of life-long learners. This has led to a move away from the dichotomous dual mode situation to a more flexible arrangement which blurs the distinction between on-and off-campus students.

The uptake of technology for educational purposes has been strongly encouraged in some sectors of the university while others have lagged behind. More widespread use of technology is placing the spotlight on teaching and learning. Furthermore, with the move towards more flexible, student-centred learning, a much sharper focus is being brought to evaluation processes.

Three large-scale technological developments at Deakin were undertaken to increase flexibility, produce productivity gains and improve outcomes for students. They are the CAL accounting programs, Deakin Interchange and the Psychology Electronic Warehouse.

One of the problems experienced by academics teaching Accounting to first and second year students was that an inordinate amount of tutorial time was spent teaching simple accounting processes and procedures, leaving too little time for exploration and discussion of the broader issues relating to the practice of accounting. A further problem was that some students grasped the concepts quite quickly, while others needed more practice on specific procedures. In other words, students' needs varied considerably and could not be adequately met in the reduced tutorial sessions available. Moreover, there was a limit to the amount of paper-based practice which could be undertaken by students.

A technological solution to these problems was sought through the development of unique computer-assisted-learning (CAL) programs for first and second year Accounting students. They have provided a way of learning which simulates the real accounting process and provides limitless opportunities for students to practise accounting procedures. The software can be used in non-linear, interactive ways and provides layered feedback ranging from hints to explanations of answers. Through random generation of unique case studies, the software allows customisation of problems in ways which enable students to practise in their areas of weakness. Using either single issue or multi-dimensional problems, students can choose the dimensions of the case study they wish to practise and can switch off any features of the program which they do not need. Hypertext support material is provided to help students use the program effectively and enable them to access content-related material to check their knowledge

In 1994, CQAHE funding was allocated to Deakin to expand and improve the existing text-based Tutorial Electronic Access System (TEAS) which provided access to various applications and electronic services to support teaching and learning. The intention was twofold: to develop a graphical user interface so that the system would be more user-friendly, and to provide a more sophisticated conferencing system for group communications. Other initiatives were underway at the same time. In particular, ITS saw the need for a common desktop throughout the university incorporating a centralised system for software distribution and version control. It was thought that a standardised system would reduce the need for support.

The decision was taken to combine these initiatives into one project with the overall intention of developing a single, integrated, university-wide system which would meet various educational, technical, research and administrative needs across the university. Deakin Interchange was based on Mandarin software from Cornell University and provided a suite of software for easily accessing a variety of networked services at Deakin University. Access could be gained either by direct network connection on campus or via modem from off campus locations.

The Psychology Electronic Warehouse makes use of integrated technologies incorporating Interactive MultiMedia (IMM) and Computer-Managed-Learning (CML). It is being developed for first year psychology students and has both teaching and administrative functions. Experience at Deakin suggests that many psychology students find it difficult to link research methodology with specific topics, usually seeing it as just another, statistics-related topic, rather than a process undertaken to better understand phenomena relating to any topic. As detailed by Hinchy and McNolty (1997), the Warehouse uses a module-based, problem-solving approach to link the teaching of research methodology with specific content areas. It is based on a seven stage model of research methodology adapted from Graziano and Raulin (1993). Through the CML component, student performance can be monitored, and research data can be aggregated then returned to students for analysis and interpretation.

A great deal has been learnt from evaluations of these major developments. (See Rice, 1997a, 1997b, 1997c) In particular, there were lessons learned about the contingency-based nature of flexible teaching and learning, the nature of technology development, technology implementation and integration into courses and evaluation processes themselves.

The move towards technologically-mediated teaching and learning meant that academic staff found themselves teaching in unfamiliar territory. Technology forced them to think about what they do and why, and to consider how effective their practice has been in facilitating students' learning. It required them to think about new, more flexible ways of teaching and assessing students.

Over the past five years, there has been a gradual process of change at Deakin, building on the practices and traditions of the past. Technology development and implementation for flexible learning has had to take account of prevailing institutional circumstances; it is contingency-based and depends on a complex 'web' of interactions between people and other variables.

The technological imperative has forced the institution to consider changing structures which were put in place for an industrial model of education. It is now recognised that inflexible structures need to be dismantled and more flexible arrangements established which are appropriate for the 'information age'. It has been difficult to deliver flexible, technologically-mediated courses in an environment where administrative structures constrain flexibility.

In respect to technology development, a key lesson arising from evaluations is the importance of thoroughly testing the software products at beta and pilot stages before releasing it for mainstream use. This was done with the CAL accounting software and two of the Electronic Warehouse workshops. Subsequently, there were fewer problems when students used the software. On the other hand, Deakin Interchange and one of the Warehouse workshops was not sufficiently well tested. Unrealistic time constraints meant that development was rushed. As a result, students experienced many more technical difficulties and needed much more support.

Notwithstanding the importance of extensive testing, it is not always possible for software developers to anticipate all the incompatibilities which might arise between Deakin's software and configurations on users' computers which can be quite unique. Such a situation occurred with Deakin Interchange which was initially incompatible with Macintosh Powerbooks. Similarly, the Electronic Warehouse did not initially work on NT machines. However, there comes a time when the software has to be released on a bigger scale. A balance must be found between proper testing of versions and the need to release the software.

Ensuring that content was produced on time was a major difficulty for developers of the CAL software and the Electronic Warehouse. Developing educational technology requires close cooperation and understanding between a number of stakeholders, including discipline experts, software developers, educational developers and database experts. With such varied backgrounds and levels of expertise, it was sometimes difficult to reach agreed understandings about educational uses of the technology. Staff were not always able to articulate what they wanted the technology to do because they were not sufficiently aware of its possibilities. Similarly, technical experts generally did not have the educational or discipline expertise to suggest ways of using the technology for learning purposes. Often there was an impasse. From a software development point of view, it was important to build a prototype as quickly as possible to enable teaching staff to see the technical possibilities. They were then in a better position to determine whether it could meet their educational objectives.

Deakin's experience demonstrated that developing quality software takes a long time and is a costly exercise. In fact, it inevitably took longer than anticipated because of the unexpected problems which arose during development. This was particularly noticeable when developing for dual platforms, as was the case with Deakin Interchange and the Electronic Warehouse. Schedules for release often had to be extended, or the software was released before it was properly tested.

Allied to this, developers found that there was a need to set parameters for the product being developed. The number of features built into the software needed to be limited since the possibilities were endless and there was a tendency to continue building new improved versions, possibly adding to the cost. Educational specifications needed to be clearly spelt out and there came a time when the product had been refined enough.

Because of the costs involved in software development, there is no need to 're-invent the wheel'. Any software developed for educational purposes should be unique and offer features that previous software has not offered. The randomisation and customisation features built into the CAL software ensured that it was unique, and therefore attractive to academics teaching accounting in other institutions. (It is currently being used by the University of Queensland and the University of South Australia.)

Developers have learnt that when writing code, they had to be thinking about re-using it. It is more cost effective to write 'smart' code which can be used in several ways. Code used in the Electronic Warehouse and indeed the template which has been constructed can now be used for other appropriate discipline areas.

Implementation of technology is a risky business. It requires an 'act of faith' on the part of the institution, a step into the unknown and has to be seen as an experimental and exploratory process informed by continual evaluations. Yet, with all three developments, this was seldom recognised by teaching staff and students who expected that everything would function properly. Understandings about the nature of technology development and implementation were somewhat limited and there was a level of naivety about their expectations. When implementing technology, it is important therefore to ensure that staff and students are aware that things seldom go to plan in the first year of implementation. Furthermore, it is important to have contingency plans in place in the event that the technology fails. As Gilbert (1996) notes, we should expect a low success rate for complex new products and be more realistic about claims made about the educational value of the software.

Research has clearly shown that unless the use of technology is mandatory, students will not use it. Therefore, in a number of courses at Deakin, academic staff who were serious about wanting students to use the technology made it a compulsory aspect of their study. CMC through Deakin Interchange was mandatory for commerce and education students. CAL was mandatory for accounting students and the Electronic Warehouse was mandatory for psychology students. However, evaluations have indicated that it is a mistake to mandate the use of technology too soon. For example, off-campus commerce students were unable to meet the requirements set for them because there were too many technical problems with the software, and remote access arrangements were unreliable. Other arrangements had to be made. On-campus students in psychology, education and accounting experienced considerable frustration and confusion when trying to complete requirements. Where mainstreaming is done incrementally, the fall out from technical failures is minimised.

Many of the problems which arose during implementation of the technologies were due to inadequate infrastructure and poor planning at the institutional level. Evaluations have shown that, for online learning, networks need to be reliable and have sufficient bandwidth for staff and students to download materials efficiently. Users also need access to hardware with sufficient memory and hard disk space and need assurances that standard hardware specifications will be established and supported. Adequate server capacity must be available and our experience suggests that when servers are devolved to Faculties rather than remain centrally controlled, there are fewer problems for users.

A very strong message arising from evaluations is that activities in campus laboratories must be properly coordinated, particularly for first year students. Labs need to be fully equipped and manned by academic and technical support staff. It is sensible to allocate time for particular groups of students to use the labs, rather than leave access to chance. Access to computers and to the software should be taken for granted by students who are required to use them for their courses.

Clearly, demand for support inevitably exceeded the capacity of the institution to respond, particularly when many of the problems were due to students' lack of experience with basic use of computers. This is not unique to Deakin, it is a world wide phenomenon, and as articulated by McClure, Smith and Sitko (1997), new ways of organising support must be developed. Students are sometimes the forgotten group of stakeholders in the teaching and learning process. During the implementation of technology, they are the 'guinea pigs'. They frequently experience anger, frustration, confusion and uncertainty when new technology is being implemented. They also experience a lack of control over their work (as do academic staff). Their studies can be severely disrupted when they encounter technical problems which they cannot solve for themselves. At the first level, there is a need for accurate, detailed documentation about how to install and use software.

Geoghegan (1996) suggests that there is a chasm between early adopters and mainstream users, such that when technology is mainstreamed, the type and extent of support needed is more comprehensive and demanding. This is particularly evident when there has been insufficient staff training. At Deakin, successful implementation and student support often depended on the commitment of the software developers and key staff with vested interest in its success. More investment in training before mainstreaming would have reduced the need for support.

Implementation works best when there is a consultative and supportive approach taken by staff, who are attune to students' reactions and have plans in place for responding to problems which arise. Consideration needs to be given to anticipating difficulties which might occur and to ways of dealing with them. When students are kept fully informed about what is happening, and when they sense that their needs are being considered, they are more likely to be positive about the experience of using new technology, and are more likely to cope with the technical mishaps.

Whether or not student assessment requirements should be embedded in the technology is an issue which needs careful consideration. If there are technical problems which delay students' study schedules or cause anxiety, it is clearly disadvantageous. On the other hand, students who have to spend time using the technology want some credit for their efforts. In the initial stages of implementation, it may be preferable not to grade the work, though there can be hurdle requirements. Providing the technology is robust, linking assessment to it sends the message that it is an integral and important part of their course and is therefore the best way to integrate the technology into mainstream teaching and learning.

Conducting evaluations in educational environments has always been a difficult, complex endeavour. Conducting evaluations of technological developments is even more so because of the many new variables which can affect the learning experience. In addition to evaluating content, pedagogies, assessment strategies, style of teaching and so on, evaluators have to examine access to the technology, its stability and reliability, bandwidth and modem speed, functionality, costs, user training and support. The interrelationships between educational and technical factors increase the complexity of the environment and the resultant evaluation.

Evaluations of technology-based learning at Deakin significantly highlighted the politics of the institution because there were multiple stakeholders with vested interests in particular aspects of the development and delivery process. As well as educators themselves, there were software developers, educational developers, ITS experts, database experts all of whom had a keen interest in ensuring that their work was seen in the most positive light. When things went wrong, there was a tendency for groups to emphasise the failings of other groups and to rationalise their own actions. This is apparently quite common. Thompson (1997) noted that a number of tertiary institutions have difficulty reporting failure and problems in the context of implementing technology.

When conducting evaluations, there are always multiple stories to be told, and different perceptions of the reality of the experience, sometimes giving rise to contradictory evidence. In such a situation, it may not be politic for an internal evaluator to be frank because it is not always in the best interests of fostering change if groups of stakeholders are humiliated or found wanting by evaluation findings. This leads to a sanitisation of findings which places the institution and groups of stakeholders in the best possible light.

The power relationships between groups and even within groups sometimes causes conflict about how to respond to evaluations. If there is a lack of consensus about solutions, or if certain powerful groups choose to ignore recommendations, evaluation does not improve anything. Decision-making processes can become paralysed and the status quo is maintained. The level of resources assigned to evaluation can indicate how much importance is placed on the outcomes. Moreover, the professional standing of the evaluator can affect the extent to which recommendations are implemented.

It is clear that evaluation processes can be facilitated by establishing a framework within which evaluations take place. Such a framework should clarify aims and objectives, list all stakeholders and include policies which deal with operational aspects regarding responsibilities for data gathering, ownership and control of the data and ethical issues. The roles and responsibilities of all involved in evaluation processes should be clearly articulated and agreed upon. The framework should also specify major areas of concern, key questions to be asked, methodologies to be used and appropriate timelines for examining particular evaluation foci. Consideration also needs to be given to ways of determining the effectiveness of evaluation processes, particularly the extent to which they enabled improvements to be made to teaching and learning. With these procedures in place, there is less chance that evaluation will be compromised by the prevailing political climate.

Clearly, technology works best when there is careful strategic planning, realistic schedules and phased implementation at the institutional level. At Deakin, the establishment of institutional infrastructure, comprehensive staff and student training and support mechanisms have lagged behind the technology push from early adopters. The mainstreaming of CAL, Deakin Interchange and Electronic Warehouse technology highlighted the need for strategic thinking and significant institutional change.

Erhmann (1996) argues that to be deemed successful, technology needs to be both valuable in terms of its educational potential, and viable in terms of its continued long term attraction to users. Each of the software developments has been potentially valuable, but only the CAL software can be said to be viable at this stage. Deakin Interchange is being phased out because the Mandarin software is no longer viable, having been developed prior to Windows 95. It will be replaced with the Deakin Learning Toolkit CD-ROM. It is too soon to judge the long term viability of the Electronic Warehouse.

(c) Mary T. Rice

The author(s) assign to ASCILITE and educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author(s) also grant a non-exclusive licence to ASCILITE to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the ASCILITE 97 conference papers, and for the documents to be published on mirrors on the World Wide Web. Any other usage is prohibited without the express permission of the authors.