Full Paper

Thao Lê, Quynh Lê

T.Le@utas.edu.au, qle@leven.appcomp.utas.edu.au

University of Tasmania

Though the Web has become a familiar name in many professions and disciplines, it is in the field of education that the Web has started to open many interesting promising windows. The names and titles such as Ed Web, Educational Multimedia, Hypermedia and other similar new terms reflect the rising interest in the Web. Tertiary education has recently become the Web target for exploration in teaching and learning. This leads to the awareness of the significance of evaluation in educational multimedia. Evaluation of software can be carried out at any time during the process of construction or after it has been introduced to a group of users. The advantage of having the evaluation during the process of construction is that weaknesses or defects can be identified early and rectified before the product is completed. This will prevent further complicated problems when the software has been produced and introduced to users. In the context of CAL, software developers and teachers can work collaboratively to enrich the quality of CAL software. Ultimately, learners are the most important players in the evaluation process.

The popularity of the Web and the incredibly rapid growth of the Web technology could not be imagined a decade ago. The Web is no longer the domain of computer scientists but belongs to what can be referred to as 'the Web community of members of divergent backgrounds'. Ellis (1996) views this community as 'gardeners' landscaping the Web who are striving to achieve new dynamic , creative environment; in this case a networked, global hyperlinked electronic environment rather than a physical environment. Ellis (1996:1) refers to the Web as a new and evolving paradigm.

Courseware has been introduced to conduct off-campus learning and to contribute to the notion of virtual university (Oliveira 1992; Siviter & Brown 1992; Linn 1992). Teachers are no longer the only resource for teaching and learning. Computers have started to be used by teachers and learners for different learning activities. Recently with the development of multimedia and the Internet, terms such as educational multimedia, hypermedia, Web-based learning, courseware etc. are used to designate learning activities which involve specific computer tools and facilities.

Gilbert (1996:22) points out various ways the Web can be used to enhance teaching. The following four main components of effective Web-based teaching are described:

• Structure: Learners need to be presented with a learning objective and then guided along their learning path.
• Interactivity: Hypertext provides a substitute for reading through a textbook.
• Feedback: Teachers can provide learners with feedback on their learning tasks.
• Reflection: Learners need the opportunity to reflect on their learning and adapt their learning and conceptions in the light of that reflection.

Newmarch (1997:204) points out convincingly that there has been a massive rush over the last few years to place courseware on the Web. It is based on the cheerful assumption that this will improve student learning. Little work has been performed on testing the value of this assumption over an extended period, largely because the Web is so young. Courseware evaluation can be carried out in different ways in different contexts and for different purposes and by different kinds of people. For instance, built-in feedback devices are made available for users when they are using a software. These devices can be in the form of feedback frame, questionnaire, and e-mail facilities.

Developers of educational software take into consideration not only educational principles and learning theories but most importantly the feedback from the user, or more appropriately the learner. Feedback is an important feature in user interface design. Different writers look at different aspects of feedback: learner cognitive levels, learning styles, learning principles.

Feedback can be obtained in different ways. For instance, Pitkow and Recker (1995) uses the Web as a means of feedback. The Web can provide an excellent facility for direct feedback from users. It can be done in the form of comments and questionnaire. Collaborative work between software developers and users has also been advocated in software design and development. Robinson (1991) emphasises the effectiveness of feedback in the development of CAL in terms of factors relating to learners.

Evaluation can be made also in the form of checklist to be used by developers, teachers and learners. A checklist is a list of items which covers important points to be examined. According to Squires & McDougall (1994), many lists of criteria for assessment of individual packages have been used. They vary in content, length and style, but all have been designed in a attempt to help teachers choose software of educational value.

Evaluation of software can be carried out at any time during the process of construction or after it has been introduced to a group of users. The advantage of having the evaluation during the process of construction is that weaknesses or defects can be identified early and rectified before the product is completed. This will prevent further complicated problems when the software has been produced and introduced to users. In the context of CAL, software developers and teachers can work collaboratively to enrich the quality of CAL software.

Squires & McDougall (1994:4) introduce the concepts of formative and summative evaluation based on these two stages of evaluation mentioned previously. Formative evaluation, evaluation during development, focuses on possible modifications to the software. Summative evaluation, after publication, is concerned with the quality and variety of experiences that the software can support. In either case the evaluation involves observation of the actual use of the package by students. Both types of evaluation may be carried out by teachers, formative evaluation as trialing for software developers and summative for purposes such as writing software reviews for publication.

Ultimately, learners are the most important players in the evaluation process. It is the learner that the CAL software is constructed and their feedback decides the success or failure of software. Boyle et al (1994) conducted a student-based evaluation of their CAL system and focused on three main criteria: how easy did they find it to learn and use the system; how did they rate the main design features, and their overall assessment of their CAL system as a learning environment. Hedberg et al (1994) carried out evaluation through interviews with students and teachers and systematic observation. They found out that many insights could be gained by using this approach.

A courseware in this study was constructed and used off-class to find out how students reacted to the use of the Web in learning and teaching and particularly in relation to the linguistics-awareness courseware introduced to them in a semester. In the following part of the paper, a brief description of the courseware will be presented and the evaluation of the courseware will be discussed.

The main aim of the courseware is to teach linguistic awareness to tertiary students who need to develop general knowledge about linguistics. The courseware is of special interest to education students as language awareness is the essential aspect of language in education for prospective teachers.

The courseware consists of the following components: Content, facilitating experiences, and learning experiences. These three components represent different roles that computers can play in teaching and learning.

Content component: The four content areas are phonology, morphology, syntax and genre. Each content area consists of five items or sub-components:

• Modules: lesson-typed presentation.
• Problems solving tasks: learning by doing.
• Basic terminology: key concepts.
• Review: what has been learned.
• Key references: important readings.

Facilitating experiences: Learners are provided with activities or facilities which can be used to enhance learning.

• Stimulating questions: For arousing intellectual curiosity.
• Test: In the form of mini-test with results.
• Feedback: About the courseware for improvement.
• References: Including key references and others.

Learning experiences: The focus is on practising academic genres with tutors in a virtual classroom. This function of the courseware includes:

• Genre work: practising genres.
• Resources: for individual and group conferencing.
• On-line help: common problems.

As this was our first experience with courseware development, we were very conscious of the significance of the evaluative process in the development and implementation of the courseware. We treated equally the significance of both the formative and summative aspects as each plays an important role in the evaluation of the courseware.

In regards to the formative aspect, two strategies were employed to ensure that users were involved during the development of the courseware. Firstly, a small reference group was formed to oversee the courseware development. It consisted of a computer specialist, an educational linguist and two students. The reference group met regularly to discuss the progress of the construction of the courseware. Secondly, a feedback form was placed in the courseware itself during the trial stage of introducing the courseware to students. The feedback form was made simple and open-ended so that users could give comments on any points or aspects of the courseware which concern them. On the basis of the feedback received, it was evident that users tended to give comments on specific features of the courseware such as references, font, on-line help, frame etc. Their feedback was considered by the reference group for modification.

Basically, the questionnaire included two parts: the first one was about personal characteristics of the participants and the second about their perceptions and attitudes. The questions were chosen to determine users' perception (1) about CAL and the Web and its significance in education, (2) and their evaluation of the Web-based courseware in terms of implicit criteria drawn from items in (1). There is a close correspondence between items in (1) and (2). For instance, they are paired as follows:

• The Web provides useful interaction with others. (1)
• This course provides useful interaction with others. (2)
• A courseware should promote independent learning. (1)
• This courseware promotes independent learning. (2)

The pairing is based on the assumption that the criteria implicitly given in (1) will be used to evaluate the courseware - that is (2).

The construction of the questionnaire is based on the guidelines given by Burns (1990:293). It was available on the Web. The advantage of having the questionnaire as part of the courseware on the Web was that the participants could fill in the questionnaire straight away after browsing through the courseware. The availability of the questionnaire on the Web could therefore re-enforce their perception of the courseware. There were several advantages of Web-based questionnaire over the conventional method of using hard copies:

• There were no complaints about loss of questionnaire.
• If mistakes were made during the filling of the questionnaire, the respondents can erase and use another form.
• No costs were involved in printing and posting.
• If the respondents were unsure about any aspects about the courseware when filling the questionnaire, they could check the courseware immediately to reinforce their views.
• No extra time was needed for returning the questionnaire to the researcher. This could be a factor which discouraged respondents to return the questionnaire in non-electronic handling.

However, there were also disadvantages concerning the use of Web-based questionnaire administration. They included:

• The respondents were discouraged to participate due to unfamiliarity with computers.
• The availability of computers could be limited.
• Uninvited respondents could participate if they chose to do so.

The statements on the questionnaire were based on a framework of six categories, as outlined below:

• The Web provides useful interaction with others.
• The Web can play an important part in education.
• The Web can replace the traditional face-to-face teaching.

• Colour on screen should be clear and soothing to the eyes.
• Sound should be used only when appropriate.
• Navigation points should be provided for users.

• A courseware should be intellectually stimulating.
• A courseware should be flexible for learners.
• A courseware should provide fun for learning.

• This courseware is important in education.
• This courseware can replace traditional face-to-face teaching.
• This courseware provides useful interaction with others.

• The colour in this courseware is clear and soothing to the eyes.
• Sound in this courseware is used only when appropriate.
• Text in this courseware is neatly and appropriately presented.

• This courseware is flexible for learners
• This courseware provides fun in learning.
• This courseware provides self-monitoring of learners.

Participants choose one out of five possible responses to each statement within the range of strongly agree, agree, uncertain, disagree and strongly disagree, with the assignment of scores in the range of 5, 4, 3, 2, 1 respectively. In addition to the 41 questions, there was a question which included comments from users.

Students and staff at the University of Tasmania were invited to participate in the questionnaire research. Responses from 61 participants were received via the Web. The participants include:

• Status: 35 undergraduate (57.37%), 10 postgraduate (16.40%) and 16 staff (26.23%).
• Field: 39 education (63.93%), 22 non-education (36.07%).
• Web familiarity: 6 high (9.84%), 22 moderate (36.07%) and 33 little (54.09%).

The analysis shows that while the range of scores for each item extends from 1 to 5, the median scores are on the upper end of the scale. Generally, this means that Agree responses to items far outweigh those of the Disagree, Strongly Disagree type. These results show clearly that most participants are in favour of the Web and the courseware, as the number of items with a response score of 4 or 5 far outweigh the number of items receiving a response score of 1 or 2. Of the 41 items, 38 items had a median of 4 or 5, while only three items had a median of 3. The response indicates a high expectation of a courseware in terms of intellectual stimulation, challenging, interactivity, and purposefulness.

The quantitative evidence (median was in the range 4 and 5) shows that the courseware was positively valued However, due to the limited number of respondents, the positive perception needs to be explored further, perhaps in a follow-up study.

The quantitative results are positive partly, if not primarily, due to the fact that the consultative process with some 'users and helpers' was highly valued in the development of this courseware. From the beginning to the end of the development of the courseware, a number of students and staff were invited to comment on certain aspects of the courseware in progress. Though opinions varied according to individuals' perception of user interface design, their feedback was useful for the team to review the progress. An on-line feedback was an integral part of the courseware as it was always available on the screen.

At the onset of the development of the Web-based courseware, the following assumptions were made:

• Learners' attitudes toward the Web in general would be reflected in the way they evaluate a specific Web-based courseware.
• Both the formative and summative components of courseware evaluation are essential in courseware development.
• 'Learner friendly', not user-friendly, is the key concept in courseware evaluation.
• Courseware cannot be divorced from learners' learning discourse. A virtual classroom can exist but not a virtual learning discourse.

As previous stated, this was our first experience with courseware development, we were very conscious of the dynamicity of the real discourse of the learners. We assigned significance of the evaluative process to the development and implementation of the courseware. We treated equally the significance of both the formative and summative aspects as each plays an important role in the evaluation of the courseware.

The Web has been available in tertiary education and it is most appropriate to make use of this modern technology to introduce innovative changes to the traditional way of teaching and learning. With the rapid growth of multimedia and hypermedia, their temptations have attracted tertiary teachers to introduce the electronic ways with learning. It is not just the power of the Web which is the focus of attention. Evaluation is an important part in courseware development. Without evaluation, courseware developers are not fully ware of their strengths and weaknesses and this can result in loss of time and energy for constructing software which is not appreciated by users. In the context of educational courseware, evaluation enables developers to improve their products to ensure that they are educationally innovative and effective.

(c) Thao Lê, Quynh Lê

The author(s) assign to ASCILITE and educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author(s) also grant a non-exclusive licence to ASCILITE to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the ASCILITE 97 conference papers, and for the documents to be published on mirrors on the World Wide Web. Any other usage is prohibited without the express permission of the authors.