Full Paper

Angela Carbone, Matthew Drago

angela@cs.monash.edu.au, mattd@cs.monash.edu.au

Department of Computer Science

Ian Mitchell

ian.mitchell@education.monash.edu.au

Faculty of Education

Monash University

In 1994, the Dean of the Faculty of Computing and Information Technology (FCIT) approached the Dean of Education for assistance in tackling a perceived problem in the teaching of computing in the Departments of Computer Science and Software Development. The main concerns were high failure rates, a low flow of students into higher degrees and a perception of a wide variation of teaching skills of the tutors (Mitchell, Macdonald, Gunn, & Carbone 1996).

As a result, a collaboration team know as ìEdprojî was established. It composed of staff members from the two FCIT departments and the Education Faculty. During 1995, Edproj sought to establish the present teaching and learning structures and dynamics in first year programming. This involved interviews with lecturers, tutors and students, observations of lectures, discussion classes and laboratories and close evaluation of the content of the courses.

Observations made by Edproj revealed that the quality of some teaching was less than a minimum acceptable standard. Data from student interviews and surveys indicated that the tutorials and labs were their most important learning situations, yet little effort was made to train those in the front line of student contact. This had serious negative consequences on the studentsí learning.

Edprojís initial findings also showed a common paramount emphasis by the students to get the set tasks done, meet the requirements and then stop. Concepts which were not stated for assessment were ignored. Tasks intended by lecturers to be gateways to exploration and reflection were seen by students merely as hurdles to be negotiated.

As a result, Edproj developed a training programme which tutors and demonstrators attended at the beginning of the year (Carbone, Mitchell & Macdonald 1996). The programme comprised of an initial 3-day workshop, and fortnightly meetings aimed at assisting new tutors and demonstrators in their role. At the fortnightly meetings, teaching strategies were discussed, and feedback regarding the lessons and the set tasks were recorded and transcribed into a World Wide Web document (Carbone & Mitchell 1996). The document, although containing useful information was difficult to maintain and another method of recording and capturing individualís teaching strategies was needed. As a result a web based database application was designed.

This paper describes the database known as ìTeachToolsî, designed to capture and analyse teaching skills, and review the set tasks. TeachTools supplements the work of a project commissioned by the Committee for the Advancement of University Teaching in June 1995, titled Reflections on University Teaching (CAUT 1996). The database includes information, as provided by tutors, on the teaching practice itself in the context in which it occurs, and includes their review of the set tasks with suggestions for improvements. This information is intended to be used to refine the tasks set in the course and improve teaching strategies to motivate students and enhance their learning and understanding.

The database design consists of four components; Configuration and Administration of the Database, two collection areas; the Fortune Line and Question Evaluator, and the Analysers to present the collected information.

The TeachTools database is configured with details about the administrator, the subject under review and the Laboratory and Tutorial tasks for evaluation. The administration process requires a request to become an administrator, and adding details of users to enable them to record their teaching strategies and review the set tasks. More specifically the data contains:

• information about the administrator and the subject under evaluation. This requires specifying the type of teaching staff, (ie. tutors and/or demonstrators).
• input for the Question Evaluator. The input consists of questions to ask tutors, demonstrators and students about the laboratory and tutorial tasks set for evaluation (which can accept either numerical or textual based responses).
• input for the Fortune Line. The input consists of questions to ask tutors and demonstrators about their teachings strategies.
• the names of teaching staff and their corresponding sessions.

The Tutor Fortune Line allows staff to self evaluate their teaching on a weekly basis and to record their teaching strategies used in each lesson. Staff also have the ability to view the previously entered self evaluations and comments, and edit these comments. Figure 1 illustrates the self-grading and evaluation component of the Fortune Line. Staff select one of their assigned classes and a week (from the session identification box) in which to review their performance. Staff record and reflect on their teaching strategies, enter their own comments about the lesson and give their lesson a rating.

The Question Evaluator component of the package is used to collect and store data about the set tutorial and programming exercises, from three sources; students, tutors and demonstrators. The information collected can be used to highlight difficult concepts to teach, the time spent on individual questions and which of the set task are helpful in aiding the students learning and understanding. The application requires each user to identify themselves then to select a task to evaluate. Once identification is complete, the task and questions are displayed for the staff to answer. Figure 2 provides an example of a practical exercise sheet and Figure 3 the corresponding question evaluation sheet.

The two information collection processes described above require separate analysation applications; The Fortune Line Analyser and the Question Analyser.

The Fortune Line Analyser, currently work-in-progress, will be used for viewing multiple fortune lines. Users will be able to select a subject taught in a particular year, resulting in a list of staff types and names so that the user then selects a staff type and one or more names of staff members to graph the self-evaluation ratings and to view comments of prior staff.

The Question Analyser, is used by the administrator to display staffís responses to the tasks. The analyser displays a grid showing which users have responded. There are multiple grids representing the staff types and students. Selection of each grid is achieved through a choice list utility.

Responses can be viewed by individual staff entry or by task title (by week). The Question Analyser can display individual responses to a given task. This will result in a new window appearing which will display the responses in a book like manner, in which the next response is found over the page. The Question Analyser can list all staff responses for a particular task. Results are displayed in a window showing the questions that the administrator set and the responses. For the value based questions, the number of people that selected each value is displayed. For the text-based responses, a button can be clicked which will result in a book window displaying the responses one at a time.

The database system is implemented in mSQL (v2.0) (a miniature version of the SQL standard) developed by Hughes Technology. The database, although implemented without the full functionality of the complete SQL standard, is still a powerful database server, designed specially for use as a web based database.

The Fortune Line and Question Evaluator use the mSQL database server as a backbone to the entire system. The Fortune Line and Question Evaluator components also use dynamically generated HTML forms based on HTML template files. The user completes the forms, which are then sent to a cgi-script that stores the information in the database.

The Question Analyser is an applet written in Java (v1.02), tested on Netscape (v3.0). Due to security restrictions, a textual based cgi mSQL server was developed which performs the query on behalf of the Java applet.

The final database application comprises of many tables for each subject. There are two main tables that all the subjects use; the Subjectsí Table and the Teachersí table. These tables hold information about the types of users and the subjects themselves. Apart from these tables, other tables are maintained for individual subject use.

The subject table stores information about the administrator (entered in the request form). It contains the following fields described below:

The teachers table holds further information about the subject such as; the staff type for a particular subject. Its fields include:

For each staff type and subject, there are three tables that comprise the Fortune Line Application:

• A table to hold a user code and the password for staff members.
• A table to hold information about the sessions taken by each member, such as the user code, session details (ie. the day, the time and room number of a session) and a session identification code.
• A table used by the Fortune Line application to store teaching strategies.

This component of the package uses a further four tables, in conjunction with the first two of the tables described above (in the Fortune Line Application Section). The two tables above are used for staff identification throughout both applications. The tables used by the Question Evaluator application are used to keep records of the actual tasks, when they are taught and the responses given by both staff and students.

TeachTools has been integrated in the review process for first year, Computer Science (semester 2). First year Computer Science, is divided into two components; Data Structures and Algorithms, and Computer Systems. Tutorials and practical classes run on a weekly basis and the tasks given in these settings are set by the lecturer. At the commencement of semester 2, only the Data Structure and Algorithm tasks were ready for insertion into the database for review.

On a weekly basis, tutors and demonstrators reflected on their teaching practice. They used the Fortune Line to rate their lesson and reflect on the teaching in terms of: the context in which it occurred and were required to frame what they did in their lessons, as well as how the students responded in terms of the learning that was and was not occurring. They also provided feedback on the tasks in the Data Structures and Algorithm component of the course, constantly modeling how thinking about the task and how issues involved in learning could make teaching far more informed and purposeful.

After integrating TeachTools in the second semester of the first year course, insights were obtained into its effectiveness by examining the Fortune Lines and the Question Analyser. Improvements to TeachTools were also suggested.

It was not clear from the start how committed staff would be at recording their teaching activities. However, most tutors and demonstrators were self evaluating their teaching, on a weekly basis, using the fortune line facility. Below (Figures 4.1-4.4) are examples of the Fortune Lines from demonstrators teaching Computer Science.

TeachTools provides staff with the opportunity to make improvements to teaching and learning by the quality of their contributions. To date contributions have been very rich in ideas and recorded vignettes characterised by very high levels of energy, commitment and engagement. Below are staff comments that are very reflective about the lesson as well as their own teaching.

Apart from the recording of the more general reflections and analyses of events, another very important type of contribution includes providing feedback about the set tutorial and programming exercises. A snap shot of the Question Analyser (Figure 5) provides evidence of the amount of feedback received, at a particular point in time mid-way through the semester. It indicates which tutors and demonstrators have reviewed the tasks and which tasks they have reviewed. Their comments can be viewed by clicking on a highlighted cell in the grid.

The purpose of the Task Analyser is to detect problems in the curriculum and task design as well as to highlight particular constructs that students found difficult with possible reasons for this. Feedback and results accumulated have been successful at highlighting the difficulties that students face and ways in which tasks can be improved. Two examples below report on the feedback given to the tasks on "Dynamic Memory" and "Sorting and Searching methods".

Rating 8 (Good)

Further Comments It was a good class and I think everyone enjoyed it.

Main Concept Pointers - for about an hour at the start of the class.

Rating 5 (Indifferent)

Further Comments Machines broke down so students did their work on paper. Actually, it didnít go too bad, as I got students writing big programs, starting from small bits then combining the small components.

Main Concept Loops and Arrays in MIPS

• Students did not know when to allocate and deallocate memory
• Students donít seem to know why they should use malloc instead of a static array.

• Possibly design a question that leads students into making mistakes in order to see the consequences if they donít use malloc()
• The sheet should have a series of small questions which causes students to use dynamic memory allocation rather than one large question which requires only one malloc call. I advise to remove part 3, and make smaller questions. Part 2 is good.
• Part 3 should have really been taken out. Although it did require student to use malloc, it was really more of a problem on loops, or arrays, rather than a problem on dynamic memory.

• If learning sorting and searching algorithms were the objective, then this task was quite unsuccessful. Students just copied out the sorting algorithms with little understanding. The most difficult concept was reading from a file.
• A lot of the concepts in the task were lost in that the students werenít expected to write any of the sorting or searching procedures themselves. The marks were exceptional, but I donít know if people really followed what they were doing.
• Most students just copy the code from the lecture notes. They found the routine C functions such as fscanf(), fgets() difficult.

• The prelim was good, but maybe more should have been asked. Less code given, more hand simulations and explanations of how the sorting and searching techniques work.
• Provide code for file input. Do not provide code for searching and sorting. Increase the amount of prelim to include searching a sorted list.

In its first stage of operation several deficiencies in TeachTools were noted. Improvements include adding extra functionality to change staffís sessions and passwords, modifying the questions, making the request form simpler (perhaps multi staged) and refining the overall package to the needs of the administrators.

Teaching in universities, and other environments, can be a difficult job. The job is even more difficult when there is little in the way of formal training for teachers.

TeachTools has been used in semester 2 of 1997, to collect and analyse data relating to front line teaching and the set tasks. Self-reporting in teaching is geared to stimulate others to take the kinds of risks that are routine in others classes. Capturing the teaching experiences of previous tutors and demonstrators and the types of learning problems students encounter can be useful.

The flexibility of the application ensures that with little modification, it can be used in other departments. This would create an extensive database that could be used by teachers from many disciplines to enhance their teaching technique and to improve the tasks, so that they are seen by students as gateways for further explorations on the concepts intended by the lecturer.

The authors wish to thank all the first year tutors and demonstrators for reflecting on their teaching, reporting their strategies in the fortune lines, and reviewing the set tasks. The authors would also like to thank the first year Computer Science lecturers Dr. David Albrecht and Associate Professor Ingrid Zuckerman for allowing their task to be reviewed.

(c) Angela Carbone, Matthew Drago, Ian Mitchell

The author(s) assign to ASCILITE and educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author(s) also grant a non-exclusive licence to ASCILITE to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the ASCILITE 97 conference papers, and for the documents to be published on mirrors on the World Wide Web. Any other usage is prohibited without the express permission of the authors.