Measuring learning and increasing productivity with the Computer Aided Testing System

The Computer Aided Testing System (CATS) is a software package designed for interactive and classroom monitoring of student learning at any level for any subject. Tests are efficiently generated and marked. Diagnostics include the production of concept maps, learning curves and National Profiles reports. It is a powerful classroom management tool for teachers.

The Computer Aided Testing System (CATS) is a software package designed for interactive and classroom monitoring of student learning at any level and for any subject. This classroom management tool offers:

• rapid production and marking of tests.
• extensive diagnostics and reports.
• reports indicating achievement and progress in the National Profiles.
• handling of multiple classes and subjects.
• flexible teacher control over the assessment process.
• membership of a question item bank sharing scheme.

• improved classroom management efficiency.
• reduced stress for teachers.
• detailed and extensive feedback to students, parents and administration.
• improvement in student motivation and interest.

• students should not be disadvantaged in comparison with non-computer based testing;
• full use should be made of the computer based medium; and
• instructor control over the testing environment should be maximised (Ring, 1993).

• the user interface should enable the program to be easy to learn and use.

The problem

In today's schools there is a great diversity of teacher methodologies and availability of resources. In some situations there are competent teachers using sophisticated information systems with highly motivated students while there are other teachers struggling with poor student behaviour and inadequate resources.

Teachers are expected to know the current level of achievement of each student in comparison to the course objectives, what may be hindrances to learning and if the students are actually progressing.

Students on the other hand are more highly motivated when they know in detail how they measure against the objectives, are given rapid feedback regarding attempts at learning, are given opportunity to improve, and are assured that they are actually progressing.

With the recent introduction of the National Profiles in Australia, teachers have the opportunity to comprehensively map the progress of students. It would be of considerable advantage to teachers to be able to have some assistance in making judgements about students' levels of achievement and their progression through the profiles. It would be even more advantageous if this could be automatically achieved during the normal classroom assessment process and results be made available immediately on demand.

To achieve this, much assessment must occur, be recorded and analysed. Teacher time involved may be extensive.

The analysis of assessment items and the subsequent transformation of this into effective diagnostic decision making are most often the first tasks to be abandoned or shortened under the pressure of a normal classroom teaching load.

The CATS program has been developed to meet these needs.

Features of the Computer Aided Testing System

1. Allows teachers to produce tests quickly. For example, a test of 20 questions is easily selected from an item bank and printed in 10 minutes. A filter on the item bank allows selection of questions by concepts or other classification.

2. A balanced set of questions can be achieved by considering and ranking them by knowledge, comprehension or higher order classification, degree of difficulty, and discrimination factor.

3. Tests are classified as interactive, pre-tests, diagnostic or assessment tests, making it possible to quickly assess the current level of student achievement and the progress they are making.

4. Hard copy class tests are can be rapidly marked using the easy data input facility. For example, a class of 30 students with a test of 20 questions takes about 10 minutes to mark.

5. Extensive analyses of results are immediately available for:

   • individual students.
   • the whole class.
   • the questions.

   Question analysis indicates validity of the questions.

6. The results of the analysis include:

   • individual percent.
   • individual responses.
   • class average.
   • class scores distribution graph.
   • scores for knowledge, comprehension and higher order graph.
   • scores for concepts.
   • concept maps for correct and incorrect responses.

   This information allows teachers to rapidly diagnose strengths and weaknesses in the class (or individuals) and make effective judgements for revision, reteaching or course evaluation.

7. Learning curves and learning factors are rapidly generated. This information indicates if students are progressing in their knowledge, their understanding and their development of thinking skills.

   The learning curve is the best fit line (least squares regression) of the average concept achievement over time. It is generated by considering the aggregate of successful encounters with each concept over the period of recorded testing. It is not merely an analysis of test scores.

   The learning factor is defined as the slope of the learning curve. A factor of 0 indicates that no learning has taken place, +1 or above indicates maximum learning.

   

   A learning curve indicating good progress.

8. Concept maps are rapidly produced for each test and show the ranked by frequency linkage of concepts in the test items. This gives a visual representation of the content of a test. When used in conjunction with test results, it indicates strengths or weakness in individual or whole class learning. Concept maps are included in student result reports.
   

   Concept map of a test on legumes.

9. An editor allows questions to be written or modified by the teacher.

10. Diagrams or pictures can be viewed with the test items.

11. Class lists of students are stored. Students have individual passwords to allow access to their own files during interactive testing. This allows control of how students use the interactive system. For example a student may be working on a set of sequential tests and return the next day to the same place, or the teacher may designate that a student have tests of randomly chosen questions.

12. In interactive mode, student tests are immediately marked and a range of diagnostics is available.

13. A report may be generated by the teacher for a class or individual at any time. This may be used to assist report writing or in interviews with students or parents.

14. A graphical report of achievement of National Profiles gives a good indicator of the level and progress individual students are making and the overall position of the class. This is accumulated as a normal part of the testing process from those test items that are linked to the profiles.

A typical classroom scenario

The student viewpoint:

Experience has shown that students enjoy doing the tests, particularly those with questions based on diagrams and pictures. Rapid feedback is important. During lesson time students use the computers in groups of two or three and work through the formative interactive tests set for them. Much discussion ensues as they debate the relative merits of the choices presented. Answers are not given but on completion of the test they are presented with a range of diagnostics which assists them to deduce which questions are giving difficulties. They may repeat the test and are only allowed to proceed with the next test when they obtain a perfect score for the current test. There is great enthusiasm when the learning curve shows improvement.

As an assessment test approaches, the students are aware of the results they may expect and know the areas they need to concentrate on to achieve best results. The teacher is in demand to explain specific weaknesses.

Weaker students are enthusiastic to progress because they know that the teacher is concerned about an improvement in their learning curve rather than the actual score in tests. More able students are motivated to keep the learning curve increasing.

The teacher viewpoint:

For example, the pre-test consists of 30 questions and covers most identified concepts in the objectives, It has knowledge, comprehension and higher order questions with a range of difficulty and discrimination factors. At least some questions are included which are linked to National Profiles. It takes about 20 minutes to prepare and print out ready for photocopying. The summary is perused and adjustments made to provide a balanced test.

The test is given in the first lesson of the new area of learning and students mark their answers on the prepared answer sheet. The test is completed quickly, the teacher sets a small task for the class, types the responses into the computer and advises students of the results. Individual result sheets are printed out before the next lesson for each student showing marks, percent, responses and a concept map showing the strengths. Students who are absent sit the test interactively in the next lesson and results are immediately available.

Diagnostic tests are sometimes administered in the same way and sometimes taken interactively. When students are progressing at different rates or on different topics then interactive tests are set to meet their needs and are taken when ready. At the end of the day the teacher scans the student files to gain a knowledge of attempts students have made and how they have achieved. Some diagnostic tests have been used to evaluate the effectiveness of learning tasks set for homework. A class report produces a concept map for the whole class showing the area of maximum difficulty . With this, the teacher plans a short targeted re-teaching of the concepts least understood by most students in the class.

In this class, the measurement of learning is an important factor of the assessment process. Students of all abilities who see specific evidence that they are progressing well are encouraged to apply themselves more diligently to their studies.

At the end of the learning unit, an analysis of the National Profiles shows that students have made progress in their total learning against this standard although no specific testing has been carried out to achieve this.

Conclusion

References

ACER Newsletter. (1993). Towards National Agreement on Learning Targets. (76) .

Neill, N. T. (1993). Computer based Testing with Question Mark Professional. Computer Education, (74) 23-26.

Ring, J. (1993). Characteristics of state of the art computer administered testing systems. Australian Educational Computing, 8(2) 25-28.

Author: Barry Peek, Department of Education and Children's Services, South Australia Please cite as: Peek, B. (1994). Measuring learning and increasing productivity with the Computer Aided Testing System. In J. Steele and J. G. Hedberg (eds), Learning Environment Technology: Selected papers from LETA 94, 234-237. Canberra: AJET Publications. http://www.aset.org.au/confs/edtech94/mp/peek.html