Full Paper

Thomas C. Reeves Ph.D.

treeves@coe.uga.edu

University of Georgia

James M. Laffey Ph.D.

cilaffey@showme.missouri.edu

University of Missouri

Mary R. Marlino Ed.D.

marlino@page.ucar.edu

University Consortium for Atmospheric Research

In higher education, many existing applications of technology, such as computer-based tutorials, content and instruction are encoded by specialists such as instructional designers into predefined educational communications intended to transmit knowledge to students. Students are expected to receive these communications passively with occasional artificial interactions to let the computer know they are ready to receive more information. In this approach, students are expected to learn "from" technologies which have been cast in the role of surrogate instructors.

An alternative approach involves using computers and other technologies as "cognitive tools" that students learn "with" in a cognitive partnership. Cognitive tools refer to technologies, tangible or intangible, such as written language, mathematical notation, and computer software, that enhance our cognitive powers during thinking, problem-solving, and learning. Cognitive tools have been around ever since primitive humans used piles of stones, marks on trees, or knots in vines to calculate sums or record events. Something as simple as a grocery list or as complex as calculus can be regarded as a cognitive tool in that each allows us to "off-load" memorization, calculations, or other mental tasks onto "technology." Computers are extremely powerful cognitive tools. When software programs are used as cognitive tools in higher education, students use software to analyze complex problems, solve difficult tasks, access information, interpret and organize their personal knowledge, devise unique solutions, and represent what they have learned to others.

Jonassen and Reeves (1996) summarize the theoretical foundations for using software programs as cognitive tools:

• Cognitive tools are most effective when they are applied within constructivist learning environments.
• Cognitive tools empower learners to design their own representations of knowledge rather than absorbing the representations preconceived by others.
• Cognitive tools can promote the deep reflective thinking that is necessary for meaningful learning.
• Cognitive tools enable mindful, challenging learning rather than the effortless learning promised but rarely realized by other instructional technologies.
• Cognitive tools should be applied to tasks or problems defined by learners with the support of their teachers.
• Cognitive tool use for education should be situated in realistic contexts with results that are personally meaningful for learners.
• Cognitive tools can enable intellectual partnerships in the form of distributed cognitive processing.

This paper describes the use and effects of cognitive tools within an undergraduate engineering course at the U. S. Air Force Academy (USAFA) in the USA. In addition to traditional knowledge and skill objectives, the course focuses on "higher order" outcomes such as "framing and resolving ill-defined problems," "communicating with multiple media," "exhibiting intellectual curiosity," and "developing a rich conceptualization of engineering." This course integrates concepts and skills from all five engineering disciplines taught at USAFA. Learning opportunities within ENGR 110 are situated within the context of a challenging scenario: the establishment of a research station on Mars. Three major tasks are included the course: 1) getting to Mars, 2) constructing a research station on Mars, and 3) operating an energy plant there.

The design of the ENGR 110 course was guided by the evolving literature on situated cognition within authentic learning environments (Brown, Collins, & Duguid, 1989). Herrington (1997) identifies nine characteristics or elements of a situated learning model as:

• authentic context that reflects the way knowledge will be used in real life
• authentic activities
• access to expert performances and modeling of processes
• multiple roles and perspectives
• collaborative construction of knowledge
• coaching and scaffolding
• reflection
• articulation
• authentic assessment (p. 44)

At USAFA, all cadets have their own computers in their rooms with access to many electronic resources such as word-processing, spread-sheets, multimedia presentation software, e-mail, and the World Wide Web (WWW). We viewed these software programs as "cognitive tools" in the sense described above. There were many opportunities to use these cognitive tools within the course. For example, at the end of each task, cadet teams make formal military briefings attended by their peers, instructors, and others from USAFA. Microsoft PowerPoint, a presentation package, was used by the teams to support their briefings. This tool enabled the cadet teams to represent what they had learned to others.

The course also involves "hands-on" projects such as building and flying model rockets. The data collected from the flights is entered into spreadsheets, and the cadets add other data to calculate what kind of rocket and fuel would be needed to support the Mars Mission. An elaborate WWW site was constructed to support cadet access to the wealth of information around the globe about Mars and space travel. In the third task, cadets collaborated in the development of computer models of the power plants they would construct on Mars to support their research stations. The use of these and other cognitive tools was an integral component of the overall situated learning environment.

ENGR 110 was first offered in the Fall Semester of 1995 to 42 freshmen cadets at the U. S. Air Force Academy (USAFA). Cadets participating in this prototype course were selected at random from an entering class of more than 1,000 cadets. Based upon a review of the research literature on cognitive assessment (cf. Merluzzi, Glass, & Genest, 1986) within the context of undergraduate engineering programs, the following assessment and research strategies were utilized: 1) Reflective Judgment Exercises, 2) Self-Assessment Questionnaires, 3) Concept Maps, 4) Focus Groups, 5) E-mail Journals, 6) Observations, and 7) Individual and Group Interviews with Faculty. Thirty-nine sophomores enrolled two sections of ENGR MECH 120, an introductory mechanical engineering courses, served as a control group. A central facet of the study was a quasi-experimental comparison of the cognitive outcomes of the new situated learning course (ENGR 110) and the traditional, instructor-led engineering course (ENGR MECH 120).

Important results were found with respect to the four cognitive outcomes listed above (Framing and Resolving Ill-defined Problems, Communication, Intellectual Curiosity, and Conceptual Understanding), but there is only space in this paper to describe the results with respect to problem-solving. The most important finding was that the cadets enrolled in ENGR 110 increased their ability to frame and resolve ill-defined problems whereas the students enrolled in the control courses did not. Three types of data were collected to assess cadets' ability to frame and resolve ill-defined problems as well as to assess their perceptions of problem-solving as a major focus of the ENGR 110 course:

(1) The Reflective Judgment Exercise (RJE) (King & Kitchener, 1994) provided evidence that freshmen cadets improved their abilities to frame and resolve ill-defined problems as a result of their participation in the ENGR 110 course.

(2) Cadets self-reported (through focus group sessions, end-of-course questionnaires, and e-mail surveys) increased awareness of problem-solving as a focus of the ENGR 110 course as well as improved abilities to frame and resolve ill-defined problems.

(3) During interviews, ENGR 110 faculty reported they had observed cadets engaging in problem-solving during the course.

* Statistically significant pre- to post-test differences p < .00001.

The Reflective Judgment Exercise (RJE) (King & Kitchener, 1994) was administered to all ENGR 110 cadets (freshmen) as well as to a comparison group of ENGR MECH 120 cadets (sophomores) at the beginning and end of the Fall 1995 semester. The pre-test problem involved a "sortie" scenario and the post-test involved a "desert survival" scenario. The reflective judgment exercises were all scored by an independent consultant, using a procedure which kept her blind to the cadets' membership in the 110 or MECH 120 courses. Table 1 presents the pre-test and post-test results for the cadets enrolled in two sections each of ENGR 110 and ENGR MECH 120. On the pre-test, the ENGR MECH 120 sample of sophomores (N=39) in the comparison group achieved a mean score of 2.23, and the ENGR 110 freshmen (N=42) achieved a mean of 2.02. These differences are not statistically significant. Both the ENGR MECH 120 (N=22) and the ENGR 110 (N=35) cadets scored higher on the posttest, but only the ENGR 110 cadets showed statistically significant gains which can be attributed to the experiences of the course. The differences in the ENGR 110 scores are also educationally significant in that they represent a shift from generally "deficient" problem-solving to generally "satisfactory" problem-solving. The score differences between the ENGR 110 and ENGR MECH 120 classes are also statistically significant. The RJE instruments are scored using a 1 to 5 scale in which "1" represents deficient (D) problem-solving, "2" represents D+ or S- scores, "3" represents satisfactory (S) results, "4" represents S+ or E- scores, and "5" represents excellent (E) problem-solving. The ENGR 110 cadets improved their RJE scores by approximately one standard deviation.

Although the improvements in RJE scores for the ENGR 110 students are impressive, there were two limitations in the administration of these tests that should be kept in mind when interpreting the findings. First, there was a loss of subjects from pre- to post-test, especially in the ENGR Mech 120 classes (see Table 1). Second, pre-tests were given as an in-class assignment on the first day of the course and the post-tests were given as an extra credit question on the final exam. Merluzzi, Glass, and Genest (1986) describe other challenges involved in this type of cognitive assessment.

Three focus group sessions were held during the semester with groups of cadets in ENGR 110. Each session was held at the end of one of the three major "Mars Mission" tasks that guided the project-based learning activities in the course. The sessions were recorded and subsequently transcribed. The focus group sessions provide evidence that cadets saw the ENGR 110 course as different from other courses in terms of its emphasis on applying knowledge to solve ill-defined problems and in the requirements placed by faculty on original thinking and constructing their own solutions rather than memorizing the instructor's solution.

At the beginning and end of the semester, cadets in ENGR 110 completed self-assessment surveys. These instruments used a Likert scale with 6 response options ranging from "Strongly Disagree" to "Strongly Agree." On the post-course questionnaire (n=40), 85% of cadets agreed (15% disagreed) with the statement, "My abilities to frame and resolve ill-defined problems have improved as a result of this course." Although this finding in and of itself may not seem especially important, it complements the other data we found through our triangulation approach to assessing whether students improved in their ability to frame and resolve ill-defined problems as a result of their participation in this prototype engineering course.

Following each of the three tasks, cadets used e-mail to respond to a set of questions about the course and the specific project just completed. After the first project, cadets were asked an open-ended question about what they had learned from the project. Eighteen of the 37 cadets (48%) identified that they had learned that engineering problems are ill-defined, complex, and require reasoning, and/or that engineering requires making assumptions, experimentation, and learning from errors. Following the final task, cadets were again asked what they had learned. Twenty-five of the 32 respondents (78%) identified recognizing that problem solving was a key part of engineering as what they had learned. Twelve cadets focused on the complexity of doing real world engineering. Nine recognized that engineering requires going beyond formulas and equations. Four mentioned learning how to apply technical information to solve ill-defined problems. The pattern of evidence in these surveys clearly supports the overall conclusion that students learned that framing and resolving ill-defined problems was a major part of the engineering process.

At several times during the semester and in a group interview near the end of the course, faculty and course observers were asked if they felt ENGR 110 contributed significantly to the ability of cadets to frame and resolve ill-defined problems. There was clear consensus that the presentations made by the cadets three times during the semester provided evidence of effective problem-solving. To make an effective presentation, the cadets had to pull together complex aspects of the problem, decide how to focus their presentation, and respond to questions. Faculty also commented, however, that cadets often failed to consider all aspects of the problems with which they were confronted. For example, the faculty observed that significant factors that would add to the complexity of the solutions proposed by cadets were often over-looked. The consensus of the faculty was that, although cadets had improved in their ability to recognize problems as ill-defined, there was much room for improvement in terms of the actual problem-solving strategies they used.

This improvement in framing and resolving ill-defined problems evidenced by the RJE was also seen systematically in the ENGR 110 activities and the briefings cadets made to report these activities. Overall, we interpret these findings as representing a developmental shift from deficient to satisfactory problem-solving. The major factors in these developments appear to be the ability of the cadets to go beyond the facts given in the written case and their ability to identify assumptions other than those specified in a given problem statement. The improved problem-solving ability indicated by the RJE results corresponds to instructional practices in ENGR 110 which required cadets to think about whether their answer or solution to a problem was sufficient or "best." These results also correspond with evidence we found in the self report data that cadets improved their understanding that engineering problems are complex and that there may not be just one "right" answer to a given problem. It is our belief that these instructional practices would not have been feasible without the use of cognitive tools within the context of a situated learning environment.

It is clear that the nature of educational outcomes at USAFA and many other institutions of higher education is undergoing profound change. There is increased attention to outcomes related to higher order thinking skills, curiosity, creativity, and communication, as well as continued attention to traditional knowledge and skills (Reeves & Okey, 1996). The results of this study indicate that these higher order outcomes can be achieved via the implementation of situated learning environment in which cognitive tools play critical roles. Based on this and other studies, we think we have learned at least three lessons. First. technology is best used as a cognitive tool to learn with rather than as a surrogate teacher. Second, pedagogy and content matter most; technology and media are only vehicles, albeit essential ones. And third, our future efforts to use media and technology in higher education must be guided by much more rigorous research and evaluation than in the past.

This study was conducted by Dr. Laffey of the University of Missouri and Dr. Reeves of The University of Georgia under subcontract with the University of Colorado at Denver and the U. S. Air Force Armstrong Laboratory. Other members of the assessment team included Dr. Mary R. Marlino and Mr. Curtis Hughes from the Center for Educational Excellence at USAFA, Mr. Kevin Oliver, a doctoral student intern at USAFA from The University of Georgia, and the USAFA faculty involved in designing and implementing ENGR 110. The guidance and support of COL. M. L. Smith and his colleagues at USAFA was invaluable in this research.

(c) Thomas C. Reeves, James M. Laffey, Mary R. Marlino

The author(s) assign to ASCILITE and educational and non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author(s) also grant a non-exclusive licence to ASCILITE to publish this document in full on the World Wide Web and on CD-ROM and in printed form with the ASCILITE 97 conference papers, and for the documents to be published on mirrors on the World Wide Web. Any other usage is prohibited without the express permission of the authors.