Assessing who is learning and how

J. Michael Spector
Learning Systems Institute
Florida State University

Most would agree that it is important to assess the progress of learners so as to help learners identify areas in which they might focus attention and to help instructors and designers identify areas in which they might consider refinements in future offerings. Likewise, most would probably agree that within the educational technology community there has been a trend to use new and innovative technologies to support ever more complex learning situations. Assessing the progress of learning with regard to simpler learning tasks using well established technologies is a problem that is well understood. However, when learning involves complex and ill-structured problem solving tasks (e.g., crisis management, environmental planning, engineering design, medical diagnosis, etc.) and when powerful technologies are available to support learning, it becomes a significant challenge to determine how individual learners are doing and whether particular instructional approaches, strategies and technologies are effective.

In this presentation, an assessment methodology relevant to this problem will be presented and discussed. The methodology is based on the notion that how an individual thinks about representative ill-structured problems will reflect relative levels of comprehension and competency. Annotated causal influence diagrams are used to capture problem conceptualizations and used to indicate progressive development of understanding based on previous conceptualizations and those of highly experienced problem solvers. A Web-based tool will be demonstrated along with preliminary results of using this methodology and the tool.