A case study of a course based on student creation of learning objects

Terry Anderson and Norine Wark

Athabasca University

#320 10030 107 St

Edmonton AB Canada  T5J3E4

Ph 780 497-3421

Fax 780 497-3416

terrya@athabascau.ca 

norinew@athabascau.ca

A common report from anecdotal writing over many generations of educators is that it is the teacher who usually learns the most during the process of gathering content materials, designing, teaching and evaluating student performance. In this project we address this issue by developing an innovative instructional design in which collaborative groups of students working at distance create, share and assess learning content (in the form of learning objects) with their peers through online learning portals. The results of this process are assessed via surveys, discussions, reflective essays and peer evaluations. We conclude that instructional models based upon student construction of content and orchestration of learning activities can reduce instructor workload, provide opportunity for students to acquire new skills while increasing their subject content knowledge, and create a lasting legacy of re-usable learning objects.

How can today's educators successfully mesh the education and training demands of a rapidly changing global knowledge society, the socioeconomic constraints of students, institutes, and governments and technocentric, consumer orientated student body? One answer lies in the creation of innovative, flexible instructional course designs aimed at creating active learning communities in which the students take on the major roles of constructing, sharing and teaching the course content. In this paper we detail such a design and provide descriptive evaluation data obtained from surveys of students' perception, reflective essays and our own perceptions of the quality and costs of developing and teaching a graduate course using this design.

This action research project addresses the research question - what are the social, pedagogical, economic and communication technology factors that are most critical for satisfaction, high perceptions of learning and cost effective development to students engaged in collaborative, project based learning activities in a graduate level educational course?

The movement from the industrial to the information age challenges traditional institutes and the pedagogies that they uphold. Traditional institutes produce learners who "...are not asked to take responsibility for their own learning - they do not set learning goals, ask questions to direct learning activities, assess their learning strategies and approaches, or reflect on what they have learned" (Grabinger and Dunlap, 2001: 1). According to the American National Science Foundation, these institutes produce workers who 'go out into the workforce ill-prepared to solve real problems in a cooperative way, lacking the skills and motivation to continue learning" (p. iii). This lack of involvement in the design and implementation of their own learning results in low levels of 'metacognition' or the skill of learning how to monitor their own thinking and learning processes, and the ability to transfer knowledge from one situation to the next (Grabinger & Dunlap, 2001: 2). Global economy demands a new kind of worker - one that is flexible, autonomous and group oriented, possesses critical thinking skills, can solve a range of problems, is capable of multitasking and is committed to lifelong learning. (Keegan, 1996; Collis and Moonen, 2001, Grabinger and Dunlap, 2001; Springer, Stanne and Donovan, 1997.)

Educators need a pedagogy, or instructional theory, upon which to construct their courses (Seels, 1997). This provides a foundation from which all aspects of course design evolve. Like many others developing collaborative learning models for online delivery (Duffy & Kirkley, 2004; Campbell, 2003), the MDE 663 course design is based on a constructivist pedagogy. "Constructivists argue that knowledge is not so much discovered, or transmitted intact from one person to another, as it is created or 'constructed' by individuals attempting to bring meaning and coherence to new information and to integrate this knowledge with their previous experience." (Rourke and Anderson, 2001: 2.) Inherent in this definition is the idea that learners need to interact with others by sharing, debating and discussing ideas "...to integrate and elaborate knowledge in ways that facilitate higher-order learning." (p. 3.)

A core theme to a constructivist instructional theory is 'learner-centeredness', meaning that courses are designed with learner attributes and choice in mind. Collis and Moonen, 2001, propose a 'flexibility and activity' framework in which they define traditional, existing and future instructional designs by placing them in one of four 'Quadrants' based largely upon the degree of learner-centeredness. Traditional instructional designs of Quadrant I are characterized as being less flexible (i.e. having less student choice regarding time, content, instructional activities, resources, delivery and logistics) and having activities that primarily center around independent knowledge acquisition. Quadrant II course designs are similarly less flexible: student participatory activities exist, but are primarily structured by instructors. More student choice is afforded in Quadrant III, but the main activity usually remains focused on independent knowledge acquisition. Quadrant IV exemplifies ideal or future instructional designs, which are flexible and are based on participatory or primarily contribution-oriented activities.

Other constructivist authors build their instructional designs around a model of active student learning as well. For example, Sfard (1998) describes a 'participation-oriented' model. Dopper and Dijkman (1997) and Simons (1999) present theirs as 'action learning'. Kearsley and Shneiderman (1998) capture active learning via their 'engagement theory'. Collis and Moonen, 2001, define their model in terms of 'the contributing student'. Grabinger and Dunlap, 2001, propose the concept of 'Rich Environments for Active Learning' (or 'REALs'), based on 'intentional learning'.

Clearly, constructivist designs maximize the amount of active learning in which students are engaged. How can educators "engage learners in dynamic, authentic learning activities that increase their control and responsibility over the learning process while they learn problem-solving and collaborative skills and content" (Grabinger and Dunlap, 2001:2)? Grabinger and Dunlap's REALs (2001: 2) offer 5 instructional strategies that they believe generate such learners:

Once educators have developed instructional strategies that revolve around active student learning, they must decide what content and media resources to use. Contrary to normal practice in which a major task of the instructor is to create or select content for study, Collis and Moonen, 2001, argue that there should be little, if any, instructor-developed content included in courses based on an active (or 'participatory') student learning model. Most content is researched and/or generated, synthesized, presented, discussed and evaluated by the students themselves. Instead of focusing on content, the instructor works to develop communication strategies that enable students to find, present and share information and thus accumulate knowledge in an effective manner. Therefore, attention must also be paid to communication and collaborative workspace technology choices.

Putting all of these variables together creates a complex, dynamic learning environment where instructors are no longer patriarchs (Springer, Stanne and Donovan, 1997), but mentors and active learning role models. Students are no longer passive recipients of knowledge, but contributing members of a learning community. Net-based technologies offer a myriad of communication and collaboration options designed to support knowledge building communities, while allowing participants to shift time and space constraints. Is it possible to use these technologies and the pedagogical insights above to create cost and learning effective courses?

This study examines data from the experience and output of 17 Athabasca University (AU) graduate students enrolled in the Masters of Distance Education MDE 663: Emerging issues in educational technology during the 2003 fall semester. The 13-week course was broken into three sections – a series of introductory lecture/discussions presented by the instructor and guest speakers (6 weeks), weekly 'Open Houses' at the team portal sites (4 weeks) and an assortment of reflection and concluding activities (3 weeks). The learning management system (LMS) used was WebCT, version 3.1. Two-hour synchronous audio-graphic sessions supported by the net-based Elluminate vClass (or virtual classroom) were offered once a week. The main student assignment during the term was the creation of a REAL or 'learning portal' that focused on one emerging issue related to educational technology. The term ‘portal’ has a number of definitions and connotations, many associated with net based marketing. We use the term to mean an entry point to a multidimensional educational site used for exploration, information and learning activities. The students developed these portals as multi-level learning objects to share and teach their colleagues during a one-week ‘Open House’. The students were given opportunity to use any technologies they wished to support small team (i.e. 4 or 5 people) collaboration, create portals and host their 'Open House'. Technologies used by portal teams included Blogs, Learning Management Systems, Content Management Systems, custom web sites, quiz and game generators, and asynchronous and synchronous chats and forums.

Then topics chosen for the 2003 version of the course were chosen by the instructor, based upon their currency, potential educational impact and high interest. They were:

The assessment of the course consisted of both a collaborative (all team members received the same grade) and individual assessment. An instructor evaluation of the team portal was used to calculate 40% of the student grade. Individual assignments include answering a comprehensive question presented by other portal teams, evaluating other portals and reflecting on the individual and group learning process.

This action research project used case study methodology involving five major instruments to gain an understanding of the learners' interpretation and evaluation of the process and technologies used in the course. These instruments included:

Each of these instruments is described in further detail below.

The group interest and skills survey was used at the beginning of the semester to help the instructor place students into portal teams. Students were asked to assess personal interest and expertise needed to create each portal project. The results of this survey were used to create portal teams and to compare students' self-perceptions of topic interest and individual group role skills at the beginning of the semester to their term end reflection papers, questionnaire and instructor perceptions.

Each student was required to submit a 1,000-word reflection essay at the conclusion of the course. The data from the reflection essay was synthesized into four broad areas of discussion: individual learning, group learning, technology issues and efficacy of the course pedagogy. The results were used to examine growth in student learning, metacognition, group dynamics, technological concerns and issues, and perceptions of the course's design. These results were triangulated with the other research tools to ensure a greater degree of accuracy in interpretation.

Individual and Group Evaluation Form

The individual and group evaluation form was part of the reflection assignment. It was broken into three broad areas: fulfilling individual roles (using the role titles and descriptions established by the instructor at the beginning of the term), timely completion of tasks and general group functioning. Each student was asked to evaluate their own performance, as well as that of the other members in their portal group. This form assigned a Likert scale of 1 - 10, with 1 showing little or no evidence of contribution, 5 indicating average contribution and 10 demonstrating exceptional contribution. The results were used to compare individual reflection essays to Likert scale scores assigned in the individual/group evaluation form to verify consistency between the two documents in terms of individual, group, technological and pedagogical issues. These findings were used to compare student perceptions and skills over the course of the term to ensure accuracy in research interpretations.

Term End Questionnaire

Students voluntarily completed the term end online questionnaire (posted at: http://survey.icaap.org/html/mde663.htm with final results at: http://survey.icaap.org/html/results.htm), after the course was finished and they had been informed of their grades in order to meet ethics review guidelines for informed, non-coercive consent. It was broken into 5 main sections: background information; social, pedagogical and technological factors; and a conclusion. This questionnaire consisted of Likert scale questions and anecdotal comments. The results were used to record and compare student perceptions and evidence of student learning, metacognition, group dynamics, technology choices and course pedagogy throughout the term.

Instructor/Researcher Insights

The instructor and teaching assistant gained many insights as the course evolved from an idea to its first implementation. Student development, difficulties and recommendations enabled the instructor to not only evaluate evidence of student understanding and performance, but also the structure of the course itself. Reflection and metacognition at the instructor level was enhanced by subsequent research findings. The results of these discoveries were used to re-evaluate and revise various components of the course's design, and to subjectively assess the value of the underlying pedagogy.

Results

Group Interest and Skills Survey

The group interest and skills survey allowed the instructor to see which topic of four chosen by the instructor was of most interest to students. Students also indicated in which of the five roles (i.e. project manager, webmaster, instructional designer, graphics developer and resource gatherer) they had greatest interest and skill. A number of students said they had careers that required them to play one or more of these roles in daily life. Interestingly, some students choose roles and topics with which they had considerable expertise and background knowledge already; others specifically choose unfamiliar territory to develop new knowledge and skills.

Reflection Essay

Analysis of the student reflection essays provided a wealth of data related to their experience of the course. Suggested guidelines for the essay requested students to comment on personal learning, technology issues and various collaborative and individual activities in which they participated.

Most students reported positively on their personal learning during the course. One commented: "At one point, I wrote to my colleagues stating that there is no one who has learned more than me in this course and I mean it. I learned so much in this time period that my learning curve was almost vertical." For many the content area was new and exciting, though a few reported problems in following all discussion and activities due to their limited background in the subject area.

Group process issues revolved around shared purposes, goals and outcomes, co-operative behaviour/interdependence (e.g. trust, planning, support), and individual accountability. As expected, most students noted that group cohesion developed through the process of portal creation. The equitable sharing of workload created some problems but also increased group empathy for absent members or those with personal constraints on their capacity to participate. Only one student felt that absences and unequal work distribution eroded group cohesion. Three students reported positive group collaboration efforts, nine noted both advantages and disadvantages of group process, and seven reported group capitalization on each others' experiences and strengths, expressing appreciation of each other's differences. Individual accountability was a big issue to some. The portals were generally completed by dividing tasks between different members and assembling finished components into an integrative whole (jigsaw style). Since this model emphasized individual accountability, it allowed the groups to progress without need to negotiate all content. However, this strategy likely limited the in-depth discussion and critique that results with more collaborative development models.

All but one of the students spoke positively about the course pedagogy, with one reporting,

"The learning design proved effective in terms of team members' construction of knowledge and project deliverables for several reasons. First, course expectations and assessments were clearly defined (sense of community was a valued outcome). Second, user-friendly tools (communication, Web design) and hardware were available. Third, mature, self-directed, experienced MDE program graduate students performed an authentic exercise. Fourth, the team participated fully."

The unhappy student felt that the content was too advanced and suggested that readings should be available before the course began. She also reported serious communication technology problems. Another student suggested that instructor-imposed deadlines for partial project completion might keep groups more focused, and therefore, more organized and cohesive.

Individual and Group Evaluation Survey

All students submitted an individual and group evaluation survey with their reflection essay. The results indicated that two of the groups had assigned roles as suggested by the literature provided by the instructor and used in creating the portal groups from the interest and skills survey at the beginning of the term. Another group 'self-organized', allowing members to change roles based on time, expertise and available resources because they believed that this facilitated group consensus and coherence. The final group developed new roles for team members.

As in many course with heterogeneous groups of students, their perception of value of the course was not homogeneous. Two students felt the course was dry, and that the content too difficult to absorb or assimilate. They had technological and time management problems, which they felt were instructional design issues. They were resentful of the time they had to spend online, and were particularly unhappy with the weekly synchronous sessions. One reported family time conflicts. Other group members saw things differently, pointing out these individuals' lack of attendance at meetings, inattention to group emails, and irresponsibility in getting tasks completed on time and in a scholarly manner.

These discrepancies point to the problem of using self-evaluations for assigning performance marks for the course. Many high performance and successful students are suspicious of design formats that force them to rely on other students who may not share their commitment, organization style, aptitudes or aspirations for high marks. These students are often reluctant to volunteer for collaborative work, however, such collaboration is inherent in much work required in the distributed, global workplaces common to many professions. Thus, we feel that even forced participation is a valid component of a high quality master’s degree experience

Term End Questionnaire

Thirteen of the seventeen MDE 663 students (or 76%) responded anonymously to the term end questionnaire. The questionnaire results are available at http://survey.icaap.org/html/results.htm. The questionnaire addressed a number of social participation factors in the course. The students indicated a slightly greater sense of belonging to the MDE 663 class than most other MDE classes. Most disagreed that it would have been easier to decide portal group roles had they been in a face-to-face setting. They were slightly pleased that the instructor picked groups and most felt that the workload had been shared equitably amongst team members. Although students expressed satisfaction with the group process, few actively planned on taking courses or communicating further with other portal group members as a result of friendships made during the process.

Perhaps not surprisingly (given the overwhelming predominance of individual learning and assessment in their current and past educational programs) students generally felt that they learned slightly better and achieved slightly better grades when working independently. They had very mixed views on questions comparing the workloads of independent study, and small and large group work. This probably reflects the variation in amount of group work performed based upon individual skill level, interest and commitment.

When asked to compare this course to others in the program, students indicated that the workload was slightly greater, but that the amount of learning was about the same. Perhaps it is not surprising that given the novelty and challenges of communicating at a distance, 62% of students found the course more stressful while only 8% found it less stressful than other courses in the program. Despite this, 62% indicated that they would take another course offered using this collaborative framework, whereas only 16% would not. It is important to note that these students are studying the theory and practice of distance education, so their level of expertise and interest in novel pedagogies and technologies is likely to be greater than that of students from other disciplines.

The questionnaire's next section looked at communication and information technologies used within portal groups. The technologies perceived as most useful for communications were the synchronous audiographic technologies of vClass (used for both full class and informal team meetings) and the ubiquitous asynchronous email. Here is one such comment,

And

In sharp contrast, here is one respondent's view:

1 = never 2 = rarely 3 = occasionally 4 = frequently 5 = extensively

Table 1: Section DIII a - Frequency of communication tools use

The course contained a variety of learning activities and features. These are summarized below, along with the perception of value attributed to them by students.

1=not useful, 2=rarely useful, 3=occasionally useful, 4=often useful, 5=very useful

Table 2: Section DIII b - Value of communication and learning tools

The questionnaire concluded with an opportunity to make comments about any aspect of the course and/or research project. Comments by seven respondents showed a very positive feeling about the learning quality and relationship between students and the instructor, and within the portal teams. A member of one team stated,

This respondent's sentiment is echoed in another's statement as well,

Three students confessed that they liked learning independently more than they did in groups, although one thought they had learned a lot in this class. One student said,

Discussion

One means of evaluating a course design is to assess its construct validity by comparing it to the theoretical model upon which the design is based. Grabinger and Dunlap, 2001, offer a model for such an evaluation based upon their REALs. This model suggests students should practice taking ownership over their own learning. "To be intentional learners, students must identify learning deficiencies and strengths, make and implement plans, develop metacognitive awareness, and revise those plans and actions based on that awareness" (p. 3). The MDE 663 term begins with the interest/skills survey, a tool meant not only to help develop balanced groups, but to help students evaluate their interests and abilities. Course and project goals are presented on the WebCT site, as well as in the first and last synchronous sessions. The group project's aim is to produce a portal on an emerging educational technology issue where a WWW-based audience of educators could deepen their understanding of new educational technology applications. Students evaluate each other’s sites in order to develop their critical thinking skills and to insure that they spend time learning the content by participating in activities presented by the other teams. At the end of the term they write a reflection paper, detailing their awareness of individual and group learning, technology issues and course pedagogy. The instructor does not use the self and group evaluation for student assessment purposes. Rather the self-assessment is used to enhance metacognition - forcing students to assess their own performance, that of other individuals and the team as whole. A danger in assigning course credit for this evaluation is the challenge students have in reliably rating each other and themselves in the absence of clear criteria and an assessment rubric.

Students report mixed feelings on benefits of participating in group activities. Overall, they characterize themselves as learning slightly better individually than in small groups. Sixty percent of their grades come from individual assignment submissions, but all assignments are generated from group portal activities. Only one of four groups appears to exemplify a strong, positive learning community throughout the course. The other three indicate periods of time when communication breaks down, resulting in frustration. The capacity for teams to empower leaders to emerge and initiate activities to resolve communications breakdowns seems critical to effective group process.

Constructivist and situated knowledge theorists argue that students need to work on realistic problems that hold relevance to their daily lives. Such problems help them to develop richer, deeper knowledge structures that are more likely to be transferred to new situations. "Ill structured, complex problems require a team approach that provides natural opportunities for learners to seek out information, test and refine their ideas, and help each other understand the context" (Grabinger and Dunlap, 2001: 3). This encourages interaction, collaboration and social negotiation of meaning. MDE 663 is based on four complex emerging issues that are likely to have major impact on the development of distance and distributed learning systems. The course design expects students to use whatever project-based, collaborative and presentation technologies they can to produce a portal that is cohesive, user-friendly, interactive and results in effective learning. The reader is invited to assess the success of the portals in achieving these goals by visiting the portals at http://cde.athabascau.ca.

'Dynamic, generative learning activities' (Grabinger and Dunlap, 2001) require projects that are based in an authentic context, use multimedia, allow for experimentation and testing and encourage research. MDE 663's unique structure enables each term's content to change as newly emerging educational technology issues arise. The collaborative work helps students develop social skills in online environments. Grabinger and Dunlap's final requirement is reflection and self-assessment. The MDE 663 design requires students to individually reflect on course content, processes, tools and resources. Thus, the real assessors of learning are the students, because they evaluate each other's projects as well as reflect on their own group and individual progress.

Collis and Moonen (2001:119) list a number of criteria and suggestions for courses that exemplify the pedagogical principles of a Quadrant IV course. Comparing this list to the MDE 663 design in the table below reveals that the MDE 663 course design virtually mirrors their ideal course structure. The only exception is that the WebCT site does not currently sport a 'frequently asked questions' section.

Table 3: Collis and Moonen's , 2001, Ideal WWW-based course site vs. MDE 663

The experience of designing, building, facilitating, and evaluating this course produces a host of tentative conclusions, lessons learned, recommendations and suggestions for further developmental research.

From the review of the literature, and now confirmed by our own experience in this course, we conclude that an instructional design based on student construction and sharing of knowledge in a web based portal is perceived to be a valued and worthwhile learning experience. Students report this design provides as much or more learning, challenge and skill acquisition as other courses in the program do. This is consistent with findings by Springer, Stanne and Donovan, 1997, who report that, "the main effect of small-group learning on achievement, persistence, and attitudes among undergraduates in Science, Mathematics, Engineering and Technology (SMET) was significant and positive'. They base this conclusion on a metanalysis of 383 research studies of collaborative projects engaging SMET students from 1980 to 1996. Given this positive assessment of learning effectiveness, we can begin to look at learning efficiency by examining the cost of the course.

Unlike many other web-based designs for relatively small numbers of learners, this design is cost effective based on three major features. First, the time to create the course is modest. The use of a full-featured LMS provides a platform for relatively easy display of content, facilitation of discourse, scheduling of activities and tracking of student progress. This design can easily be re-used during subsequent courses. Changing the design to a different set of issues in this or another discipline requires a modest amount of preparation. This includes development of:

Secondly, the course has potential to save development costs in other courses because the legacy portals are re-usable learning objects. On the final week of class, the students worked in teams to 'metatag' and enter their portals into three major learning portals (Adlib, CAREO and Merlot), thus facilitating their re-use not only within our program, but also by teachers and students globally. The portals can be accessed and enhanced in subsequent courses, where these subjects may be refined and explored in greater detail. They can also be used in courses for which the portal content is only of minor use by informal learners investigating important emerging topics on their own volition, and as models and exemplars for subsequent courses' teams using a similar instructional design.

Thirdly, the course design, with its emphasis on student management of learning activities during the final weeks of the course, reduces instructor time commitments. The instructor’s role during the final six weeks of the course becomes one of a supportive class participant, allowing reallocation of time to assessment activities which increase (as is common in most formal education courses) near the end of the course.

There are also a number of important lessons learned that we will use in subsequent courses. We must remain cognizant of the vast differences in student backgrounds and experiences in large public programs. Team support compensates for this to some degree, but background information must be available to scaffold learners with the necessary minimal subject matter, collaboration and technical resources needed to support exploratory learning. The MDE 663 assessment scheme is heavily weighted towards the term's end. This provides too little opportunity for formative assessment earlier in the course, with subsequent capacity for remediation and targeted support by the instructor. The submission of a team based portal plan and perhaps a subject matter assignment earlier in the course would provide this capability.

For many students working collaboratively is challenging, and the logistics of mediated communication increase the risk when the course is offered at a distance. Although our data indicate that students do not feel that a face-to-face gathering is critical for success of this design, there is little doubt that formal and informal opportunities for both real time and asynchronous interaction are important. Students report feeling slightly more stressed by MDE 663's collaborative structure than they do most MDE courses. Allan and Lawless (2003) report similar findings, saying that the students they studied want to meet the perceived expectations of other group members, have time conflicts and/or spend more time on work than anticipated and have to wait on others' work before they can do theirs. Most MDE 663 groups indicated communication or collaboration breakdown at some point during the development of their projects.

New research questions spring from this course and subsequent investigation project. For example, would the inclusion of more background content, group dynamics and process skills, and technical trouble-shooting resources at the beginning of the course help develop more cohesive groups? How can changes in the timing and activities of course assignments affect group organization and instructor opportunities for remediation? What happens if the instructor imposes more control over the structure of groups by, for example, asking teams to choose a leader and/or submit plans for the project soon after they begin working on it? Do such impositions reduce the amount of flexible learning to a point where the course no longer fits Collis and Moonen's, 2001, Quadrant IV or Grabinger and Dunlap's, 2001, REALs? Or should we offer this scaffolding until students are more experienced with learning by this pedagogy? Would inclusion of a 'frequently asked questions' section on the LMS site improve student autonomy and reduce instructor response time enough to warrant the time needed to create and update it? How much would any of these changes positively alter student perceptions of learning efficacy?

This research project shows that MDE 663 emulates the ideals of a constructivist paradigm, and typifies Collis and Moonen's, 2001, futuristic flexibility-activity Quadrant IV framework. MDE 663 students come from Quadrant I course backgrounds, modeling the skills and profiles of learners who are masters at knowledge acquisition. Yet, despite the fact that few report having worked in collaborative online groups, they rate small group learning almost as high as they do independent learning. This project also shows that, with little instructor direction or interference, most students at this level demonstrate the necessary individual and group skills of an emerging Quadrant IV learning community.

Ngeow, K. Y. (1998). Enhancing student thinking through collaborative learning.