Associate Professor Elizabeth Murphy, Ph.D.
Memorial University
emurphy@mun.ca

Tami Jerome
Cape Lazo Middle School
tjerome@sd71.bc.ca

Students’ critical analysis of their participation in and contribution to course-based online discussions represents a means to assess their performance at the end of the discussion. It can also serve as a means to direct participation and thinking and to improve their performance in the discussion. This paper presents the case of a graduate level online course in which criteria are provided to students to direct and self assess their participation in and contribution to a text-based asynchronous discussion. One student’s analysis is presented outlining the quantity of her participation in terms of number of postings and the quality in terms of the types of claims and grounding provided. The paper concludes with some implications for practice and research.

From an instructor’s perspective, the benefits of online discussions may create an enthusiasm for their incorporation into courses and no doubt help contribute to an increase in their use. However, incorporation of online discussions into courses most often requires of instructors that they assess students’ contribution to this discussion. In many cases, such assessment can represent an onerous task particularly when class sizes are large and individual discussion contributions are numerous. For example, if each student in a class of forty makes only one posting per week in a twelve-week course, the total number of postings to be evaluated at the end of the course will be just short of 500.

The rationale for student involvement in the process of assessing participation in an online discussion relates to a need to divert some of the responsibilities of this onerous task away from the instructor and towards the student. More importantly, however, the rationale derives from the premise that students can improve their performance in online discussion forums by reflecting on, analyzing and being critical of how they participate. This process can serve a dual purpose of assessment on one hand, as well as a means of directing participation and thinking on the other.

Surprisingly, little has been written on students' self-assessment of participation in online discussion although a number of studies have addressed issues related to student participation and satisfaction (see Rovai & Barnum, 2003; Ruberg, More & Taylor, 1996; Sain & Brigham, 2003; Williams & Pury, 2002). Likewise, numerous studies have focused on content analysis of online discussions to determine if intended outcomes such as critical thinking, knowledge construction or argumentation were achieved (see Aviv, 2001; Aviv, Erlich, Ravid & Geva, 2003; Campos, 2004; Clulow & Brace-Govan, 2001; Fahy et al. 2000). Rovai (2000; 2003) has shown some interest in the issue of students' assessment of their own participation using a three-point grading scale to rate and value certain contributions over others. Salmon (2000) takes a different approach by requiring of students that they use the contents of their postings to illustrate their writing and that they keep a reflective diary of their contributions.

This paper adds to the yet small body of literature that focuses on investigating ways students might be involved in grading or assessing their contributions to an online discussion. A case of an online graduate course is presented in which students are provided with criteria according to which they must analyze and assess their participation in and contribution to the course’s discussion. The paper presents one illustrative example of a student’s analysis of her contribution to a discussion. The paper begins with a description of the purpose and requirements for the discussion as they were communicated to students. Implications for learning and research are presented.

The discussion was part of an online, 12-week course entitledIssues and Trends in Educational Computing offered in the context of a Masters’ program in Education. The course which was comprised of four modules was housed in the WebCT^TM learning management system. The purpose of the discussion which counted for 20% of the students’ total course grade was to construct and share knowledge. To achieve this goal, students were presented with specific requirements regarding participation. Each student had to make a minimum of two postings for each of the course’s four modules except in module one where they were required to make three postings. This was the minimum requirement for participation, however, students could post more than this amount. As a result, the final number of postings for the four modules excluding those made by the instructor was 420 or an average of 28 postings per student. Students were expected to limit the length of their postings to approximately 100 words. This requirement was designed to discourage students from posting short essays and to promote dia-loguing and multi-loguing instead of mono-loguing. In this regard, these requirements were meant to promote active sharing of knowledge between students.

In relation to the types of postings, for each module, students were required to make at least one original posting and one reply or follow-up to a posting made by another student. Original postings were important in terms of encouraging students to think originally and independently and to come up with new perspectives on the issue or topic under discussion. Follow-up postings or responses were also important in terms of promoting critical thinking on the part of students and ensuring that they were able to appreciate and evaluate alternative perspectives. The requirement for follow-up postings also emphasized the interactive nature of the discussion and the importance of knowledge sharing.

Students were provided with instructions at the beginning of the course for how they would be graded on the discussion. They were informed that, once all four modules had been completed at the end of the course, they would have to complete an analysis of their contribution to the discussion forum. In relation to participation, they were instructed to determine how many postings they made and how these were distributed over the modules i.e. how many were for module 2, how many were for module 4 etc. and how many were original versus follow-up postings. They were also instructed to do a word count using the word count features in their word processor to determine the average length of their postings and to determine both the shortest and longest postings made.

Their self-analysis also had to show evidence of efforts to construct knowledge and to advance the discussion by putting forth grounded claims. A claim was defined for them as the argument they were trying to make such as: "Professional development programs do not always show how best to integrate technology". For this exercise, students had to rely on an adaptation of Toulmin’s (1958) models of argument to classify their claims according to whether they represented 1. fact or empirically verifiable phenomena; 2. judgment or opinions, attitudes and values; or 3. policy or advocating courses of action. Next, they were required to identify the grounds of their claim. These could consist of statistics, quotes or findings from relevant literature, logical argument or anecdotal evidence.

To complete the analysis, students were instructed to first use the WebCT^TM search feature to locate all their postings and to select those from the four modules only. Students may have also made postings in other areas such as in a folder set up for non-course-related social discussion or in a folder for asking questions regarding assignments. The messages in these folders were not included as part of the evaluation. After locating all postings for the four modules, students were then instructed to use the compile feature of the WebCT^TM discussion forum to collect all of their postings into one text file that they could download to their own computer. This file could then be opened in any word processor.

Students were subsequently required to use a spreadsheet program to present the results of their analysis in pie charts or bar graphs. Finally, after presenting these findings related to their analysis, they were expected to interpret, evaluate and discuss to what degree they did or did not advance the discussion and promote knowledge building and sharing. For this task, they were expected to provide some specific examples with quotes and refer as well to the charts and graphs. In addition, they were asked to reflect on how they might have improved their participation in the discussion in order to promote more sharing and construction of knowledge. This analysis had to be presented in an essay format and submitted to the instructor at the end of the course.

The following section of the paper describes one student’s attempts at analyzing her contributions to the discussion forum. Excerpts from her analysis are provided as examples. The presentation of the student’s analysis is followed by a discussion of this approach to the assessment of students’ postings to online discussion forums in the context of university-based courses.

The student began her analysis of the discussion with quantified descriptions of her participation. In the first figure presented here below, the student presents the number of postings and illustrates and compares the proportions of postings or messages that were original as opposed to messages that were responses to the postings of others. The figure shows how the majority of the student’s postings fell in the follow-up category as opposed to being original messages.

[Archives editor note: Image file for Figure 1 missing in the original]

Figure 2 illustrates how the student’s postings were distributed across all four modules. The student was expected to explain the distribution. In her case, the highest number of postings was in Module 1, likely as a result of the general nature of the topic for the module which might have invited more participation on the part of students.

The student’s interest in the Module 3 topic resulted in a larger proportion of postings as compared to two of the other modules. In the following excerpt, the student explains how the Module 4’s emphasis on collaborative, small group work and discussion made it appear as if she had discussed less in this module:

The requirement for the analysis included, not only a focus on the total number of messages, but on the individual messages themselves in terms of the length. The student calculated her shortest posting at 12 words while her longest posting was 648 words. The overall average length of her postings was 182 words. The student’s postings varied in length and she captures this variety of length in Figure 3 in which she classifies postings into word count ranges. She then notes that the majority of her postings were in the 0-99, range reflecting the course guideline which encouraged discussants to keep their postings in the range of 100 words.

In relation to the requirement to analyze the content of postings, the student identified a total of 47 claims in her 29 postings with longer messages often containing more than one claim. The distribution of the types of claims in each of the various modules is illustrated in Figure 4.

The figure illustrates that the majority of claims were judgment/value based (64% overall), followed by fact claims (23% overall), and then policy claims (13% overall). The student also provided some examples from her own postings of the three types of claims. As a claim of fact, the student presented the following example: "Some research indicates that teachers that collaborate notice improvements in the achievement levels, behavior, and attitude of their students…" For the judgment or value claim, the student presented the following example:

The following example was provided for a policy claim or claims advocating courses of action that should be undertaken: "Did you send a letter to your MP? I agree that this is the time to take action…"

The student then categorized each of her 47 claims according to the type of grounding she used to support each claim. These groundings were classified as evidence in literature, anecdotal or reasoning. The distribution of the types of grounding used in each of the various modules is illustrated in Figure 6. The student noted that the majority of her claims were grounded in anecdotal evidence (43%), followed by evidence in literature (34%), and then by reasoning (23%).

The student identified the type of grounding for each of three main types of claims. 100% of the fact claims were grounded in evidence in literature while the judgment/value claims were largely based in anecdotal evidence (56%), followed by reasoning (27%), and then by evidence in literature (17%). For the policy claims, 50% were based in reasoning and 50% were based in anecdotal evidence. As an example of providing evidence from the literature, the student presented the following message excerpt:

The following is an example she provided for anecdotal evidence: "One of the most beneficial professional development activities that I have been involved in was the BC Ministry of Education’s Information and Communication Technology Initiative…" For reasoning, the student presented the following excerpt: "By investing financial resources, standards can be better established, resources can be created, and educators can be trained in how to create and use learning resources…" The student justified her reliance on anecdotal evidence in the following excerpt presented in her essay:

In the following excerpt from her analysis, the student comments on the value of grounding her claims in evidence from literature:

In the following excerpt, she justified her reliance on anecdotal evidence as follows:

In reflecting on her contribution to the forum, taking into consideration both the length of messages and the claims they contained, she noted that she could have adopted a more effective approach that might have met with more interest by other students:

Discussion

The case presented in this paper illustrates how students can be involved in assessing their participation in and contribution to online discussions that are part of university-based courses. The analysis is designed to provide students with an opportunity to retroactively focus attention on the types, quantity and quality of participation and contributions in the context of an online discussion. The student’s assessment of her participation verified that she had completed the minimal requirement of one posting and one reply per module. In terms of her participation, the student was able to identify the distribution of her postings over the modules and to account for the differences between modules. In and of itself, the description of her participation did not provide insight into the quality of her postings. On the other hand, recognizing the number of original versus follow-up posts made evident the student’s engagement in sharing knowledge and in appreciating alternative perspectives.

The analysis of her claims and grounds provided more insight into the quality of the postings and allowed the student to focus on the implications of the types of the postings she had made in terms of the participation of others. The focused attention on the claims required her to become aware of behaviors and secondly to be able to understand and explain them and thirdly to appreciate what the implications of these behaviors might be. For example, the student noted that she relied on claims of judgment and value and that she had grounded 43% of her claims in anecdotal evidence. She subsequently explained this behavior and rationalized it by describing the importance of connecting with other individuals in the class on the basis of personal experience. She also referred to a comfort level with this type of claim and grounding. In another section of her analysis, she noted the implications of the longer postings she had made and came to a conclusion about how this might have resulted in other students ignoring her postings. She then noted that she should instead have made shorter postings with fewer claims.

The student’s attempt at retroactively focusing on her participation meant that she had to evaluate her strengths and weaknesses in a discussion context. The requirement that she consider the implications of her participation provided her with an opportunity to learn from her experiences and to improve her participation in subsequent discussion forums. In this regard, the self-assessment assignment should be designed with these goals in mind. Thus, if we want the students to improve their future participation in forums then we need to specifically require of them that they consider the implications of their behaviors in the forum. Considering the implications requires that they move from a descriptive level - i.e. ‘this is how I participated’ to a more analytical level i.e. ‘this is what my participation means’ and ‘here is how I can improve it’.

Students’ critical analysis of their participation in and contribution to course-based online discussions represents a means to assess their performance at the end of the discussion. It also represents a means to direct participation and thinking and to improve their performance in the discussion in general. By providing students with the expectations and assessment criteria for the discussion at the beginning, students can direct and monitor their communication and interaction behaviors to ensure that they are congruent with the stated criteria. From an instructor’s perspective, making the expectations explicit and easily interpretable by students will play an important role in ensuring that the discussion and its assessment achieve their intended purpose.

In the case of the discussion presented in this paper, students were required to share and construct knowledge by putting forth grounded claims and propositions and by responding and reacting to the claims and propositions of others. In other contexts, the discussion might focus instead on promoting problem solving, or on critical or creative thinking. The choice of intent for the discussion will vary depending on the context of the course and its participants. However, regardless of the intent, what is important is to clearly and carefully articulate from the beginning how students can effectively engage in communication and interaction behaviors that will help achieve the intended goals of the discussion. Indicators of those behaviors can be specified and, in addition, the moderator or instructor can also provide examples of best practices that model the intended behaviors.

From a research perspective, more investigations are needed of students’

self assessment of participation in online discussions. This research might draw

resources from the literature on meta-cognition as well as from the large body of

existing studies of content analysis of online discussions. Numerous researchers

have created instruments for the analysis of online discussion content (see for example Hara, Bonk & Angeli (2000); Henri (1992); Garrison, Anderson & Archer, (2001); Gunawardena, Lowe & Anderson, (1997); Jonassen & Kwon, (2001); Kanuka & Anderson, (1998); Murphy, (2004); Newman, Webb & Cochrane, (1995). These could be adapted and tested for use by students themselves both in order to summatively assess and formatively guide participation.

Toulmin, S. (1958). Uses of Argument. New York: Cambridge University Press.