Promoting metacognition through negotiated assessment

Metacognition is widely considered integral to effective learning. However environments that support metacognition can be difficult to develop. This paper proposes an approach to assessment through student contracts that are designed to both address issues of fairness of assessment as well as promote the planning, monitoring and evaluation integral to enhancing metacognition. By negotiating their assessment in ways that involve feedback that is internal, parallel and external to students, they have the opportunity to develop clearer understandings of themselves as learners and their own learning processes. The model is supported with online technology to help create an easy and confidential manner in which peer feedback can be collated. Initial findings suggest that students perceive themselves to be metacognitive, however, they may not be fully engaged in the processes that underpin this psychological state. It is proposed that online environments that support negotiated assessment expose these processes in ways that may enhance metacognitive outcomes, and lead to further research that identifies the nature of such processes and their value in the assessment approach.

Introduction

Weinstein & Mayer (1986) see all metacognitive activities as partly the monitoring of comprehension, and it would appear that this ability to monitor oneself is what distinguishes metacognitive activity from domain specific cognition. Wilson (1999) argues that the term metacognition can be used in 'vague, confusing, and often contradictory' ways and can be used to describe a range of disparate higher level cognitive skills. She goes on, however, to define metacognition as "awareness individuals have of their thinking and their evaluation and regulation of their thinking". In this definition it is both a state and a process, with 3 functions:

• Metacognitive awareness - individuals' awareness of their learning process, knowledge about content knowledge, and knowledge about their own strategies
• Metacognitive evaluation - individuals' judgments of their capacities and limitations
• Metacognitive regulation - the conscious modification of thinking using cognitive resources.

This apparently contradictory position of being a both a state and a process can be reconciled by acknowledging the dependence of metaknowledge upon domain dependent cognitive processes. In their research on metamemory, Nelson and Narens (1994) identify the relationship between the meta-level and the object level of cognition through a reciprocal flow of control and monitoring (Figure 1). While the model itself is perhaps a little simplistic, it does give some hope to those floundering in the problem of how to encourage metacognition in students. In Nelson and Naren's concept, one can view metacognition as a pet puppy - in order to grow and become strong it must be fed and exercised. The process of monitoring nurtures metacognition, likewise the activation of control processes exercises it.

Figure 1: A model of metacognition

It can be argued, then, that metacognition is not something that can be tackled as a discrete entity - just as well, since it is quite inaccessible as such - but can be enhanced through engaging at the subordinate levels of monitoring and control of thinking processes.

Blakey and Spence (1990) cite Dirkes' synthesis of much of the literature on metacognition into the following features:

• Connecting new information to former knowledge.
• Selecting thinking strategies deliberately.
• Planning, monitoring, and evaluating thinking processes. (Dirkes, 1985).

One of the ways of promoting metacognition is through assessment. Haefner (2004) describes an approach to assessment that engages planning monitoring and evaluation, through three different mechanisms of assessment feedback. These engage students in setting goals, evaluating their performance and monitoring their understandings through techniques that are internal, such as self assessment, parallel such as through peer collaboration, and external, such as tutor feedback.

This study builds on this approach by engaging these forms of feedback in a formative way, where the criteria for students judging the value of their work is negotiated over a semester. The study is based around a final year undergraduate unit in Project Management Methodology for Interactive Multimedia development. As with most final year courses (both graduate and undergraduate), teamwork is often needed to complete developmental projects that illustrate the students' technical/content skills learnt throughout the course of their study, as well as professional skills needed by industry practitioners. However, students often complain about team based assessment, as they perceive that if they work harder than others, the whole team is still given the same mark. A learning environment was developed to help promote fair and equitable teamwork, while at the same time integrating the planning, monitoring, and evaluation inherent in metacognitive development through internal, external and parallel feedback.

An approach to negotiated assessment

Context and unit assessment requirements

Students are encouraged to select their teams and allocate their own tasks based on their skill strengths. Teams are required to create a project proposal, design specification, metrics, evaluation report, post mortem and a web site. Decisions need to made on:

• Team role - each team made of 3-5 students needs a project manager, graphics designer, programmer and instructional designer. Roles can also be shared, combined or created (e.g. media designer, content developer, evaluator and tester). These details are negotiated and finalised in the first two weeks of the semester; and
• Project topic - selected by students to enhance their skills, though considered for suitability by tutors i.e. team roles, client, clearly achievable objectives value of final product;
• Clients - team members consider how to approach clients and establish what commitment and input they will give the project. The client will is requested to pass comment on the quality of the final product.

• Project Name and description;
• Team members and their roles;
• Web site URL; and
• Documentation, which consists of Project Proposal; Design Specifications; Metrics; Evaluation and Post Mortem.

• Correct focus in answering the question, with relevant facts and research supporting your perspective
• Synthesis of ideas into a cohesive solution
• Correct grammar and spelling
• A clear introduction and conclusion
• Proper referencing of information sources
• Under 500 words, excluding references

• 34% for web site development and documentation - teamwork assessment;
• 16% online problem solving - teamwork assessment; and
• 50% exam - individual assessment (must pass).

Designing the learning environment for negotiated assessment

• Authentic Tasks - team based project work is integral and uses real clients. Student liaise with clients to cost, schedule and track projects, reporting on discrepancies and developing documentation that has direct relevance in the industry. The final product and documentation is hosted on a university server for students to use as an electronic CV to enhance employment opportunities;
• Self regulated Tasks - students are encouraged to take ownership of their learning, selecting project topic, team members, team roles and responsibilities and committing to these through a "Team Contract". This enables students to make decisions about what skills they want to focus on and develop. These are negotiated at the beginning of the semester with peers, clients and tutor; and
• Reflective Tasks - students review their contributions on the "Team Contract" when they submit each assignment. Also, online self/peer assessment journals can be used to continually monitor their performance during the semester. These can submitted confidentially by students to report on their peers progress, which gives tutors extra useful information when deciding on how to moderate marks.

Figure 2: Learning environment design

Within this learning environment, students use the "Team Contract" (Table 1) to negotiate their assessment items. Each row in the Team Contract represents a key assessment point that students can consider how heavily they want to contribute to, based on their aims for future employment and current skill sets.

If there are four students in a team, then each student should contribute 25% of the overall marks. However, this is not mandatory, and students can specify how much of the "assessment pie" they want! This team based negotiated assessment process is completed in two separate stages. Students are to consider:

• Estimated Contributions - at the beginning of the semester students commit to completing a series of tasks (approximately 25% of the total in a team of four students) by specifying which assessment tasks they want to contribute to, as well as the predicted quality they're aiming for;
• Actual Contributions - when each of the team assignments are submitted, the Team Contract is re-submitted. Students now complete the "Actual Contributions" columns, with a review of what marks they actually contributed to, and also with a rating of the actual quality they presented. This is agreed to by the whole team and the tutor, and then submitted for marking. The reviewed assessments and quality standards agreed to here are then used to help distribute marks in the team using tutor led peer assessment sessions.

Beyond the issues of fairness and equitability of marks distribution, however, this negotiation also involves students in planning their learning, by setting goals and estimating their performance both in terms of outcome (mark) and process (quality of work), they are then required to evaluate these goals against actual achievement. Through a cyclical process and through the internal, parallel and external feedback mechanisms of peer, tutor, and self assessment, students are engaged in a continuous process of self monitoring.

Assessment Items	%	Name 1	Name 2	Name 3, etc
		EM    EQ    AM    AQ	EM    EQ    AM    AQ	EM    EQ    AM    AQ
Online Tasks	16
Project Proposal	10
Design Specifications	5
PM Doc 1	2
Application Development	5
Presentation & Online CV	2
Evaluation Report	3
Metrics Report	3
Post Mortem	2
PM Doc 2	2
Total	50
We agree that the assigned assessment values & quality are acceptable (Tutor ____________________)
Signatures
EM = Estimated Mark, EQ = Estimated Quality, AM = Actual Mark, AQ = Actual Quality

The implementation of the team contract is based on planning, monitoring and evaluation (Dirkes, 1985). By week 3 students must plan and negotiate with their team members which assessment items they are responsible for ("EM" in Table 1), as well as predicting the quality of these ("EQ" in Table 1). As the semester progresses, students are required to continually monitor their own performance in terms of their stated plans as well as their team members' commitments (as agreed to in the contract). If they feel the team isn't progressing as agreed, they can inform the tutor through the peer assessment tool. As well as ongoing modification of their initial plans, students formally evaluate their performance and that of their peers when the assessment item is submitted. They do this by entering the actual mark and quality of their contributions into the team contract ("AM" and "AQ" in Table 1).

Rationale for the development of a means of negotiated assessment and initial support for student contracts

Question		SD	D	N	A	SA	Metacognitive Category	See Figure
1.	I know how I learn best	0	0	3	10	3	General Metacognition	2
2.	I know what I am good at as well as the things I have difficulty with	0	0	1	10	5	General Metacognition	2
3.	It is useful for me to think about my studies before I go to class	0	0	2	8	6	Planning	3
4.	I find it useful to set myself goals for learning	0	0	3	11	2	Planning	3
5.	I understand concepts better when I imagine them in practice	0	0	1	5	10	Monitoring	4
6.	It is important for me to find ways of applying what I am studying to real settings	0	0	2	9	5	Monitoring	4
7.	Making notes helps me understand what I am studying	0	1	4	9	2	Monitoring	4
8.	I find it helpful to compare my ideas with other students to make sure I'm on the right track	0	0	3	8	5	Evaluation	5
9.	I compare what I've achieved in learning to what I planned to achieve	0	2	4	8	2	Evaluation	5
10.	Using the student contract will help me plan my learning	0	0	6	10	0	Value of student contracts	6
11.	Using the student contract will help me monitor my performance	0	0	3	12	1	Value of student contracts	6
12.	Using the student contract will help me evaluate my performance	0		3	12	1	Value of student contracts	6
13.	Using the student contract will help me understand myself as a learner better	0	2	4	10	0	Value of student contracts	6
SD = Strongly Disagree, D = Disagree, N = Neutral, A = Agree, SA = Strongly Agree

As can be seen from Table 2, students on the whole perceived themselves to be operating at a reasonably high metacognitive level. Figures 3 to 7 show the extent to which they support each concept of overall metacognition, planning, monitoring, evaluation, and the value of the student contract as a tool to enhance their monitoring processes.

Figure 3: Perceived overall metacognition (Q 1-2)

Figure 4: Perceived planning (Q 3-4)

Figure 5: Perceived monitoring (Q 5-7)

Figure 6: Perceived evaluation (Q 8-9)

The positive overall response suggests that most students perceive themselves to be operating at a high level with regard to their ability to plan, monitor and evaluate their learning processes. The higher reported response to monitoring indicates that most students are comfortable with engaging in strategies that involve monitoring their understandings although as Table 2 (question 7) demonstrates, one student

Figure 7: Perceived value of student contracts (Q 10-13)

disagreed with the concept of note taking as a useful approach. It was also intriguing that two students disagreed with the statement regarding the evaluation of goals (question 9). While comparing what one has achieved with what one has planned to achieve is integral to the process of metacognitive monitoring it would appear that for some students at least, it is not highly valued.

The student contracts, as with the concepts inherent in metacognition also received support overall. Again, some disagreement was received, in particular to the statement regarding the value of the tool as a means of making students better learners (question 13). It would appear that while all subjects can see how the strategy will engage them in planning, monitoring and evaluation, the generalisation of these processes to a general state of metacognitive awareness is not considered an integral outcome of negotiated assessment for all of the students.

Conclusions about negotiated assessment as a tool for promoting metacognition

Initial findings suggest that students perceive themselves to be operating at a reasonably high metacognitive level. They can also see the value of a tool for negotiated assessment in promoting self monitoring. Nevertheless, the accuracy of student perceptions must be questioned, when at least two students fail to see the value of evaluating performance against goals; an activity integral to the planning, monitoring and evaluation feedback loop that is self monitoring. This is further exacerbated by a tendency for a couple of students to fail to draw the link between the student contracts and the processes that underpin them, with a general concept of metacognition involving enhanced self awareness in the learning process.

Further study will be required to explore the actual patters of self monitoring in which students engage. This may involve analysis of the discourse involved in negotiating assessment to identify the actual nature of self monitoring compared to the students' self perceptions. While this represents the beginning of a project that is necessarily iterative and exploratory over a period of time, initial findings suggest that negotiated assessment through an on line student contract system has the potential to provide a means to expose the processes of assessment that will ensure self monitoring on the part of students, that should lead to outcomes that are fair, representative of the work that students engage in, and above all, lead to the development of students' awareness of the processes that underpin their own learning.

References

Boekaerts, M. (1997). Self-regulated learning: A new concept embraced by researchers, policy makers, educators, teachers, and students. Learning and Instruction, 7(2), 161-186.

Collis, B. (1997). Supporting project-based collaborative learning via a World Wide Web environment. In B. H. Khan (Ed.), Web-Based Instruction (pp. 213-219). New Jersey: Educational Technology Publications.

English, S. & Yazdani, M. (1999). Computer-supported cooperative learning in a virtual university. Journal of Computer Assisted Learning, 15(2), 2-13.

Haefner, T. (2004). Assessment as a magnification of internal, parallel, and external reflection. Action in Teacher Education, 25(4), 14-19.

Nelson, T. O. & Narens, L. (1994). The role of metacognition in problem solving. In J. Metcalfe & A. Shiminura (Eds.), Metacognition (pp. 207-226). Cambridge: MIT Press.

Jacobson, R. (1998). Teachers improving learning using metacognition with self monitoring learning strategies. Education, 118(4), 579-589.

Klemm, W. R. & Snell, J. R. (1996). Enriching computer-mediated group learning by coupling constructivism with collaborative learning. Journal of Instructional Science and Technology, 1(2). [verified 30 Sep 2004] http://www.usq.edu.au/electpub/e-jist/docs/old/vol1no2/article1.htm

Pressley, M., Van Etten, S., Yokoi, L., Freebern, G. & Van Meter, P. (1998). The metacognition of student scholarship: A grounded theory approach. In D. J. Hacker, J. Dunlosky & A. C. Graesser (Eds), Metacognition in Educational Theory & Practice (pp. 347-366). New Jersey: Lawrence Earlbaum Associates Inc.

Schraw, G. (1998). Promoting general metacognitive awareness. Instructional Science, 26, 113-125.

Weinstein, C. E. & Mayer, R. (1986). The teaching of learning strategies. In M. Wittrock (Ed), Handbook of research on teaching (pp. 315-327). New York: Macmillan.

Wilson, J. (1999). Defining metacognition: A step towards recognising metacognition as a worthwhile part of the curriculum. Proceedings AARE Conference, Melbourne. [verified 30 Sep 2004] http://www.aare.edu.au/99pap/wil99527.htm

Authors: Joe Luca, School of Communications and Multimedia. Edith Cowan University. j.luca@ecu.edu.au Mark McMahon, School of Communications and Multimedia. Edith Cowan University. m.mcmahon@ecu.edu.au Please cite as: Luca, J. & McMahon, M. (2004). Promoting metacognition through negotiated assessment. In R. Atkinson, C. McBeath, D. Jonas-Dwyer & R. Phillips (Eds), Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference (pp. 562-570). Perth, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/luca.html

© 2004 Joe Luca & Mark McMahon
The authors assign to ASCILITE and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to ASCILITE to publish this document on the ASCILITE web site (including any mirror or archival sites that may be developed) and in printed form within the ASCILITE 2004 Conference Proceedings. Any other usage is prohibited without the express permission of the authors.