Opening Pandora's box of academic integrity: Using plagiarism detection software

Academic integrity issues are currently a major focus of concern at most tertiary institutions. This paper details the strategic framework for work at the University of Tasmania (UTAS) for management of these issues. It focuses on the introduction of plagiarism detection software which has served to highlight the wide variety of issues associated with academic integrity and the importance of embedding good practice on the part of both staff and students. The paper reports on the Pandora's box of implementation issues - legal, workload, training and support - that have emerged and the strategies being used to manage these, as part of the project. It recommends the use of a model which focuses on an educative approach to the management of academic integrity, as well as including mechanisms for identifying and discouraging plagiarism, and where it occurs, proceeding against it as academic misconduct. Many of the issues raised by the project have challenged the 'comfort zones' of students, staff and university academic administration. These are being managed both through the approaches being used in the pilot and the project governance adopted.

Background

For the purpose of this paper the term 'academic integrity' is used in a broad sense, referring to mastery of the art of scholarship. Scholarship involves researching, understanding and building upon the work of others and requires that credit is given where it is due and the contribution of others to your own intellectual efforts is acknowledged. At its core, academic integrity requires honesty. This involves being responsible for ethical scholarship and for knowing what academic dishonesty is and how to avoid it.

Plagiarism, in Webster's dictionary (1993, p. 1728) is defined as "to steal and to pass off as one's own (the idea or words of another); use (a created production) without crediting the source; to commit literary theft; present as new and original an idea or product derived from an existing source".

The Centre for Study for Higher Education at the University of Melbourne suggests in their 2002 report (James, McInnis & Devlin, 2002) that academic integrity can be managed through the introduction of and commitment to four strategies, all of which are underpinned by the central principle of ensuring fairness:

1. A collaborative effort to recognise and counter plagiarism at every level from policy, through faculty/division and school/department procedures, to individual staff practices;
2. Thoroughly educating students about the expected conventions for authorship and the appropriate use and acknowledgment of all forms of intellectual material;
3. Designing approaches to assessment that minimise the possibility for students to submit plagiarised material, while not reducing the quality and rigour of assessment requirements;
4. Installing highly visible procedures for monitoring and detecting cheating, including appropriate punishment and re-education measures.

UTAS has found this a useful reference point for the management of these issues at an institutional level and in 2001 it established a working party to review our framework and make recommendations as appropriate.

The outcome of work during 2001 - 2002 of the working party can be summarised as follows:

• A statement on Plagiarism was developed, which was to be included in all unit outlines.
• A generic University assignment cover sheet was introduced, to include an attestation that the work presented is the student's own.
• Current sanctions were reviewed. The relevant Ordinance, of Student Discipline, was reviewed in 2003, for use by heads of schools (as responsible officers) in cases of academic misconduct. This recommendation was consistent with those of academic integrity expert, Jude Carroll, who recommends the development of a set of penalties and the appointment of a person at the departmental level as a responsible officer for their assignment (Carroll, 2002). At UTAS, all cases of academic misconduct that incur penalties are recorded on a central database, managed by Academic Administration.
• The Academic Registrar was identified as a reference person for heads of school, for queries on appropriate penalties. In this way we sought to manage potential issues of inconsistency in the application of penalties.
• Resources were developed to assist students and staff manage issues of academic integrity and plagiarism, both unintentional and intentional. This includes information on how to acknowledge sources and for staff, how to set assessment items which reduce the possibility of deliberate plagiarism. These resources can be found on the university's Academic Integrity website (http://www.utas.edu.au/tl/supporting/academicintegrity/index.html). It should be noted that in developing these resources and working to embed them in practice, the Flexible Education Unit (FEU) focussed on a developmental approach. We believe it is important to develop a framework which focuses on educative strategies and processes whilst covering punitive issues.

This working party also supported the introduction of an auditing mechanism in the form of plagiarism detection software, to assist in ensuring that the work submitted by students is their own.

In 2003 the FEU took up this work and applied a project management methodology to its continuation. To ensure continued high level support for the project, the Pro Vice-Chancellor Teaching and Learning (T&L) took on the role of project sponsor. The project steering committee consisted of the Director FEU as Chair, the project leader, Academic Registrar, Dean of Graduate Studies, two student and academic staff representatives, a representative from the central IT unit, plus the Library as observer. While the project focused on the introduction of plagiarism detection software - Turnitin - it also involved a further revision of policy and support issues. We were not sure what other issues might arise, administrative, policy or legal, and the role of the Steering Committee was to provide advice on their management as well as generally oversee the project.

Plagiarism detection software: Why Turnitin?

The Cooperative Action by Victorian Academic Libraries (CAVAL) supports Turnitin and provides consultancy, training and help desk services for it. Turnitin is currently used at 28 Australian tertiary institutions and in nearly 50 countries world wide, including extensive use in the UK through the JISC Plagiarism Detection Service and in the US at both the tertiary and secondary level.

What is Turnitin?

Figure 1: Turnitin report, print version format, used with permission

After a document is submitted to Turnitin, it is added to Turnitin's database. This enables a historical archive of submitted documents to be built up and included in later checking. An additional function of collecting documents in the database is the building up of references taken from printed material. Print based material is not currently available to Turnitin through any other source.

Turnitin does not identify all potential cases of plagiarism, as its database does not contain all web pages, electronic journals, published works or individually produced works and it cannot match paraphrased text. It is only one tool in an overall strategy for managing academic integrity being implemented at the university.

Because Turnitin only reports on the degree of text matching, it is necessary for individual lecturers to review Turnitin's reports to determine the actual level of plagiarism. Turnitin does not differentiate between correctly cited references and unacknowledged copying. What it does provide is a ranking of assignments, according to the level of text matching it has found with other sources, highlighting those assignments that are most likely to include plagiarism.

Project strategies

UTAS joined the CAVAL Plagiarism Detection Consortium, established across Australia and New Zealand to achieve better educational outcomes in the area of plagiarism reduction. Membership is free and the consortium provides discounts on services and software and will provide software support through its Turnitin Help Desk and initial training.

Figure 2: Turnitin report, side by side version, used with permission

Key Performance Indicators for the pilot are as follows:

• Appropriate policies and procedures to support use of Turnitin and to address cases of plagiarism
• Implementation of user administration processes prior to the pilot
• Attendance at training and usage of resources and support services
• Usage levels of Turnitin by stakeholders and assessments of usefulness.

• Approval for the introduction of an amendment to the University Statement on Plagiarism to reflect the fact that student work might be submitted to Turnitin.
• The timely development of training and support services for staff to use the software.
• Easy to use system to register and use the software.
• The support of the senior executive and academic committees for the use of plagiarism detection software.
• Student acceptance of the use of plagiarism detection software, for plagiarism detection and peer review of assignments.

Key strategies can be summarised as follows:

• Policy revision: Prior to introducing Turnitin, the University's Plagiarism Statement and assignment cover sheet were amended. For 2004, the Plagiarism Statement was changed to inform students that assignments might be submitted to plagiarism detection software. The assignment cover sheet was also updated to include this information.
• Administrative processes: Responsibility for the management of Turnitin accounts and passwords has been devolved as much as possible. The Turnitin account ids and passwords for Faculties are managed and distributed centrally. Unit coordinators are responsible for creating and distributing the Turnitin course ids and passwords for their units. Staff and students are responsible for setting up their own Turnitin user account ids and passwords. Over time, responsibility for management and distribution of Faculty accounts and passwords are likely to be distributed to appropriate Faculty representatives.
• Provision of Training: The CAVAL Plagiarism Consortium provided a train the trainer session at the start of the pilot. Training resources have since been developed for staff and students, covering both how to use the software and related academic integrity issues. For staff this has included training in strategies to discourage plagiarism and university policy and procedures relating to academic integrity. For students it has included resources on academic integrity and how this relates to using both the text and ideas of other authors in their work.
• Communications plan: This has included briefings to senior staff on academic integrity issues, the running of information sessions and the promotion of the multiple uses of Turnitin to staff and students through internal university publications.
• Conducting and evaluating a pilot: The piloting of Turnitin in self selected schools and the evaluation of its use and the associated resources. The results of the pilot are discussed later in this paper.
• Investigation of issues: Throughout the pilot communication was encouraged between the pilot participants and project leader. Issues raised either by students or staff were followed up. These included several legal issues for example: clarification of the status of Turnitin reports as evidence in disciplinary proceedings and IP issues regarding student's work being stored on third party databases off site.
• Reporting: Reports will be developed as a result of the pilot evaluation, to the University's Teaching & Learning Committee, Heads of Schools and Associate Deans T&L, on trends and issues in the use of Turnitin.
• Determining Turnitin's appropriateness for UTAS: As a result of evaluating the Turnitin pilot, the effectiveness of Turnitin within the UTAS environment for assisting with the management of academic integrity and plagiarism will be determined.
• Recommending models of use: Models of use of Turnitin for staff and students will be recommended, to ensure effective and efficient use within the UTAS environment.

Semester 1 pilot 2004

The pilot aimed to investigate issues related to:

• administration, resourcing and support of the use of Turnitin
• the use of Turnitin by students, staff and other stakeholders
• training of staff and students in issues related to academic integrity and plagiarism

The pilot was evaluated by:

• investigating difficulties experienced by participants
• investigating workload implications
• analysis of enquiries to the Service Desk, FEU Help Line and CAVAL Help Desk
• feedback from training courses
• a questionnaire for student users of Turnitin
• a focus group of students that had not used Turnitin
• a focus group of staff participants
• Turnitin's statistics report on submissions

Preliminary findings and issues - Semester 1 pilot 2004

Turnitin did not highlight all occurrences of plagiarism detected in units where the lecturer submitted the student's work. Markers in these three units identified cases of plagiarism not highlighted by Turnitin. These were not identified by Turnitin because a website was not included in Turnitin's database and copied work had been paraphrased by students. However in two of these units, Turnitin highlighted the majority of plagiarism cases. In these two units it also highlighted more cases of plagiarism than initially detected by the markers.

Staff participants were surprised that a recent Australian study by Marsden (as cited in O'Connor, 2003) had detected no significant difference in rates of plagiarism between domestic and international students. They agreed that it was easier for markers to identify plagiarism by students from non-English speaking backgrounds.

Impact on students and staff

Student workload issues

Staff workload issues

Legal issues

• concern over the protection of their copyright and IP rights, particularly since the Turnitin database was housed in the USA and not Australia.
• refusal to submit their work to Turnitin as they considered there was no requirement on them as students to do so.
• refusal to submit their work to Turnitin as it would be retained on a database external to the university.

All these issues were raised with the university legal officer. None has been considered an impediment to using Turnitin, however the need to inform students that the university will use plagiarism detection software is important. The University does this in its Plagiarism statement and educates students about academic integrity issues - such as how to cite sources. The University also has a responsibility to ensure the protection of the copyright and IP rights of its students' work stored on the Turnitin database, which is addressed through its contractual arrangements.

The Steering Committee also raised a number of issues:

• whether student agreement to the use of plagiarism detection software could be considered consent under coercion.
• the status of Turnitin reports as evidence in plagiarism cases.

These were also referred to the university legal officer and again did not pose any difficulty to continued use of Turnitin. Some, such as coercion were not issues at all, but were followed up in any case to ensure full consultation and ownership of the project.

The legal issues raised required a significant amount of time to document and follow up, however, they are likely to be a significant part of any implementation project, until the use of such tools is common place. They indicate the discomfort felt by some students as the result of introducing a tool that can identify non-compliance with standards.

Training issues

Of even greater importance is raising staff awareness of academic integrity issues such as why students plagiarise, how it is done, how to design assessment tasks to minimise the opportunity for plagiarism, as well as information about university policies and procedures regarding plagiarism. As pilot participants were new to the use of Turnitin, it is not surprising that their efforts were initially concentrated on mastering the software, before considering changes to their teaching practice. This behaviour is described by the model of technology adoption developed by Sandholtz , Ringstaff, and Dwyer (as cited in Torrisi & Davis, 2000, p. 169). Pilot participants would be considered to be in either the entry or adaption stage of this model, learning to use the technology and apply it to their current teaching practices. They will require a higher level of confidence with the tool, before being ready to look at issues such as changing assessment practices.

Staff need to be comfortable on issues such as how much needs to be copied to be considered plagiarism?. Issues related to academic integrity are frequently not black and white, as highlighted in the excellent article by Devlin (2003). Encouraging discussion on such issues and seeking agreement is an important component of academic integrity.

Students There were few difficulties experienced by students using Turnitin, indicated by no requests being received by local support services. Participants in the focus group felt that first year students and particularly international students needed to be effectively inducted into the university's policy on plagiarism and referencing standards. These students wanted information relating to plagiarism reinforced throughout the unit, not just in the unit outline and the first week of lectures. The focus group participants also reported that different schools required different referencing standards and that this caused considerable confusion for students. Participants in the student focus group were not aware of university procedures to deal with plagiarism, nor the range of penalties that could be imposed, despite promotion of them by the university. The focus group students felt that the action which had most raised awareness amongst students of plagiarism issues was an email from a unit coordinator announcing that two students had been found to have plagiarised.

Support issues

Staff also encountered relatively few problems using Turnitin, these included: issues with accessing Turnitin from a Macintosh, text files were only identified as such when the suffix .txt was added to the file name, Turnitin was not able to handle very early versions of Word documents (these had to be resaved in a new version of Word to be analysed by Turnitin), Turnitin failed to identify a piece of copied text from a website (the website URL was sent to Turnitin who included it in their 'crawl list', the contents of the site would be added to their data base within two weeks), two or more sources for a piece of matched text can cycle rather than being excluded as requested (this was a known problem and an individual request to reanalyse the document would have to be made to Turnitin), this problem was identified in test data used on the site and not reported in any student work.

Administration

Attitudes towards use of plagiarism detection software

Staff attending the focus group felt that the software could be used to deter academic misconduct; as a 'sharp hook' for discussion about plagiarism; and, assist students improve their referencing. They felt it was very important that staff and students were clear regarding what could be expected of Turnitin - that it was just one tool of many used by the university. They felt it could be promoted to staff as a way of tracking down suspect material; as an initial tool for flagging obvious cases; and, that it would provide more confidence to staff marking work, that cases of plagiarism would be detected. They felt it could be promoted to students as one way of preserving the integrity of the award they receive and give higher confidence that they will receive the mark they deserve relative to the rest of the group.

Responses to the student questionnaire (with a limited 17% response rate) indicated that participants felt Turnitin made the results of the assessment fairer (60%) and less than 20% would not recommend Turnitin to other students. However respondents indicated a very strong preference that students always know when Turnitin was to be used, always have access to Turnitin's analysis of their work and always be able to resubmit their work after seeing the initial Turnitin report. This again highlights students discomfort at the possible regulatory use of Turnitin.

Semester 2 pilot 2004

• development of a model for use of Turnitin across the university. During the second semester pilot participants will be using the following models:
  • for assignment submission, either staff will submit student assignments or students will submit their assignments, be able to view the Turnitin report and then resubmit a final version. This is equivalent to either an 'auditing' or 'educational' model of use of Turnitin.
  • for determining which reports to review, staff will either check those reports that Turnitin indicates have 50% or greater text matching (represented by a red or orange icon), or a set number of reports chosen randomly from the documents submitted.

• awareness amongst staff and students and adherence to university procedures for handling cases of plagiarism - FEU staff are working with schools to familiarise staff with strategies for minimising plagiarism and to refer students to resources for academic writing and referencing. Heads of School will be included in the second semester pilot through the opportunity to review the Turnitin reports of participants within their school. The Library also works with staff and students in this regard. In December this year the FEU will host workshops for staff on the application of penalties for plagiarism and strategies for minimising plagiarism by students.

• the inclusion of an offshore unit in the pilot and representation from all Faculties.

Conclusion and summary

Interestingly, we found that students were more aware of the extent and nature of plagiarism activity within the institution than were staff. Students were also very supportive of university initiatives to detect and punish cases of misconduct, though also considered important, improved student awareness and assessment practices. The widespread implementation of plagiarism detection software at UTAS is likely to result, at least initially, in an increased level of detection. Dealing with such an increase will be difficult for all sectors of the university community. It is important that academic staff and relevant committees are aware of this possible phenomena and its causes, so that it is not misinterpreted and briefings have been undertaken to address this.

On reflection, we can see that we were optimistic in our ambition to both introduce plagiarism detection software and encourage staff to adopt measures, in relation to the assessment practices for instance, which would reduce the possibility of plagiarism. This was too wide a range of changes to be made in one step. Staff involved in the pilot were focused on how to use the software and we are now planning a strong staff development program for 2005 to encourage staff to review teaching practice. In our view, a staged approach over a period of years is required.

Workloads and ease of use are important issues. The first semester pilot highlighted for us that staff and students will only use plagiarism detection software if it is easy to use and does not add to their workloads. The semester one pilot highlighted workload as an issue for staff when reviewing Turnitin reports, this issue had to be addressed to enable wider adoption of the software. As a result the guidelines for the extended semester two pilot includes two options for the review of reports, which would cause limited impact on staff workload.

At the end of Semester two we should have a model, with options and guidelines, which will enable the effective and sustainable use of plagiarism detection software, and is also acceptable to staff and students.

Plagiarism detection software is viewed at UTAS as one tool in a broader approach to addressing issues of academic integrity. We believe it is important to focus on the development of an educative and developmental approach with students, embedding good practice in scholarship and academic referencing. In late 2004 and 2005 more work will need to be undertaken with staff embedding this practice in curriculum and raising awareness about the importance of compliance in the application of policy and procedures and consistency in the application of penalties, this inevitably will move many staff beyond their comfort zones.

Looking to the future from a technological perspective, Turnitin is only the beginning of the application of technology to detect plagiarism. It addresses only text documents and other tools are in development and use to detect plagiarised computer code and author identification. These tools will assist in identifying cases of academic misconduct. However, relying heavily on technology to provide a "purely 'catch and punish' approach ... will simply lead to a never ending 'arms race' between the students and the university." (Carroll, 2002)

Reference list

Devlin, M. (2003). The problem with plagiarism. Campus Review, Nov 12-18 2003, 4-5.

James, R., McInnis, C. & Devlin, M. (2002). Assessing Learning in Australian Universities. Centre for the Study of Higher Education, University of Melbourne. [29 Dec 2002, verified 23 Oct 2004] http://www.cshe.unimelb.edu.au/assessinglearning/

O'Connor, S. (2003). Cheating and Electronic Plagiarism - Scope consequences and detection. Paper presented at EDUCAUSE conference May 2003, Adelaide). [28 Jul 2004, verified 23 Oct 2004] http://www.caval.edu.au/about/staffpub/docs/Cheating%20and%20electronic%20plagiarism%20-%20scope,%20consequences%20and%20detection%20EDUCASUE%20May%202003.doc

Torrisi, G. & Davis, G. (2000). Online Learning as a catalyst for reshaping practice - The experiences of some academics developing online learning materials. The International Journal for Academic Development, 5(2), 166-76.

Turnitin Tour. [20 Jul 2004] http://www.turnitin.com/static/tour/tour_master.html

Webster's Third New International Dictionary of the English Language, Unabridged (1993). Cologne: Konemann.

Authors: Sue Mulcahy. Flexible Education Unit, Office of Pro Vice-Chancellor Teaching and Learning, University of Tasmania, Private Bag 133, Hobart 7001. Sue.Mulcahy@utas.edu.au Christine Goodacre, Director, Flexible Education Unit, Office of Pro Vice-Chancellor Teaching and Learning, University of Tasmania, Private Bag 133, Hobart 7001. Christine.Goodacre@utas.edu.au Please cite as: Mulcahy, S. & Goodacre, C. (2004). Opening Pandora's box of academic integrity: Using plagiarism detection software. In R. Atkinson, C. McBeath, D. Jonas-Dwyer & R. Phillips (Eds), Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference (pp. 688-696). Perth, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/mulcahy.html

© 2004 Sue Mulcahy & Christine Goodacre
The authors assign to ASCILITE and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to ASCILITE to publish this document on the ASCILITE web site (including any mirror or archival sites that may be developed) and in printed form within the ASCILITE 2004 Conference Proceedings. Any other usage is prohibited without the express permission of the authors.