Student Perceptions of a Trial of Electronic Text Matching Software: A Preliminary Investigation
Promoting academic integrity has become a major focus in universities in recent times. Universities are expected to minimise plagiarism, collusion and cheating to maintain academic standards. Many authors have argued that in order to minimise plagiarism and collusion within a university, a range of strategies need to be implemented. A general culture of academic integrity needs to be engendered across all levels of the university; students need to be taught referencing and writing skills and be given the opportunity to practice these; academics need to design courses which reduce opportunities for plagiarism and collusion; policy needs to be reviewed and applied in a way which is workable and ensures students are treated fairly and consistently; and detection methods need to be applied (McCabe, Klebe Trevino et al. 2001; Carroll 2002; James, McInnis et al. 2002; Allan, Callagher et al. 2005).
In order to address these strategies, Flinders University is currently undertaking an integrated Academic Integrity Management Strategy (AIMS) project (Flinders University 2005; Evans and Green 2005) which has 4 overlapping elements:
The student education element involves the development of an online learning package which helps students understand academic integrity, explains its benefits, gives information about referencing and working together, and gives examples and practice questions to aid understanding. An online test will also be available. The staff education aspect aims to inform staff of the educative, and therefore preventative, aspects of the overall strategy. This includes how to educate students about academic integrity, how to design courses and set meaningful assessment tasks which minimise opportunities for plagiarism and collusion, and how to model academic integrity in teaching. A policy review is being undertaken to ensure workable, consistent and fair policies are in place across the institution. The focus of this paper is on the trial of electronic detection methods.
It was decided that the electronic text matching software, Turnitin was to be trialled on a limited multi-department basis in Semester 1, 2005 and a cost benefit analysis carried out to establish whether the university should purchase an ongoing license for the software. Turnitin is produced by a private US based company and compares submitted assignments against a database which includes archived copies of portions of the Internet, other students' assignments and some of the ProQuest database. Originality reports are generated which identify the degree to which the assignment matches the content of the database (Turnitin 2005). The trial is being overseen by a management group chaired by the Deputy Vice Chancellor (Academic) and consisting of general and academic staff, as well as the General Secretary and the Academic Rights Officer of the Students' Association. Students involved in the trial were given access to the online materials developed for students as part of AIMS and were informed in detail about the purpose of the trial. An educative, rather than a punitive approach was emphasised, where students could read their own originality reports and resubmit assignments. The term "text matching software" was used to emphasise what the software does, not for how it might be used. It was stressed to students that judgement of plagiarism was an academic decision, not a technological one.
The introduction of text matching software has many perceived benefits. These include a deterrent effect, detection of plagiarism, an educational benefit in fostering proper acknowledgment practices and ensuring institutional reputation (James, McInnis et al. 2002; Martin 2004). James et al. (2002) indicates that "installing highly visible procedures for monitoring and detecting cheating" (p. 37) should be a major strategy for ensuring fairness in minimising plagiarism. Benefits discussed in the literature appear to reflect organisational and staff perspectives, with limited reference to student views.
The cost of the implementation of such software is relatively straightforward. License, time and administration costs in implementation and training and support can all be quantified. What tends to be overlooked, but could also be thought of as a cost, is the impact of the introduction of these methods on the learning environment, including the levels of trust and anxiety (Bedford St. Martins 2001; Briggs 2003; Ottawa Citizen 2003, Edmonton Journal 2003, as cited in Martin 2004). Martin (2004) warns that "there may be serious negative effects of compulsory checking; especially reduced trust" (p.6). While this warning appears well founded, there has been little effort to measure this effect. The few that have studied this perception, report a range of acceptance from disinterest, to considerable concern, to outright hostility (Chester 2001, Freewood 2001, as cited in Carroll 2002; Gulik and Tippin 2004).
Flinders University has a strong culture of respecting the views of students. This paper reports on one aspect of the evaluation of Flinders University trial of Turnitin, where students' thoughts on plagiarism, and their perceptions of being involved in the trial were investigated before they had used the software. A post-survey is currently being conducted to establish how these attitudes and views may have changed over the period of the trial.
All students participating in the trial were surveyed prior to the trial commencement. Students were included from a number of topics (the basic unit of study at Flinders) across all four faculties. The subject areas included Law, Nutrition and Dietetics, Chemistry and Commerce, and levels ranged from first year to postgraduate students. There were a few students enrolled in more than one topic involved in the trial, most often within the same subject area, however there were some students enrolled in both Commerce and Law topics.
Members of the management group attended student introductory lectures and provided an overview of the software and the trial, information on how the software was to be used and information on where to get help. The educative uses of the software were highlighted, and students were given the opportunity to ask questions.
Surveys were limited to one page for ease of completion by students. Survey questions focussed on three main themes: student perceptions of their understanding of plagiarism and its consequences; student opinion on the university using text matching software; and student opinion on how the use of text matching software will impact on them. Students were asked to indicate their views on 5-point Likert scales. The survey also asked students to respond to open-ended questions regarding perceived benefits and concerns. Student demographic information such as age, gender, full or part-time status and whether English was the student's first language was also collected.
Survey completion was not compulsory and all surveys were completed anonymously.
Quantitative data were entered into SPSS for Windows 11.5 (SPSS Inc. Chicago, SPSS for Windows, version 11.5 2002) for descriptive and inferential analysis. Means and standard deviations were generated for the sample as a whole, while t-tests were performed to test differences between subgroups (i.e. males and females; young students, defined as those 23 or younger, compared to mature students; NESB status; full time or part time status; and topic area). Levene's test was used to check the assumption of homogeneity of variance for t-tests. Corrections were made as appropriate where indicated. Qualitative responses were coded using N*VIVO 2.0 (QSR International Pty. Ltd. Doncaster, N*VIVO Version 2.0 1999). Chi-square analyses (with Continuity Correction for 2x2 designs) were conducted to compare the frequency of qualitative responses across each of the groups.
728 surveys were received, a response rate of 74.7%. Of this, 360 students were from Commerce (response rate of 65.9%), 310 from Law (85.2%), 48 from Nutrition (92.3%) and 10 from Chemistry (76.9%).
53.5% of respondents were female. Students ranged from 17 to 64 years old (M = 23.07, SD = 7.10). 6.6% of respondents were enrolled part time, and 19.3% indicated that English was not their first language (NESB).
Student Perceptions of Their Understanding of Plagiarism and its Consequences
Most students believed that they understood what plagiarism was (M = 4.70, SD = 0.56), and that they understood the consequences of plagiarism (M = 4.54, SD = 0.76).
NESB students (M = 4.52, SD = 0.73) indicated they were less knowledgeable about the meaning of plagiarism than were the rest of the student sample (M = 4.74, SD = 0.51), t(167.04) = 4.11, p = .001 (corrected for violation of the assumption of homogeneity of variance). In addition, NESB students (M = 4.27, SD = 0.93) indicated they had a poorer understanding of the consequences of plagiarism than other students (M = 4.60, SD = 0.71), t(174.29) = 4.66, p < .001. No significant differences were found within the sample for gender, age group or study status (p > .05).
Student Perceptions on the University Using Text Matching Software
Students indicated that they understood why the university is using the software (M = 4.27, SD = 0.95) with no significant differences noted for gender, age group, study or NESB status (p > .05).
Students were mostly unsure as to whether the university should be using the text matching software (M = 3.33, SD = 1.12). Law students (M = 2.92, SD = 1.09) disagreed significantly more than other students (M = 3.63, SD = 1.05) about its use, t(635.40) = 8.78, p < .001 (corrected for violation of the assumption of homogeneity of variance).
Females (M = 3.43, SD = 1.06) felt more strongly than males (M = 3.20, SD = 1.18), t(710) = 2.66, p = .008, and NESB students (M = 3.61, SD = 1.06) felt more strongly than other students (M = 3.27, SD = 1.12) that the university should be using the software, t(706) = -3.18, p = .002. No significant differences were found within the sample for age group or for study status (p > .05).
The participants were mostly unsure as to whether text matching software is useful for students (M = 3.03, SD = 1.18). Law students (M = 2.44, SD = 1.06) disagreed significantly more than other students (M = 3.45, SD = 1.07), t(721) = 12.52, p < .001, and NESB students (M = 3.38, SD = 1.10) agreed more strongly than other students (M = 2.95, SD = 1.17) that the software would be useful to students, t(710) = -3.93, p< .001. No further significant differences between subgroups were found (p > .05).
Students were mostly unsure about their level of concern about how the university will use text matching software. (M = 3.25, SD = 1.29) NESB students (M = 3.47, SD = 1.25) were more concerned than other students (M = 3.20, SD = 1.30) as to how the university will use text matching software t(709) = -2.20, p = .028. Law students (M = 3.41, SD = 1.30) were also more concerned than other students (M = 3.14, SD = 1.27), t(719) = -2.80, p = .005. No significant differences were found within the sample for gender, age group or study status (p > .05).
Students were mostly unsure about whether they were unhappy about the use of text matching software. (M = 2.67, SD = 1.24). Law students (M = 3.06, SD = 1.24) were significantly unhappier than other students (M = 2.39, SD = 1.17), t(723) = -7.41, p < .001, and males (M = 2.86, SD = 1.22) were unhappier than females (M = 2.53, SD = 1.24), t(716) = -3.51, p < .001. No significant differences were found within the sample for gender, age, study or NESB status (p > 05).
Student Perceptions on How the Use of Text Matching Software Will Impact Them
The students were mostly unsure whether using the software would help them in their topic. (M = 2.66, SD = 1.22). Law students (M = 2.04, SD = 1.03) disagreed more that it would help them than other students (M = 3.12, SD = 1.15), t(722) = 13.06, p< .001. NESB students (M = 3.17, SD = 1.15) felt more strongly than other students (M = 2.55, SD = 1.20) that the software would assist them in the topic, t(212.26) = -5.44, p< .001 (corrected for violation of the assumption of homogeneity of variance). No significant differences were found between the other subgroups (p < .05).
Students were mostly unsure about how worried they were about the use of text matching software impacting on them personally (M = 2.61, SD = 1.31). NESB students (M = 3.02, SD = 1.33) were more worried than other students (M = 2.50, SD = 1.29), t(711) = -4.29, p < .001. No further significant differences were found between the other subgroups (p < .05).
57% (n = 416) of students offered responses to the open-ended questions. Overall, NESB students were under-represented (38.4% responded, compared with 61.2% of native English speakers, X2(1) = 22.77, p < .001), and law students were over-represented within these comments (65.8% responded compared to 50.7% of students from other faculties, X2(1) = 15.94, p < .001).
When asked to explain how the software would be useful to them, 228 students (31.3% of all students) responded. The most common response from students was that they felt the software would assist them to avoid unintentional plagiarism (n = 105; 46.1% of all valid responses). Comments included:
Students also indicated that using the software could assist them with learning (n = 67; 29.4%).
Law students were significantly less likely to suggest that the software would assist them with their learning (3.5% of all law students) than were students in other faculties (13.2%), X2(1) = 18.79, p<.001.
Students also felt that the use of the software would be useful for deterring intentional plagiarism (n=35; 15.4%) and for catching cheats (n = 15; 6.6%).
"Something needs to be done about students copying other people's work."
"Possibly useful for those students who aren't doing the work and who are getting the same marks as the hard working students."
"If some students are plagiarising work then this program will make it known. Means that other students are not disadvantaged by others cheating."
"It will at long last put everyone on the same level playing field, especially students that are renown for obtaining students' work from previous years."
Law students were significantly more likely to feel that the software would deter intentional plagiarism (7.4%) compared to other students (2.4%), X2(1) = 9.27, p = .002.
39% (n = 284) of students described concerns they had about the use of text matching software. The main concern related to a fear that plagiarism would be detected in their work when it was unintentional or coincidental (n = 94; 33.1%).
Law students were significantly more likely to be concerned about this (17.1%) than were students from other faculties (9.8%), X2(1) = 7.77, p = .005.
Students reported concerns about the company and the reliability of its software (n = 65; 22.9%), as well as the security and legality of the database that would retain their work (n = 68; 23.9%).
Law students were more concerned about their work being retained on the database (19.0%) than were other students (2.6%), X2(1) = 53.22, p < .001.
18 students (6.3%) indicated that they felt offended by the use of the software with responses ranging from defensiveness at the lack of trust to outright antagonism towards the university.
Significantly more law students (3.9%) than other students (1.2%) mentioned trust or emotive issues, X2(1) = 4.47, p = .034.
Students also indicated the use of the software would increase student anxiety (n = 17; 6.0%)
Flinders University has a strong commitment to ensuring student views are considered in decisions relating to teaching and learning and university governance (Flinders University 2004). This paper reinforces the importance of this commitment as students have demonstrated a great diversity of views and some strength of feelings about issues relating to the use of the Turnitin software.
It would seem from the quantitative data collected that students were quite unsure about what they thought of the use of the Turnitin software. This may be expected as the survey was conducted prior to the students having the opportunity to experience using the software. However, qualitative data revealed a more complex picture of student feelings, which were often expressed in strong terms.
Students identified a range of potential benefits of using the software.
Students focussed strongly on how the software would assist them to avoid plagiarism by allowing them to check or edit their work prior to submission. They felt the software would assist their learning in the development of academic skills through closer scrutiny of writing skills, essay structure and particularly referencing. Students also valued the deterrent effect the software would have on potential plagiarists and on the detection of cheats. Some authors support these potential gains, discussing detection software as a means for deterring both intentional and unintentional plagiarism (Carroll 2004; Martin 2004). Learning may occur by students having the ability to submit their work for checking (Martin 2004). The importance placed on academic conventions through the use of the software may also induce students to pay more attention to their academic writing skills, allowing new learning to take place (University of Sydney 2003; Martin 2004). McGowan (2002) believes students will benefit through the use of plagiarism detection tools by being more original and critical in their work. Students from this study support this view to some degree.
Martin (2004) considers the reputation of the university as an important rationale for use of plagiarism detection software by institutions. Of interest is that no students in our study identified this as a benefit to them. This may be one area where the agenda of an institution may not be fully understood by students and may require further explanation.
Student concerns were far more apparent in the written responses than responses outlining benefits, with some students even expressing strong concerns when asked for benefits. This could indicate that many students were feeling quite anxious about the use of the software and needed to express their misgivings.
Humes et al (2003) highlights the need for further investigation into student perceptions on privacy and intellectual property issues to see whether academic concerns are mirrored by students. Our data would support this. The range of issues causing student concern in this study included privacy and security of intellectual property and questions about the integrity and reliability of the Turnitin software.
Students were also worried that plagiarism would be detected in their work when it was unintentional or even coincidental. All of these issues raised by students could potentially contribute to feelings of anxiety and stress. Stress and anxiety as an outcome of using the software was identified specifically by a small number of students. Stress can impair student learning (McIlroy, Bunting et al. 2000; Cotton, Dollard et al. 2002) and stress levels caused by the use of text matching software needs to be seriously considered as a factor which could discourage student learning. Equally as important, some students indicated feelings of mistrust and animosity towards the university for using the software. For these students it implied that the university did not trust students and was taking a 'big brother' approach to dealing with issues of plagiarism. Although these feelings were expressed by a small number of students, it is important that they are not overlooked. Briggs (2003) discusses the moralistic attitude accompanying plagiarism and it's detection, with claims that students who feel their morals are questioned may be less likely to admit to the need for improving skills to reduce plagiarism in their work. Martin (2004) also fears negative consequences of mistrust on learning, drawing from experiences in the workplace where surveillance to reduce theft can reduce productivity in workers. Further attention needs to be paid to the extent of these feelings in the wider student body and its impact on learning, and to student attitudes to their studies and to the university in general.
Of particular note is that the benefits and costs of using the software which were identified by students differed across groups within the sample. Two groups emerged with significantly different views to the overall group of students - NESB and Law students.
NESB students in general felt they had a good knowledge of what plagiarism is and the consequences of plagiarism, but were less confident in these areas than other students in the sample. A survey of 138 international students in the UK revealed 23% did not know what plagiarism was and less than half felt they could successfully avoid plagiarism (King 2002). King's findings indicate poorer knowledge than in our study. Our results may indicate that NESB students have a greater anxiety over this issue.
This student group was also more supportive of the university using the software and felt generally that it would be useful to themselves and other students. However, they were more concerned than other students about how the university would use the software and its impact on them personally. These results would suggest that these students have a higher level of confidence in the software and its benefits to students. Results would also suggest a lower level of confidence in the interpretation of results or in their own ability to meet academic conventions. Written comments from these students did not disclose further information, however these students were less likely to write comments and so details of their views remain unknown.
The results of this study suggest that NESB students may benefit from increased support to allow them to better understand issues relating to the use of the Turnitin software and to feel more confident in their ability to apply academic conventions. With international student numbers increasing at universities (Australian Education International 2005), it is important to investigate these findings further, particularly the possible impact on the learning experience for these students.
Law students disagreed more strongly about the university using the software and were less likely to feel that it would be useful to students or themselves in their own work. They were more concerned about how the university would use the software and generally expressed that they were unhappier than other students about introduction of the software. In light of this it was surprising that these students did not express more concern about the use of the software on them personally. They were far more concerned than other students of the risk of unintentional or accidental plagiarism and of broader risks relating to security of their work and privacy issues. Law students were far more expressive in the written comments than other students with many comments expressing strong concerns about a range of issues relating to use of this software. Some comments expressed moral indignation ("This is the greatest threat to student liberty in my time at university, no matter how you try to sugar coat it.") and a clear feeling of broader implications of the use of the software ("I feel this is another step towards the de-personalisation of the academic process" "I'm not happy participating! Arbitrary outcomes, because it is computer software. Assumption of guilt of plagiarism. Privacy concerns. I don't really believe it is educational.")
These concerns and feelings indicate a lower level of trust in the university's ability to implement the program fairly, in the reliability of the software and in the software company itself. Responses would also suggest a lower level of confidence in the ability of the university staff to interpret results accurately. Management group members who oriented the Law students to the trial process were given strong impressions that students did not trust the process and felt let down by their lecturers and the university for allowing it to proceed. Comments directed to their lecturer included "I can't believe that you let this happen!" and "We're not kids. Why are you doing this?" (Green, 2005, personal communication). This level of mistrust may be reflective of the standards placed on law students regarding plagiarism, where an incident can seriously impact on a student's future career. It may also reflect the sceptical attitude the discipline may engender. These results contrast with results reported from the Law School at Staffordshire University where 82% of students supported the use of the Turnitin software and 64% felt more time should be spent on educating students about plagiarism. Staffordshire law students were more supportive in general of introducing the software (Chester 2001). The Staffordshire responses were gathered at the end of a trial period, and it will be of interest if our study finds similar results in the follow-up evaluation.
It is apparent from these results that all students and disciplines within the sample are not uniform in their views on the use of text matching software. This does not mean such software should not be used in some disciplines. Rather it may serve to caution universities to consider the individual nature of academic groups within the university and how preparatory programs may need to be individualised to meet the differing needs of students.
An obvious limitation of this study is that the sample of the trial is one of convenience and this limits generalisability to the university student population. In order to extend these finding to the university population a representative sample would be needed. Rennie and Rudland's (2003) work with medical students showed some clear changes in attitudes and behaviours towards academic dishonesty across year levels, which they attributed to experience and pressure to achieve results. The results presented here were limited due to the preliminary nature of the study, however using a more representative cross-section of students in any further investigations would allow more comparisons between year levels, across disciplines and within subgroups.
The study shows that students' initial perceptions of the use of electronic detection methods is a complex phenomenon, and one in which students' English speaking background and interdisciplinary differences play differing roles. These differences warrant further investigation. Even gender is indicated as having some role and this could be further investigated. As part of the evaluation a post-survey is being undertaken that will establish whether the initial views change after using the software. This may reveal further insights into the nature of difference between subgroups in the sample.
A key question is the significance of these student perceptions on the learning environment. A number of students identified benefits to the implementation and how these perceptions may impact on the learning environment could be further investigated. However, the study also found that there is some impact on anxiety and levels of trust. Martin (2004) hypothesises that the effect on classrooms may be similar to those found in workplaces where surveillance intended to stop stealing and other inappropriate behaviour may actually reduce productivity. Further investigation would reveal if this hypothesis has merit. If there is a risk of significant deleterious effects arising from the use of the software in certain groups, strategies will be needed to minimise these effects. In our own implementation, we emphasised an educative approach, rather than a punitive one. We also highlighted the well-considered introduction of the trial, including high-level Students' Association involvement. Nevertheless, there still was considerable variation in student views. Other approaches at mitigating these effects obviously need to be made.
Of course, as part of the evaluation of the trial, student attitude and its impact on the learning environment is only one consideration. We are also investigating the impact this trial has had on students' learning, including its effect on minimising plagiarism or collusion. This needs to be viewed in conjunction with the other aspects of the overall Academic Integrity Management Strategy, and decisions made as to whether using Turnitin software is the best way to provide the benefits that it promises.
Allan, G., L. Callagher, et al. (2005). Policies, Technology and Processes for Promoting Academic Integrity: Some Australasian Perspectives on Academic Integrity in the Internet Age. EDUCAUSE Australasia, Auckland, New Zealand.
Australian Education International. (2005). Recent Annual Statistics [Online].
Bedford St. Martin's. (2001). Technotes [Online].
Briggs, R. (2003). Shameless! Reconceiving the Problem of Plagiarism. Australian Universities' Review, 46(1): 19-23.
Carroll, J. (2002). A Handbook for Deterring Plagiarism in Higher Education. Oxford, Oxford Centre for Staff and Learning Development.
Carroll, J. (2004). Institutional issues in deterring, detecting and dealing with student plagiarism.
Chester, G. (2001). Pilot of Free-text Electronic Plagiarism Detection Software, JISC (Joint Information Systems Committee).
Cotton, S. J., M. F. Dollard, et al. (2002). Stress and Student Job Design: Satisfaction, Well-Being and Performance in University Students. International Journal of Stress Management, 9(3): 147-162.
Evans, S. and D. Green (2005). Plagiarism Detection Discussion Paper, Flinders University.
Flinders University. (2004) Education at Flinders [Online].
Flinders University. (2005). Academic Integrity Management Strategy [Online].
Gulik, C. and D. Tippin (2004). The University of Auckland's Turnitin.com trial - Semester 2, 2003: Evaluation Report, University of Auckland.
Humes, C., J. Stiffler, et al. (2003). Examining Anti-Plagiarism Software: Choosing the Right Tool: The CMC Anti-Plagiarism Software Survey, Claremont McKenna College.
James, R., C. McInnis, et al. (2002). Assessing Learning In Australian Universities, Centre for the Study of Higher Education, University of Melbourne.
King, P. B. (2002). Plagiarism; an informal investigation into international students' perceptions of the problem, The University of Birmingham.
Martin, B. (2004). Plagiarism; policy against cheating or policy for learning?, University of Wollongong.
McCabe, D. L., L. Klebe Trevino, et al. (2001). Cheating in Academic Institutions: A Decade of Research. Ethics & Behavior, 11(3): 219-232.
McGowan, U. (2002). Plagiarism or language development?, Adelaide University Learning and Teaching Development Unit.
McIlroy, D., B. Bunting, et al. (2000). An evaluation of the factor structure and predictive utility of a test anxiety scale with reference to students' past performance and personality indices. British Journal of Educational Psychology, 70: 17-32.
Rennie, S. C. and J. R. Rudland (2003). Differences in medical students' attitudes to academic misconduct and reported behaviour across the years - a questionnaire study. Journal of Medical Ethics, 29(2): 97-102.
Turnitin. (2005) Homepage of Turnitin [Online].
University of Sydney (2003). Plagiarism detection software, its use by universities, and student attitudes to cheating: a report for the University of Sydney Teaching and Learning Committee, University of Sydney.