Scaffolding Academic Integrity:
|
Table 1. Level of secondary source usage (%) within one cohort of students (2004; n =142) |
|
Secondary source usage |
Number of students (%) |
---|---|
<20 words |
30 (21.1) |
0-24% |
107 (75.4) |
25-49% |
2 (1.4) |
50-74% |
3 (2.1) |
75-100% |
0 (0) |
As Table 1 shows, only 5 students (3.5%) showed source usage above 25% and in each of these cases the system had indeed flagged a serious plagiarism problem (nine or more sentences unacknowledged) according to our criteria. The results of one of these students suggested either that he had copied from another student or that two students had colluded. Interviews with these students confirmed that intentional copying had taken place - that one student who understood the definitions of plagiarism had made a deliberate decision to act outside of these conventions - and so the action was classified as misconduct, and the student received zero for the assignment.
The four remaining students, all ESOL students, had included extensive passages in their text (nine or more sentences) that were copied without acknowledgement from another source. They had all marks for referencing (25% of the grade) and some of the marks for content (depending on the extent of the plagiarism) deducted from their grade and were issued with a warning. While it is important to be sensitive to cultural differences regarding use of source material, and while language barriers may impede the development of new skills, these students had been exposed to at least three hours of tuition on the correct use of sources in a New Zealand university as well as having available, in the study guide and text book, lengthy explanations of what was required. We therefore felt that while the errors technically fell into the category of misuse or misunderstanding, students could not convincingly claim no knowledge of what was required of them and so some penalties were imposed.
Initial results from the first stage of analysis, therefore, suggested that Turnitin had identified intentional plagiarists. Furthermore, the approach we had taken, combining the use of a plagiarism detection tool with constructive teaching on the topic of integrating secondary sources, seemed to have been effective: over 96% of students had produced work with less than 25% copy. While no valid comparisons were possible with earlier cohorts, the course coordinator felt confident that this represented a considerable decrease in the amount of plagiarism in the class.
However, closer analysis raised issues and doubts, suggesting that while a simple glance at the amount of secondary sources used might identify the intentional plagiarisers, a more in-depth approach is needed for identifying those students still struggling with the issues.
The bulk of our students (107 students - 75.3% of the cohort) fell into the category 0 - 24% source material. For step two of the analysis, we sorted these results into percentage copy categories (Table 2) and then inspected each report to identify how each student had used that source material.
Table 2. Breakdown of secondary source usage level (%) within the 0-24% category for one cohort of students in 2004 (n = 107) |
|
Secondary source usage |
Number of students (%) |
---|---|
1-5 |
58 (54.2) |
6-10 |
33 (30.8) |
11-15 |
12 (11.2) |
16-20 |
2 (1.9) |
21-24 |
2 (1.9) |
The inspection of each report showed that 18 (16%) of these 107 students demonstrated plagiarism problems. Nine students showed minor plagiarism problems (i.e. quotations not acknowledged as quotations, but treated as citations, or inclusion of less than six sentences of unacknowledged material) and nine demonstrated moderate problems (i.e. 6-8 sentences of unacknowledged material). None of these 18 students were ESOL students.
A key finding to note here is that, within this group, it was not possible to classify the extent of the problem using percentages alone. A student who included 24% secondary sources demonstrated no evidence of plagiarism; she had used quotations far too extensively for a scientific report, but her management and acknowledgement of sources was competent. By contrast, two students with only 8% secondary source material showed minor plagiarism problems. Close inspection of these reports suggested that these were cases of unintentional plagiarism, of misuse or misunderstanding rather than misconduct where, despite the educational programme, students had failed to distinguish between paraphrasing and quoting, or had failed to indicate a quotation in their notes prior to writing the essay.
What conclusions, then can be drawn from the complete analysis of this cohort of students? First, as stated Turnitin proved a simple and effective method of identifying misconduct, for locating the intentional plagiarisers who, for whatever reason, had chosen to contravene the standards of the academy. It had also effectively identified students exhibiting the more serious forms of misuse or misunderstanding.
The fuller analysis, however, also suggested that Turnitin could be a useful way of identifying students who were still struggling to understand and apply adequate referencing techniques - as long as the level of secondary source usage was not used as a method of identifying problems. Identification of problems requires extensive and time consuming analysis of reports, not a simple glance at which students fit into the categories offered by Turnitin.
With regards to the educational programme, it was concluded that while anecdotal evidence suggested that rates had fallen as a result of the combined strategy of detection and the educational programme, the rate of plagiarism could be further reduced if the educational programme were more fully developed.
The second part of the project was undertaken in 2005 in the same course but with a new cohort of first year students with a similar demographic to the 2004 cohort. Turnitin was retained as a form of detection, but the educational programme was adjusted in the light of part 1 to include an additional feature. This added feature was a "tutor clinic": a 10-15 minute individual meeting with the tutor, compulsory for every student, which was held one week before the submission of the essay. Students were required to bring a completed draft of the essay to the individual meeting, where the tutor discussed key aspects of their assignment and, in particular, looked at their referencing. Every student was asked "are you confident that you have not copied from an original source without referencing?" and "do you feel you understand the rules of APA referencing?" Tutors took notes from the meeting, which were shared with the student, and each student wrote a response to their tutor's comments about their essay. Students were expected to revise the essay in the light of the discussion and feedback.
171 students submitted their assignments through Turnitin in 2005. Again the results were sorted according to the Turnitin Similarity Index (Table 3) and then the green reports were sorted as for part 1 (Table 4).
Table 3. Level of secondary source usage (%) for one cohort of students ( n = 171) in 2005 |
|
Secondary source usage |
Number of students (%) |
---|---|
<20 words |
20 (11.7) |
0-24% |
147 (86.0) |
25-49% |
3 (1.8) |
50-74% |
1 (0.6) |
75-100% |
0 (0) |
Table 4. Breakdown of secondary source usage (%) within the 0-24% category for 2005 cohort of students (n = 147) |
|
Secondary source usage |
Number of students (%) |
---|---|
1-5 |
72 (49.0) |
6-10 |
48 (32.7) |
11-15 |
18 (12.2) |
16-20 |
4 (2.7) |
21-24 |
5 (3.4) |
This time only one student (with 52% secondary source usage) showed plagiarism rates high enough to warrant a serious reduction in grade, and other rates of plagiarism were considerably lower than in the earlier iteration (Table 5).
Table 5. Occurrence of students (n, %) within three levels of plagiarism for two cohorts in 2004 (n = 142) and 2005 (n = 171) for the same assignment |
||
Plagiarism level |
2004 |
2005 |
---|---|---|
Severe |
5 (3.5) |
1 (0.6) |
Moderate |
9 (6.3) |
5 (2.9) |
Minor |
9 (6.3) |
5 (2.9) |
Total |
23 (16.1) |
11 (6.4) |
Three significant points arise from these results. First, in this second part of the project, all the students who were identified as having plagiarism problems had attended a tutor clinic prior to submission of their assignment. However, all but one had attended with an incomplete assignment draft and had been identified by their tutors as not having completed in-line citations. Tutors discussed the issue with these students, who assured the tutor that they would attend to the needed in-line citations. The fact that they were identified in this way, and that they did not successfully complete inline citations, suggests an error not of understanding but of technique and process. Most writers of academic documents would complete in-line citations, and identify quotations, while writing the document, rather than adding them in after the draft was completed. This finding has implications for the teaching of referencing skills within academic courses.
Second, in this iteration, percentage copy was again not an adequate method of identifying plagiarism. Several students in the 16-20% range and the 21-24% range, and even one with 34% secondary source usage, did not show plagiarism problems. They did show problems of using quotations far too extensively than was appropriate for scientific writing, but they did not plagiarise. By contrast, three students in the 1-5% copy range showed minor plagiarism problems. Users of Turnitin need to be aware of the dangers of using percentage of secondary source usage as an indicator of plagiarism.
Third, none of the students in this cohort showing plagiarism problems of any kind were ESOL students. Instead, ESOL students were more likely to show over-extensive use of quotations. While this over-use of quotations may be something that needs to be addressed, it is interesting that the introduction of the tutor clinic, rather than the introduction of Turnitin, proved to be the decisive factor in almost eliminating plagiarism in our (admitted small) sample of ESOL students. Analysis of the assignments and tutor clinic sheets failed to clarify whether the tutor clinic was successful in addressing the issue because tutor-student discussion in tutor clinics addressed overseas students' understanding of referencing conventions or because the tutor clinics motivated them to undertake correct identification of source material. It would certainly be useful in future studies to clarify whether plagiarism is present or not in the drafts ESOL students take to the tutor clinic.
Like Soto and McGee (2004), we were disappointed that we were only able to reduce rather than eliminate plagiarism through the use of Turnitin as a detector combined with an educational programme. The underlying reasons for intentionally plagiarising under these circumstances, when students are aware that a detection device is being used, were not explored as part of this project . Interviews with intentional plagiarisers may be a possibility for further research.
However, our results confirm the findings of Vernon, Bigna and Smith (2001), Junion-Metz (2000), Carroll and Appleton (2001) and Malouff and Sims (1996) that multiple teaching strategies are required to reduce the plagiarism rates. Within this context we can make more specific suggestions.
The most important finding is that misuse of secondary sources is likely to be an outcome of poor processes, rather than a misunderstanding of the conventions. The second part of this study found that, with the possible exception of one student, all those with unacknowledged quotations in their assignments after the tutor clinic lacked reliable processes for integrating secondary sources into their work (i.e. adding in-line references once the document was completed, rather than as they wrote the document). This has implications for the way we teach how to reference secondary sources, suggesting that teachers should focus on process as well as on the placing of the citations and other technical aspects of referencing. It also gives a clear indication that simply referring students to handbooks on APA (or similar) referencing, or providing material in study guides which focuses on the conventions of referencing will not solve the errors made by these students. While the material on conventions is important, it needs to be supplemented either by active teaching on process, or by study material with examples and discussion of effective process, in terms of how to integrate secondary sources into an academic text. This issue of process is so closely linked to understanding the academic writing process in general, that it may be advisable to integrate such teaching into the context of a course on academic writing or writing within the disciplines where the pedagogy is focused on process rather than product (Hall, 2005).
Another key finding of this study is that a one-on-one tutor clinic in which students directly discuss their use of secondary sources with their tutor does have a considerable impact on plagiarism rates. We may speculate that the impact of the tutor clinic is two-fold. First, it has an educational impact: it allows tutors to spend time talking with students about referencing and addressing their immediate concerns; and since it is a two-way dialogue, it provides an opportunity for students to discuss any issues they are struggling with. Second, it acts as a deterrent. Research has shown (Cummings, 2002) that students are more likely to plagiarise if they are in big classes, have little contact with teaching staff, and feel anonymous. It may be easier for a student to hand in a flawed piece of work to a distant teacher with whom the student has no personal contact than to sit in front of someone who is talking about their work and say "there are no unacknowledged quotations in this assignment" when they know there is a problem. Tutors reported several incidents where students, when asked this question, looked worried and said they would check their work again, and then wrote on their tutor clinic sheets that they would address this issue.
Of particular significance was the impact of the tutor clinic on ESOL students. The literature on plagiarism as it pertains to ESOL students shows mixed results (Thompson and Williams, 1995; Hyland, 2001, Soto and McGee, 2004) but this study found that, after completing a tutor clinic, ESOL students were less likely to plagiarise than native speakers of English. This confirms the findings of Soto and McGee, whose work also focused on a large science class, rather than an English language class. They state:
At first, we expected ESL students to have more difficulty with technical text and therefore to plagiarise more than native speakers. However, most of the students who plagiarised in our study were native English speakers....We speculate that native speakers opted for expediency, perhaps thinking they could easily blend their own text with the plagiarised material....ESL students, perhaps through experience, feel uncomfortable blending their writing with improperly paraphrased technical text. (2004, pp. 45-6)
Our study's findings, that ESOL students were more likely to over-rely on quotations, certainly supports their suggestion. In our study, however, it is clear that the tutor clinic made a real difference to the likelihood of ESOL students plagiarising. It is not clear whether the tutor clinic, and facing up to a tutor, helped to motivate these students or whether the tutor clinic itself was a vital part of the tuition - or whether, as discussed above, these two factors worked together. Further research is needed to focus this issue. Ideally this research would take place in a class with a larger component of ESOL students - our sample is too small for us to draw confident conclusions.
Two findings are significant in relation to Turnitin as a tool for working with plagiarism. Because of the way the tool is designed, to categorise assignments according to percentage copy (and, in this study, without distinguishing between plagiarised material and acknowledged quotations), there is a danger of educators relying on percentage copy as an indicator of plagiarism. This study has shown that this is an inaccurate measure of plagiarism. Some students with only 5% copy showed signs of minor plagiarism, while students with 34% copy showed no evidence of plagiarism. Turnitin, then, should be used with caution, and as a complex rather than simple tool. Failure to recognise that Turnitin is a blunt instrument that can only be effective if used with care and subtlety could lead academic staff, especially those working from a detect and punish perspective only, to punishing students who are guilty only of excessive use of quotations (which is poor style rather than misconduct) while allowing students with substantial plagiarised copy, to remain undetected . Scrutiny of individual reports is required for an accurate assessment of plagiarism.
A second outcome of this study in relation to Turnitin is the finding that it is a valuable source of information for educators teaching into the field of academic writing skills, and in particular those teaching students how to use secondary source material. By examining reports in detail, teachers can identify the kinds of problems students are experiencing in employing secondary sources as part of their work, and then teach directly to those skills. The process is time consuming, since all Turnitin reports need to be scrutinised, but the colour coded reports make analysis by the teacher easy and clear, and allow students to see immediately, under direction from the teacher, where their problems lie.
What, then, are the implications of this study for tertiary institutions concerned with matters of academic integrity?
First, universities should be concerned with academic integrity, and hence with the detection of plagiarism, through their policies and procedures; and a detection device such as Turnitin is a useful device as part of a procedure in deterring students from engaging in intentional plagiarism. This study has shown that Turnitin is an effective system for identifying problems with plagiarism, but only if used with care and subtle observation. However, it is essential that its limitations be acknowledged. Some limitations are acknowledged and well documented, such as its inability to identify sources material from written materials or to identify the author (Carroll and Appleton, 2001). This study, however, suggests that its limitations are more pervasive than this, that its method of identifying plagiarism (through percentage copy and then categorisation) is flawed and may allow unacceptable amounts of plagiarism to pass undetected while flagging problems with assignments which do not contain plagiarism. If Turnitin is used, universities should be putting training in place, which includes reference to this design flaw and how academic teachers should handle the problems of using percentages alone. More research is needed on how to more efficiently manage the flaws in this detection system.
Second, universities need to integrate into their strategy and policies for dealing with plagiarism an educational strategy, based on sound pedagogical principles. These issues have been documented elsewhere, and include issues around assignment design and submission of drafts. However, this study has pointed to two useful elements of such an educational strategy. First, active teaching on use of secondary sources should include a focus on process, not just on conventions. Further research is required to ascertain whether the ideal place for such teaching is a dedicated academic writing course, or whether such instruction would be just as effective within a content course. And, second, the study suggests that a short, one-on-one tutor clinic can have a dramatic impact on rates of plagiarism, especially by ESOL students. Again, more research is needed with a larger sample size and broader demographic base. It would also be interesting to experiment with different forms of one-on-one tuition - for example, it would be useful to ascertain whether voluntary or compulsory meetings with a learning advisor (in a Student Learning Centre or Learning and Language Service) would have a similar impact to meeting with a course tutor.
Earlier in this paper we described Turnitin as a "blunt instrument that can only be effective if used with care and subtlety". We might extend this metaphor to the broader context of dealing with perceived increases in the rates of plagiarism. Plagiarism is a complex matter, difficult to define (Price, 2002), and a consequence of a range of student attitudes and skills. This study has shown that both detection and a multi-focused education approach should be a part of the strategy for dealing with this complex issue, and that both aspects of this strategy need to be designed and handled with care and discrimination, as an acknowledgement of this complexity.
We would like to acknowledge the tutor team for the Communication in Sciences course for their engagement in, and discussion of, the tutor clinics: Elizabeth Gray, Joy Green, Louise Folster, Judith Moore, Megan Stace-Davies, and Marie Silverwood.
Anderson, R D 1995, 'Science, technology and education: the challenge to education', Agricultural Science, vol. 8, pp. 37-40.
Angelova, M & Riazantseva, A 1999, 'If you don't tell me, how can I know?: A case study of four international students learning to write the U.S. way', Written Communication, vol. 16, pp. 491-525.
Ashworth, P & Bannister, P 1997, 'Guilty in whose eyes?: University students' perceptions of cheating
and plagiarism in academic work and assessment'. Studies in Higher Education [Online], vol. 22, pp. 187-204.
Available: Academic Search Premier [Accessed 2004, September 22].
Brown, B S 1995, 'The academic ethics of graduate business students: A survey', Journal of Education for Business, Jan/Feb, pp.151-156.
Cameron, J 2004, 'George Bush cheats so why can't I'? Chaff, vol. 70, no.18, p.16.
Carroll, J and Appleton, J 2001, Plagiarism, A good practice guide [Online].
Available: http://www.jisc.ac.uk/uploaded_documents/brookes.pdf [Accessed 2004, November 28].
Cummings, R, Maddux, C D, Harlow, S, & Dyas, L 2002, 'Academic misconduct in undergraduate teacher education students and its relationship to their principled moral reasoning', Journal of Instructional Psychology, vol. 29, pp. 286-296.
Franklyn-Stokes, A & Newstead, S E 1995, 'Undergraduate cheating: Who does what and why'? Studies in Higher Education, vol. 20, no. 2, pp.159-172.
Fulwood, S 2003, 'Plagiarism playing by the rules: in the academic world, in music, and even in church, what constitutes plagiarism is under new scrutiny after journalism's wake up call', Black issues book review, Sept - Oct.
Graham, M A, Monday, J, O'Brien, K, & Steffan, S 1994, 'Cheating at small colleges: An examination of student and faculty attitudes and behaviours', Journal of College Student Development, vol. 35, pp. 255-260.
Gray, F E, Emerson, L, & MacKay, B R (in press, 2005). 'Meeting the Demands of the Workplace: Science Students and Written Skills', Journal of Science, Education and Technology.
Hall, J. (2005). Plagiarism across the curriculum: How academic communities can meet the challenge of the undocumented writer. In Across the
Disciplines: Academic Perspectives on Language [Online].
Available: http://wac.colostate.edu/atd/articles/hall2005.cfm [Accessed 2005, August 30].
Hyland, F 2001, 'Dealing with plagiarism when giving feedback', ELT Journal, vol. 55, no. 4, pp. 375 - 381.
Junion-Metz, G 2000, 'The e-plagiarism plague', School Library Journal, vol. 46, no.9, pp.43 - 44.
Malouff, J M & Sims, R L 1996, 'Applying an employee-motivation model to prevent student plagiarism', Journal of Education Business, vol. 72, pp. 58 -61.
Park, C 2004, 'Rebels without a clause: Towards an institutional framework for dealing with plagiarism by students', Journal of Further and Higher Education, vol. 28, no. 3, pp. 291 -306.
Price, M 2002, 'Beyond "Gotcha!": Situating plagiarism in policy and pedagogy', College Composition and Communication, vol. 54, no.1, pp. 88 - 114.
Rennie, S C & Crosby, J R 2001, 'Are "tomorrow's doctors" honest? Questionnaire study exploring medical students' attitudes and reported behaviour on academic misconduct', British Medical Journal, vol. 322, pp. 274-5.
Soto, J G & McGee, E 2004, 'Plagiarism avoidance', Journal of College science Teaching, vol. 33, no. 7, pp. 42-48.
Thompson, L C & Williams, P G 1995, 'But I changed three words! Plagiarism in the ESL classroom', The Clearing House, vol. 69, no. 1, pp. 27 - 30.
Vernon, R F, Bigna, S & Smith, M L 2001, 'Plagiarism and the web', Journal of Social Work Education, vol. 37, no. 1, pp. 193 - 196.