back to contents page

In 2003, the Faculty of Health & Behavioural Sciences at our university offered a series of postgraduate research seminars. The series of seminars, selected by a three-person faculty team, received a mixed reaction - some seminars attracted a large proportion of the students whereas others were poorly attended. Thus, it was decided to continue the seminar series in 2004, but to undertake some formative evaluation to determine the students' preferences for the seminar topics. It was important to gain input from as many students as possible, and to allow maximum latitude for responses. It was decided to conduct a two-stage evaluation process: nominal groups with a subset of postgraduate research students; followed by a survey of all students based on the groups. It was apparent from the results of this study that the students' preferences for seminar content were quite dissimilar to those anticipated by academic staff. This study demonstrates the advantages and disadvantages of the nominal group technique in investigating information and skill needs among postgraduate research students.

Introduction

In January 2003, it was decided to introduce a series of seminars for research students in the Faculty of Health & Behavioural Sciences (FHBS) at our university. The FHBS includes four departments: Biomedical Sciences, Nursing, Psychology, and Public Health. The series of seminars, selected by a three-person team, received a mixed reaction - some of the seminars attracted a large proportion of the students whereas others were poorly attended. Thus, it was decided to undertake a small-scale formative evaluation to determine the appropriate topics for the 2004 seminars.

The 2003 seminars

The team responsible for the development of the seminars included the author, the Dean of the Faculty, and the Head of the Department of Nursing. The seminar topics were chosen during a brainstorming session between the three team members following informal discussions with current research students and a review of seminars currently offered elsewhere in the university (to minimize duplication). Additionally, it was decided to include two open forum sessions where the research students could discuss any issues of interest to them. The 2003 seminar series is summarized in Table 1.

19 March	Your thesis — Planning your thesis research and making it a part of your life
30 April	Training and development — what’s available at UOW and how to find it
28 May	Critical appraisal (evaluating research articles)
02 July	Writing journal articles (& choosing target journals)
06 August	Forum — open to any issues students want to discuss (with faculty representative from each department)
10 September	Career planning — deciding where you want to go and knowing how to get there
15 October	Writing a conference abstract & presenting your research at a conference
19 November	Writing ethics applications
17 December	Writing fellowship applications

The purpose of the seminars was two-fold: first, to provide students with additional knowledge and skills to assist them in obtaining their research degree; second, to provide them with an opportunity to interact and share ideas with other research students. An important component of the networking was seen to be interaction between students across the four departments and at varying stages of completion of the research degree. For this reason, it was decided to make the seminar topics generic to the FHBS with input from the four departments to ensure that department-specific issues were addressed. Wherever possible, this was achieved by having a senior faculty member from each department attend and participate in the session.

Attendance at the 2003 seminars was considerably lower than expected, averaging less than 10 students per seminar, and there was considerable variation in the number of attendees across the different topics. For example, the session on training and development attracted only 5 students, but three times as many attended the session on thesis planning, and the session on writing journal articles was standing-room only.

Planning for the 2004 seminars

Importantly, the feedback from attendees at all of the 2003 sessions was extremely positive, with informal discussion with the students after each session confirming that they gained valuable information from attending.

Therefore, it was decided to continue the seminar series in 2004, but to undertake some formative evaluation to determine the students' preferences for the seminar topics. The logistics of obtaining the views of a diverse body of students, many of whom are part-time and study off-campus, led to the obvious conclusion that a survey was the only viable method for collecting the evaluation data. However, the concern was that in developing such a survey, the questions and response options would be limited by preconceived ideas of "appropriate" seminar topics.

For this reason, it was decided to conduct a two-stage evaluation process. First, two nominal groups with a subset of postgraduate research students to develop the questions and response options for the survey. Second, a survey of all FHBS postgraduate research students.

The Nominal Group Technique (NGT)

The Nominal Group Technique (NGT) is designed to facilitate collaborative and democratic decision making (Delbecq, Van de Ven & Gustafson 1975). In its "pure" form the NGT consists of six steps:

1. Individual generation of ideas
   
   
2. Recording of all participants' ideas (in a round-robin format)
   
   
3. Group discussion of all generated ideas (to organize the list and remove duplications)
   
   
4. Preliminary vote to select the most important ideas
   
   
5. Group discussion of the vote outcomes (including additions and further merging of overlaps)
   
   
6. Final voting on the priority of items
   
   

However, as will be discussed in the following section, there are many variants on the NGT when it is used in practice.

Use of the NGT in education research

The NGT has been used in educational settings to investigate a wide range of topics, including characteristics of expert associate teachers (Boudreau 2000), homework communication strategies in remedial education (Nelson et al. 2002), primary teachers' beliefs about families' competence to contribute knowledge to their child's education (Moseman 2003), evaluation of one-to-one teaching in general practice (Duggan & Cox 1999), selecting topics for a BSN leadership course for Registered Nurses (RNs) (Waddell & Stephens 2000), and occupational health nurses' perceptions of their education and training needs for the "new public health agenda" (Bamford & Warder 2001).

As suggested above, there are numerous variations in the application of the Nominal Group Technique in practice.

The Nelson et al. (2002) study did not use the NGT to generate ideas, but rather to rank and evaluate the effectiveness and feasibility of 44 potential strategies generated in an earlier focus group study. As described by the authors, the NGT started at the third step (group discussion of each idea) and was preceded by all participants being provided with summaries of the previous studies (i.e., the list of potential strategies from which to choose). The result of the process in this case was agreement on 14 strategies (of the original 44) which were seen to be the most effective and feasible.

The Duggan and Cox (1999) study combined NGT and focus group research to develop an evaluative tool for assessing the effectiveness of trainers of general practitioner registrars. The NGT commenced with individual generation of ideas, with trainers and registrars writing down the characteristics of good teaching sessions. The NGT was then modified to utilize a "snowball" technique (Duggan & Cox 1999), with participants discussing their lists in pairs and adjusting them accordingly, then the pairs combining into increasingly larger groups repeatedly undergoing the same process, and so on until the whole group combined with one list. Finally, each individual marked on their own copy of the comprehensive list the five characteristics they personally felt to be the most important.

The Waddell and Stephens (2000) study used the NGT to involve RNs in the selection of topics to be covered in their course, following a similar process to Duggan and Cox (1999). In this study, the RNs were asked to individually list their three most important topics for the course, then to share their list in pairs, and again in a group of four, resulting in a composite list for each group of four RNs. These lists were then posted on a wall, and students were given coloured dots with which to "vote" for topics. The final stage was a group discussion to reach consensus on the final topics for the course. It is interesting that Waddell and Stephens (2000) emphasise that the key advantage of the NGT is the "incorporation of the ideas and desires of all the learners, regardless of their levels of assertiveness or extroversion," yet students did not get the opportunity to present their ideas until after the four-person teams had reached "consensus" (which could potentially result in ideas from the less assertive students being excluded from the consensus lists).

Advantages and disadvantages of using NGT in education research

The primary advantage of the NGT over other strategies is the enhanced opportunity for all participants to contribute ideas and to minimize the domination of the process by more confident or outspoken individuals (Vella, Goldfrad, Rowan, Bion & Black 2000). Other advantages (Nelson, Jayanthi, Brittain, Epstein & Bursuck 2002; de Ruyter 1996; Brahm & Kleiner 1996; Fox 1989; Butler & Howell 1980; Moore 1987; Delbecq, Van de Ven & Gustafson 1975) include:

• The generation of a greater number of ideas than other group processes.
  
  
• The generation of more creative ideas than other group processes.
  
  
• The ease of interpreting the results (as ideas are generated, voted on/ranked, and evaluated at the session itself).
  
  
• A greater sense of accomplishment for members (as the results are available immediately after the session).
  
  
• The minimal resource requirements (a venue, facilitator, whiteboard, paper and pens).
  
  
• The comparatively efficient use of time.
  
  

However, it is important to recognise the disadvantages, or limitations, of nominal groups (de Ruyter 1996; Brahm & Kleiner 1996; Fox 1989; Butler & Howell 1980; Moore 1987; Delbecq, Van de Ven & Gustafson 1975). These include:

• The limited number of topics and issues that can be covered (tend to be single-topic sessions).
  
  
• The limitation of idea generation to the meeting itself (i.e., no opportunity for participants to think about the issue in depth and generate additional ideas in their own time.
  
  
• The need for participants to feel comfortable with, and remain within, a very structured group process.
  
  
• The lack of anonymity, which may limit participants' willingness to express their views.
  
  
• The necessity for all members to be capable of, and comfortable with, expressing their ideas in writing and then communicating them verbally to the group.
  
  
• The time commitment required from participants, and the necessity for them to attend a specific location at a given time, which may limit participant numbers.
  
  
• The lack of generalizability of the results to the wider population due to the specific characteristics of the participants (both in terms of who is nominated to attend, and who agrees to participate.
  
  
• The limited nature of the data (i.e, in terms of number of respondents) often requires a follow-up survey or other quantitative methodology prior to making final decisions about an issue.
  
  

Stage 1 - The nominal groups

It was decided to conduct two nominal groups: one with students at the beginning of their degree, to determine their perceived information/development needs; and one with recently completed students to determine both the information/development experiences they had found most useful during their degree, and the information/development experiences they felt they had missed out on.

Each of the four department heads was asked to nominate four students - two who had recently completed their research degree (or were about to submit) and two who had commenced their research degree in 2003. These 16 students were contacted by email to ask for their participation in what was described as a small-group discussion for the planning of the 2004 seminar series. Six of the eight students in each group agreed to attend the session; reasons for refusal were being overseas or otherwise unable to get to the campus (two students) and being too busy with the thesis submission process (two students). An additional student cancelled on the day of the session due to family illness, leaving a final sample of five early-stage and six late-stage students. All of the participating students were given a free movie pass as a thank you, but they were not told of this until after they had agreed to participate.

In order to retain the integrity of the NGT process - particularly in relation to ensuring input from all of the participants - stage one was designed to closely follow the initial steps of the NGT process. The later stages were altered to incorporate the survey in stage two of the study.

On arrival at the venue, participants were welcomed and given the opportunity to introduce themselves to the group (i.e., their name, the department they were from, the degree they were undertaking, and how long they had been enrolled in their degree).

This was followed by an informal discussion about the 2003 seminars, focusing on how (or whether) the participants were informed of the dates and topics for the seminars, and the logistics of the seminars (e.g., appropriateness of the timeslot chosen). The nominal groups process was then explained to the participants, and they were given the question to be considered: "What topics do you think should be included in the 2004 seminar series." Participants were encouraged to think as broadly as possible about the information or skills they felt they needed to complete their degree, and to include topics which may not lend themselves to the current 90-minute seminar but could be provided by other means (e.g., longer sessions or externally-provided seminars). This was seen as important to ensure that topics of interest to the students were not excluded simply because they did not appear amenable to the 90-minute format.

Individual generation of ideas - participants were provided with a sheet of paper and a pen and given 15 minutes to independently record all of the topics they would like to have included in the 2004 seminar series, or as an adjunct to the seminar series.

Recording of all participants' ideas - after the 15 minutes writing time, participants were invited to volunteer one item from their list in turn (i.e., in a round-robin format) until all of their ideas were listed on the whiteboard.

Group discussion of all generated ideas - once all of the ideas were listed on the whiteboard, they were discussed by the group to organize the list and remove duplications.

Preliminary vote to select the most important ideas - participants were provided with a sheet of paper and asked to list the three topics from the board which they thought were the most important. Group two (late-stage students) were asked to make two lists: one listing the three topics they thought would be most valuable to them now; and a second listing the three topics they thought would have been most valuable to them earlier in their research degree.

Group discussions of the vote outcomes were not held, nor was there final voting on the priority of items, as the intent was to incorporate all of the items which received one or more votes in the survey to be distributed to all postgraduate research students (phase two).

Results

Session 1 - early-stage students:

After removal of duplications and combination of overlaps, a total of 16 seminar topics were nominated by the group. The topics and the number of votes they received are listed in Table 2.

Session 2 - late-stage students:

After removal of duplications and combination of overlaps, a total of 22 seminar topics were nominated by the group. The topics and the number of votes they received, both for "sessions I would like now" and "session I wish I had at the beginning" are listed in Table 3.

Table 2: Topics and votes from Session 1 (early-stage students)

How to work with your supervisor (stress, motivation, conflict) and how to get help if there’s a problem	3
Overview of a thesis — length, structure	2
Job opportunities	2
Advanced English skills for thesis writing	2
General introduction (e.g., email, SOLS) — including network opportunity	1
Funding sources — conferences, trips, etc. (inc. external funding)	1
Postgraduate funds — how much do I have and how much control do I have over how its spent?	1
How to apply for research grants	1
New and updated software	1
What is an ideal PhD and how do you achieve it? (inc. timing, management, publications)	1
Overall picture of the department and faculty (e.g., who to contact, how to get copy cards, etc.)	-
Writing skills — proposal, literature review	-
Experiences of late-stage postgrad students (inc. time management)	-
Library information	-
Getting extensions on scholarships	-
OH&S requirements for collection of data from subjects (e.g., sterilizing of equipment)	-
TOTAL	15

Table 3: Topics and votes from Session 2 (late-stage students)

	EARLIER	NOW
Publishing in journals/academic writing	-	5
Conference presentations	-	4
How to start thesis writing (when, where, how) — inc. staying on track & writer’s block	3	3
Jobs after your thesis	1	2
Writing grant applications	-	2
Project management (inc. setting milestones and measuring achievement)	3	1
‘Selling your thesis’ (punchy writing)	-	1
Supervisor-student relationships (working styles, etc.)	3	-
Rights and obligations of PhD students (inc. resources)	2	-
Conceptual frameworks for thesis	1	-
Library skills — advanced database searching techniques	1	-
Processes for thesis submission and examination	1	-
Lifestyle/emotional management of PhD	1	-
Software packages (what there is and how to use them)	1	-
Stats consulting service — when can you see them & what can they do	1	-
Purpose of a PhD — remind me why I’m doing this	1	-
Time management	-	-
Academic writing	-	-
Intellectual property	-	-
Data management — logistics (e.g., backups) and university requirements	-	-
Stats refresher course	-	-
Study participant management (recruitment, retention etc.)	-	-
TOTAL	19	18

Stage 2 - The survey

Method

A questionnaire was designed, based on the outcomes of the nominal groups, to canvass the views of all HBS postgraduate research students. The questionnaire was posted on the faculty website, and emails were sent to all postgraduate research students asking them to take part in the survey. This email included detailed instructions for saving, completing, and returning the questionnaire. A reminder email was sent three days later.

The questionnaire contained two main sections:

Logistics of the seminars:

• Preferred length for the seminars (response options: one hour; 1.5 hours; two hours; no preference)
  
  
• Preferred distribution for the seminars (response options: session time only; in session and out of session; no preference)
  
  
• Preferred time of day for the seminars (response options: lunchtime, 12.00 start; early afternoon, 2.30 start; late afternoon, 4.30 start; no preference)
  
  
• Preferred day for the seminars (response options: Monday; Wednesday; Friday; no preference)
  
  

Seminar topics:

The seminar topics generated by the nominal groups (phase one) were listed, and respondents were asked to indicate their likelihood of attending each seminar ("assuming the seminars were held at a time suitable for you"). For ease of completion, the seminar topics were grouped into four categories, based on the discussion from the nominal groups: The research degree; Writing your thesis; Research skills; Administration and resources. Respondents were also invited to nominate any additional seminar topics they would like to see offered which were not included in the list.

Results

A total of 25 usable surveys were returned. Eleven of these were from students in the Biomedical Sciences department; with the remainder from Public Health (7), Psychology (6), and Nursing (1). Fourteen of the respondents (56%) were enrolled full-time and 11 part-time; 22 (88%) were enrolled as domestic students and three as overseas students. The majority of the respondents (64%) had been enrolled for less than three years, based on year of first enrolment (i.e., six enrolled in 2003, five in 2002, five in 2001, three in 2000, four in 1999, one in 1998, and one in 1993 with an extended leave of absence). Sixteen of the respondents (64%) had completed their undergraduate studies at the University of Wollongong, five at another Australian university, and four overseas.

Logistics of the seminars

Length: There was a clear preference for short seminar length, with 14 respondents (56%) expressing a preference for one-hour seminars, six for 90 minutes, one for two hours, and the remainder no preference.

Distribution: Opinion was divided on whether the seminars should be held only during session (six respondents), or in and out of session (eight respondents) - with eleven respondents (44%) stating that they had no preference either way.

Time: Opinion was also divided in relation to the best time of day for the seminars. Approximately even numbers of respondents stated a preference for seminars to be held at 12.30 (eight), 2.30 (seven), and 4.30pm (seven), with the remaining three not expressing a preference. As would be expected, part-time students (who are generally employed) were more likely to express a preference for the later afternoon seminars (six part-timers versus one full-timer).

Day: One-third (eight) of the respondents stated that they would prefer the seminars to be held on Wednesdays (as they were in 2003), six that they prefer Fridays, and 11 had no preference. Interestingly, none of the respondents expressed a preference for Monday seminars.

Seminar topics

For each of the listed topics respondents were asked whether they would "definitely attend," "probably attend," "possibly attend," or "definitely not attend." Table 4 lists the seminar topics in order of popularity (defined as the greatest proportion of students expressing an intention to definitely attend). A mean score for the total sample was calculated for each topic (shown in column 2), where 3 equals "definitely attend" and 0 equals "definitely not attend." Column three shows the actual number of respondents who stated that they would definitely attend each of the seminars. As can be seen from the table, these two measures result in a similar prioritization of topics, although there are some small differences due to the distribution across the other response options.

There were a total of six topics for which more than half of the respondents reported that they would definitely attend a seminar. These were: process for submission & examination (17); writing skills (selling thesis) (16); overview of thesis structure (15); how to start thesis writing (14); what is an ideal PhD (14); and writing grant applications (14). Importantly, the 2003 seminar series included only one of these topics (writing grant applications). There were a further four topics for which ten or more (i.e., 40% or more) of the respondents said they would definitely attend: developing conceptual framework (12); job opportunities & career planning (11); publishing in journals (11); and time management (10). Two of these four topics were included in the 2003 seminar series (career planning and publishing in journals).

Table 4: Preferences for the seminar topics

	Mean	Would definitely attend N (%)
process for submission & examination	2.6	17 (68%)
overview of thesis structure	2.6	15 (60%)
how to start thesis writing	2.4	14 (56%)
writing skills (selling thesis)	2.4	16 (64%)
developing conceptual framework	2.3	12 (48%)
what is an ideal PhD	2.2	14 (56%)
job opportunities & career planning	2.2	11 (44%)
publishing in journals	2.2	11 (44%)
time management	2.1	10 (40%)
writing grant applications	2.1	14 (56%)
project management	2.0	8 (32%)
how to work with your supervisor	2.0	9 (36%)
statistics refresher course	2.0	9 (36%)
conference presentations	1.8	8 (32%)
software packages	1.8	7 (28%)
intellectual property	1.7	5 (20%)
lifestyle/emotional management of PhD	1.6	5 (20%)
advanced library skills	1.6	6 (24%)
data management logistics	1.5	2 (8%)
rights & obligations of PhD students	1.4	5 (20%)
statistical consulting service	1.4	5 (20%)
study participant management	1.3	1 (4%)
picture of dept. and faculty	0.7	2 (8%)
thesis writing for NESB	0.6	4 (16%)
general introduction	0.5	1 (4%)

The following section reviews the responses to the proposed seminar topics by the four broad focii under which the seminars were categorized (the seminar titles and categories are exactly as they appeared on the survey form). Given the small sample sizes, it was not possible to conduct any meaningful statistical analyses of differences between sub-groups of respondents. However, respondents were dichotimised into: new (commenced 2003) versus old (commenced prior to 2003); and UOW undergraduate versus non-UOW undergraduate degree. Where the data identified apparent differences between the sub-groups, these are discussed below.

1. The research degree

Four of the seven topics in this group were included in the respondents' top ten.

What is an ideal PhD and how do you achieve it? (inc. duration, time management, publications): Fourteen (56%) of the respondents reported that they would definitely attend this seminar, and a further five that they would probably attend. Only two respondents reported that they would definitely not attend a seminar on this topic. All six of the "new" students reported that they would definitely attend (compared to eight of the 19 "old" students).

Time management (inc. experiences of late-stage postgrad students): Ten (40%) of the respondents reported that they would definitely attend this seminar, and a further nine that they would probably attend. Only one respondent reported that they would definitely not attend a seminar on this topic.

Processes for thesis submission and examination: Seventeen (68%) of the respondents reported that they would definitely attend this seminar, and a further six that they would probably attend. None of the respondents reported that they would definitely not attend a seminar on this topic.

Job opportunities and career planning: Eleven (44%) of the respondents reported that they would definitely attend this seminar, and a further nine that they would probably attend. Again, none of the respondents reported that they would definitely not attend a seminar on this topic. Five of the six "new" students reported that they would definitely attend (compared to six of the 19 "old" students).

2. Writing your thesis

Four of the five topics in this group were included in the respondents' top ten.

Overview of a thesis - length, structure: Fifteen (60%) of the respondents reported that they would definitely attend this seminar, and a further nine that they would probably attend. None of the respondents reported that they would definitely not attend a seminar on this topic.

Developing/choosing a conceptual framework for your thesis: Twelve (48%) of the respondents reported that they would definitely attend this seminar, and a further eight that they would probably attend. Only two respondents reported that they would definitely not attend a seminar on this topic.

How to start thesis writing (when, where, how) - inc. staying on track & writer's block: Fourteen (56%) of the respondents reported that they would definitely attend this seminar, and a further five that they would probably attend. None of the respondents reported that they would definitely not attend a seminar on this topic.

Writing skills (proposal, literature review, selling your thesis with 'punchy' writing): Sixteen (64%) of the respondents reported that they would definitely attend this seminar, and a further four that they would probably attend. Only two respondents reported that they would definitely not attend a seminar on this topic.

3. Research skills

Two of the seven topics in this group were included in the respondents' top ten.

Publishing in journals (inc. how is journal writing different to thesis writing): 11 (44%) of the respondents reported that they would definitely attend this seminar, and a further nine that they would probably attend. Only two respondents reported that they would definitely not attend a seminar on this topic.

Writing research grant applications: 14 (56%) of the respondents reported that they would definitely attend this seminar, and a further two that they would probably attend. Only two respondents reported that they would definitely not attend a seminar on this topic.

4. Administration & resources

None of the topics in this group were included in the respondents' top ten.

Discussion

It is apparent from the results of this study that the students' preferences for seminar content are quite dissimilar to those anticipated by the development team. Only three of the students' top ten topics had been offered in the 2003 seminar series and a number of the preferred topics were outside the content areas that were under consideration for the 2004 series. It is evident from the disparities between the list generated by academic staff (i.e., the 2003 seminar series) and the list generated by the students, that staff are perhaps not as aware of the students' needs and preferences as was previously thought.

There are several possible explanations for these disparities and it is important to recognise these in making decisions about the seminar topics to be addressed. The first explanation is that the development team was incorrect in their assumptions about the topics that would be of value to students - a reasonable supposition given that the team's views were based on their own experiences and feedback from the research students they had supervised in their own areas of expertise. The second explanation is that the students are not aware of the gaps in their knowledge and the areas in which they need assistance - again a reasonable supposition as it is common ground that people often "don't know what they don't know." The truth is probably a combination of these two explanations, and thus decisions about future seminar topics need to incorporate both the students' perspectives of their needs and interests and the academics' perspectives on the knowledge and skill gaps of the institution's research students.

As a result of this process, a series of recommendations were made - and subsequently acted upon - in relation to the seminar logistics and topics. As the students were deemed to be the best judges of their availability to attend the sessions on specific days and times, the 2004 seminars were scheduled in line with the students' responses. In terms of the specific topics to be covered, a list was developed based on the survey responses, feedback from previous seminars (particularly attendance levels), and consultation with academic staff.

The ability to draw conclusions from this process reflects the advantages and disadvantages of nominal group techniques discussed in the introduction to this paper.

The use of NGT enabled us to generate a greater number of ideas for potential seminar topics than were previously considered, and the topics were considerably more diverse than had been anticipated. Further, the final stage of the NGT (voting on ideas and collating the votes) enabled us to end the session with a comprehensive, but not overly long, list of seminar topics for inclusion in the survey. Informal feedback from the participants subsequent to the session confirmed that they found the process interesting and were satisfied with the outcomes and with their feelings of involvement in the process.

However, the limits on the number of issues that can be covered meant that the study was able to focus only on the seminar topics themselves, and not on other issues that may influence students' participation in the seminar series. It is acknowledged that there are a number of other factors that could influence students' attendance at the seminars that were not specifically addressed in this study.

These include such things as distance from campus and ease of transport to the seminars; career and family responsibilities; perceptions of "belonging" to the Faculty as opposed to identifying solely with the department; and existing attitudes about the value of attending groups seminars.

Other limitations of the study due to the nature of NGT research include:

• The lack of opportunity for participants to think at length about knowledge and skills they would like to develop in the context of their degree due to the need to provide immediate feedback at the session
  
  
• The possibility that participants may not have expressed all of their perceived needs due to the lack of anonymity or to any discomfort with the formal group process
  
  
• The potential for some participants' responses to be limited by their ability to express their thoughts in writing and/or verbally (particularly some of the overseas students)
  
  
• The lack of generalizability of the results to the wider student population as the participants were nominated by their department heads (thus possible selection bias) and attended voluntarily (thus potential response bias)
  
  

In conclusion, this study demonstrates the value and the limitations of the nominal group technique in investigating information and skill needs among postgraduate research students. The use of the nominal group technique enabled us to obtain an extensive and varied list of seminar topics - which we were then able to use as the basis for a survey of postgraduate students. Further, through this process it became evident that steps needed be taken to address the needs of a sub-group of students for information on the remaining topics: including providing students with information on where such seminars are offered outside of the faculty series (e.g., the library, the Student Learning Centre); and ensuring that written information is provided in the student handbooks. However, a number of questions were left unanswered by the process, including other reasons for (non) attendance, attitudes towards the seminars in general, and other options for delivery of research skills training. Future research could address these issues, as well as surveying students who have completed degrees to obtain retrospective opinions on the value of seminar topics.

References

Bamford, M & Warder, J (2001). Occupational health nurses' perceptions of their education and training needs to meet the new public health agenda using the nominal group technique, International Journal of Lifelong Education 20(4), 314-25.

Brahm, C & Kleiner BH (1996). Advantages and disadvantages of group decision-making approaches, Team Performance Management: An International Journal 2(1), 30-35.

Boudreau, P (2000). L’expertise d’un enseignant associe, McGill Journal of Education 35(1), 53-70. [English abstract only]

Butler, LM & Howell, RE (1980). Coping with Growth: Community Needs Assessment Techniques. Washington State University. WREP 44.

de Ruyter, K (1996). Focus versus nominal group interviews: a comparative analysis, Marketing Intelligence & Planning 14(6), 44-50.

Delbecq, AL, Van de Ven, AH & Gustafson, D (1975). Group techniques for program planning: A guide to nominal group and Delphi process. Glenview IL: Scott Foresman.

Duggan, S & Cox, J (1999). Evaluating one-to-one teaching in general practice, Medical Teacher 21(1), 61-66.

Fox, D (1989). The improved Nominal Group Technique (INGT), Journal of Management Development 8, 20-27.

Moore, CM (1987). Group Techniques for idea building. Newbury Park, CA: Sage.

Moseman, CM (2003). Primary Teachers Beliefs about Family Competence to Influence Classroom Practices, Early Education and Development 14(2), 125-153.

Nelson, JS, Jayanthi, M, Brittain, CS, Epstein, MH & Bursuck, WD (2002). Using the Nominal Group Technique for homework communication decisions, Remedial and Special Education, 23(6), 379-386.

Vella, K, Goldfrad, C, Rowan, K, Bion, J & Black, N (2000). Use of consensus development to establish national research priorities in critical care, British Medical Journal 320(7240), 976-980.

Waddell, DL & Stephens, S (2000). Use of learning contracts in a RN-to-BSN leadership course, The Journal of Continuing Education in Nursing 31(4), 179-184.

back to contents page