ASET-HERDSA 2000 Main Page
[ Proceedings ] [ Abstracts ] [ ASET-HERDSA 2000 Main ]

Diverse feedback mechanisms focus multi-disciplinary team improvement

Paul Wellington
Department of Mechanical Engineering
Caulfield Campus, Monash University
University graduates in the areas of engineering, design and business are being employed by companies increasingly pressured by global competition, rapid changes in technology, changes to organisational structures and procedures and if these graduates are to succeed, the courses from which they graduate must prepare them appropriately.

Since the mid 1990s, engineering, marketing, accounting and industrial design students at Monash University have participated in project based learning teams, acting as consultants to selected companies on new product development projects. A continuous improvement process has been established to ensure that course objectives are met, courses adapt to reflect changing employer requirements and student motivation and performance are enhanced.

The purpose of this paper is to examine the structuring of the project and the use of student assessment with the additional use as feedback for continuous subject improvement. The paper will discuss the use of reflective reports completed by students at the end of the project, feedback from the companies for whom they consult, research findings from independent observers and information provided by graduates once in the workforce. As a result, changes in facilitator guidance, company and project selection and team and peer assessment have been and continue to be improved.

Introduction

The 1990s have seen many developments in business practice including the trend to globalisation, the increasing use of information technologies and electronic communications, the replacement of hierarchical structures with flatter ones and a recognition that expert, multidisciplinary teams are faster and more effective in developing new products and ideas than traditional approaches.

New product development is a critical activity for businesses if they are to survive and prosper and effective product development requires the joint efforts of marketing, design, accounting and engineering personnel. The current trend is to bring these personnel together in teams which require input from market researchers about perceived market place needs, from designers to generate feasible concepts, from engineers to determine feasible manufacturing processes and ways of achieving quality at minimal cost and from the accountants to ensure that the expenditure is appropriate for projected sales.

In the mid 90s, 3rd and 4th year Monash students in accounting, marketing, industrial design and industrial engineering were formed into multidisciplinary teams to act as consultants on new product development for client companies. The purpose of this paper is to evaluate the effectiveness of this program based on student responses via reflective reports, graduate perspectives, company feedback and independent observation and research.

Generic skills unmet by traditional approaches in higher education

Since 1980, Finniston (1980), Wearne, (1984), Williams (1988) and Johnson (1996) have carried out extensive reviews of engineering education in Britain and Australia. Each report has concluded that engineering courses do not provide graduates with the level of communications or managerial skills which industry, professional associations, governments or indeed the graduates themselves, consider necessary. Neither does there appear to have been a significant improvement in these areas over the nearly 2 decades between the Finniston and Johnson reports.

Davis and Broadbent had similar concerns about Industrial Design courses (1987) stating "Particular encouragement be given to design education initiatives which have ready access to relevant industrial resources and place design education in a suitable multi-disciplinary context." They also stated that "that students of design, engineering, management and marketing develop the knowledge of contiguous disciplines and the multidisciplinary skills needed for cooperative participation in the innovation process".

Such problems have also been observed in business courses. Albrecht (1994) and Kuczynski (1996) identified lack of communication, decision making, leadership and problem solving skills as well as difficulties in working with other disciplines, in accounting and marketing graduates respectively.

The 1998 Dearing Report (1998), recommended many changes to higher education in the UK, based in part on the changing nature of the workplace and in part on the increase in global competition. Paragraph 4.32 specifically advocates closer working with industry, greater flexibility in organisational structures so as "to give students the opportunities and skills to work across disciplines and to develop generic or transferable skills."

Action and problem based learning

Action and problem based learning (PBL) are terms commonly used to describe learning which focuses on students learning as a result of their own actions or experience, often focused on specific problems or challenges, rather than the more common lecturer centred didactic style.

An overview of the benefits of PBL has been given by Boud and Feletti (1997) with more focussed analyses of engineering courses given by Woods et al (1985), for business by Laughton and Otterwil (1998) and for design by Lines (1997).

Senge (1992), Berndes and Stanke (1996), Pintrich and Schunk (1996) and Rotter (1966) are amongst the major contributors to exploring team dynamics. They highlight the higher levels of achievement which groups can attain, the increased motivation and ways of optimising group dynamics.

It is the central tenet of this thesis, that by exposing students at the undergraduate level to such team experiences, that they will ultimately become more effective professionals because they

The Monash Multidisciplinary Project

Structure

Monash staff in Engineering, Business and Design came together in 1996 with experts in educational process and peer assessment to develop the concept of multidisciplinary team projects in which each student would play an appropriate role in consulting to industry on product development. It was apparent to the staff coordinators that the organisation, student assessment, staff roles, industry liaison and project evaluation would require a PBL approach and authentic assessment based on their contributions to the group and success in achieving the goal. It also became apparent that continuous monitoring and improvement of the program was essential if the motivation level and achievement of objectives were to be maintained.

For the Design and Engineering students, this subject was compulsory, while for Marketing and Accounting students it was an elective promoted by enthusiastic staff. Teams were established with five Industrial Engineering, one Industrial Design, two Accounting and two Marketing students being a typical composition due to the numbers of available students. Teams comprised 8 to 10 students so as to provide a broad range of perspectives and maintain a size where each team member could chair and minute 1 meeting during the semester.

Learning objectives

The learning objectives as stated by Wellington (1996) were that students should be able to:
  1. Formulate, through group interactions, solutions to business problems which require the integration of design, manufacturing and marketing solutions.
  2. Separate engineering, accounting and marketing problems into solvable elements; explore solutions mindful of the influence each discipline area has on the other.
  3. Demonstrate understanding of manufacturing design and the possible need for redesign;
  4. Exhibit committee chairperson, secretarial and recording skills;
  5. Negotiate responsibilities within a group to ensure effective project management;
  6. Organise communication systems to ensure effective project management;
  7. Compile, present and defend a syndicate report on the project;
  8. Assess personal and peer performance in achieving individual and group objectives;
  9. Value the complexity of issues and range of people affected by the introduction of new products and technology,
  10. Appreciate the degree of involvement necessary in the decision making process in a typical industrial situation.

Assessment

When the program was developed, it was agreed by the supervisors that individual and team performance should be given equal weighting, but that there should be an element of peer assessment to minimise the possibilities of 'passengers'.

Assessment was structured to reflect these objectives with facilitator assessment of contributions by each student to committee processes, interim and reflective reports accounting for 50% of the total with the company and facilitator awarding the other 50% to the group on the basis of their final presentation and report. This group assessment was awarded equally by the facilitator and company representative, but modified by a peer assessment factor as discussed by Brown Parker et al. (1997). The students evaluated their peers 3 times during the semester - the first and second time the 360° feedback was used formatively and the third summatively (Table 1, Figure 1). This tool was designed to help students improve their group performance, ensure that all students contributed and also introduced them to an increasingly common approach to peer assessment in industry.

Table 1: Peer Assessment Questions, 1996.
    To what extent on a 1 to 5 scale does the student you are evaluating

  1. Makes constructive comments.
  2. Keeps other informed
  3. Provides verbal feedback which is helpful
  4. Focuses discussion on key points.
  5. Participates in team meetings.
  6. Builds a strong sense of teamwork and group purpose.
  7. Encourages the group to improve processes.
  8. Cooperates with other to improve problems.
  9. Motivates others to work hard.
  10. Treats everyone with respect.
  11. Listens to suggestions from others.
  12. Accepts help from others.
  13. Follows up on delegated tasks to gain results.
  14. Organises work to reduce waste, delay and repetition.
  15. Keeps an open mind when presented with new information.
  16. Maintains focus on the assigned project.



Figure 1

Figure 1: Peer Assessment Feedback, 1996 -'98.

Organisation

A preliminary information session was held in May with the first formal meeting at the start of semester 2 focussing on committee skills and development of positive relations with the sponsoring company, including confidentiality and safety issues. Representatives of host companies introduced the project topic. In the second week of the program, students divided their projects into equal sized areas of responsibility and usually designated one of their members as project manager. Schedules for chairperson and minutes secretary roles were negotiated, and contact lists of names, telephone numbers and e-mail addresses were prepared. In the third week students began the task of information gathering at their host company.

Each problem was many faceted and needed to be divided into appropriate segments, with one student either volunteering or being assigned responsibility for each, a necessary part of the group problem solving process. The groups submitted two interim reports, with final reports and a formal presentation given to members of their sponsoring company at the end of semester.

Twenty three projects have been run in the 4 years since inception with feasible solutions proposed for each problem. Two teams worked in parallel on some problems, usually generating quite different solutions. Several companies have implemented a number of the recommendations, or have identified problems in their own procedures. One company, a small cycle manufacturer, showed a doubling in productivity after implementing many of the recommendations.

Analysis of student feedback

Feedback was obtained formally through individual reflective reports written by each student, which commented on what they had learned from the course. They also analysed the project challenges, suggested improvements to its organisation, and expressed their feelings about peer assessment. The main points over the first 4 years were:
  1. The multi-disciplinary nature of the project. This had been highlighted as being a major innovative element and key objective of the project. It was encouraging to have a strong endorsement of this in student responses with more than 70% in each year giving positive or very positive endorsements with less than 5% being negative.

  2. The groups and meeting structure. While the majority expressed enthusiasm about the use of a formal meeting structure, a significant minority had reservations about the practice in their particular group, although few students in the more successful groups had criticisms of the process. Problems with group leadership were identified as an issue in some groups.

  3. Dividing up the problem. This would seem to be a major aspect capable of further attention, not only in this subject but in tertiary courses in general. Although only a minority of students complained about role division in their group, supervisors observed that weaker teams had real problems in evenly and effectively dividing the problem.

  4. Skills developed. Most believed that the project enhanced their communication and problem solving skills, gave them a more realistic perspective which would help them as graduates.

  5. Value of industry contact. While most students were positive about involvement in a real industrial project, numerous responses expressed frustration that the sponsoring company did not provide the information desired. In some instances the students failed to ask appropriate questions of the right people, in others, companies were unwilling to provide commercially sensitive information, and in a third category, companies wanted students to gather data from new sources. Several students realised that often the information was not known; a surprise for those accustomed to being given all necessary information required to solve academic problems. Those who circumvented such blockages found it a major learning experience.

  6. Peer assessment. While many students were in favour of the notion of peer assessment, many found it a chore and significant numbers disagreed that it ensured even contributions from all team members. Only about a third of students found the feedback useful. It is apparent that this was an area requiring increased attention by the facilitators.

Feedback from independent observers

To extend the multidisciplinary concept further, it seemed appropriate to involve 'experts' on teamwork to analyse the group dynamics and recommend improvements. Accordingly 6 Masters of Applied Psychology students have acted as observers/ researchers in 3 of the 4 years the project has run. Only small numbers of such students were available, but they have furnished the organising committee with a number of pointers for improvement.

Healy and Bankier (1997) made the following observations.

  1. The level of supervisor intervention had a significant effect on the group dynamics with less participation being desirable.
  2. Interdisciplinary conflict became noticeable at moments of tension.
  3. Higher performing group members addressed the whole group to a greater extent than their less effective peers.
  4. The most successful groups were the more homogeneous.
  5. Quieter students, especially overseas students, participated more when the higher performers were absent.
  6. Gender differences were less apparent where the highest performing members were female.
  7. Better performing groups minimised disciplinary differences and volunteered more explanation about their discipline to their colleagues.
Reynolds and Nolan (1998) identified problems with the 360° feedback generated by the peer assessment process. They observed that the students seemed confused with its goal and that students did not avail themselves of the opportunity to discuss their feedback with the consultants. They also evaluated the leadership seminar which had been a 1 day elective program held over the mid-year vacation. They reported that most students found it to be useful in terms of understanding themselves and others, learning about leadership styles, group problem solving, planning personality traits and found it helped in confidence building. They recommended that it should be longer and more comprehensive and undertaken by more students, preferably the whole group. They also looked at leadership styles and found that the workshop had mainly helped assist students further develop their natural style, rather than adopting new styles.

Feben (1999) found that Social Identity was an effective predictor of team efficacy. Those groups which had high levels or increasing levels of group identity, showed that their members achieved higher levels of satisfaction, commitment and organisational citizenship behaviour (OCB), and positive change in attitude to groups. Self rating correlated with group performance in one measure but peer ratings seemed to have little effect. Individual items such as self rating, global satisfaction, positive change in attitude to group work, preparation for future group work and effort correlated with successful outcomes as measured at the project's end. Surprisingly no groups showed major signs of burnout over the 18 weeks of the project.

One unexpected finding was that the group culture showed little change after the first 5 weeks, with only student's ratings of their own performance increasing with time. Group identity satisfaction, commitment, peer rating, general attitude to group work and OCB showed no time effects.

Attempts to measure the effectiveness of leadership were un-testable in 1999 due to unclear leadership roles in some groups.

Feedback from graduates

A 1998 survey of the graduate engineers indicated that 90% were working in multidisciplinary teams and were using their committee skills to at least a moderate extent. All believed that the project had improved their communication skills and felt that it had also improved their personal organisational skills at least moderately and 70% felt that it had been significant in getting them their current position. Most were working in manufacturing organisations and dealing more with other engineers, production workers and some with information technologists rather than business graduates, although anecdotal evidence, indicates that many graduates are more likely to work with business graduates as their careers progress.

Feedback from client companies

While the companies were largely impressed by the reports and presentations, they did indicate some key areas in which performance could improve, including justifying all recommendations in terms of detailed cost analysis and market surveys. The Engineering students were seen as needing to quantify their statements and explore all feasible processes, not just the first option. The marketers needed to explore all areas of product use and buyer preference so as to define the exact market requirements. The Accountants had to fully understand the company's accounting system and be able to take into account all relevant cost considerations.

Changes which have been implemented

Peer assessment and reflective reports

While student reflective reports indicated agreement with the philosophy of peer assessment, few approved of the time to fill out the forms answering 16 questions 3 times a year about 7 or more colleagues. Hence, the number of questions was cut down first to 8 and subsequently to 4 questions and the 360° feedback which had been developed from an industry oriented package, is now being presented in a simpler bar graph format (see Tables 1 and 2, Figures 1 and 2). Advisors from Psychology have contributed to this change and feel that relatively little information has been lost.

Table 2: Peer assessment questions, '99.
    Please rate each team member including yourself between 1 and 5 on the following 4 items.

  1. Reliability
  2. Cooperation
  3. Initiative
  4. Citizenship



Figure 1

Figure 2: Peer Assessment Feedback, 1999

The still disappointing aspect of the peer process, was that few students availed themselves of the information provided in the formative feedback. The supervisors, the staff member of their own discipline (usually not their group supervisor), the psychologist/observer and the campus community services counsellors were all introduced to students as being available for consultation, but few students have made use of such services. It may be that in future, supervisors actively recommend or require at risk students to consult one of the above.

While the reflective reports have provided a number of insights, facilitators feel that there is substantial scope for improvement and keeping of weekly diaries is seen as one option which will be implemented in 2000.

Supervisor's roles

In more recent years, supervisors have chaired the first group meeting and provided a set of minutes to provide style examples to their groups - an innovation which has seen a marked improvement in committee operation.

Following company criticisms, supervisors have encouraged teams to address the hard questions and give more specific recommendations rather than superficial comments. Many students voiced concerns about companies 'with-holding' information, but supervisors have more recently been able to relate how earlier groups have dealt with such problems.

Some companies were less than impressed by students addressing the company's "problems" and advice to students about their reports and presentations has subsequently emphasised an increased level of tact.

Team building

Following Feben's recommendations, a greater emphasis will be put on team building activities in the first 2 weeks in 2000. This will include students completing a Myers Briggs inventory which will be discussed with them after it has been analysed in week 3. The leadership seminar which was developed in response to Healy and Bankier's report and was endorsed by Reynolds and Nolan, has been optional in the last 2 years but is to become part of the course in 2000 as they recommended.

With at least 1 new supervisor becoming involved in 2000, an introduction to facilitation process for supervisors will hopefully ensure a uniform approach to supervision. The role of facilitator rather than tutor is one which does not come easily to many academics and further efforts are needed to develop this skill.

Conclusions

The Monash Multidisciplinary project has addressed many of the concerns expressed by authors who have analysed the deficiencies of tertiary professional education, providing students with an opportunity to apply their disciplinary skills to real problems in a realistic team based structure, while also developing a wide range of communication, interpersonal and group skills. Most of the feedback has been highly positive about the basic concepts of a multidisciplinary, team based problem solving experience. However, problems with implementation of peer assessment, group leadership, aspects of team forming and student understanding of the "real world" have been identified and been addressed. The use of the range of sources of analysis and feedback has enabled continuous improvement to be made, a process which will continue and hopefully provide even more improvements as feedback comes from more experienced graduates who have benefited by their involvement.

References

W. Albrecht (1994). An Accounting Curriculum for the Next Century. Issues in Accounting Education, Fall 1994

S. Berndes and A. Stanke (1996). The Organisational Environment of Concurrent Simultaneous Engineering [CSE]. In Concurrent Simultaneous Engineering Systems, BullingerH.-J. and Warschat J. Springer, London.

D. Boud, G. Feletti, (1997). The Challenge of Problem Based Learning. Kogan Page

J. Brown-Parker, I. Thomas, and P. Wellington (1997). Peer assessment of student performance: Measuring congruency of perceptions in a multidisciplinary team. Research and Development in Higher Education. 23rd Annual Conference, Higher Education Research & Development Society of Australasia, Adelaide, SA, 8-11 July, 1997.

B. Davis and J. Broadbent (1987). The Responsiveness of Tertiary Education to the Design Needs of Australian Industry. CTEC, AGPS.

R. Dearing (1998). Higher Education in the Learning Society. HMSO. http://www.leeds.ac.uk/educol/ncihe/

D. Feben, (1999). Leadership and Social Identity in the Temporary Work Group: A preliminary Exploration Using Multidisciplinary Student Teams. Monash internal report.

M. Finniston (1980). Engineering Our Future. Cmnd 7794, HMSO, London.

B. Healy and G. Bankier. (1997). 1996 Multi-Disciplinary Project at Monash University. Monash internal report

P. Johnson (1996). Changing the Culture: Engineering education into the Future. Institution of Engineers, Australia, Canberra.

A. Kuczynski (1996). Marketing Graduates Lacking Skills. Australian Professional Marketing Institute, St. Leonards, Australia

D. Laughton and R. Ottewill (1998). Laying Foundations for Effective Learning from Commissioned Projects in business Education. Education + Training V40, No 3.

R. Lines (1997). Collaborative Approaches to Teaching and Learning. Proc of Conference Emergent Paradigms in Design Education, Sydney,10-13 July

P.R. Pintrich and D.H. Schunk (1996). Motivation In Education. Theory, Research and Applications. Merrill, Prentice Hall Columbus, Ohio.

K. Reynolds and G. Nolan (1998). Consultant's report on the Multi-Disciplinary Project 1998. Monash internal report.

J.B. Rotter (1966). Generalized Expectancies For Internal Versus External Control Of Reinforcement. Psychology Monographs, 801, 609.

P.M.Senge (1992). The Fifth Discipline. Random House, Australia, Sydney.

S. Wearne, D. Pugh, N. Eley, F. Uemura, W. Vaags, O. Solem, (1984). Managerial Skills and Expertise used by Samples of Engineers in Britain, Australia, Western Canada, Japan, The Netherlands and Norway. Report TMR 152, Uni. of Bradford.

P. Wellington (1996). Preliminary Information for Multidisciplinary Industry Based Projects IND 3322 and MKT 3631. Monash University class notes.

B. Williams. (1988). Review of the Discipline of Engineering. Canberra, AGPS.

D. Woods, C. Crowe, T. Hoffman; J. Wright (1985). 56 Challenges to Teaching Problem Solving Skills. Chem 13 News, Dept. of Chemistry, Univ. of Waterloo, Waterloo, Ontario, Canada.

Author: Paul Wellington, Monash University
Phone (03) 9903 2156 Fax (03) 9903 2766 Email paul.wellington@eng.monash.edu.au

Please cite as: Wellington, P. (2001). Diverse feedback mechanisms focus multi-disciplinary team improvement. In L. Richardson and J. Lidstone (Eds), Flexible Learning for a Flexible Society, 710-719. Proceedings of ASET-HERDSA 2000 Conference, Toowoomba, Qld, 2-5 July 2000. ASET and HERDSA. http://www.aset.org.au/confs/aset-herdsa2000/procs/wellington.html

[ Pre-conference abstract ] [ Proceedings ] [ Abstracts ] [ Program ] [ ASET-HERDSA 2000 Main ]
Created 28 Oct 2001. Last revised: 4 Apr 2003. HTML: Roger Atkinson
This URL: http://www.aset.org.au/confs/aset-herdsa2000/procs/wellington.html
Previous URL 28 Oct 2001 to 30 Sep 2002: http://cleo.murdoch.edu.au/gen/aset/confs/aset-herdsa2000/procs/wellington.html