|
| [ ASCILITE ]
[ 2004 Proceedings Contents ] |
The design of student assessment for the online context can be a disquieting experience without the support of colleagues and a group of peers from across a range of disciplines with whom to share ideas. This paper reports on interviews conducted with academic "peak performers" as well as some educational design practitioners about the current practices of designing for online assessment. A transdisciplinary approach to educational design is proposed for further exploration.
A long standing associated role of educational designers has been that of academic staff development - empowering academic staff to design, develop, evaluate and reflect upon the quality learning experiences of their students. While this facilitation role is commonly acknowledged by educational designers themselves, even in terms of being agents of social change, Schwier et al. (2004, p.94) state "it isn't enough to work quietly and effectively in the shadows". Increasing demands on all academic staff in the tertiary sector have meant that an educational design method which seeks to work collaboratively across disciplines would leverage the creative inputs of a range of staff and ensure the sustainability of educational design activities while preserving academic standards.
An earlier paper (O'Reilly, 2003) describes an action learning approach to educational design in which academic staff are brought together from across diverse disciplines to work in small teams comprised also of reference librarians and technical support staff in the process of designing, developing, evaluating and reflecting upon strategies for online assessment. This paper builds on that earlier work through an examination of academic "peak performers" across a range of disciplines, and educational design practices in four regional universities within eastern Australia. Discussion considers the potential of educational designers to provide leadership in moving beyond the comfort zone to explore the issues emerging from designing across a widening curriculum and the concomitant production of transdisciplinary knowledge.
The role of educational design in these partnership relations is of primary interest here, since it could be described as having a bird's eye view of disciplinary practices within higher education institutions. This paper offers insights from the author's professional experience of facilitating collegial interaction at a regional university where educational design resources are rather too scarce to be exclusively based within one disciplinary context - an approach which it seems is echoed at other regional universities. The educational design process is explored in the context of designing assessment for the online environment as a strategy to support both classroom and distance education. The focus on assessment rather than on the design of all teaching and learning elements has meant that a deep exploration has been possible in relation to assessment and its connection to teaching and learning as a whole.
University academics can be facilitated into a shared context for knowledge development and thus find themselves involved in shared praxis. Educational designers are well placed to facilitate such interdisciplinary or transdisciplinary activities for creative collaborations to address complex questions of learning and assessment design. The costs of ignoring commonalities in this kind of educational design process can be considerable such that escalating workloads as well as the proliferation of "islands" of design activities can be the result.
Another perspective on the construction of knowledge is provided by Klein who considers the "nostalgia for lost wholeness" as one of the pressures on traditional knowledge development in the disciplines and speaks of explorations of interdiscplinarity as "evoking a common epistemology of convergence" (Klein, 1990, p.11). New knowledge development processes have also been discussed by Gibbons and associates since the 1990s. Gibbons agrees that disciplinary specialism has long been seen as a secure and reliable way to advance knowledge - and calls this "Mode 1 knowledge" (Gibbons, 1997, p.7). Mode 1 knowledge is "generated within a disciplinary, primarily cognitive context," while "Mode 2 knowledge is created in broader transdisciplinary social and economic contexts" (Gibbons et al., 1994, p.1).
By comparison Mode 2 is more socially accountable and reflexive (Gibbons, 1997, p.9). It is characterised by being produced in the context of application and having a transdisciplinary function. Typically this kind of research and knowledge production occurs when temporary networks of people come together for a purpose and who disperse once the problem is either defined or solved. By virtue of such a multidimensional approach, a greater sensitivity to the impact of research is conceived from the outset. "Mode 2 does more than assemble a diverse range of specialists to work in teams on problems in a complex, applications oriented environment. To qualify as a specific form of knowledge production it is essential that inquiry be guided by specifiable consensus as to appropriate cognitive and social practice." (Gibbons, 1997, p.11)
The transdisciplinary approach to educational design being suggested in this paper, illustrates the integration of a diversity of disciplinary perspectives into the problem focused action research practice of collaborative design, particularly for assessment as it is facilitated in the online context.
During this data collection, the process of locating exemplary online assessment design led to the identification of several academics who were known as the early adopters in their universities, and who had by now found themselves mentoring and influencing others. As Rogers and Shoemaker (1971) reported, "the earlier adopters are less dogmatic than later adopters. Dogmatism is a variable representing a relatively closed belief system…" (p.187). As the following case studies will show, these peak performers were not rigid at all and were even inspired to reflect creatively within the course of the research interviews!
From these theoretical beginnings as linear behaviourist models of practice, thankfully, changes in the professional practice of instructional design have accompanied the emergence of contemporary learning theories. Recent changes evident in the profession in Australia also include a renaming of the role in many institutions to that of "educational design" (Bird, 2002). This is often attributed to the new processes required when considering greater flexibility in learning and teaching, and when including the use of the Internet with its resultant technological considerations in pedagogical design.
Since the constructivist learning theories with their implications for teaching and learning have been on the horizon (Kuhn, 1970; Wittgenstein, 1965), impacts on educational design models of practice have been widely considered (Duffy & Cunningham, 1996; Jonassen, 1994). Willis (1995) and more recently Crawford (2004) also discuss theoretical models based on constructivist learning theories and thus emphasise the iterative and the negotiated nature of the educational design process.
Willis (1995) states that general education design experts are a "myth" and designers who can effectively work from a constructivist interpretivist perspective must be immersed in the discipline in order to be of genuine assistance. This is not a position easy to support in Australian practice as evidenced by the data that follows. In the regional universities examined in this study, educational designers are not appointed to work as discipline based advisers. Rather, as Willis (1995) goes on to say, educational designers do not expect to maintain control of the outcomes. They work as consultants and facilitators to subject matter experts who actually design the instructional material themselves.
Crawford's (2004) contribution to the field builds on the previous work of Willis (1995) in that is does not eschew complexity in practice, and provides another non-linear model that depicts the continuous nature of educational design, making explicit the critical role of evaluation within the scope of practice. The Crawford (2004) model also refers particularly to the online context for learning and teaching.
Likewise, the impetus for transdisciplinary educational design in partnership with teams of subject matter experts from diverse disciplinary backgrounds has emerged at Southern Cross University from the need to find sustainable development of effective, efficient and pedagogically sound assessment designs for the online context. In this context, questions of authentic, discursive, semiotic and reflexive assessment (Duffy & Cunningham, 1996) can also be collegially and comprehensively explored. This study also asks - in the context of designing assessment for online, are educational designers exploring creative and recursive approaches to practice? Are transdisciplinary educational design activities being undertaken?
Beginning from a voluntary web survey conducted in four regional universities in eastern Australia between June 2002 and June 2003, questions of how and why assessment was being designed for online implementation were explored. A small number of academic staff responded from the target institutions - University of Southern Queensland, University of New England, Charles Sturt University and Southern Cross University. Where respondents indicated agreement to a follow up interview, these were conducted by phone during first semester, 2004.
Follow up interviews were conducted to explore the nature of assessment design and to follow up on questions of educational design input to the design process. Furthermore, in order to identify those academics who are designing for online assessment and who are considered by their peers as inspirational, the snowball method was used as the process of sampling additional academic staff. In this way each academic interviewed about their own approach to designing assessment for online also suggested a colleague they considered to be carrying out innovative work in this regard. In all, 22 interviews were conducted and Academic cases 1 and 2 reported below, were derived from this sample.
Questions arising on the level and nature of educational design input to the process of designing assessment for online led to the need to interview educational designers themselves. A purposive sample of educational designers in these four regional universities was interviewed in small focus groups (videoconference, teleconference or face to face), and in one case by one to one discussion on the phone. The recorded interviews were transcribed and where on two occasions, recordings failed, reference was made to written notes taken at the time. All four Design cases reported below were derived in this way. A draft of this paper was circulated to those concerned for checking and all feedback has been incorporated.
In the meantime, two action learning cycles have been completed with staff at Southern Cross University (semester 2, 2002 and semester 2, 2003) in which cross disciplinary action learning sets sought to reflect upon the design, development, implementation and evaluation of their assessment of students for the online environment. Some results of these explorations are discussed in O'Reilly (2003) where transdisciplinary teams of five staff worked together with an educational designer to produce subject specific assessment designs for three fields of study (in each of two action learning cycles). Academic case 3 reported here is derived from this aspect of the research.
The method of sampling has meant a very selective process and the interviews reported in this paper include (a) those academics who are interested in reflective practice and those open to innovation in assessment as core to teaching and learning, and (b) a self selected group of educational design practitioners at the same four regional universities from which the sample of academic staff was drawn.
Academic case 1
Interviewee 1, an academic from the disciplinary field of education who was co-teaching a group of 50-60 students, reported on decisions made to design online collaborative activities and software based activities for assessment of students. In the process of designing these assessment regimes (one undergraduate and one postgraduate) this academic valued opportunities to consult with and seek validation from her primary team teaching colleague who was also qualified to provide technical support. When questioned on the influences upon their assessment design, Interviewee 1 referred to reflections on past teaching experience as well as students' feedback. Influences from the relevant literature encountered since the introduction of these subjects in 1999 included key publications about online communities of practice and the process of scaffolding student learning in the context of group work. In addition, discussions with another 'critical friend' colleague were important to the creative process of designing challenging and effective assessments and as such, illustrate a multidisciplinary approach (Jantsch, 1970).
An educational designer did have some involvement in development of subject outlines and a follow up role included verification of alignment between assessment activities and stated subject outcomes, as well as advice on developing appropriate marking rubrics. There were no further opportunities provided or organised for this academic to consult with a cross disciplinary team of colleagues, though once this question was raised in the interview this most open to suggestion academic considered an additional disciplinary input she would seek in her next revision of these subjects. She commented… "teams of assessment consultants sounds like a good idea… In terms of principles of collaborative practice… it would be nice to look at feedback from the business world".
Academic case 2
Interviewee 2, an academic from the disciplinary field of engineering who also manages the team teaching of 300 students, named the design, development, implementation and review of assessment in his subject as a "multi-disciplinary team" approach. Seven academic staff from two faculties (engineering and science) are involved and supported by educational design and technical support staff. Regular team meetings have been a feature since the introduction of this subject in semester 2, 2002, and the strengths of this collaboration, which brings together a diversity of perspectives from within the engineering and science domain, are highlighted in the comment "people with different expertise bringing complimentary strength to the team". Due to the scale of the initiative, and in accordance with problem based learning, the "staff team also need to function as a team, similar to a student team". Any differences between staff experience, capabilities, relative levels of contribution and workload allocations must also be managed.
In Jantsch's (1970) terms this example reflects an interdisciplinary approach and opportunities for input from a broader disciplinary base were explored in interview. This academic proposed the value of future consultation regarding the design of assessment with staff from psychology and perhaps the field of arts.
Academic case 3
Interviewee 3, an academic from the field of management and commerce, has moved from solo activities in design, development, implementation and evaluation of assessment strategies through a period of working in cross disciplinary design teams, to ongoing reflective practice activities.
From 1999, when the multi-disciplinary approach to design, development, implementation and evaluation of assessment for online was supported by university executive, this academic began to maximise opportunities to consult with colleagues from other disciplines, educational design, technical design and support staff. Following participation in an in house online staff development workshop, Interviewee 3 took on board the suggestion by library staff (also involved in the workshop) that they be included in the teaching process. His teaching innovations continued to develop through collegial exchange of ideas via school based workshops and university wide seminars hosted by the Teaching and Learning Centre.
This case contrasts with the first two cases described as this academic participated in the cross disciplinary action learning sets as described elsewhere (O'Reilly, 2003). On interview for this component of research, Interviewee 3 commented that he saw his involvement in a cross disciplinary action learning set for creative design of online assessment, as supplementary to his personal (broadly consultative) approach. He appreciated that through action learning "suggestions were made to pursue ideas… ideas became clearer particularly regarding the group activities [assignment]". He stated that "not enough is facilitated like this" in order to provide more staff with the opportunity to consider multiple perspectives on their own approaches to assessment, and to conceive of their work in transdiscplinary terms.
Design case 1
Two "instructional designers" were interviewed from Interviewee 1's institution and they both agreed that no real crossdisciplinary or transdisciplinary design work has been occurring within their university as exemplified by the comment that "largely most of the design work has not been team related". Design activities occur in the traditional mix of one to one and faculty based methods. Classification of designers at this university is that of general staff rather than academic, hence their title "instructional designers". This classification further contrasted with the usual involvement of educational design staff in that the process of design was often restricted to the latter end of the writing process i.e. the creative planning and design of assessments would have already occurred prior to the involvement of instructional designers. Academics with innovative ideas on teaching, learning and assessment are required to find their own creative support networks either within the university or through published literature. However, if instructional designers have established good relations with academic staff they can be called upon to advise, provide ideas and informal feedback. There has recently been recognition for university-wide teams in some development initiatives such as a pilot project focused on learning designs, and these teams are mooted to include instructional designers, IT services staff and production staff in future.
The expanded role for these instructional design practitioners include project based responsibilities such as "media manager" and "coordinator of MCQ testing tools". Critical feedback from instructional designers is regularly sought by teaching staff to inform future improvement of classroom teaching practices. Academic staff development occurs in both formal and informal settings at this university, where these events support an exchange of ideas from across a range of disciplines. Cross campus exchanges of ideas occur on a more formal basis around identified issues of interest.
Design case 2
The educational design role at Interviewee 2's institution was described by four (academic) members of the team as primarily a centralised service. Each of the four educational designers worked with academic staff within a number of disciplinary contexts and described the current institutional emphasis as being on a CD and print based "hybrid model". Online design features were utilised in off-campus study packages and, to some extent, for the assessment process in both submission and feedback stages. An emerging model of assessment for online was still to be discussed at a management level, but its limited use for the moment was described as being determined by authentic and relevant learning contexts.
The educational designers at this university also each carried additional portfolios such as academic staff development, involvement in a university-wide graduate attributes initiative, teaching and learning inductions for new staff (conducted centrally and thus transdisciplinary), and the development of tertiary preparation courses. It is through the tertiary preparation courses such as those on English language ability for overseas students and those on advanced information literacy for postgraduate students, that a transdisciplinary perspective was brought to the design process. Staff development seminars and workshops also provided the most convenient forum for cross fertilisation between disciplines around specific topics and "on a needs basis". Otherwise the model of educational design at this university was more of a traditional mix of one to one and "groups from Faculty" as determined by project needs.
Design case 3
A focus group of two educational designers was conducted in person at Interviewee 3's institution, where the role is an academic one based within the Teaching and Learning Centre. Each of the educational designers interviewed described their work as primarily taking place on a one to one basis in conjunction with academic staff designing their teaching, learning and assessment. One of the educational designers commented that she works with the more problematic cases as the "innovators need us least".
The online context for assessment is only incorporated where special or authentic objectives prevail, such as in the case of programs delivered to students who travel for their employment and need to study and carry out assignments while in transit. Design work was also said to occur with small groups and school based groups of staff. There were no crossdisciplinary or transdisciplinary design activities reported (beyond those of the researcher's own) as these were said to be "not the bread and butter of our educational design work".
As with earlier cases, additional staff development responsibilities for educational designers at this university generate opportunities for crossdisciplinary dialogue amongst academic staff.
Design case 4
Though no academic staff emerged as exemplars from this institution due to the sampling method used, Design interviewee 4 is included to provide a designer's perspective for all four universities in the study.
The instructional designers in this institution work as general staff from a centralised Teaching and Learning Centre through a series of arrangements covering individual support, funded faculty projects, and large cross faculty projects supported via annually awarded VC's teaching development grants. Collaborative development teams including academic, instructional design, library and technical staff are occasionally convened for development of teaching, learning and assessment strategies within generic academic support or study skills packages.
The question of combined degrees in this university has raised the possibility of cross disciplinary collaboration in assessment design but such initiatives are not yet evident. There are however, some cases where units core to multiple programs provide assessment options according to disciplinary focus of the students e.g. a geology subject may have different assessment options for geography students as compared to those indicated for the science students.
Further investigation is needed to discover how the large cross faculty projects approach the shared process of assessment design at this university. One example given of a collaborative project was where "both groups [science and education] were quite knowledgeable about teaching strategies and theory, and one of the scientists had done some additional studies in the area of assessment, so it was probably successful for other reasons in terms of the educational developer input…".
Interviews conducted with academic staff who were identified through a process of snowball sampling to be innovators in online assessment, revealed that while they consulted to some extent with their discipline based colleagues and an educational designer (particularly with designers classified as academic staff), they did not often consult with colleagues from outside their own disciplinary area. These academics were also found to be reflexive and actively engaged in reflections on practice.
Teams of academics who are reportedly brought together for design, development and implementation of assessment strategies, tend to only include a small cross section of allied disciplines as well as technical advisors. Although librarians seem to be commonly involved in the provision of preparatory courses across disciplines, the inclusion of reference librarians in assessment development teams was not found in any initiatives other than the researchers' own work (O'Reilly, 2003). The desirability of their inclusion at a design stage is reinforced by a national exploration of librarians' role by Doskatsch (2003, p.113) who states librarians need "confidence to collaborate with faculty in designing learning activities…"
Through purposive sampling, groups of educational designers at four regional universities in eastern Australia were interviewed. It was found that the predominant experience of crossdisciplinary and transdisciplinary dialogue within any "intellectual communities" tends to occur at academic staff development seminars and workshops. Some transdisciplinary design initiatives were reported such as tertiary preparation, information literacy and language proficiency courses, though some tailoring for specific disciplines is often embedded within the teaching strategies of such courses. As a general rule, educational designers did not feel in a position to convene crossdisciplinary and transdisciplinary design meetings. However, it may be that educational designers could be further encouraged in their own reflective practices in order to be of some assistance to the peak performers who at this point in time seem to get by with a minimum of educational design support.
"The rubric of survival in academic research is changing from 'publish or perish' to 'partnerships or perish'." (Gibbons, 1997, p.16), and although this statement refers to the conduct of research, in accordance with this sentiment by Gibbons, there is an imperative for universities to consider partnerships in educational design processes. In the absence of evidence to show widespread adoption of constructivist interpretivist, recursive and reflexive models of practice, it is being proposed here that the creative process of educational design be strengthened and sustained by transdisciplinary partnerships in certain design activities such as assessment.
In this small study, transdisciplinary partnerships have been reported as an effective way to inspire innovations in assessment (Academic case 3) and seemed to the academic peak performers interviewed as promising greater benefits in the collaborative design process (Academic cases 1 and 2). If we are to implement Mode 2 knowledge production through the educational design process, we can bring staff together in reflective contexts to solve the design of assessment for online for each individual, while at the same time finding a consensus on 'cognitive and social practice'.
It is suggested in the writings of Klein (1998) that opportunities to bring together individuals for creative cross fertilisation and innovations also result in convergent epistemologies, however this assertion needs further investigation. At the base of all such explorations is the welcome suggestion that creative assessment design by staff can stimulate enhanced learning by students. One thing is certain, the educational design practices of today at some Australian universities might still show enduring signs of their roots, but there is now increasing evidence to suggest that a more transdisciplinary problem focused action research practice of collaborative educational design warrants much more investigation for the creativity and intellectual leadership it promises.
Bird, J. (2002). Changing Workscapes: Designers and Developers in Australian Universities 2002.Unpublished manuscript, University of Southern Queensland.
Booth, R., Clayton, B., Hartcher, R., Hungar, S., Hyde, P. & Wilson, P. (2003a). The development of quality online assessment in vocational education and training, Volume 1. NCVER. [verified 23 Oct 2004] http://www.ncver.edu.au/research/proj/nr1F02_1.pdf
Booth, R., Clayton, B., Hartcher, R., Hungar, S., Hyde, P. & Wilson, P. (2003b). The development of quality online assessment in vocational education and training, Volume 2. NCVER. [verified 23 Oct 2004] http://www.ncver.edu.au/research/proj/nr1F02_2.pdf
Crawford, C. (2004). Non-linear instructional design model: Eternal, synergistic design and development. British Journal of Educational Technology, 35(4), 413-420.
Dick, W. & Carey, L. (1978). The Systematic Design of Instruction. Glenview, IL: Scott, Foresman and Company.
Doskatsch, I. (2003). Perceptions and perplexities of faculty-librarian partnership: An Australian perspective. Reference Services Review, 31(2), 111-121.
Duffy, T. & Cunningham, D. (1996). Constructivism: Implications for the design and delivery of instruction. In D. Jonassen (Ed.), Handbook of Research for Educational Communications and Technology. New York: Macmillan Library Reference USA.
Frost, S., Jean, P., Teodorescu, D. & Brown, A. (2004). Research at the crossroads: How intellectual initiatives across disciplines evolve. he Review of Higher Education,T 27(4), 461-479.
Gagné, R. & Briggs, L. (1974). Principles of Instructional Design. New York: Holt, Rinehart and Winston.
Gibbons, M. (1997). What Kind of University? City Campus: ACU.
Gibbons, M., Limoges, C., Nowotny, H., Schwartzman, S., Scott, P. & Trow, M. (1994). The New Production of Knowledge. London: SAGE.
Hersey, P., Blanchard, K. & Johnson, D. (2000). Management of Organizational Behavior: Leading Human Resources (8th Edition). UK: Pearson Education.
Høstaker, R. (2000). Policy change and the academic profession. In M. Henkel (Ed.), Transforming Higher Education: A Comparative Study (pp. 131-158). London: Jessica Kingsley Publishers.
James, R., McInnes, C. & Devlin, M. (2002). Assessing Learning in Australian Universities. [29 Nov 2002, verified 23 Oct 2004] http://www.cshe.unimelb.edu.au/assessinglearning/index.html
Jantsch, E. (1970). Towards interdisciplinarity and transdisciplinarity in education and innovation. In G. Michaud (Ed), Interdisciplinarity: Problems of Teaching and Research in Universities (pp. 97-121). Nice: Centre for Educational Research and Innovation, OECD.
Jonassen, D. (1994). Thinking technology: Toward a constructivist design model. Educational Technology, 34(3), 34-37.
Klein, J. T. (1990). Interdisciplinarity. Detroit: Wayne State University Press.
Klein, J. T. (1998). The discourse of interdisciplinarity. Liberal Education, 84(3), 4.
Kuhn, T. (1970). The structure of scientific revolutions (2nd ed.). Chicago, IL: University of Chicago Press.
Mager, R. (1984). Preparing instructional objectives (2nd ed.). Belmont: Lake Publishing.
Morgan, C., Dunn, L., Parry, S. & O'Reilly, M. (2004). The Student Assessment Handbook - New Directions in Traditional and Online Assessment. London: Routledge Falmer.
O'Reilly, M. (2003). Using cross-disciplinary action learning sets when designing online assessment. In Interact, Integrate, Impact: Proceedings 20th ASCILITE Conference. Adelaide. http://www.ascilite.org.au/conferences/adelaide03/docs/pdf/375.pdf
Ravn, J. (2004). Cross-system knowledge chains. Systemic Practice and Action Research, 17(3), 161-175.
Rogers, E. & Shoemaker, F. (1971). Communication of innovations: A cross-cultural approach. New York: Free Press.
Schwier, R., Campbell, K., & Kenny, R. (2004). Instructional designers' observations about identity, communities of practice and change agency. Australasian Journal of Educational Technology, 20(1), 69-100. http://www.ascilite.org.au/ajet/ajet20/schwier.html
Willis, J. (1995). A recursive, reflective instructional design model based on constructivist-interpretivist theory. Educational Technology, 5-23.
Wittgenstein, L. (1965). The blue and brown books. Harper Collins.
| Please cite as: O'Reilly, M. (2004). Educational design as transdisciplinary partnership: Supporting assessment design for online. In R. Atkinson, C. McBeath, D. Jonas-Dwyer & R. Phillips (Eds), Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference (pp. 724-733). Perth, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/oreilly.htm |
© 2004 Meg O'Reilly
The author assigns to ASCILITE and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author also grants a non-exclusive licence to ASCILITE to publish this document on the ASCILITE web site (including any mirror or archival sites that may be developed) and in printed form within the ASCILITE 2004 Conference Proceedings. Any other usage is prohibited without the express permission of the author.