Margaret Haughey
University of Alberta
haughey@ualberta.ca
&
Bill Muirhead
University of Ontario Institute of Technology
bill.muirhead@uoit.ca

In the K-12 education sector, learning objects are seen as important in providing quality resources for teachers and learners but there has been little formal research on the assessment of learning objects based on the qualities that would be important for K-12 teachers. In this paper we describe the developments in the K-12 sector, the arguments around learning object characteristics and the development of an assessment profile. We applied this instrument in two separate analyses of learning objects and found it useful in identifying characteristics of importance to teachers.

Although there is an extensive and ever-growing literature about learning objects, the clarity of the term continues to be elusive (McGreal, 2004). The various approaches to learning objects attempt to meet two common objectives: (1) to reduce the overall costs of digital resources and (2) to obtain better learning resources (Wiley, 2003a). Downes (2001) contends: "the economics are relentless. It makes no sense to spend millions of dollars producing multiple versions of similar learning objects when single versions of the same objects could be shared at a much lower cost per institution" (¶ 4). Equally strongly, it can be argued that the provision of learning objects provides better access to quality resources and supports enhanced learning outcomes. Duval, Hodgkins, Rehat, and Robson (2003) explain the purpose of learning objects as "to increase the effectiveness of learning by making content more readily available, by reducing the cost and effort to produce quality content, and by allowing content to be more easily shared" (¶ 1). These two purposes, effectiveness and efficiency, receive differing emphases from different sectors.

Wiley (2000) provides an overview of the development of the term learning object and comments that "the proliferation of definitions for the term ‘learning object’ makes communication confusing and difficult" (p. 5). He illustrates his contention with a number of examples of different terms developed by various repositories for learning objects, such as, "pedagogical documents" (used by ARIADNE), "educational software components" (ESCOT), and "online learning materials" (MERLOT). Similarly, in reviewing papers for a ED-MEDIA pre-conference workshop, Duval and his colleagues (2003) concluded that "quite a few papers devoted considerable space to a discussion of the exact definition or nature of learning objects as a concept" and that "many groups seem to be grappling with issues that relate to the pedagogically sound use of learning objects" (p. 2). Two years later, the pedagogical issues unresolved but pedagogical criteria now identified by IEEE, the literature has moved on to focus on databases and repositories.

Differing perspectives have been taken to the concern with producing learning resources that can be used more than once. In general, some definitions have focused on the "object" part of the term while others have emphasized the "learning" aspect. Learning objects do not have value or utility outside instructional contexts. Their value is in their application to classroom settings and to online environments where teachers may or may not be present. As a result, learning objects are designed to help teachers perform these functions:

Mortimer (2002) contends that while we may be surprised that there is no single definition of learning objects and that the term holds different meanings for different people, most learning objects have four characteristics. First, each learning object has a learning objective, content and activities that support the objective and assessment activities that reflect that expectation. Second, they usually take less than 15 minutes to complete. Third, the content is metatagged to some set of standards, and finally, the object can exist on its own and be provided to the learner in a just-in-time and as-needed fashion. However, there is still disagreement about these components. Merrill (2002) insists that objects without a learning design component are knowledge objects only. Until recently metatags did not include any instructional design components and McGee (2003a) points out that "most educational metadata do not include attributes for evaluation, suggesting that documenting of learning occurs outside the learning object experience" (¶ 4). In addition, she commented that "there are still many unanswered questions, particularly in the area of pedagogical design" (¶ 1).

Most of the pedagogical questions have focused in two areas: first, whether a specific learning design should be used in designing the object and second, the issue of reusability. The pedagogical issue was raised as more designers adopted a constructivist approach to learning. Some developers were concerned that too much specificity increased the size of the object and thereby reduced its reusability. Some wanted to include resources but not provide a learning design within the object. Others felt the previous learning models were adequate. None of these views includes the desires of an end user. The most recent position has been one approaching pedagogical neutrality. McCormick (2003) reflecting his experiences with CELEBRATE argues that efforts to include a specific pedagogy are doomed to failure and advocates for "the development of LOs with sophisticated, high quality media representations of content, around which teachers build learning activities and assessment" (¶ 2). Reusability has been of major interest over this past year, much of it focused on metatagging fatigue and the development of alternatives and the issues surrounding databases and repositories. While this is of particular concern in other sectors, in the K-12 sector the development of high-quality approved curriculum materials that meet the learning needs of students is generally welcomed by teachers so that reusability is less of a concern.

The overall development of learning objects is still much of a ragged front. Much of this development in Australia, Canada, Europe and the US, has occurred in the post-secondary sector, mainly in universities and much of the research focuses on issues in an adult setting. There are a number of examples of learning object repositories or research-based working groups that are trying to research, design, collect and disseminate learning objects. The important point to note is that it is important that there be sufficient use and reuse of objects in practice to help inform the pedagogical theories and technical standards which have been identified. The development of learning objects is a complex undertaking and a number of institutions who formed consortia to help share development have revealed that after the first flush of enthusiasm it has been difficult to sustain interest without having a central guiding body to ensure sufficient practical application (EOE History, 2003, ¶ 10).

The context for employing learning objects differs between the K-12 sector and postsecondary sector and affects the design of learning objects. Unlike more adult settings where learning objects can be used by learners independently, much of the current efforts in the K-12 sector have been to design learning objects for face-to-face classroom settings. This change in context and the requirement to design for incorporation into pre-existing learning strategies has created additional complexities for designers who must take into account the culture and climate of school classrooms.

In Australia and New Zealand, Canada, the USA, the UK and Europe, there have also been developments in the K-12 sector. For the USA curriculum, there are over 18 general databases of what could loosely be identified as learning objects, most with minimal meta-tagging. These might be considered more like online repositories that contain artifacts, websites, and lesson notes, as well as learning objects. They range from sites with sample learning problems to Education World and from SMETE (Science, Mathematics, Engineering and Technology Educational content and services) to the Apple Learning Exchange. This latter repository contains a range of lesson plans, lesson starter ideas, student projects, virtual field trips, videos and interviews.

In Canada the initiative for the development of K-12 learning objects has been led by the work of the learnalberta.ca portal. It is now being used as the basis for development of a national portal under the Committee of Ministers of Education, Canada (CMEC). LearnAlberta is being developed to serve not only the K-12 but also the post-secondary and lifelong learning sectors. Much of the development of K-12 objects has been in the areas of mathematics and science and the total output is still relatively small. In 2003, Ontario announced its development of a similar portal that would be connected to the national initiative. In a review of learning object repositories for CANARIE, a national, Canadian government-funded, research committee, Porter, Curry, Muirhead and Galan (2002) noted that the projects were not yet at a sufficient stage of maturity to continue to advance without central leadership. They confirmed that an integrated focus was essential to prevent fragmentation in what is an expensive development initiative. Some of their analyses are appropriate outside the specifics of their project. For example, they noted the need to develop a critical mass of users and assessors and proposed the development of communities of practice to realise these educational goals. The notion of communities of practice can be traced back through the work of the MERLOT consortium in the US which uses peer-reviews of learning objects as the basis for inclusion. Additionally, they noted the importance of training and skill-development for teachers and the development of end-user applications as a further development in the use and reuse of learning objects.

In the UK, in January, 2003, the government announced a web portal, CurriculumOnline, which is designed "to give teachers easy online access to a wide range of digital learning materials, which they can use to support their teaching across the curriculum" (¶1). At this time the development of the portal is still in the early stages. A European initiative, the European SchoolNet (www.eun.org), is a partnership of over 26 European Ministries of Education interested in the educational use of ICT (information and communications technology) for policy-makers and education professionals. Based on the Canadian SchoolNet (www.schoolnet.ca), it coordinates discussions and activities among teachers, students, policy makers and commercial vendors. One of its projects was CELEBRATE (2002-2004).

A large scale project, CELEBRATE was designed to examine how learning objects can enhance teaching and learning in European schools. It was funded by the European Commission’s Information Society Technology program (IST) and included ministry personnel, university researchers, large educational publishers, content developers and technology suppliers. "Its key aim is to provide a large-scale test-bed or practical demonstration of how schools from across Europe can use, adapt, reuse and develop Learning Objects" (¶ 5 FAQ). In the demonstration phase (January 2003-May 2004) teachers accessed reusable learning objects, and also the tools and a virtual learning environment to make their own objects, build courses, and communicate and collaborate with other developers. Over 1400 learning objects and 25 simple authoring templates were developed by commercial publishers and Ministries of Education. They were mainly in Mathematics, Science and Language with smaller numbers in other subjects. While assessing the demonstration project as generally successful, the evaluation team (McCormick, Scrimshaw, Li, & Clifford, 2004) concluded that the program would be sustainable, "only if at each stage of its development:

In Australia and New Zealand, under the initiative of The Le@rning Federation the development of learning objects while still in its early phrases is more extensive. There has been development of objects in a variety of areas, and initial feedback from teachers about their viability. In addition, the work of the Federation in exploring international exchanges of learning objects through "The Oklahoma Exchange" (2002) has helped document the difficulties faced in such an exchange. In particular, the work identified the importance of learning objects reflecting the Australian teachers’ pedagogical approaches to learning and the limited utility of objects which come from a transmission learning model rather that from one which has embraced the new learning principles. This makes the Australian project all the more important since this is a finding that is particular to the K-12 sector and unresearched elsewhere.

The evaluation of learning objects is a comparatively new concern as the quantity of learning objects has grown and the development of learning object repositories has come about to allow for greater ease in finding and using objects for both classroom and online instruction. The growth in the number of learning objects, the multiplicity of authors, their increasing diversity of design and their availability to trained and untrained educators has generated interest in how to evaluate them and which criteria to use to make judgments about their quality and usefulness.

The need to evaluate learning objects requires the development of criteria to be used in judging them. Vargo, Nesbit, Belfer and Archambault (2003) developed a Learning Object Review Instrument or LORI to evaluate learning objects. The LORI approach uses the following 10 criteria when examining learning objects:

The criteria were drawn from a review of pertinent literature on instructional design, computer science, multimedia development and educational psychology. Each measure was weighted equally and was rated on a four point scale from "weak" to "moderate" to "strong" to "perfect". The LORI process involved both individual and group rating of learning objects. Reviewers are drawn from the discipline in which the object is intended for use. This "peer review process" closely mirrors other practices found in the postsecondary sector. It is unclear if this is or should be a requirement in the K-12 area.

The criteria used by Merlot (www.merlot.org) to review learning objects for acceptance in its learning object repository include many of the same criteria used by Vargo et al.(2003). The Merlot process employs both individual evaluation (peer review) and referral to standards for learning objects. The standards or guidelines are an attempt to help reviewers assess materials submitted by faculty. The criteria used by Merlot reviewers fall into three broad areas:

Peer reviewers use a five star scale. The scale describes a continuum from one star denoting "material not worthy of use" to a five star rating representing "excellence all around". Like the LORI process, reviewers are drawn from the discipline for which the material is meant to be used.

More recently, the Collaborative Learning Object Exchange (CLOE) based at the University of Waterloo in Ontario, Canada, has developed a peer review process for material developed through a collaborative initiative among the university and college sectors in Ontario for inclusion in a provincial learning object repository. This review process closely follows the MERLOT criteria but differs in the range and number of questions used in the review process. The Merlot criteria use a set of more than 30 individual questions requiring detailed answers, while the CLOE criteria use a smaller set of questions/criteria. The CLOE process also places the review process under the responsibility of an Editor-in-Chief who is responsible for managing the peer review process involving two categories of reviewers—instructional designers and subject matter experts. Reviewers in both processes are asked to evaluate the learning objects on the merits of quality of the content, its effectiveness as a teaching tool and its ease of use. The 14 items which comprise the CLOE criteria are listed below:

Two significant reasons why the number of questions was reduced were to expedite the peer review process and to reduce the criteria to those deemed most critical to the successful use of the learning object.

Different criteria have been developed for evaluation of software and interface design. Nielson (2003) developed ten criteria for examining user interface design and computer-user interaction. These can be used when evaluating learning objects. In the "Ten Usability Heuristics" of interface design at www.usit.com/papers/heuristic_list.html Neilson closely parallels the Merlot and CLOE criteria but adds items that are important to learning in a technology enhanced learning environment. They include the following five criteria that directly bear on learning object design.

The standards/criteria developed by MERLOT, CLOE, Vargo et al. (2003) and Neilson (2003) were developed for the postsecondary sector. There has been less attention paid to criteria for learning objects used in the K-12 sector. This lack of published material is best explained by the relatively few school level projects in the area of learning object development across the globe. The relative newness of projects such as the Le@rning Federation (AU), Curriculum Online (UK), CELEBRATE (EU) and LearnAlberta.ca (CA) has so far not attracted the attention of researchers. The lack of research has resulted in the development of project-specific criteria for review of learning objects. Another result has been the use of the criteria from the postsecondary sector and a growing concern regarding what should be added or subtracted for evaluating K-12 learning objects.

A number of critical points for K-12 educators do not appear in lists of postsecondary criteria. For example, special requirements associated with child development and learning context do not appear. Learning objects used in a school or with school-age children require that attention be paid to concerns such as learner motivation and how the learning object supports independent learning and minimizes instructor intervention. In addition, other criteria not common in lists of postsecondary criteria include the need to clearly delineate what prior learning is required for use of specific learning objects, criteria regarding the concern over cultural bias, and the requirement that learning objects incorporate values that are consistent with prevailing community standards while also ensuring that the content is age appropriate.

Postsecondary criteria also do not address the necessity for learning objects to include pedagogical information about implementation, information regarding multiple instructional settings, or information about using learning objects within existing instructional contexts. While help files are useful and necessary, guidelines about how to use learning objects are more beneficial for teachers. At this stage of adoption of information and communication technology, teachers require support and guidance concerning how best to use learning objects and computer technology in their teaching practice.

The criteria for evaluating learning objects were drawn from four sources: A) the CLOE draft guidelines, B) the Le@rning Federation Soundness Specification, C) the rating scale previously used by Vargo et al. (2003) in their Learning Object Review Instrument (LORI) and D) criteria developed with respect to the special concerns of the K-12 environment. The resulting Learning Object Evaluation Instrument (LOEI) was developed to examine school level content. The scale used in evaluating each component and of the subsequent total for each object IS NOT meant to provide a comparison among learning objects. The comparison of learning objects is fraught with complications when not all objects conform to similar designs or intended use. Rather, this scale allows a reviewer to determine the integrity, usability, learning, design and values focus of each object in and of itself.

The 14 criteria used in this instrument and the principles they reflect are listed below:

Integrity

Our analyses to assess the quality of the instrument and determine its utility in providing information for K-12 teachers occurred at two different times over a two year period. We worked with The Le@rning Federation’s learning object repository because it contained the most complete set of objects covering the greatest variety of subjects available designed for the K-12 sector. First, we used the instrument to assess 22 learning objects drawn from The Le@rning Federation’s databank of learning objects. Based on the outcome of that analysis we made minor refinements and then used it to assess a further 14 learning objects one year later.

We received web access to each object and its metadata for a limited period of time and independently used the criteria to do the assessments. We began by reviewing two objects. This allowed us to assess the clarity of the criteria and check the ease of use of the instrument. When we compared our analyses we agreed that the LOEI was easy to use and found a high degree of congruence in our assessments. We then used the criteria to review each object independently and did not assess one learning object against another.

Attempts to review learning objects are fraught with complexities not found in assessing other non-digital educational content. Learning objects are multifaceted, incorporating curricular content developed to meet state and national curriculum frameworks. Developers and designers face the need to design objects that are small (disaggregated) yet adaptable in design to allow them to be combined or recombined with other objects to create learning sequences (repurposing). However the disaggregated nature of learning objects requires that "content" be as flexible as possible in its design to encourage teachers to utilize objects across a variety of instructional settings. Hence, the task faced by learning object developers is to design learning materials that can stand alone (disaggregated) and be used in a variety of learning environments (repurposed). Therefore, an evaluation process must be sensitive to the overall goals that designers and developers have for such digital assets as well as the constraints upon designs.

Learning objects differ from more traditional learning materials in a number of important ways. First, learning objects use a variety of media sources including text, graphics, sound, video and music. Unlike textbooks, learning objects are not developed for use in a universal format. While technical specifications for learning objects include minimum standards for operating systems, browser settings and necessary Internet browser "plug-ins", the variety of hardware and infrastructure found across schools and the diversity within schools regarding classroom computers (e.g., large or small screens, high or low fidelity speakers)and differing levels of connectivity (i.e., bandwidth) add to the challenges that designers have in creating materials that can be universally accessed and used. This issue is compounded when educators expect to "repurpose" and "personalize" learning objects for individual teaching styles. In addition, the desire to "localize" learning objects for specific communities of learners only adds to the diversity and intricacy of design and technical considerations faced by designers. While technical standards address the notion of a common platform, with recommended settings and common infrastructure, experience suggests that the computer equipment used can have an effect upon the experience of the end user. With this knowledge as a basis this review we undertook to examine all the learning objects in a variety of Internet Browsers (Netscape 7.1, Opera 7.11 and Internet Explorer 6.0).

Learning objects also differ from more traditional educational content in that if the educational object is disaggregated sufficiently it will more likely be reused. This tension and/or desirability between large and small learning objects, self contained learning objects and medium sized learning object "chunks" is still less than straight forward for optimum utility. This tension between large and small objects is best explained in Figure 1.0, Instructional Context and Learning Object Design, which illustrates this tension and the requirement that learning objects, more often than not, require a context or "instructional wrap" to ensure that they enhance learning and can be used by instructors within a learning sequence.

Figure 1. Context and learning object design

From "Designing for reuse and versioning" by M.Thorpe, C. Kubiak & K. Thorpe in A. Littlejohn (Ed.), Reusing online resources, p.113. Copyright 2003 by Kogan Page. (2003).

Adapted with permission of the authors.

Concern for small flexible objects presents educators and developers with difficult decisions regarding how much content and learner supports should be incorporated into a learning object. This is an issue with many objects for the K-Year 12 environment where students may lack academic skills and/or experience to fully interact with learning objects in a purposeful manner. The challenge is "not too large" or "not too small" but "just right" to paraphrase Goldilocks in the well known nursery rhyme. In short, the size of object has enormous influence on how best to use learning objects. The learning objects we reviewed showcase current thinking in the area of size, complexity and flexibility. While some were quite large and self sufficient and existed as independent learning activities, other objects were best used within a contextual framework where teachers provided the instructional support for their use. Rather than see this as a criticism, the diversity of learning objects should be viewed as more analogous to a research and development strategy, where new understandings are created through the process of design, build, use, revise, use and document.

The variety of learning objects provided by The Le@rning Federation highlighted the many kinds or types of "architectures" of learning objects that are emerging to address the specific needs of school-aged children. The approach to presenting content and the instructional strategies imbedded in the learning objects are congruent with practices found in Australian classrooms and incorporate best practices found in the literature on teaching in the various curriculum areas. For example, in some learning objects the design of the object, and the tasks presented to students to complete inside the object, reinforce the scientific method. Learners are asked to review the task, make an appropriate selection, make a prediction about the likelihood of success, test their prediction, compare their prediction with that observed in the simulated experimental setting and test their choice in a simulated situation. If unsuccessful, they are again led back to step one and if successful, on to the next task. The learners were led through an experimental design while the learner makes decisions and observes the effect that particular variables have upon the phenomena being observed. Such learning objects exemplify the criteria embodied in the "Integrity" standard.

A variety of architectures concerning functionality and the treatment of content existed within disparate curricular areas. Some of the learning objects incorporated significant third party content resulting in a "narrative" design. The result was a much larger multimedia intensive learning object. In others, the learning design was best described as "explore and practice" where external third party content was not used and little multimedia audio or video (except animation) was employed. The two approaches highlighted both diversity of design approaches but also the use of specific architectures or models for specific curricular areas. This use of different learning approaches with specific curricula demonstrates how the developers have begun to build new knowledge about approaches to the development of leading edge learning objects. The variety of learning activities within learning objects included actions such as repurposing and augmenting story lines, creating personal/individual reports, recording experimental results and writing songs or poems.

The use of game-like designs shows how gaming environments can be used for educational purposes. Educational gaming involves use of the narrative story line, complex, user-controlled navigation, the design of immersive environments, and inclusion of success indicators. As computer games continue to grow in popularity it is likely that learners will expect learning environments to include such activities. Early research suggests that boys in particular can benefit from such gaming environments (Foreman, 2004).

Some learning object designs incorporated interdisciplinary experiences across curricular areas. In each, the pedagogical design and the subject matter content ensured they could be repurposed for use in a variety of curricular areas. This approach to designing learning objects for interdisciplinary reuse illustrates the potential of learning object design where in addition to small and generic designs which enable reusability interdisciplinary design enables reuse and repurposing across classroom settings, disciplines and age levels.

Size or granularity is often considered to be in inverse relationship to complexity in the design of learning objects. However, the development of learning objects which are targeted for senior students suggests that the more advanced the audience and content, the larger some learning objects may become. Within the academic literature most authors suggest that learning objects should be small and focused on smaller portions of domain specific knowledge. It is thought that small learning objects encourage reusability and repurposing by teachers. However, learning objects for senior students which contain more advanced subject matter while also retaining the desired pedagogical features for K-12 learning objects, are likely to become more complex and may grow in size and density.

As larger objects generally include more content components and predetermined pedagogies there is an increasing risk that this may reduce the instructional approaches employed by teachers. Consequently it is a matter of balancing competing approaches between larger more comprehensive learning objects which can border on units or modules and smaller learning objects that may not address all aspects of the knowledge or skills desired but result in learning objects that are more easily repurposed and reused. As additional features are added to learning objects, such as Help Buttons, demonstrations, and greater use of multimedia, the overall mass of learning objects may also grow. Without comprehensive field testing to gather data about the optimum size of learning objects, answers will remain illusive.

All online learning materials require clear instructions for using the learning object. In many of the learning objects, this was attended to through a combination of available help files or in "explanatory" objects that accompanied the learning objects. However, the question of whether it is best to include this material into the object thereby making it larger or to continue to disaggregate the files into different learning objects remains unresolved.

All objects involve a level of familiarity with computers and with input devises such as keyboards, mice, tab key functions and space bars. Certainly, it will be inevitable all students will one day have familiarity with computer technology. But today that is not true. Therefore, it is highly recommended that a learning object be developed to focus on how to use and interact with learning objects. This learning object would not only support accessibility, but also usability and navigation.

Clear learner navigation through the object is essential for student independence in working with learning objects. One important area is to ensure that a system status icon is included in all learning objects. In most cases the first step in using the learning object requires the learner to click on a button or link to begin loading the applet. Not all objects include a system status button to allow the user to see the progress of loading. This may cause confusion regarding feedback about the status of the connection or the operation of the object.

Attention to navigation design is equally important. A repertoire of functional capabilities should include the use of "Help" buttons throughout the object, "Back and Forward" buttons, and the use of "demonstrations" to instruct students about how to interact with and make use of the learning object. The use of "Loading Buttons" in most learning objects to inform users how long and how fast the object was loading from its source (repository, local server etc.) demonstrates the growing understanding surrounding "interoperability" and the disparate connectivity found within many jurisdictions. Feedback on successful progress through the learning objects that included the use of both audio clues and visual clues to signal to the learner appropriate or inappropriate movements and/or decisions was also important.

All learning objects incorporated specific learning objectives in their design. However, the objectives were not always as explicit to learners or to teachers as they could be or perhaps should be. The lack of detailed curricular information regarding what particular learning outcome the object was meant to address requires attention as to how to embed this information or metadata into the object. Without this vital information, teachers will be required to make instructional decisions regarding where these objects should be used within an instructional setting without benefit of curriculum maps and the onus would be upon teachers to deduce the learning outcomes for which the objects were designed.

Students also need clear learning objectives regarding the purpose of the objects. In many of the objects this was presented but some provide learners with directions concerning how to use the object but they did not specifically state what the students would "learn" or what outcomes they would achieve when finished with the learning object.

One criterion of the LOEI was the necessity to clearly identify academic level, necessary technical abilities, and specific demographics for learners to interact with the learning object. This information should appear not only in the online repository but also in the learning object itself. This recommendation for additional information to be available about the learning object could be addressed through the addition of a teacher help file in the object. Alternatively, the information could be included in the online site through the development of a teacher user file that could be downloaded and made available to teachers.

It is essential to provide teachers with information regarding what prior knowledge learners must have to successfully complete the tasks presented in the learning object. The design of one object required the learner to have prior knowledge about a topic and the object did not expose or present information about it prior to asking learners to do a task based on that information. However, once learners successfully complete the tasks they were exposed to enrichment information. As a result, learners must have significant prior knowledge to use this object. However, nothing in the information found in the repository notes, or in the introduction to the object or in the object control files provide information about this. It is conceivable that this omission is purposeful and that this object is meant to be used to test prior knowledge or as an assessment learning object. Nevertheless, without some instructions the object could conceivably be used inappropriately by learners or teachers.

One of the chief benefits attributed to the development and use of learning objects is the ability of multimedia technology to present students with learning opportunities not easily replicated in classroom environments. The inclusion of the criterion item, "The technology helps learners to engage effectively with the concept/skill/ideas," was incorporated into the LOEI instrument to provide some assessment about the use of technology to support expanded learning opportunities. Objects provided learners with opportunities to manipulate variables not as easily presented in the classroom. Moreover, the objects also allowed students to return to the object many times to review the skills presented or to reinforce new learning. Some learning objects also demonstrated the potential that sharing "object design shells" can have in supporting the laddering of skills from the simple to the more advanced.

Learning objects present opportunities to extend and expand learning activities to beyond the confines of the object itself. Therefore, one of the design considerations and evaluation criteria used in LOEI is to assess to what extent the learning object structures information content in order to scaffold student learning. Objects could use help or hint files, hypertext or pop-up instructions to provide just in time assistance. Scaffolding that encouraged self-assessment and reflection is also encouraged. Many objects included scaffolding that provided additional information or next steps hints. In this regard, we found that the objects reviewed did not provide links to external resources. This lack of reference to outside resources was no doubt intentional, allowing the objects to be "repurposed" by a variety of teachers and used in a variety of curricular areas and settings without referral to "other" more particular and less general resources. Not referring to external resources in the learning object alleviates the task of constantly reviewing the external links to ensure that Internet sites do not change their addresses or post inappropriate content. One solution to these twin problems is to set up mirror sites for external materials/resources thereby maintaining secure Internet addresses while also ensuring that content does not change without their knowledge.

The addition of an external reference site could be used to build a shared resource site/repository of online materials from educators. This site could include tips concerning implementation, lesson plans built around the specific learning objects, a site for sharing student work, and a location where groups of educators could develop and share "peer reviewed" set of links for use by students and teachers when using objects to build learning sequences or activities. This community resource would assist learners and teachers to provide enrichment activities.

The inclusion of multiple challenge levels is a desirable design feature which serves the dual purpose of providing additional activities while also scaffolding learning within a single object. It also serves the purpose of further reinforcing learning experienced in the initial set of learning activities. Still used relatively infrequently, this option should become a new functional instructional design feature that may be incorporated in other learning objects

Learners require feedback mechanisms to ensure that they can self assess whether they have acquired new skills, knowledge or attitudes presented by the learning object. Not all objects contained assessment opportunities. In some, tasks are supplied and feedback regarding correct answers is immediately presented. Hints are provided. Some learning objects help learners review where they have succeeded or where they have made mistakes. One design feature not all objects shared allowed students to print their answers and engage in off-line activities through the construction of a jigsaw puzzle or students could print their results and refer to the data at a later date. Print functions facilitate students in capturing data generated within the object and provide the opportunity to record and capture project materials or to store and reflect on the materials in the learning object. The challenge for designers will be to expand this capability to link online with offline activities.

A proposal which arises from the postsecondary and corporate training sectors, is that learning objects be designed to be used without direct instructor involvement in either face-to-face or online learning environments. For this reason one criterion for evaluating learning objects is their ability to stand alone i.e., be a "chunk" that does not require direct teacher administration thereby enabling students to work independently but with some ongoing interaction between instructor-learner-object. Many of the learning objects had this balance. They could be used by students in a classroom setting independent of teacher administration, yet they relied upon teachers to weave these objects into existing teaching practices. In some objects where there is an insufficient introduction to the learning objectives more can be done to meet this criterion.

The appropriateness of language, image and sound is a key criterion in learning objects for the K-12 sector. Educators expect that the materials will be free of racial or gender stereotyping and that the images reflect the variety of groups in the society. Care must be taken to include appropriate language use and to reflect the community’s social norms.

It cannot be over-emphasized how important comprehensive help and documentation files are to good design of learning objects. Learners should not be confronted with inadequate assistance regarding how to use a learning object. Help should also be contextual and provide specific assistance for the task being undertaken. This often requires effort and therefore greater cost, yet it is critical to "good design" principles. Help files need to have sufficient information about how to move through the learning object.

Good design includes an engaging interface. The use of figures to personalize the objects for students is often included. The use of text-to-voice features to reinforce the text on the screen is another desirable feature. It reinforces instructions and expands the ways in which students can interact with the learning objects. This also contributes to greater accessibility for learners with diverse needs.

Learners with diverse needs require special attention. In some cases it may be a physical handicap while in other instances the challenge may be intellectual. Designs that use alternative input strategies such as keyboards rather than relying exclusively on a mouse, or navigating across screens using the tab button, or hearing rather than reading instructions support diverse students. So too does the use of animated graphics to help students navigate through learning objects or assist learners to use objects without relying upon highly developed reading skills. The increased use of audio instructions and of animated instructions can only increase the accessibility of these objects to "students at risk". This use of audio and visual clues is important for both low literacy students who may find text based feedback difficult to comprehend or for visually impaired students who find text-based directions difficult to see.

The final item in the evaluation instrument concerns the extent to which learning objects are "appropriate for community and cultural affiliations, including language, dialect, reading and writing". This can be the most difficult and potentially the most problematical of the criterion to quantify. Yet to ignore or to skip over this criterion is inappropriate. Consequently, it was included in the LOEI instrument. In evaluating learning objects using this criterion, the reviewers asks the question, did the learning objects portray a particular group, or individual, or use stereotypical representations of individuals or groups or include graphical designs or voice-overs that could cause concern in the widest variety of learning settings or among the cultural groups in the country of origin of the learning object? Within the K-12 sector where learning objects are seeking approval for curriculum use, the appropriate use of language and cultural norms and the importance of inclusion and representation of diversity are values that must underlie the content and processes involved. Other criteria include minority language rights, recognition of indigenous peoples’ rights and environmental agreements.

From our analysis of the objects, three topics stand out. These are (1) the accessibility criterion; (2) the student interface and (3) pedagogical issues.

From discussions with developers we became more cognizant of the complexity of practical design issues involved in addressing accessibility criteria. Incorporating all accessibility features into all learning objects made the learning objects more complex to develop, more costly to create and in limited cases adversely affected the overall design of the learning object. An alternative was to develop separate purpose-built learning objects for learners with specific special needs. Everyone subscribes to the principle of universal design but unresolved practical issues remain: Should all adaptations be included in every object or are some adaptations for specific needs best addressed through purpose-built objects? Will some students choose the adaptations to avoid working through the object directly? These questions are best resolved through actual observations with teachers and learners.

The Le@rning Federation had completed some preliminary assessments of learning objects with teachers (Snapshot 2, 2004) who were involved in an introductory workshop prior to trialling the learning objects in their classrooms. Teachers worked in pairs to evaluate 30 learning objects and then participated in a general discussion of their utility for their classrooms. One of the most common concerns for teachers is the level of language used. This was also the case in this instance. The large majority of learning objects were considered suitable and probably going to be used but some teachers were concerned that in some objects the language level was too high or that it was too text-based and required extensive reading.

In a subsequent field review across 14 schools (Lake, Phillips, Lowe, Cummings, Schibeci, & Miller, 2004) teachers demonstrated how they adapted learning objects to suit the particular needs of individual children. Being able to loop back through the object to repeat sequences or view alternatives was viewed as a way to allow for different levels of difficulty which enhanced the object for the teacher. All K-12 learning objects have to be able to meet a range of learner competencies and learning styles. The provision of language level options and alternative pathways then were two design features which could receive particular attention from designers and should be included under this criterion. Commonsense suggests a balanced approach to the issue of accessibility. This leaves developers with the issue of addressing additional accessibility criteria on an object by object basis.

As we interacted with the learning objects we tried to remain aware that we were imposing adult perceptions on materials that would be used mainly by children. Kenworthy (1993) has proposed eight guides to assist in high quality interface design. These involve keeping the cognitive load appropriate to the age and grade level of the learner, and avoiding dividing attention between text and graphics. Instead he suggests that audio rather than text should accompany images. He proposed multiple representations, color, icons and animations to direct attention and enhance communication, and concept maps, table of contents and other visual means for assisting knowledge retrieval. Practice exercises to encourage rehearsal, realistic simulations, exercises that reinforce and sustain interest and use of concrete words with audio support to assist with abstract terms are other points Kenworthy identified.

We went back to the field trials data from Lake and his colleagues (2004) and found that there were some instances of students commenting on the importance of variety, especially when they reused the object a number of times. We also reviewed comments from teachers who found that rather than answering the question students put in any keyboard character in response to a prompt so that the program would continue to the next screen. Engagement, interactivity and immediacy were all identified as important to students.

Equally, we remain convinced that other students have had little access with computers and need to be helped to develop a repertoire of skills surrounding the loading and use of learning objects. This may mean that for the first iteration of learning objects, additional help buttons and prompts are needed to assist these learners.

One of the major advantages of learning objects is their interactivity. In reviewing the CELEBRATE project, McCormick and his colleagues (2004) noted that "there is an interplay between the affordances of a learning object’s pedagogy and the pedagogical practices the teacher is able to construct, given her pedagogic competence in using learning objects, and her underlying conceptions of learning" (p. 134). This range in professional competence based on familiarity with learning objects was also evident in the report from the Le@rning Federation’s field trial. Teachers unfamiliar with their use provided learning objects as stand-alone unconnected activities undertaken by all students at the same time or used them as a whiteboard activity in a teacher-directed full-class lesson. Teachers with greater experience with learning objects embedded them in a sequence of activities that encouraged student-oriented individual and group learning.

McCormick’s (2004) argument is that teachers will use learning objects in alignment with their own pedagogical orientation and that using specific orientations may be a detriment to use of the object or the object will be placed in the larger context of the teacher’s own orientation. He contends that "learning objects can enhance active learning by students, collaboration, authentic material and activity, and provide multiple perspectives in knowledge" but goes on to point out that regardless of intention "It is clear that the locus of control or design of pedagogy is in the hands of the teacher" (p. 134).

At the same time much of the educational literature supports a change from a more teacher-directed to student-oriented inquiry model and the integration of technology is no longer seen as an optional skill for teachers or learners. Perhaps the provision of learning objects will help resolve this false dichotomy. As Parrish (2004) notes,

In conclusion, we recognize that this is a fruitful area for further inquiry. Each of the three dimensions discussed above require further research and learning objects in general will benefit from more field-based research involving students and teachers in the process.