Geraldine Clarebout and Jan Elen
Center for Instructional Psychology and Technology
Katholieke Universiteit Leuven, Belgium
E-mail: Geraldine.Clarebout: geraldine.clarebout@ped.kuleuven.be
Jan Elen: Jan.Elen@ped.kuleuven.be

Abstract

The aims of the present studies were to test the generalizability of the modality and dialogue effect to open learning environments, previously found by Moreno and Mayer (e.g., Moreno, Mayer & Lester, 2001) with an agent providing metacognitive support. As an extension, the agent’s continuous presence effect on learning was also tested. Three studies were conducted. In all studies participants interacted with an open learning environment on an ecological problem. No confirmation was found for the modality or dialogue effect, nor did the agent’s presence have any effect on students’ performance.

Introduction

Recently, pedagogical agents have gained increased attention in educational research (e.g., Atkinson, 2002; Baylor & Ruy, 2003; Moreno & Mayer, 2000). Pedagogical agents are animated characters designed to operate in an educational setting for supporting or facilitating learning (Shaw, Johnson & Ganeshan, 1999). Research on these agents deals with the agents’ contribution to learning and the impact of different agent’s features. For example, Lester, Converse, Kahler, Barlow, Stone, and Bhoga (1997) performed a study to measure agents’ affective effects on learning. The best results on learning were found with an agent that was fully animated, while a mute agent resulted in the least learning effects. Moreno and colleagues did different studies on the effects of agent features on learning (Moreno & Mayer, 2000; Moreno, Mayer, & Lester, 2000; Moreno, Mayer, Spires & Lester, 2001). They studied for instance whether a pedagogical agent’s image, language style, and modality affected learning results. Their studies reveal that using an agent communicating in a personalized way with spoken voice rather than using neutral language and on-screen text results in better performance. The image of the agent did not have any effect. In 2002 a review-study was published (Clarebout, Elen, Johnson, & Shaw, 2002) illustrating that in research agents were used that mostly provide content support and that mostly act as a coach by providing hints and feedback to the learners when problem solving tasks were executed. The review revealed a clear lack of studies on the use of agents in open learning environments.

Open learning environments

Instructional design theorists (Jacobson & Spiro, 1995; Jonassen, 1997) have proposed the use of open learning environments to foster the acquisition of complex problem solving skills. In open learning environments learners are encouraged to look at problems from different perspectives in order to reach an adequate solution. The target problems are as ill-structured problems for which no single solution exists, but that have to be looked at from different perspectives for reaching an adequate solution. Considering multiple perspectives is claimed to stimulate learners’ cognitive flexibility, and to enable learners to deal with new complex problems more easily (Jonassen, 1999; Krems, 1995; Spiro, Feltovich, Jacobson & Coulson, 1991). In line with this, several other authors (Barab & Duffy, 2000; Honebein, Duffy, & Fishman, 1993) suggested to make the problem as cognitively demanding as it is in the real world. In open learning environments a variety of support devices are provided to scaffold learners in their knowledge construction process. Learners control the use of these devices. They have to determine themselves how to handle the environment, how to use the different support devices, decide when sufficient understanding has been acquired, when an adequate solution has been found, etc.

In other words, open learning environments assume learners to autonomously initiate the interaction with the learning environment. Unfortunately not all learners are inclided or able to do so. As Perkins (1985) already indicated, learners do not always grasp the ‘opportunities’ offered by an environment, either because they do not recognize them or because they are not willing to do so. Additionally, learners have to be aware of the opportunities and have to take them (Perkins, 1985)

Pedagogical agents and open learning environments

Pedagogical agents could make learners aware of the opportunities presented to them. Pedagogical agents can provide advice to the learners on the tools to be used, and can explain the tools’ functionalities. This allows for the environment to remain as open as possible, and hence still encouraging the acquisition of complex problem solving skills.

However, research on pedagogical agents has until now mainly used closed learning environments, environments that confront students with a clear-cut problem for which only one single solution exists and for which only one perspective has to be considered to reach an adequate solution. As such, this study can be seen as a replication study of Moreno and Mayer’s 2000-study. It is studied whether the dialogue (language style) and modality effects can also be found in more open learning environments.

For at least two reasons it can be doubted that effects found with more ‘closed’ learning environments can be generalized to open learning environments. First, in open learning environments learners have ample control. Tabbers, Martens, and van Merriënboer (2004) for instance showed that the modality effect is not found when learners control the pace of instruction. Second, in open learning environments, the role of the agent may differ from the one in more pre-structured learning environments. In the studies of Moreno et al. (2001), the agent provides domain specific information, while in the experiments presented here, the support provided by the agent is metacognitive, directed towards tool use. Tool use is a problem specific for open learning environments. Research shows that students hardly make (adequate) use of tools and consequently do not optimally benefit from these open learning environments (e.g., Crooks, Klein, Jones & Dwyer, 1996; Land, 2000; Oliver & Hannafin, 2000). As a result, in the present studies, not only the effect of the agent’s features on performance will be studied, but also the effect on tool use.

As an extension of these replication studies, a study is reported in which an additional pedagogical agent feature is studied: the continuous versus discontinuous presence of the agent.

First the three effects are discussed in more detail. Next, the three experiments are reported on. The general discussion critically reflects on the results found in the different experiments.

Dialogue effect

Moreno and Mayer (2000) studied whether the use of personalized messages in a multimedia science lesson best promotes deeper learning. The personalized messages, using first and second person points of view resulted in deeper learning than the unpersonalized or neutral messages. Moreno, Mayer, and Lester (2000) showed similar results. Moreno and Mayer (2000) explain these results by referring to the cocktail-party effect (see for a review: Arons, 1992) and the self-referential effect (Klein & Kihlstrom, 1986). The cocktail party effect refers to people’s capacity to attend one conversation while being able to detect their own name in a separate conversation. The self-referential effect relates to findings that retention is facilitated when people can process information by relating it to aspects of themselves.

In the Moreno et al.-study (2000), it is argued that the beneficial effect of the personalized messages is a confirmation of the conversational hypothesis (dialogue effect) that stresses the importance of the social and individual processes involved in knowledge construction through conservation (Brennan, 1990). This hypothesis is contrasted to the transmission hypothesis, which states that communication involves encoding an idea into a signal, transmitting this signal and decoding the signal (Reddy, 1993).

Experiment 1 and 2 investigate whether the dialogue effect can be replicated, and hence whether the conversational hypothesis also holds for a pedagogical agent in open learning environments.

Modality effect

In the Moreno et al.-study (2000, 2001), a modality effect with respect to the effect of the pedagogical agent on learning was found. The group who had a pedagogical agent communicating through voice demonstrated higher ratings for recall and transfer than students who learned with an agent communicating through on-screen text. These findings were congruent with previous findings on modality effects in multimedia learning (Mayer, 2001; Moreno & Mayer, 1999; Mousavi, Low, & Sweller, 1995). In 2002, Moreno reports on the use of pedagogical agents in virtual reality learning environments and came up with similar results. An agent using narration to communicate resulted in higher scores on recall and transfer tests than an agent using on-screen text (Moreno, 2002). Atkinson (2003) and Mayer, Dow, and Mayer (2003) confirm this modality effect. Mayer et al. (2003) used an environment with an agent explaining the functionality of an electrical motor and where students had control over the pace of instruction. Their confirmation of the modality effect contradicts the findings of Tabbers et al. (2004) who did a similar study with pace control but without an agent. In their study students had to learn about developing a blueprint for a training program, based on the skills hierarchy of a complex skill. Tabbers et al. found no evidence for a modality effect.

This modality effect is related to the split attention effect (e.g., Mayer, 2001; Mousavi et al., 1995; Sweller & Chandler, 1994). This effect occurs when learners have to divide their attention among multiple information sources and have to integrate these sources to reach understanding of the material. If these information sources are both requiring visual information processing, this may overload the visual information-processing channel. Providing some of the information in an auditory format reduces the load of the visual channel.

In experiment 2 the comparison is made between a group working on an ill-structured problem with an agent communicating through narration and a group working on the same problem but with an agent using on-screen text. In line with the split-attention effect one could expect that an agent using narration frees some spaces from the visual working memory, and hence makes more room for tool use.

Presence effect

Cognitive load theory suggests that “effective instructional materials facilitate learning by directing cognitive resources towards activities that are relevant to learning rather than toward preliminaries to learning” (Chandler & Sweller, 1991, p. 293). Learning imposes germane cognitive load on the learner. This is the effort required to construct and store schemata into long-term memory (Kirschner, 2002). Since working memory is limited in capacity, it is claimed that instructional interventions should reduce extraneous cognitive load, which is the effort of learners to process irrelevant (for learning) information (Chandler & Sweller, 1991; Kirschner, 2002). Whether extraneous cognitive load is induced remains an empirical question. Moreover, in order to assess the presence of extraneous cognitive load, both the cognitive load itself and learning results are to be considered. Otherwise a distinction between germane and extraneous load cannot be made (van Merriënboer et al., 2002; Tabbers, Martens, & van Merriënboer, 2001).

In experiment 3 it is studied whether an agent who is continuously present, results in higher cognitive load than an agent who disappears when not saying anything. The hypothesis, based on cognitive load theory is that in the first condition the agent poses additional extraneous cognitive load on the learner. The agent may form an element that participants have to actively ignore when not contributing to instruction. While in the second condition this extraneous cognitive load is decreased since the agent disappears when not contributing to instruction. However, an alternative hypothesis could be that the agent being continuously present might be a reminder for students to use tools and hence increase germane cognitive load.

Experiment 1

Experiment 1 examined the role of the agent’s language style (i.e., dialogue effect) by comparing learning outcomes of participants working with an agent using a personalized communication style with participants working with an agent using a non-personalized communication style. Based on the results of Moreno et al. (2000), the personalized agent-group is hypothesized to perform better than the neutral agent-group and to use more tools.

Method

Participants and design: The participants were 42 secondary education students (14-15 years old), with 22 serving in the personalized agent-group and 20 in the neutral agent-group. All participants were pupils at the same school, recruited from different study programs. Participation in the study was voluntary, but participants knew they would be rewarded with a movie-ticket.

Materials: the computer program, developed with Macromedia Director, called STUWAWA (studying tool use with and without agents; Clarebout & Elen, 2004) confronted students with an ecological problem. The reason for choosing this topic was because it was known that students did not have any experience with it. Participants were asked to come up with the most ecological drinking cup on a music festival, considering also financial and security issues. Because students had to consider different perspectives and that there is no uniform solution, the problem can be considered to be ill-structured (Spiro et al., 1991).

To make the task authentic, a real person (on video) introduced the problem. This person represented a member of the neighborhood committee who requested help to solve the garbage problem in their garden after a music festival.

In order to solve the problem, participants had access to all sorts of information. Different videos were included with different persons giving their opinion about the topic. Through these persons, participants could access documents that related to a specific person’s perspective. Additionally, they had access to all information at once through an information-list. This list gave the titles of all available documents within the program. By clicking on a title participants got access to that specific document. Apart from this information-list tool, other tools were available to the students (information resources, cognitive tools, knowledge modeling tools, performance support tools, etc.). Table 1 provides an overview of the different tools in STUWAWA.

Table 1: Description of available tools in the environment

While working with the program participants also had the possibility to take notes n their personal note-space (“persoonlijke werkruimte” in Figure 1). This space was always accessible, also when handing in their solution. Figure 1 presents a screen dump of the main screen of the program.

Figure 1: Screen dump of STUWAWA

For solving the problem participants received assistance of Merlin (Figure 2), a pedagogical agent (Microsoft Agent). He directed participants’ attention towards the available tools. He explained the functionalities of the different tools and when students indicated to be willing to hand in their solution, he reminded them that they could consult their own notes at any time. Merlin took the initiative to deliver support; students could not ask him questions.

Figure 2: Merlin, the pedagogical agent

In the personalized agent-condition, Merlin used the first and second person (‘I’ and ‘you’) to communicate with the participants, while in the non-personalized condition the third person was used.

To gain insight in participants’ performance and tool use, different instruments were used. First, a questionnaire gathered information on tool use, participants indicated whether and how often they used a certain tool. Although log files would have given more accurate insight into tool use, due to technical problems this was not possible. In experiment 2 and 3 however, log files were used.

To measure participants’ performance, two different instruments were used: the solution to the problem and an assessment of problem solving skills. First an analysis was made of the solution to the problem. Participants received one point for each argument provided as well as for each counterargument. One point was subtracted for those arguments that contradicted participants’ choice. Participants received one additional point for each of the three perspectives considered.

Procedure. Participants were involved in two sessions of maximum 50 minutes. In a first session participants were randomly assigned to a condition and introduced to the STUWAWA environment. First the assignment as well as the environment were explained by discussing an example problem in the complete group. Next, participants worked individually on the specific problem.

The post-test and the tool use-questionnaire were administered in the second session. After administration of the instruments, participants were thanked and received their movie ticket.

Concerning the analyses, due to the rather limited number of participants separate analyses were made for tool use and performance. A first series of analyses were ANCOVA’s to determine the effect of the conditions on tool use. For each tool (8 in total) an analysis was performed. Secondly, for scoring participants’ performance, participants’ solution was scored by two raters and interscorer reliability was calculated. Next, a one-way ANCOVA with condition as independent variable and the score on the solution as dependent variable. In all ANCOVA’s time-on-task was used as a co-variate.

Results and discussion

In experiment 1, participants in the personalized group did not differ significantly from participants in the neutral group for tool use. For none of the tools a significant difference was found. Apparently, language style does not influence students’ tool use.

For performance, an interscorer reliability was found of .873. The solutions for which no agreement was reached were looked at by the two raters after which agreement was reached.

The ANCOVA on participants’ solutions revealed a significant difference between the two groups (F(1,35)=5.84, p≤.05, eta²=.15.). Participants in the neutral group (mean = 5.75; sd = 1.53) outperformed participants in the personalized group (mean =4.95; sd = 1.50). The effect sizes shows that the condition explains 15% of the variance in the solutions. This is a large effect when following Cohen’s criteria (1988).

The results of experiment 1 contradict the conversational hypothesis. Moreover, the effect that was found was opposite to previous findings of Moreno et al.. Participants with a neutral agent offered better arguments in their solution than participants with a personalized agent. As such, also the transmission hypothesis was falsified.

Experiment 2

Experiment 2 investigates also the dialogue effect by comparing one group with a personalized agent with a group with a neutral agent. These two agents use both narration and are identical as in experiment 1. A third agent condition was introduced to study the modality effect, namely an agent using a personalized way of speaking through on-screen text. In line with Moreno and Mayer, the hypothesis with respect to the modality effect states that an agent using narration will result in better performance (and tool use). However, the results of Tabbers et al. (2001) might raise some doubt about the generalizability of these findings.

Method

Participants and design: Participants were 61 first year educational sciences and psychology university students, with 22 serving in the personalized narrated group, 19 in the neutral narrated group and 20 in the on-screen personalized group. Students participated on a voluntary basis. All first year educational sciences and psychology students had received an e-mail with a request to participate to this study. 61 students replied. All participating students received a movie-ticket as an incentive.

Materials: The materials were identical to experiment 1. Except that the questionnaire on tool use was not used, instead automatically generated log files were kept of students’ activities on STUWAWA. Each tool click was registered in an Access database; consequently not only the number of tool consultation, but also the time spent per tool was logged.

Procedure: Participants were involved in one session. The session was almost identical to the first session of experiment 1. It differed in the respect that participants did not go through the first problem together in order to increase the relevance of the explanation provided by the agent.

For the analysis, the two groups with an agent using narration were compared to gain insight in the dialogue effect. To study the modality effect the two groups with a personalized agent were compared. Statistical techniques were identical to experiment 1, except that also time spent on tools was considered as a dependent variable.

Results and discussion

For experiment 2, no dialogue effect was found for tool use. Participants in the personalized group used a similar amount of tools during a similar amount of time as the neutral group. No differences were found neither for the solution or for the post-test between the personalized and neutral group. In this experiment the conversational hypothesis could not be confirmed. With respect to the modality effect, one effect was found for consulting the technical help (F(1,38) = 5.36; p≤.05, eta² = .13), where the on-screen text group did use this tool (mean= .22; sd = .43), while the narration group did not use this technical help function (mean = .00, sd = .00). Groups did not differ with respect to their solution..

Experiment 2 confirms the transmission hypothesis and not the conversational hypothesis, it does not seem to matter whether the agent uses a personalized way of communicating or not. No modality effect was found for performance, only for the use of technical help tool. In this study a split attention effect could not be revealed.

Experiment 3

In experiment 3, two groups are compared that differ with respect to the continuous presence of the agent. In the continuous group, the agent is continuously present, also when being mute. In the non-continuous group, the agent only appears when he has to say something and disappears when not talking. Cognitive load theory suggests that the non-continuous group will outperform the continuous group. In the continuous group, the agent may increase extraneous cognitive load. On the other hand, a continuously present agent might be a reminder for participants to use tools and consequently result in more tool use and better performance.

Method

Participants and design: Participants were 47 first year social science students, with 24 serving in the continuous group and 23 in the non-continuous group. All first year social science students were asked to participate through an information session in class, the researcher asked the students personally whether they would like to participate. The lecturer additionally encouraged the students to participate. Students received a movie-ticket as incentive. The design was identical to experiment 2.

Materials: The materials were identical to experiment 2, except that after working with STUWAWA, students also received a question on cognitive load. They had to indicate to what extent they invested mental effort to solve the problem. The same instrument was used as van Merriënboer et al. (2002). A nine-point likert type scale was used from ‘very very little mental effort’ to ‘very, very much mental effort’.

Procedure: The same procedure and analyses were used as in experiment 2 with one exception. The additional instrument on cognitive load was administered immediately after working with STUWAWA and prior to solving the post-test.

Results and discussion

Three effects were found of condition on tool use, namely for the amount of drawing tool use (F(1,45) = 6.94; p≤.05; eta²=.14), for the time spent on the drawing tool (F(1,43)=5.59; p≤.05; eta²=.12) and for the time spent on the. information list (F(1,43)=4.28, p≤.05; eta²=.10). Participants in the non-continuous group consulted more (mean = .32; sd = .49) the drawing tool than participants in the continuous group (mean= .00; sd = .20). They also spent significantly more time on this tool. However, it should be noted that the amount of consultation was very small in both groups and that the time spent in proportion to the overall time did not exceed 1%. In the non-continuous group this was 0.6% and for the continuous group this was even less, 0.05%. With respect to the information list, the continuous group spent a significant larger proportion of their time on consulting this tool (mean=15.38%, sd=13.57%) than the non-continuous group (mean = 7.83%; sd =8.89%).

No differences were found between the two groups for their solution to the problem. No differences were found in cognitive load for the two groups.

It seems that the presence of the agent, even when not contributing to the learning process does not cause additional cognitive load, be it germane or extraneous.

General discussion

The three experiments focused on different agent features that have been discussed in multimedia design. The results of these studies do not confirm the results found by Moreno et al. (2001) with respect to either the dialogue or the modality effect. The effect found in experiment 1 for language style was opposed to those found by Moreno et al. The neutral group outperformed the personalized group with respect to their solution. No effect was found for post-test scores. It could be that language style is less important for more open learning environments and an agent who does not give content specific information. Moreno et al. refer to the self-referential effect to support their findings, meaning that retention is facilitated when participants can process information by relating it to aspects of themselves. However, in these open learning environments retention is not the goal, focus is more on problem solving skills. It could be that when the focus is not on learning specific content information, the dialogue effect does not occur or, it might be that depending on the kind of advice provided by the agent a dialogue effect occurs or not.

Another possible explanation could be that the personalized agent induces a connection with students’ personal perspective, interfering with the different perspectives to be considered in open environments. This would explain the results found in experiment 1. Given the fact that in this experiment younger participants were used, one could wonder whether this detrimental effect may especially be of influence for younger children.

With respect to the modality effect, no effects were found on performance. An explanation could be that the agent did not cause a split attention effect. It could be that also this split attention effect is sensitive for the kind of information provided by the agent. In studies of Moreno et al. (2001), Mayer et al. (2003) and Sweller and Chandler (1994), this split attention effect occurs when a specific process has to be learned, for instance, when a diagram has to be integrated with an explanatory text. In the problem that was dealt with here, no process had to be learned. The integration pertained to linking the explanation of the agent to a specific icon on the screen.

In experiment 3 research on pedagogical agent’s features was extended. The influence of the agent continuous presence was studied. No effects were found on performance or on cognitive load as indicated by participants. A continuously present agent did not cause more cognitive load than an agent not continuously being present. Some effects were found on tool use. The effect on the time spent on the information list might suggest that the continuous present agent forms a reminder for students to consult information or it might be that participants needed more time to process the information due to the presence of the agent. The latter would suggest that even when participants did not indicate differences in cognitive load that the agent did cause some additional cognitive load. For the drawing tool, the opposite effect was found, the non-continuous agent group consulted more and during a longer period the drawing tool. However, this tool was consulted very little, on average less than once per participant.

The results presented here suggest that the effects found by Moreno et al. (2001) are not necessarily generalizable to open learning environments with agents providing metacognitive support. It could be argued that the complexity of open learning environments causes cognitive ‘overload’ and that changing the agents’ modality or dialogue style are merely details not noticed or not perceived as relevant by the learners, and consequently not causing additional cognitive load. All this is in contrast to ‘closed’ environments, where the agent is more prominently present. On the other hand, Sweller (2006) argues that these effects only occur in high cognitive load settings. A possible explanation could be that students just get used to it. Compared to studies of Mayer (2001) for instance, the duration of the intervention in this study was long. Or, participants may be used to split their attention over two visual information resources through their experience with reading subtitles on Flemish television (Elen & Van Gorp, 2005).

With respect to pedagogical agents-research, the results of the studies presented here raise questions about the focus of pedagogical agents-research. Given the findings that an image does not really matter (e.g. Baylor & Ryu, 2003; Choi & Clark, 2006), it may be wondered whether pedagogical agent-research should not primarily focus on the agent’s role instead of the agent’s appearance. In line with this, Baylor, Cole, Graesser, and Johnson (2005) also questioned the efforts to try making a pedagogical agent as human-like as possible. It seems that when students are engaged in a task, they do not take notice of an agent’s appearance as such. Of greater importance seems the kind of support and the role the agent should take to support learners.

Acknowledgment

The authors want to thank Richard Mayer and Jeroen van Merriënboer for their fruitful comments on the plans for, and results of the presented experiments, as well as Jaak Billiet for his help in recruiting participants.

References

Arons, B. (1992). A review of the Cocktail party effect. Journal of the American Voice I/O Society, 12, 35-50.

Atkinson, R. K. (2002). Optimizing learning from examples using animated pedagogical agents. Journal of Educational Psychology, 94, 416-427.

Barab, S. L., & Duffy, T. M. (2000). From practice fields to communities of practice. In D. H. Jonassen & S. M. Land (Eds.), Theoretical foundations of learning environments (pp. 1-23). Mahwah, NJ: Lawrence Erlbaum Associates.

Baylor, A. L., Cole, R., Graesser, A., & Johnson, W. L. (2005, July). Pedagogical agent research and development: Next steps and future possibilities. Pannel discussion at the international conference on Artificial Intelligence in Education, Amsterdam, The Netherlands.

Baylor, A., & Ryu, J. (2003). Does the presence of image and animation enhance pedagogical agent persona? Journal of Educational Computing Research, 28(4), 373-395.

Bransford, J. D., & Stein, B. S. (1984). The ideal problem solver: A guide to improving thinking, learning and creativity. New York, NY, : W. H. Freeman & Company.

Brennan, S. E. (1990). Conversation as direct manipulation. In B. Laurel (Ed.), The art of human-computer interface design (pp. 393-404). Reading, MA: Addison-Wesley.

Chandler, P., & Sweller, J. (1991). Cognitive theory and the format of instruction. Cognition and Instruction, 8, 293-332.

Choi, S., & Clark, R. E. (2006). Cognitive and affective benefits of animated pedagogical agents in learning English as a second language. Journal of Educational Computing Researc, 33(2), 455-480.

Clarebout, G., & Elen, J. (2004). STUWAWA: Studying tool use with and without agents. In L. Cantoni, & C. McLoughlin (Eds.), Proceedings of ED-Media 2004, World Conference on educational multimedia, hypermedia & telecommunications (pp. 747-752). Norfolk, VA: AACE.

Clarebout, G., Elen, J., Johnson, W. L., & Shaw, E. (2002). Animated pedagogical agents: An opportunity to be grasped? Journal of Educational Multimedia and Hypermedia, 11, 267-286.

Clark, R. E., & Choi, S. (2004, April). A review of experiments on the effects of animated pedagogical agents and suggestions for improved research design. Paper presented at the annual meeting of the AERA, San Diego, CA.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates.

Crooks, S. M., Klein, J. D., Jones, E. E., & Dwyer, H. (1996). Effects of cooperative learning and learner-control modes in computer-based instruction. Journal of Research on Computing in Education, 29, 109-123.

Hayes, J. R. (1981). The complete problem solver. Philadelphia, PEN: The Franklin Institute Press.

Honebein, P. C., Duffy, T. M., & Fishman, B. J. (1993). Constructivism and the design of learning environments: Context and authentic activities for learning. In T. M. Duffy, J. Lowyck, & D. H. Jonassen (Eds.), Designing environments for constructivist learning (pp. 87-108). Berlin: Springer-Verlag.

Jacobson, M. J., & Spiro, R; (1995). Hypertext learning environments, cognitive flexibility and the transfer of complex knowledge. Journal of Educational Computing Research, 12, 301-333.

Jonassen, D. H. (1997). Instructional design models for well-structured and ill-structured problem-solving learning outcomes. Educational Technology Research and Development, 45(1), 65-94.

Jonassen, D. H. (1999). Designing constructivist learning environments. In C. M. Reigeluth (Ed.), Instructional design theories and models. A new paradigm in Instructional theory. (Vol.2, pp. 215-239). Mahwah, NJ: Lawrence Erlbaum Associates.

Kirschner, P. A. (2002). Cognitive load theory: Implications of cognitive load theory on the design of learning. Learning and Instruction, 12, 1-10.

Klein, S. B., & Kihlstrom, J. F. (1986). Elaboration, organization, and the self-reference-effect in memory. Journal of Experimental Psychology. General, 115, 26-38.

Krems, J. (1995). Cognitive flexibility and complex problem solving. In P. A. Frensh & J. Funcke (Eds.), Complex problem solving: The European perspective (pp. 201-218). Hillsdale, NJ: Lawrence Erlbaum Associates.

Land, S. M. (2000). Cognitive requirements for learning with open-learning environments. Educational Technology Research and Development, 48(3), 61-78.

Lester, J. C., Converse, S. A., Kahler, S. R., Barlow, S. T., Stone, B. A., & Bhoga, R. S. (1997). The persona effect: Affective impact of animated pedagogical agents. In Proceedings of the CHI97 conference (pp. 359-366). New York, NY: ACM Press.

Mayer, R. E. (2001). Multimedia learning. Cambridge: Cambridge University Press.

Mayer, R. E., Dow, G. T., & Mayer, S. (2003). Multimedia learning in an interactive self-explaining environment: What works in the design of agent-based microworlds. Journal of Educational Psychology, 95(4), 806-813.

Moreno, R. (2002). Pedagogical agents in virtual reality environments: Do multimedia principles still apply? In P. Barker, & S. Rebelsky (Eds.), Proceedings of ED-MEDIA 2002 (pp. 1374-1379). Norfolk, VA: AACE.

Moreno, R., & Mayer, R. E. (1999). Cognitive principles of multimedia learning: The role of modality and contiguity effects. Journal of Educational Psychology, 87, 317-334.

Moreno, R., & Mayer, R. E. (2000). Engaging students in active learning: The case for personalised multimedia messages. Journal of Educational Psychology and Technology, 92, 724-733.

Moreno, R., Mayer, R. E., & Lester, J. C. (2000). Life-like pedagogical agents in constructivist multimedia environments: Cognitive consequences of their interactions. In J. Bourdeau, & R. Heller (Eds.), Proceedings of ED-MEDIA 2000. World conference on educational multimedia, hypermedia and telecommunication (pp. 741-746). Charlottesville, VA: Association for the Advancement of Computers in Education.

Moreno, R., Mayer, R. E., Spires, H. A., & Lester, J. C. (2001). The case for social agency in computer-based teaching. Do students learn more deeply when they interact with animated pedagogical agents? Cognition and Instruction, 19(2), 177-213.

Mousavi, S. Y., Low, R., & Sweller, J. (1995) Reducing cognitive load by mixing auditory and visual presentation modes. Journal of Educational Psychology, 87, 319-334.

Newell, A., & Simon, H. A. (1972). Human problem solving. Englewood Cliffs, NJ: Prentice Hall.

Oliver, K. M., & Hannafin, M. J. (2000). Student management of web-based hypermedia resources during open-ended problem solving. The Journal of Educational Research, 94(2), 75-92.

Perkins, D. N. (1985). The fingertip effect: How information-processing technology shapes thinking. Educational Researcher, 14, 11-17.

Reddy, M. J. (1993). The conduit metaphor: A case of frame conflict in our language about language. In A. Ortony (Ed.), Metaphor & Thought (2^nd ed., pp.164-201). Cambridge: Cambridge University Press.

Shaw, E., Johnson, W. L., & Ganeshan, R. (1999, May). Pedagogical agents on the web. Paper presented at the third international conference on autonomous agents, Seattle, WA. [Online]. Available: http://www.isi.edu/isd/ADE/papers/agents99/agents99.htm

Spiro, R. J., Feltovich, P. J., Jacobson, M. J., & Coulson, R. L. (1991). Knowledge representation, content specification and the development of skills in situation-specific knowledge assembly: Some constructivist issues as they relate to cognitive flexibility. Educational Technology, 31(9), 22-25.

Sternberg, R. J. (1996). Cognitive psychology. Fort Worth: Harcourt Brace College Publishers.

Sweller, J., & Chandler, P. (1994). Why some materials are difficult to learn. Cognition and Instruction, 12, 185-233.

Tabbers, H. K., Martens, R. L., & van Merriënboer, J. J. (2001). The modality effect in multimedia instructions. In J. D. Moore, & K. Stenning (Eds.), Proceedings of the 23^rd annual conference of the cognitive science society (pp. 1024-1029). Mahwah, NJ: Lawrence Erlbaum Associates.

Tabbers, H. K., Martens, R. L., & van Merriënboer, J. J. (2004). Multimedia instructions and cognitive load theory: Effects of modality and cueing. British Journal of Educational Psychology, 74, 71-81.

Tobias, S. & Everson, H. T. (1996). Assessing metacognitive knowledge monitoring. [College Board Report no. 96-01]. New York: College Entrance Examination Board.

van Merriënboer, J. J., Schuurman, J. G., De Croock, M. B., & Paas, F. G. (2002). Redirecting learners’ attention during training: Effects on cognitive load, transfer test performance and training efficiency. Learning and Instruction, 12, 11-37.