Daniel Surry

David C. Ensminger

University of South Alabama

dsurry@usouthal.edu

In this paper, we describe the results of a study into the conditions that facilitate the implementation of innovations. According to Ely (1990, 1999), there are eight conditions that facilitate the implementation of instructional innovations. The current study operationalized each of the eight conditions for both business/industrial settings and for educational settings. The goal of the study was to determine which of the eight conditions were perceived to be most influential in facilitating implementation by those working in business and in education. Results of the study can be used to increase the utilization of new programs and technologies.

The field of instructional design is often described as having five phases. These phases are analysis, design, development, implementation, and evaluation. The first letter of each phase forms the widely-used acronym "ADDIE." Each of these five phases has been the focus of research. For example, in the analysis phase, researchers have looked at task analysis procedures and the importance of learner and context analysis. In the design phase, research has been conducted on feedback, motivational design, and performance objectives. In the development phase, studies have been done on issues such as screen design, media usage, and media selection. In the evaluation phase, research has been done on program evaluation, return on investment, and usability testing. Most instructional designers are familiar with the major activities, and the related research, in each of these phases.

The implementation phase, however, remains a mystery to many in our field. There are no specific activities that instructional designers can point to as occurring in the implementation phase. The implementation phase is often dismissed with the simplistic prescription that after development, a product is implemented prior to evaluation. There are three main reasons for this lack of understanding. First, the vast majority of the research into the implementation of innovations has been conducted outside the field of instructional design. Much of the research related to this area has been conducted in fields such as rural sociology, communications, advertising, and engineering. While there is a fairly large body of research related to educational change, relatively little has been written about the implementation of specific instructional products. Second, instructional designers, like professionals in many technical fields, have a determinist mindset in regard to technology (Surry & Farquhar, 1997). This determinist mindset results in the belief that technically advanced and instructionally sound products will be desired by end-users solely because of their technological superiority. Many instructional designers have bought into the misguided cliché that "if you build a better mousetrap, the world will beat a path to your door." Third, research into the implementation phase has been overly focused in the initial adoption of innovations. The initial adoption of an innovation, while an essential part of the innovation decision process (Rogers, 1995), does not ensure the implementation and long term use of the product.

Donald P. Ely is one of the few who has done extensive research into the implementation of instructional innovations. His research (1990, 1999) has shown that the existence of certain conditions tends to facilitate the implementation of an innovation. These conditions are:

1. Dissatisfaction with the status quo: an emotional discomfort that results from perceiving the current method as inefficient or ineffective. This condition does not have as much influence as the other seven (Ely 1990, 1999).
2. Knowledge and Skills: an assessment of the current level of skills and knowledge of the product users. Ely reports that this condition consistently ranks as one of the most influential conditions among the eight (Ely 1990, 1999).
3. Adequate Resources: the amount of resources currently available to successfully implement the innovation. Resources include finances, hardware, software and personnel (Ely 1990, 1999).
4. Time: adequate time and compensated time for users to become educated and skilled in how to use the innovation. This condition refers not only to the organization’s willingness to provide time but the users’ willingness to devote learning time for implementation (Ely 1990, 1999).
5. Rewards or Incentives: the existence of incentives that motivate users to employ the innovation, or rewards provided by the organization for those who do use the innovation (Ely 1990, 1999).
6. Participation: the involvement of key stakeholders in decisions that relate to the planning and design of the innovation. The condition refers to all stakeholders but emphasizes the participation of product users (Ely 1990, 1999).
7. Commitment: the perception by users that the powerbrokers of the organization (i.e. Presidents, CEO, Vice-Presidents) actively support the implementation of the innovation (Ely 1990, 1999).
8. Leadership: an active involvement by immediate supervisors in assisting the users in implementing the innovation (Ely 1990, 1999). This includes providing support and encouragement to users, as well as role modeling use of the innovation.

These conditions have been demonstrated to apply to both technological and non-technological innovations. They have also been shown to traverse institutional and cultural boundaries. Although presented independently, these conditions are interrelated. They affect each other by either supporting or undermining one another, (Ely, 1990, Ensminger, 2001). Ely does not present a specific model for implementation. However, by addressing these eight factors during the adoption phase and development phase, change agents can increase their chances of successfully implementing an innovation.

There were two purposes of the study described in this paper. The first purpose was to test the assumptions underlying Ely’s theory of the eight conditions of implementation. We wanted to see if practitioners working in business and education felt that each of the eight conditions were important factors in implementation. The second purpose was to determine if there were differences in the perceived importance of the conditions by those working in business and industrial organizations and those working in educational organizations.

In this section, we will describe the methodology used in this study. We will describe the participants and instruments used in the study. The procedure for data collection and analysis will also be described. In accordance with standards for research involving human subjects, the methodology and instruments described in this section were submitted to, and approved by, the Institutional Review Board of the University of South Alabama.

Participants in this study were solicited by sending a message to an internet mailing list for instructional technology professionals. Anyone from the mailing list who wished to participate in the study was included in the sample. A total of 92 people responded to the questionnaires. 36 people responded to the business questionnaire and 56 responded to the education questionnaire.

Two questionnaires were used in the study. One questionnaire was developed for business situations and one was developed for educational situations. Participants who were employed in business were asked to complete only the business questionnaire, those working in education were asked to complete only the education questionnaire.

Each of the questionnaires presented two hypothetical innovation scenarios. Each of the innovations scenarios contained eight questions – one for each of Ely’s eight conditions. Each question required a response on a five point semantic differential scale. The scale ranged from "very easy to implement" to "very difficult to implement." Some of the questions were worded in a positive manner reflecting the presence of a condition, while some were worded in a negative manner reflecting the absence of the condition. Therefore, each questionnaire was made up of 16 implementation questions and their five point response scales. The complete questionnaires are included in Appendix A and B of this paper.

Prior to the study, the questionnaires were reviewed by our colleagues and by managers from both business and educational organizations. Our colleagues provided feedback about the design and structure of the questionnaires and about the wording of the questions. The managers validated that the hypothetical innovation scenarios used in the questionnaires were appropriate and practical. Managers also made suggestions for improving the questions by altering the wording to avoid confusing or vague questions.

After the questionnaires were reviewed and revised, they were posted as forms on the World Wide Web. A home page was set up for the study. The home page explained the purpose of the study, provided informed consent information for the participants, and provided links to either the business or education questionnaire. The main page and the questionnaires were housed on the College of Education’s server.

Responses from the questionnaires were sent via an anonymousemail function to our email addresses. We used a free web-based form service toprocess the forms and provide for the anonymous emails. The emails were printedand also saved in electronic format. The printed emails were then used to enterthe responses into a statistical analysis software package. Responses on thesemantic differential scales were assigned a number with 1 representing"very easy to implement" and 5 representing "very difficult toimplement." Responses for negatively worded questions were reverse coded toensure consistency in data entry (i.e., for analysis purposes 1 alwaysrepresented a strong "pro-condition" response and a 5 alwaysrepresented a strong "anti-condition" response). Once all of the datawere entered into the statistical analysis software package, the electronicemails were deleted.

A variety of descriptive and inferential techniques were used to analyze the data. Frequency counts, graphs, and mean plots were the primary descriptive techniques used. Analysis of variance was the primary inferential technique used.

In this section, we will provide a brief overview of the most important results of our study and discuss the implications of those results. We will also discuss limitations of the current study and possible areas of future research.

As noted above, 36 participants responded to the business questionnaire. Because each participant was asked to reply to two implementation scenarios, the N used for statistical analysis in this section is shown as 72. Table 1 shows the descriptive statistics for the business group. A lower mean represents stronger agreement that the condition is perceived to be an important factor in facilitating implementation.

Table 1. Descriptive statistics for each of the eight conditions by respondents to the business questionnaire.

Table 2. Rank of the eight conditions for business respondents

Based on the results shown in Table 1, we were able to rank order the eight conditions. This ranking is shown in Table 2. Time, Leadership, Resources, and Skills & Knowledge were closely grouped as the four most important factors in facilitating implementation for the business group. This shows the business group perceived that, in order for an innovation to be successfully implemented, workers need time to learn how to use the innovation, leadership from immediate supervisors, supporting resources, and the skills and knowledge needed to use the innovation effectively. Two of the conditions, dissatisfaction with the status quo and participation, had means above 2.5, representing neutral perceptions about the condition.

The questionnaire also asked for demographic information related to each participant’s age level, employment level, and educational level. We used these to determine if there were any intra-group variables that resulted in different perceptions about the conditions. We used a one way analysis of variance to compare the means for the three demographic variables on each of the eight conditions. For the business group, we discovered two statistically significant results. One was for the demographic variable "Age Level" and the condition "Rewards and Incentives". The second was for the demographic variable "Employment Level" and the condition "Skills and Knowledge".

Table 3. ANOVA comparing means by age level and the condition"rewards and Incentives exist"

Figure 1. Mean plot by age level and the condition "rewards and incentives exist"

Table 3 shows the ANOVA comparing the means by Age Level and the condition "Rewards and Incentives Exist." Figure 1 is a mean plot representing the same data. Table 3 and Figure 1 seem to show that middle age workers, those between 36 and 50 years of age, perceive that rewards and incentives are a more important condition than younger or older workers. That’s a fairly interesting result, although the relatively small number of respondents in each age level make the results difficult to generalize. More research is needed to determine if there are, in fact, differences in perceptions by different age groups for this condition.

Table 4 shows the ANOVA comparing the means by employment level and the condition "Skills & Knowledge." Figure 2 is a mean plot representing the same data. Table 4 and Figure 2 suggest that respondents who identify themselves as "staff" have different perceptions about the importance of skills and knowledge than respondents who identified themselves as middle or lower management. Once again, that’s a fairly interesting result that is difficult to generalize due to the relatively small numbers in each category. As in the previous example, more research is needed to determine if there are, in fact, differences in perceptions by different employment levels for this condition.

Table 4. ANOVA comparing means by employment level and thecondition "skills and knowledge are present"

Figure 2. Mean plot representing means by employment level and the condition "skills and knowledge are present"

Fifty-six participants responded to the education questionnaire. Because each participant was asked to reply to two implementation scenarios, the N used for statistical analysis in this section is shown as 112. Table 5 shows the descriptive statistics for the education group. As with the business group, a lower mean represents stronger agreement that the condition is perceived to be an important factor in facilitating implementation.

Table 5. Descriptive statistics for each of the eight conditions by respondents to the education questionnaire.

Based on the results shown in Table 5, we were able to rank order the eight conditions. This ranking is shown in Table 6. Resources, participation, and skills & knowledge were closely grouped as the three most important factors in facilitating implementation for the education group. This shows the education group perceived that in order for an innovation to be successfully implemented, workers need supporting resources, a sense of ownership in the decision making process, and the skills and knowledge necessary to use the innovation effectively. Only one of the conditions, commitment, had a mean above 2.5, representing neutral perceptions about the condition.

Table 6. Rank of the eight conditions for education respondents

The education questionnaire also asked for demographic information related to each participant’s age level, employment level, and educational level. As with the business questionnaire, we used these to determine if there were any intra-group variables that resulted in different perceptions about the conditions. We used a one way analysis of variance to compare the means for the three demographic variables on each of the eight conditions. For the education group, we did not discover any statistically significant results.

A total of 92 participants responded to the questionnaires – 36 for the business and 56 for the education. Because each participant was asked to reply to two implementation scenarios, the N used for statistical analysis in this section is shown as 184. Table 7 shows the descriptive statistics for the combined group. As with the individual groups, a lower mean represents stronger agreement that the condition is perceived to be an important factor in facilitating implementation.

Table 7. Descriptive statistics for each of the eight conditions by respondents to both the business questionnaire and the education questionnaire

Based on the results shown in Table 7, we were able to rank order the eight conditions. This ranking is shown in Table 8. Resources, Skills & Knowledge and time were closely grouped as the three most important factors in facilitating implementation for the combined group. This shows the combined group perceived that in order for an innovation to be successfully implemented, workers need supporting resources, the skills and knowledge necessary to use the innovation effectively, and time to learn about the innovation. Only one of the conditions, commitment, had a mean above 2.5, representing neutral perceptions about the condition.

Table 8. Comparing the rank of the eight conditions for combined respondents, business respondents and education respondents.

Table 8 also shows the relative importance of each of the eight conditions for each group. The education and business groups differed dramatically on their perceptions of time and participation. As shown in Table 8, Time, is the most important condition for the business group was only ranked seventh by the education group. Table 8 also shows that participation, the second most important condition in the education group was the least important condition for the business group.

In an attempt to discover less obvious differences, we compared the means of each group on each of the eight conditions. A one-way analysis of variance was used to compare the means. The results are shown in Table 9.

As shown in Table 9, there were statistically significant differences between the means of the education group and the business group on four of the eight conditions. Time and participation, the two conditions with differences observed in Table 8, did have statistically significant differences. The other two conditions with statistically significant differences were dissatisfaction with the status quo and commitment. Table 10 shows the means for each group on each of the four conditions.

Table 9. ANOVA comparing responses from business and education on each condition

Table 10. Comparing means from business and education on the four Conditions with statistically significant differences

There were three important limitations in this study. First, the relatively small size of the sample is a threat to the study’s validity. Only 36 people completed the business questionnaire and only 56 people completed the education questions. The second limitation is that higher education was disproportionately represented in the education sample. Relatively few workers from the K-12 sector replied to the questionnaire. The third limitation is that the study used hypothetical implementation scenarios instead of actual scenarios.

The results of this study, and the questions that arose from the study, suggest that future research in the area of implementation would be useful. We plan to replicate this study with a larger, more representative sample. We also plan to do more research in the area of intra-group variables. Based on this study, we hypothesize that intra-groups variables play an important role in implementation. We will begin by identifying major intra-group variables for both educational and business settings. There are likely many more than the three we used in this study. Length of time at the worksite, experience with technology, and a variety of personality variables likely lead to different perceptions about implementation. Once the major variables have been identified, we will develop a methodology to study them in more detail. Another area of future research related to this topic is to determine perceptions in regard to actual "real world" implementation situations. This study used hypothetical situations.

This study produced three main results. First, the study tended to validate Ely’s theory of the eight conditions that facilitate innovation. All of the eight conditions had low means in this study, representing agreement that the condition does facilitate implementation for both the education and business groups. Second, the results of the study support our hypothesis that there is a difference in the relative importance of the eight conditions between educational and business settings. Four out of the eight conditions resulted in statistically significant differences between the business and education groups. Third, the results of the study suggest that there are important intra-group variables that affect the perceptions of group members in regard to the eight conditions. For the business group, respondents at different age levels and employment levels differed in their perceptions of the importance of different conditions. No intra-group variables were found to exist in the education group although we hypothesize that such variables do exist.