Questioning the teacher's comfort zone with online learning: A 4 year case study of students' perceptions of online learning

Students in a first year course evaluated the contribution to their learning of online technologies over the period 2001-2004 through their Student Evaluation of Learning and Teaching responses. A minimum eighty percent believe that (1) online learning has enhanced their ability to complete tasks, at least 60% believe that (2) online learning has helped their ability to learn independently, and 67% believe that (3) their knowledge of the course was enhanced through the online course experiences. But there has been a very significant decline from 2002 to 2004 in the numbers who believe (1) and (2). There has also been a 20% decrease from 2001 to 2004 in the number who believe that (4) the use of Discussion Boards (assessable and non-assessable) was useful to their learning, and this decline is very significant (p=0.0001). Students' reasons are explored through qualitative research methods to reveal that students commencing University are seeking social learning experiences which connect them with their fellow learners. An uncomfortable gap between teacher's valuing and student's valuing of online learning is exposed - whilst 80% of students perceive that completing tasks was aided by online learning only 66% of teachers at their University believe that overall online learning benefits their students.

Introduction

In 2003 The University of Adelaide funded a University-wide study into factors predicting the use of online learning technologies by teachers (Shannon and Doube, 2003). When teachers who had used web based teaching were asked about the impact on their students of 15 items, ranked on a 7 point Likert scale from 1, very decreased to 7, very increased, on several items - attendance, continuance in course, continuance in program - more than half the teacher respondents considered that web based teaching had no impact, and on another two items - summative grades and critical thinking & problem solving - almost half of respondents considered that web based teaching had no impact (Shannon and Doube, 2003b, p.483). The principal impact teachers responding believed online learning had for their students was in the development of IT skills (77%), and independent learning (67%) whilst more than half of respondents believed that there was a positive impact on enjoyment while learning (55%) discipline area knowledge (55%) communication skills (54%), and time management (53%). Understanding what students believe are the impacts of online learning on their learning is the purpose of this study.

Method

The Course evaluated is a Semester1, Level 1, core course in a professional degree program. Enrolments include every new student in the program. Up to 88% are school leavers (2002), and up to 15% are international students (2004). The course aims to

• Provide a holistic, integrated introduction to the University, the School, and the subject area of people interacting with design and the environment.
• Engage students in active learning
• Provide a nurturing learning environment where mistakes can be made, and where there is an emphasis on critical thinking, making, and reflection, as an iterative cycle.
• Emphasise the development of communication skills and the relationships between them: manual sketching, computer modelling, plain English writing, listening, argument formulation and delivery, and reflective, summary writing with current industry standard tools (Course Handout, 2004)

There is no explicit aim to acquire generic computer skills - this is subsumed to the emphasis upon the development of communication skills and the relationship between them. The overarching graduate attributes for the Program are the gaining of current professional skills, and principally the ability to think critically and respond creatively. Familiarity with industry level computer skills is considered essential.

There are three assignments in the course designed to assess skills, values and knowledge. First, an iterative Discussion Board assignment "e-research" takes place over 4 weeks. Students form into groups of 5-6 around current topics in the built environment - 25 annually renewed topics for which new online resources are provided (Shannon, 2001a). Students work as individuals, but as part of a topic group. Students commence with a session in Week 1 on using Discussion Boards, after which all their interaction with peers, group members, Tutor, and Coordinator takes place on the Discussion Board. There are 4 assessed Discussion Board entries - Week 1, 2%, Week 2, 3%, Week 3, 5% and Week 4, 12%. In weeks 3 and 4, students must respond to their peers, and in Week 4, summarise all the arguments and research of their Topic Group. In 2004 the assessable Discussion Board had 451 student entries.

The second topic "e-role play", concerning divisive issues in the local built environment, invites tutorial groups to form 5 groups around 5 topics, and then asks each individual to select a stakeholder role within that topic. The assessment is an integrated illustrated PowerPoint presentation which highlights each stakeholder's role, values and understandings (Shannon, 2001b). Accompanying this 4 week assignment is a non-assessed Discussion Board for groups to use to share information, and build up their knowledge of their peers' points of view. In 2004, the non-assessable Discussion Board had 143 student entries.

The third topic is the design of a Bird Hide at Banrock Station in the Riverland in South Australia. It includes a Field Trip, and draws on students' designing and evaluation skills (Shannon, 2001c).

Results

As the Tertiary Entrance Ranking rose, from 2001 to 2004, the class size rose at the same time in response to over enrolment, acceptance of full fee paying local students, and an increase demand from in overseas students. The withdrawal rate and the average class mark appear to have stabilised in 2004. Two sample T tests for independent groups have been conducted to ascertain significance for Table 1. Tests at each stage were carried out for differences between variances and the appropriate p-score was calculated.

Questions	2001 R=73 N=88 83% response Mean, Median, Std Deviation [N of + responses Likert 5-7] % of +ve responses	2002 R=71 N=103 69% response Mean, Median, Std Deviation [N of + responses Likert 5-7] % of +ve responses	2003 R=65 N=109 60% response Mean, Median, Std Deviation [N of + responses Likert 5-7] % of +ve responses	2004 R= 77 N= 107 72% response Mean, Median, Std Deviation [N of + responses Likert 5-7] % of +ve responses
1. Completing tasks was aided by the online course materials	5.7, 6, 1.1 [64] 87.7%	6.0, 6, 1.2 [62] 87.4%	5.6, 6, 1.0 [57] 83.8%	5.5, 5, 1.5 [63] 79.7%
2. Independent learning was helped through the online learning experiences	5.2, 5, 1.2 [54] 74%	5.4, 5, 1.1 [59] 80.3%	5.1, 5, 1.1 [45] 66.2%	4.8, 5, 1.3 [48] 60.8%
3. My knowledge of the course was enhanced through the online course experiences	5.2, 5, 1.4 [56] 76.8%	5.5, 6, 1.2 [59] 83.1%	5.0, 5, 1.1 [47] 69.1%	5.2, 5, 1.3 [53] 67%
4. The Discussion Boards were useful to my learning	5.3, 5, 1.2 [55] 75.3%	4.9, 5, 1.3 [44] 62%	4.6, 5, 1.3 [39] 57.4%	4.5, 5, 1.6 [44] 55.6%

Question	2001 R= 94 YES Response Results as %	2002 R=109 YES Response Results as %	2003 R=106 YES Response Results as %	2004 R= 68 YES Response Results as %
Are you familiar with using a personal computer (PC)	91	99	98	100
Are you familiar with using Windows	88	97	99	99
Have you used electronic mail	84	94	97	100
Are you familiar with the World Wide Web as a user "browsing"	91	97	99	99
Are you familiar with the World Wide Web as a web site creator	17	17	19	22
Do you have a home based Internet connection at your term time address?	69	66	82	88
Are you familiar with using Photoshop?	48	61	59	57
Have you used CAD Programs such as AutoCAD?	18	23	21	25

Performance Indicator	2001	2002	2003	2004
Class Size	88	106	117	112
TER Score*	68.1	71.25	73.8	77.7
% School leavers	71.6	88	78.9	n/a
% withdrawal rate	3	3	6	6
% Average mark	69.6	67.9	64.5	64.5
* First year class, Tertiary Entrance Ranking cut off for access to the Program and course

Discussion

Questions of significance: Table 1

Results for Question 2: "Independent learning was helped through the online learning experiences" show no significant difference between 2001 and 2002. There is a weak significant drop from 2002 to 2003 (p = 0.0485). There is almost a significant drop from 2003 to 2004 (p = 0.0683). However the difference between 2002 and 2004 is very significant (p = 0.004). This shows a drop over the time frame 2002-2004.

Responses to Question 3: "My knowledge of the course was enhanced through the online course experiences" reveal no difference between 2001 and 2002. There is a very significant drop from 2002 to 2003 (p = 0.0018). There is no significant difference between 2003 and 2004.

Finally in Question 4: "The Discussion Boards were useful to my learning", there is a significant drop from 2001 to 2002 (p = 0.0283) There is no significant drop from 2002 to 2003 nor from 2003 to 2004. If we compare 2001 and 2004, there is a very significant drop (p = 0.0001).

Completing tasks

Whilst 80% (minimum) of students agree that completing tasks was aided by the online course materials:

The amount of information on MyUni is fabulous - other courses should take note of [Teacher's] great MyUni information ...

... the best thing about this course was the online course materials

more students in 2004 than in 2001 do not agree that the online environment aids task completion - and see the requirement to access online information as a disadvantage to them - causing a significant decrease in agreement with this proposition 2002-2004 (p=0.0082) and citing:

[There should be] less reliance on students looking at MyUni [Blackboard] and more handouts instead. Accessing MyUni takes too much time

Independent learning

In 2002, 2003 and 2004, students were asked to separately rate "My ability to work independently is being increased" which revealed that in 2003 and 2004 more students believed their ability to work independently is being increased by the course than believed their independent learning was helped through the online course experiences. Is this potentially an uncomfortable outcome for the teacher who has prepared all the online course experiences? Perhaps the results rather reveal that this wired, internet savvy generation of school leavers (see Table 2) do not specifically single out online learning experiences as being relevant to increasing independence -possibly the overabundance of prepared online material militates against the very learning activities which they constitute as 'independent'. Table 4 shows Mean, [median] and standard deviation for a Likert 7 point scale where 1 =strongly disagree and 7 = strongly agree plus the number of respondents (%) who agreed to strongly agreed with the proposition in the Questions about their independent learning. There is a very significant (p = 0.0004) decline in the number of respondents who agree, from 2002 to 2004 that their independent learning was helped through the online learning experiences. There is no significant drop from 2002 and 2003. There is also no significant drop from 2003 to 2004, although it is nearly so (p = 0.0683) - the drop is only significant from 2002 to 2004. Considering the question: "My ability to work independently is being increased", there is no difference between 2002 and 2003, but is almost a significant drop from 2003 to 2004 (p = 0.0676).

Question	2002 R=71 N=103 69% response	2003 R=65 N=109 60% response	2004 R=77 N=107 72% response
My ability to work independently is being increased	5.3 [5] 1.2 71.8%	5.3 [5.5] 1.2 79.4%	5.0 [5] 1.2 67%
Independent learning was helped through the online learning experiences	5.4 [5] 1.1 80.3%	5.1 [5] 1.1 66.2%	4.8 [5] 1.3 60.8%

Students' open ended comments:

... [the course] started to introduce new tools and experiences to us.
... so much to learn about [meant] an increase in productivity.

associate the structure of the course, and the requirement for high quality group work and interactivity but independent assessment of assignments with an increase in their ability to work independently without explicitly mentioning the online learning experiences. However the requirement for weekly Discussion Board submissions is singled out as a tool for managing one's own learning relating to time management:

I found the e-research Discussion Boards quite useful as it makes me do my work every week and not in the last minutes, and I can compare my work with others once I've finished mine"

Knowledge acquisition

Knowledge acquisition is a key dimension of this and most other University courses (Graduate Attributes Working Party, 2002). Seventy five percent of teacher respondents to the University-wide study agreed to strongly agreed that they had experienced an increase in time spent on preparation and delivery of content for their web based course (Shannon and Doube, 2003b, p 484). This was exceeded only by work on administering an online course (82% agreement) and closely followed by increased time on preparation of teaching (69% agreement). The students in this course are clearly the beneficiaries of this teacher time. Once again - is this an uncomfortable result? - that teachers who are "hanging back" and waiting for conclusive evidence that online learning will result in learning improvements are not advantaging their classes' learning and knowledge acquisition? In the University of Adelaide survey 55% of teachers thought discipline area knowledge would be increased, 31% thought there would be no change, and 5% thought it would decrease for their students as a result of introducing online learning. 9% did not know. Overall, only 66% of respondents who had used online learning believed it had overall benefited their student (Shannon and Doube, 2003b, p 483). But the minimum number of students over 4 years who agreed to strongly agreed that their knowledge of the course was enhanced through the online course experiences is 67%, much larger than the 55% predicted by teachers.

Knowledge acquisition is critically related to students' engagement in active learning (Prosser and Trigwell, 1999) and students' open ended answers tell about their active learning through the online course experiences to gain knowledge:

The material is interesting and engaging and the assignments are more interesting then in other degrees; one can be more creative instead of rote learning.

Thorough information online - good material (films, article references) to support assignments

Group interaction and online research skills [were the best aspects of this course]

... through the e-research [assignment], we can begin to draw influences from other designers and study the built environs around us.

Interactivity

Students' open ended responses give their reasons both for valuing and not valuing Discussion Boards, whether assessed or non-assessed. Students comments valuing assessable Discussion Boards included:

Good introduction to Discussion Board tool. A good way to start the first assignment. I found it interesting to view fellow Discussion Board responses.

Looked where other people got their information from

Discussion Boards are useful for getting feedback and ideas from peers.

Since this was our first assignment, some of us were not sure what to write [but] information shared was similar to other people.

I used it to look at other people's understanding of the topics included.

Assessed standard of writing expected by teacher.

I used other group's inputs as inspiration for my own.

Some students did not value assessable Discussion Boards

No more Discussion Boards!! I'd rather hand in an essay than little 300 word entries.

They didn't really work too well. It wasn't very easy to chat on, due to that no-one replied for about a week, so the ideas went stale.

I didn't use them very well. It helped me to share information with group members. But in my group's case we passed on the majority of our info in our own time.

[to improve my learning] Not as many Discussion Boards, I would prefer to hand up work

Students valued assessable Discussion Boards as a means to "finding out what information I should be writing, the style and to communicate with others in my group". They were important in exposing standards for assessable written work for their first university assignment, but one student makes the point that they need to be able to see their peers' feedback comments (not marks) to complete this process: "It would be good to read the comments on others' work, but not their scores" despite the policy being that all tutor feedback is mounted on the Discussion Boards, and only marks emailed individually. Students coming from a school environment where marks and feedback are private to the individual perhaps are unaware that the feedback is the tutor's response on the Discussion Board to each entry. This point of view is further supported by the comments of students who did not value assessable Discussion Boards being based on their preference for "hand[ing] in an essay" or "hand[ing] up work" suggesting that they do not see assessable Discussion Boards as either an "essay", nor "work".

Students' comments on valuing non-assessable Discussion Boards (for e-role play assignment):

I found I communicated more as a group. This discussion board was more discussion like I thought.

[I used them] for showing of info and ideas between group members

I could see what stance & point of view other people in the group had and also other Tutes.

To discuss the assignment with peers and to organise times to meet up for further discussion. Also to share useful information.

To share and work out where others in the group were at.

To help get an idea of opposing arguments of my topic and view other group's work [who are] doing the same topic to see how that group was doing it.

I could see what stance and point of view other people in the group had, and also other Tutes.

Once I had some knowledge of its uses my group used it to plan meetings and receive feedback. It was very helpful.

This Discussion Board was useful in helping others as people's roles interact and interconnect with each other.

Students who did not value non-assessable Discussion Boards (for e-role play assignment) said:

No-one else studying my Topic used it very extensively so I didn't find it helpful for the assignment

Would have been better to meet with group in person - more effective communication.

Didn't use it much cause other people in my group didn't use it either.

Positive themes emerging are the interaction between group members being enhanced; the setting and monitoring of standards as a means of self evaluation despite the non-assessable nature of the Discussion Board, indicating that students found intrinsic value in their use. Students experienced learning isolation if others did not join in, and may have held a preference for face to face meetings as more effective, although they frequently reported using Discussion Boards to establish these meetings.

The purpose of the Focus Group conducted in 2001 was to understand more fully the impact of this new way of learning on students. Eight students participated. Seven of the eight explained that their favourite part of the course was the Banrock Field Trip, for its more 'hands on" approach, but one explained that:

The e-role play, about stake holders and stuff. That was some good experience like being able to communicate people, find out their opinions and stuff.

For the e-role play I played Jane Lomax-Smith [the Lord Mayor] and I e-mailed her and I got a fairly prompt reply.

To complete the Discussion Board Assignments, focus group participants said that their principal learning task was "using the internet" ..."because they were ... pretty recent issues, so I didn't think the library would have anything on them, so just newspapers [online] and internet were my main resources." Some participants agreed that having the availability of references and submissions on the "net" was useful:

It's good being able to hand it in on the internet. (several voices agree) That's really helpful. (Several comments together: Rather than coming in to the uni to do it. Yeah.)

... because I don't have the internet at home, but that was no problem...with gathering information I think I'd probably use the internet more than anything else as well. Especially like on the website where [the teacher] has the references and everything, how she puts those down you just click on it and go straight to the page ... that's really, really helpful. It's probably the most helpful thing.

However, not everyone agreed, citing the complications which arise when they have poor Internet access.

I didn't really like having it on the internet. Sometimes like my server's not that great, ... it screws up, so I don't have it at home and my timetable, when I ... by the time I finish I just want to go home, I don't really want to do like extra back up here, but if I had it at home [Id do] more. So I'd prefer it if it was on hard copy, a lot more. I didn't really manage my work that well.

Students then discussed the notion of looking at others' work online, and how they felt about the opportunities that Discussion Boards and other internet based hand ins gave to view others' work, concurring with the more recently gathered open ended responses that viewing the work of others assists in setting standards.

But if you ever do look on the net it's usually just to ... I find it's just to sort of if you're not sure about something just to make sure that you know how to do it properly.

Some students felt strongly about the difference between talking to their tutor to receive feedback and receiving feedback online and asynchronously.

[Discussion Boards] It was a good idea, but it wasn't really helpful cos you had to wait a couple of days to get a simple reply. Like it would've been good if you could just catch up with the tutors and have a quick talk like in 5 minutes, and then you wouldn't have to wait to get your reply to do your work.

Like why talk to people online, over a computer, when you can go and do it in person. I mean ... everyone's forgotten how to communicate with people I think. It's a disgrace.

Staff resourcing issues impact on students learning in this regard. All tutors in the course are hourly paid casual staff, and the coordinator is half time:

I was thinking it would be really helpful if we had people we could just go and talk to during the day, like the lecturers and tutors just don't make themselves available enough. ( Yeah that's sensible ) I've tried to catch up with people, and just sort give up, I don't even bother now, I just ask my mates cos it's just too much of a hassle trying to catch up with anyone.

The vital social aspect of learning for first year students (McInnes, James and Hartley, 2000) is exposed in the following comments about using Discussion Boards in lieu of face to face discussion in tutorials:

It's easier to go and talk to people in person than about it on the computer. That's the way I've got most of my information. Rather than talking about it online, actually going up and asking people questions, asking your friends and stuff. I think at the beginning of the course, that was also a way to like you know, meet new people cos it was an excuse you know to get a conversation started, and I think that was really ... really good. This Discussion Board, I think it discourages people from doing that and I don't like it.

It's good having a discussion which is more than 2 way as well, having the whole group there at once.

With the online thing, I only know half the people's names still, and the only reason I would say which parts of the course I got most out of was the [Banrock] trip, is because we were all interacting, you know, like, meeting people. And at least we were actually talking to them, otherwise if it's just on the computer it's just a name.

The declining evaluation results for the usefulness of Discussion Boards to my learning are of concern in that students in 2004 are more computer literate than their peers in 2001, so the reasons for their concern relate more to the a-synchronicity aspects not allowing a genuine discussion, and not viewing their 300 or 500 word researched and referenced Discussion Board entries as work or equivalent to an essay. The mode of working to prepare a Discussion Board entry is the same as preparing an essay, but the online submission and display does not yet sufficiently engage students in genuine discussion.

I just thought it was like a bit of a marking part of the assignment where the teachers have a look at what you've found out and they mark it and I didn't actually realise it was supposed to be used for asking each other questions and actually discussing stuff, which I s'pose that's what the name says, but it wasn't.

Conclusion

Further, this case study reveals that at least two thirds of the students believe that their knowledge of the subject/course is improved through the online course experiences whilst studying at a University at which only 55% of their teachers engaged in online learning thought discipline area knowledge would be increased for their students. There is an uncomfortable 'gap' between students' evaluations of the online learning environment, and the beliefs of some of these teachers because 80% of these students perceive that completing tasks was aided by the online course experiences, and only 66% of teachers at their University believe that overall online learning benefits their students. At the same time 75% of teachers with online courses also agreed to strongly agreed that they had experienced an increase in time spent on preparation and delivery of content for their web based course, but the argument can now be mounted that students are benefited by this investment in their course task completion, and knowledge acquisition, in this course.

Acknowledgements

Q: This course could be changed in the following ways to improve my learning?
A: Less evaluations.

Profound thanks to students who willingly complete evaluations to improve teaching inputs and to John Petkov, Head, Applied Statistics Unit, UniSA Whyalla Campus, who provided statistical analyses.

References

Graduate Attributes Working Party (2002). University of Adelaide Graduate Attributes. http://www.adelaide.edu.au/DVC/students/draft_graduate_attributes_program.html [30 Jul 2004]

Graham, C., Cagiltay, K., Lim B.-R., Craner, J. and Duffy, T. M. (2001). Seven principles of effective teaching: A practical lens for evaluating online courses. The Technology Source, March/April. http://ts.mivu.org/default.asp?show=article&id=839 [30 Jul 2004, verified 26 Oct 2004]

McInnes, C., James, R. and Hartley, R. (2000). Trends in the First Year Experience in Australian Universities. Evaluations and Investigations Programme, Higher Education Division, DETYA. Canberra: AGPS. [verified 26 Oct 2004] http://www.detya.gov.au/archive/highered/eippubs/eip00_6/fye.pdf

Prosser M. and Trigwell, K (1999). Understanding Learning and Teaching: The Experience in Higher Education, Open University Press, 325 Chestnut Street, Philadelphia, PA 19106

Shannon, S. (2001a). Course resources. [30 Jul 2004] http://www.arch.adelaide.edu.au/games/e-research/

Shannon, S. (2001b). Course resources. [30 Jul 2004] http://www.arch.adelaide.edu.au/games/e-roleplay/

Shannon, S. (2001c). Course resources. [30 Jul 2004] http://www.arch.adelaide.edu.au/games/bird_hide/

Shannon, S. and Doube, L. (2004). Valuing and using web supported teaching: A staff development role in closing the gaps. Australasian Journal of Educational Technology, 20(1), 114-136. http://www.ascilite.org.au/ajet/ajet20/shannon.html

Shannon, S. and Doube, L. (2003a). Factors influencing the adoption and use of web-supported teaching by academic staff at the University of Adelaide. Report prepared for the Deputy Vice Chancellor (Education) & Provost, supported by a University of Adelaide Learning and Teaching Development Grant. http://www.adelaide.edu.au/hr/development/academic/Final%20Report.pdf

Shannon, S. and Doube, L. (2003b). Factors impacting on the adoption and use of web-supported teaching by academic staff. In Interact, Integrate, Impact: Proceedings 20th ASCILITE Conference, pp 476-485. Adelaide, 7-10 December. http://www.adelaide.edu.au/ascilite2003/docs/pdf/476.pdf

Shannon, S. and Doube, L. (2003c). Predicting the future use of web-teaching by academic staff. In Interact, Integrate, Impact: Proceedings 20th ASCILITE Conference, pp 696-699. Adelaide, 7-10 December. http://www.adelaide.edu.au/ascilite2003/docs/pdf/696.pdf

Shannon, S. and Leverett, K. (2004). Course resource and material website http://www.library.adelaide.edu.au/guide/arch/BEI/2004/index.html [30 Jul 2004]

Authors: Dr Susan J. Shannon, School of Architecture, Landscape Architecture and Urban Design, The University of Adelaide, ADELAIDE 5005. susan.shannon@adelaide.edu.au http://www.arch.adelaide.edu.au/~sshannon/ Susan practised architecture until 2000 when she joined the School of Architecture as a Course Coordinator. With a PhD in learning and teaching and assessment in architecture, she has a particular interest in evaluating learning outcomes, particularly related to the introduction of computers in learning. Please cite as: Shannon, S.J. (2004). Questioning the teacher's comfort zone with online learning: A 4 year case study of students' perceptions of online learning. In R. Atkinson, C. McBeath, D. Jonas-Dwyer & R. Phillips (Eds), Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference (pp. 831-840). Perth, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/shannon.html

© 2004 Susan J. Shannon
The author assigns to ASCILITE and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The author also grants a non-exclusive licence to ASCILITE to publish this document on the ASCILITE web site (including any mirror or archival sites that may be developed) and in printed form within the ASCILITE 2004 Conference Proceedings. Any other usage is prohibited without the express permission of the author.