Conference logo
[ ASCILITE ] [ 2004 Proceedings Contents ]

Differences in the quality of learning outcomes in a F2F blended versus wholly online course

Nian-Shing Chen
National Sun Yat Sen University
Craig Zimitat
Griffith Institute for Higher Education
Griffith University
Many researchers have focused on the question of whether or not there is a difference in learning effectiveness between online and on campus courses. Using the SOLO taxonomy, we explore the quality of learning outcomes of students enrolled in "Computer Network and Internet" classes offered by an institution in wholly online and face to face (blended) learning modes. Students enrolled in the F2F course made much less use of the e-learning resources than those in the virtual course. Although there were significant differences between the two classes with respect to pre-test scores, at the end of the semester, there was no significant difference in post test scores or SOLO ranking. This is not to say that the two classes resulted in the same individual improvement in learning outcomes. This methodology, if adapted to include more assessment items and larger cohorts of students might be a useful model for evaluating the pedagogical effectiveness of various e-learning courses.

Introduction

The last decade has seen a proliferation in degree programs and courses offered wholly online, or on campus with supplementary support provided online. Student's experiences of online learning has generally been positive though in 2002, fewer than 6% of students took online courses for credit, with only half reporting that the course was a worthwhile learning experience (Jones, 2002). With the proliferation of online courses there have been concerns about the quality of the course offerings and the educational outcomes achieved by participants (Barbera, 2004; Benigno & Trentin, 2000). Often evaluations of online courses tend to focus on comparisons with traditional courses, rather than examine the real issue of whether quality learning is happening in online courses.

The pedagogical effectiveness of computer based and online learning experiences has been under active research for many years (Hartley, 1996). Whilst pre- and post-tests have often been used as measures of instructional effectiveness, they often focus on recall if facts and when used alone do not reveal sufficient information about the quality of learning, or depth of understanding reached by students. This is in part because of different types of assessment in the different courses, or because assessment items do not involved higher order learning. One framework for considering the quality of learning outcomes is the Structure of Observed Learning Outcomes (SOLO) taxonomy (Biggs & Collis, 1982). Bloom's taxonomy can be used for planning examinations, but SOLO is best used to review learning outcomes (Biggs, 2002). The five level SOLO taxonomy, when applied to student responses to examination questions, can be used to see if the learner has remembered only a number of points without relating them in a meaningful way or has built meaningfully related structures in the newly learned material, that has been extended by relationship with prior knowledge related to the topic. Thus the primary goals of this study were to use the SOLO taxonomy to explore the quality of learning outcomes from students completing a wholly course with those completing a parallel on campus course.

The context of this study

The Computer Networking and Internet course is offered by the National SunYat Sen University (NSYSU), Taiwan in two parallel modes. Those enrolled in the virtual course (n=62) were largely industry based working adults taking the course through NSYSY online whilst those enrolled in the blended course (n=30) were postgraduate Masters students at the MIS Graduate School, NSYSU. The virtual course (wholly online) was based upon a pure e-learning model: a weekly, two hour interactive lecture delivered though desktop videoconferencing (from Australia) was followed by an interactive one hour tutorial conducted by synchronous chat with electronic whiteboard facility with follow up discussions amongst students facilitated through an asynchronous discussion forum. The F2F course was taught on campus through two hour video based lectures, supported by the same synchronous tutorial hour and an asynchronous discussion forum offered to the virtual class. The on campus course used tutored video instruction (Dutra et al, 1999) where the teaching video was the recorded lecture from the virtual course, replayed, with the additional support of a faculty member to further explain and clarify matters raised by the students in the classroom. Our particular interests related to the educational effectiveness of these two courses.

Table 1: Computer networking and internet course descriptions

F2F Class (Blended Course)Virtual Class (e-Course)
  • Weekly 3 hour lecture in a classroom
  • Weekly 1 hour online real time tutorial
  • Final report presented in class
  • Weekly 2 hour online real time class
  • Weekly 1 hour online real time tutorial
  • Online final special project
  • Online discussion board and queries
  • Homework assignment
  • Small group report discussion
  • Online discussion board and queries
  • Online small group discussion
  • Online small group report discussion

Methods

At the beginning of semester, a progress test (PT) (Whelan et al, 2002) with 16 MCQ and two essay type questions was administered to students enrolled in both courses. The progress tests, designed using the SOLO framework, contained questions designed to assess knowledge ranging from that expected of undergraduates entering the course to knowledge well beyond that expected of graduates of the course. This design of the examination was such that no student enrolled in the course would gain a 100% result, but that all students could improve their test scores during the course. There was a range of questions at each SOLO level, though most were at higher levels indicative of understanding. Each question was given a number of marks equivalent to its level of understanding required to answer it, according to SOLO. Items reflecting attainment of uni-structural understanding (characteristic of SOLO Levels 1-2) carried 1 mark, those reflecting multi-structural understanding (SOLO Level 3) carried 2 marks, and those reflecting relational or extended abstract understanding (SOLO Level 4-5) carried 3 marks. Student examination scripts from the essay type questions were scrutinised using the SOLO and rated independently by two teachers and agreement reached through moderation (Zimitat & McAlpine, 2003). The PT score was determined by summing marks for each correct item in the test. Differences in scores on the pre- and post-PT were examined for the two classes.

Results

Students in the F2F class were able to view and participate in the online activities on their own volition, however their participation was limited. Students in the F2F made an average of 8 discussion forum posts compared with an average of 52 posts in the virtual class. Students in the virtual class participated and reviewed Office Hours video on demand an average of 48 times throughout the course, compared with average of 8 viewings by those in the F2F class. The number of course pages accessed by students in the virtual class was two fold higher than that of F2F students.

The pre-PT results of the students in the F2F class were significantly higher than those enrolled in the virtual class (Table 2). The students in the virtual class had lower scores, but there was a greater range of scores than that seen in the F2F class. On the post PT, students in the F2F had higher mean scores, but there was no significant difference between the mean scores for the two classes. In the case of post PT scores, there was a greater range of scores in the F2F class than in the virtual class. The SOLO based analysis of questions in the pre-test revealed that students in the F2F class had a deeper understanding of the subject matter on entry to the course compared with the students in the virtual class (p<0.05) (Table 3). On exit from the course, there was no significant difference between the groups in terms of their depth of understanding of the examination topics. This is not to say that the overall learning 'effect' of the two courses was identical. Plotting each student's performance on the pre and post questions (Table 4) shows a clustering in the lower right corner. This reflects increased post-test scores compared with the respective pre-test scores. In the Virtual and F2F classes, 33% of students increased one SOLO rank from SOLO 3 to SOLO 4. It is noteworthy that some students in the F2F class did not reach SOLO 3 and some regressed.

Table 2: Pre- and post-PT test results for the two classes

Progress
Test		ClassStudentsAverageStandard
Deviation
Pre*Virtual class3749.864917.6575
F2F class2160.238112.1363
PostVirtual class3760.135115.0225
F2F class2163.809520.8509
* Significant difference between e-class and Blended class t=-2.387, df= 56, p<0.020

Table 3: Pre- and post- SOLO scores of Virtual and F2F Classes

SOLO RankVirtual Pre
(n=37)		Virtual Post
(n=37)		F2F Pre
(n=21)		F2F Post
(n=21)
113%00% 00%00%
238%00% 15%314%
32157%13 35%838%15%
41130%23 62%1152%1571%
513%13% 15%210%
Mean SOLO score3.213.683.573.76

Table 4: SOLO Change matrix. Changes in individual pre- and post- SOLO ranking

Table 4

Discussion

There were three important finding arising from this study. First, students enrolled in the parallel courses in this study achieved comparable learning outcomes. Second, the use of SOLO has potential for uncovering the depth of learning occurring in (online) courses. Third, even virtual classes that mirror traditional F2F courses can achieve sound learning outcomes with a diverse range of students.

Learning outcomes for the two courses were comparable when SOLO was used to inform assessment planning and review learning outcomes. The use of grades in evaluating educational outcomes is based upon the often incorrect assumption that the course content is adequately sampled in the assessment regimen and assesses understanding rather than lower order cognitive skills (Biggs, 200). In this case, multiple choice questions and two open ended questions used a pre- and post-tests provided some standardisation for comparison between the two courses. However there were other assessment data that could have been analysed using SOLO to determine more broadly the extent of learning achieved in this course. But since different assessment tasks were used in the two courses, for the purposes of direct comparison between the courses it was inappropriate. If the exercise was to be repeated, it would be useful to consider the use of a broader range of assessment items common to both courses that could available for analysis.

The majority of students in the virtual class demonstrated improvements in learning compared with those in the F2F class. Given that the participants in the virtual course entered from industry, often with a limited formal knowledge of the subject matter, this was a welcome sign. Course website logs show that these students made extensive use of online materials, including the replaying of recorded lectures and the use of the discussion forum. The greatest potential for e-learning lies in harnessing the power of technology to support students from non-traditional backgrounds (Veronikas & Shaughnessy, 2004). In this virtual course, the orchestration of a range of learning experiences and resources appeared to be near optimal in supporting students less prepared for academic study.

The differences between these two courses point to some factors necessary for the educational effectiveness of online courses. Communication tools - discussion forums, email and interactive video and whiteboard - were infrequently used by students in the F2F course. However in the wholly online course they were almost over used. Morris & Zuluaga (2003) and Koory (2003) emphasised the importance of a range of mechanisms to facilitate staff and student interactions for successful learning in online courses. For students (in this case in different cities in another country), these mechanisms appeared to provide equivalent academic and peer interactions that would normally occur in the F2F classroom. Examining the relationship between these communication mechanisms and learning outcomes is an area for further research.

Acknowledgement

We would like to acknowledge the contribution of our students to this work, as well as the constructive feedback on manuscript drafts received from colleagues at the Griffith Institute for Higher Education. This study was supported by a grant (NSC92-2511-S-110-001-X3) from the National Science Council: Research and effectiveness evaluation of a digital network learning system used to teach primary school mathematics (2/2).

References

Barbera, E. (2004). Quality in virtual education environments. British Journal of Educational Technology, 35, 13-20.

Benigno, V. & G. Trentin. (2000). The evaluation of online courses. Journal of Computer Assisted Learning, 16, 259-270.

Biggs, J.B. (2002). Teaching for quality learning at university. Buckingham: Oxford University Press.

Biggs, J.B. & Collis, K.F. (1982). Evaluating the quality of learning. Academic Press: New York.

Dutra, J., Gibbons, J. F., Pannoni, R. L., Sipusic, M. J., Smith, R. B. & Sutherland, W.R. (1999). Virtual Collaborative Learning: A Comparison between Face-to-Face Tutored Video Instruction (TVI) and Distributed Tutored Video Instruction (DTVI). A Sun MicroSystems Research Technical Report. [verified 22 Nov 2004] http://www.sunlabs.com/technical-reports/1999/smli_tr-99-72.pdf

Hartley, S. et al. (1996). Enhancing Teaching Using the Internet: Report of the Working Group on the WWW as an Interactive Teaching Resource. In ACM SIGCSE Integrating Technology into Computer Science Education, Vol 28, Special Issue, 1996.

Jones, S. & Madden, M. (2002). The Internet goes to college. Pew Internet & American Life, 15 Sept. [verified 10 Oct 2004] http://207.21.232.103/pdfs/PIP_College_Report.pdf

Koory, M. A. (2003). Differences in learning outcomes for the online and F2F versions of "An introduction to Shakespeare". Journal of Asynchronous Learning Networks, 7(2). http://www.sloan-c.org/publications/jaln/v7n2/v7n2_koory.asp

Morris, E.J.S. & Zuluaga, C.P. (2003). Educational effectiveness of 100% online IT courses. In G. Crisp. D. Thiele, I. Scholten, S. Barker and J. Baron (Eds), Interact, Integrate, Impact: Proceedings 20th ASCILITE Conference. Adelaide, 7-10 December. http://www.ascilite.org.au/conferences/adelaide03/docs/pdf/353.pdf

Veronikas, S.W. & Shaughnessy, M.F. (2004). Teaching and learning in a hybrid world: An interview with Carol Twigg. EDUCAUSE Review, 39(4), 50-62. http://www.educause.edu/apps/er/erm04/erm0443.asp

Whelan, A.M., Mansour, S. & Farmer, P. (2002). Multiple choice progress examination: Comparison of traditional and PBL-based curricula. Interim analysis. PE6. Association of Faculties of Pharmacy of Canada Annual Conference, Montreal.

Zimitat, C. & I. McAlpine. (2004). Student use of CAL and associated learning outcomes. Biochemical and Molecular Biology Education, 31, 146-150.

Authors: Nian-Shing Chen, National SunYat Sen University, Taiwan. Email: nschen@cc.nsysu.edu.tw
Craig Zimitat, Griffith Institute for Higher Education, Griffith University, Brisbane. Email: c.zimitat@griffith.edu.au

Please cite as: Chen, N.S. & Zimitat, C. (2004). Differences in the quality of learning outcomes in a F2F blended versus wholly online course. In R. Atkinson, C. McBeath, D. Jonas-Dwyer & R. Phillips (Eds), Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference (pp. 175-179). Perth, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/chen.html

© 2004 Nian-Shing Chen & Craig Zimitat
The authors assign to ASCILITE and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to ASCILITE to publish this document on the ASCILITE web site (including any mirror or archival sites that may be developed) and in printed form within the ASCILITE 2004 Conference Proceedings. Any other usage is prohibited without the express permission of the authors.

[ ASCILITE ] [ 2004 Proceedings Contents ]
This URL: http://www.ascilite.org.au/conferences/perth04/procs/chen.html
HTML created 22 Nov 2004. Last revision: 22 Nov 2004.