ASCILITE 2004: McAlpine, Wilson, Russell and Cunningham

[ ASCILITE ] [ 2004 Proceedings Contents ]

An online diary as a research and evaluation tool for first year physics

Iain McAlpine, Kate Wilson, Carol Russell and Maria Cunningham
The University of New South Wales

Concerns about the quality of student learning in first year Physics courses at UNSW, particularly laboratory activities, led to a decision to introduce some new ideas about learning activities. A comprehensive evaluation of student learning in these courses was initiated to clarify the student perceptions of first year courses. An online diary, using surveys and a discussion in WebCT, was used to gather regular feedback from students. A number of student volunteers responded to the surveys and answered a range of open ended questions at regular intervals over three semesters. Open ended text responses were analysed using NVIVO software to identify themes and quantify the responses to each theme. This led to the introduction of open ended group learning tasks. The paper details the process and outlines enhancements to the courses and their effectiveness.

Introduction

The quality of the student experience of learning in the first year can be an important influence on student success and satisfaction in their undergraduate degree program (McInnis, James, & Hartley, 2000). While most students are motivated to study and have an intrinsic interest in their studies, a significant minority find the experience of university study to be uninteresting and frustrating. These students may withdraw or persist with a negative attitude to their studies. As the first year of study has a critical role in inducting students into the practices and modes of thought of their disciplines, the learning experiences at this time play a large part in shaping student perceptions of their study at university. As part of an extensive project to evaluate and improve first year physics at the University of New South Wales, WebCT was used as an online "diary" to collect data from students throughout 2002 and semester 1, 2003. Data gathered provided insight into the student experience in first year physics. Text responses to open questions were supplemented by online multiple choice questions and by face to face focus group discussions with the same group of students. Initial analysis of the data gathered each semester contributed to an action research process, in which changes were made to the course. These changes required both students and staff to take new approaches to learning processes in first year Physics courses. The online diary was also used to evaluate the effectiveness of these changes.

Student learning in physics laboratories

Science courses present a number of challenges relating to the quality of the student learning experience. These include matters such as laboratory sessions that are 'cookbook' activities, uninteresting to the students and often disconnected from the other course materials (Belcher, 2001), lectures with no real world context, and learning problems and assessment tasks that are formula driven with little connection to real world contexts (Hanson & Wolfskill, 2000). Science, however, does not have to be taught in these ways. There are a number of aspects of scientific inquiry that can be used to encourage higher levels of engagement and more active learning.

Learning in science courses can be seen as an active process of inquiry in which theoretical understanding is used to form ideas about the nature of the topic under investigation. Some approaches to science learning and teaching seek to apply the stimulus of inquiry and practical investigation to make science courses more engaging, and to lead to improved learning outcomes. Redish (1999) reports very significant improvements in physics learning by replacing lectures with the guided discovery workshop approach. The workshop physics approach combines brief lectures, small group experimentation and class discussion, using open ended problems as well as those that have specific numerical solutions (Redish, 2002). These approaches emphasise group work as an integral part of the learning process with the aim of encouraging the development of communication and teamwork skills while leading to a deeper understanding of and greater engagement with the science. A problem solving model for Physics courses developed by Heller and Heller (1999) emphasises cooperative group work in solving context rich problems. The guided discovery approach described by Redish (2002) and the Technology Enabled Active Learning (TEAL) approach developed at MIT (Belcher, 2001) all emphasise small group learning activities. The TEAL approach uses technology to support small group experimentation, and to communicate results to the whole class.

These approaches illustrate some of the problems that have emerged in science courses at other institutions, and approaches that have been taken to enhance course outcomes. The following sections outline the UNSW context and the reasons for commencing a project to improve Physics laboratory courses with an online diary and surveys as evaluative tools

First year physics laboratory courses at UNSW

Perceived shortcomings in approaches to teaching first year Physics led to a proposal to investigate students' experiences in physics classes. Anecdotal evidence from teaching staff indicated that they were dissatisfied with student performance and attitudes in some parts of the course, in particular tutorials and laboratories. Poor attendance at tutorials and lack of student preparation prior to attending tutorials led to discussions of whether tutorials should be dropped from some courses, as they are relatively expensive to run due to the high staff to student ratios required and students did not seem to be gaining very much from them. Most tutorials in first year physics courses at UNSW were at that time traditional recitation sessions where a tutor presents solutions to a previously set tutorial question sheet. In a few courses more student centred workshop approaches had been adopted, and since then workshop style tutorials have been introduced into further courses.

At the beginning of 2002, the laboratory program consisted of a set of experiments, generally of low inquiry level, in which an experimental aim and method were given, and often the (expected) result was known in advance (for example the measurement of a constant, or the total force on a body in equilibrium). The students were given explicit instructions on how to perform the experiments, which took place weekly in a two hour laboratory session, during which students recorded their results by filling in blanks in their laboratory manual. Informal student feedback showed that students considered the labs easy marks, and generally neither challenging nor engaging. This impression was supported by discussions with the demonstrators.

All courses have standard multiple choice student satisfaction surveys administered at the end, and generally the first year Physics courses rated on a par with other science subjects. However these sorts of simple student feedback surveys often fail to give any insight into students' learning experiences. Hence it was decided that an in-depth study of the student learning experience in first year physics should be carried out. The diary project was developed to meet this need.

Research methodology

Overall, this was an enquiry focussing on student learning processes as suggested by Biggs (1999). It involved gathering and analysing students' comments on how they experience 1st year physics courses, to deduce what is effective and what is ineffective for their learning in this context. The methodology broadly follows an action research approach for higher education, in which there are repeated cycles of action, observation, reflection/analysis, and planning for the next semester's activities (Zuber-Skerritt, 1992).

A detailed analysis of text to find emergent concerns in the first semester of 2002 laboratory experiments informed experiment design for the first semester of 2003. Further analysis of the combined semester 1 and semester 2 feedback has built up a picture of student learning patterns in 1st year Physics as a whole. This is providing more extensive and rigorous data - by identifying which learning issues students mention most frequently. These can be built into future routine evaluations.

Research aims

This research is part of a broader project to enhance Physics courses at UNSW. The aims for the diary project were:

To evaluate student perceptions of first year laboratory courses to provide data that may identify changes that may be needed to improve these courses.
To evaluate the outcomes of changes to courses introduced as a result of the initial stage of the project.

These aims represent two stages of the project, the initial investigation to determine student perceptions before any changes were introduced, and the first major enhancement of the course.

The online diary

Each week the students logged on to a WebCT site to record their thoughts on the week's learning experiences. The 'diary' was an online survey consisting of open ended questions, multiple choice questions, and space for general comments. The multiple choice questions asked the students to rank on a five point scale the conceptual difficulty, technical difficulty, their enjoyment of and interest in that week's laboratory experiment. Occasionally throughout the semester other multiple choice questions were used, for example in the final online survey each semester students were asked to rate to what extent the course had helped them to develop a range of generic skills. They were also asked to comment on this in two open questions.

The students occasionally used the WebCT discussion board to suggest questions which they wanted to answer, and these were also added to the survey, as were suggestions made in the focus group meetings held two or three times per semester.

Approximately 15 students were selected each semester from a list of volunteers. A small payment was offered for participation to compensate for the time required to complete the diary. The selection process aimed to achieve:

A range of courses
A range of home disciplines
A range of abilities, and
Students who were not in classes taught by the investigators.

It is always a challenge to find a representative group of students, and it is likely that those selected were more opinionated or more in need of financial assistance than the average student. But we believe that overall the students were neither more nor less able than average, since the range of learning styles and preferred learning activities described by the students indicate a varied sample. Discussions during focus group meetings indicated a range of backgrounds, from high achievement in secondary school physics programs, to no prior experience of physics at high school, and a range of approaches to their physics study from very shallow, fact and formula memorising to much deeper learning approaches.

Why WebCT?

WebCT was used for a number of reasons. First, it is the internet course environment used at UNSW, so students should have been reasonably familiar with it, and comfortable with using various features such as the survey and communication tools. It was also easy to create a "dummy" course and enroll the student participants into the course.

Second, the survey tool could be used to gather lengthy text responses to open questions, such as "what did you learn in lab this week?" and "What helped you to learn?" in electronic form so that text analysis could be carried out without the need for data entry. This is a major advantage over having students complete a "pen and paper" journal, which subsequently needs to be carefully transcribed by a researcher who must grapple with difficult to read handwriting.

A third advantage is that responses could be reviewed quickly each week so that the next week's questions could be modified and added to based on arising issues. In semester 1, 2003, students were asked if they could suggest any additional questions to add to the online diary, and typically one or two questions were added each week either as one off questions, for example those relating to mid-semester exams, or regular questions which related to ongoing course activities. This would not be possible with a paper based journal, without regularly asking the students to submit their journals and them hand them back again later, which would be inconvenient both for the students and the researchers.

A disadvantage is that students are limited to completing their diary entries only when they have internet access, however this did not seem to be an issue for the students who contributed to this study. Internet access is widely available on campus, and most of the students involved in the project had internet access at home. The flexibility of the online diary was particularly valuable to researchers, as it enabled the student responses to be viewed at leisure, without having to collect and return paper based diaries.

Data analysis

The greatest advantage of using webCT for data collection was the removal of the need for data entry on the part of the researchers, a very substantial saving in time. As student responses were gathered, the multiple choice questions could be used to supply quick, quantitative data on student experience of courses, in particular the lab course. However the bulk of responses collected were open text responses which varied in length from one or two words to over a page in response to a single question. In all, around 300 pages of text responses were collected.

These responses were analysed using the NVIVO text analysis software (Walsh, 2003). The text was coded manually on learning topics and learning issues - initially basing the codes on the students' own words and grouping these into themes as patterns began to emerge. The software allows exploration of the frequency of like comments - for example, the balance of positive and negative comments about the role of tutors, or the identification of the most common learning issue.

The researcher doing the coding analysis was from a central support unit and, although familiar with Physics education in general, had no knowledge of the design and operation of the experiments, nor what the students were supposed to learn from them. So the judgements made were on the basis of the student responses only. Although the analysis process may have missed a few comments, the pattern of comments is likely to be reliable, especially for those issues where there were no specific question prompts.

The diary project ran for three consecutive academic sessions and, with each iteration, the questions became more focused on the concerns already identified, while still offering scope for open comment. One advantage of using the NVIVO software was that, increasingly, the question structure could be used to code these responses automatically.

With a large and continuing survey, using the NVIVO software to analyse online diaries would reduce manual analysis work. However, ongoing qualitative analysis in this depth may have limited returns in relation to the time and costs.

In this context the main advantage of using the software was in providing systematic and quantified evidence in a discipline where practitioners are unfamiliar with qualitative methods - without restricting the initial research to multiple choice questions. Future multiple choice questionnaires can now be focused on issues that students themselves have raised in this context, rather than on the initial assumptions of teachers.

Results for semester 1

In semester 1 of 2002, students answered questions on the laboratory experiments they had done each week, and what they had learnt from it. Table 1 shows the top ten learning issues mentioned in order of decreasing frequency, with a breakdown of the number of specifically positive and negative comments for each.

Table 1 represents a classification of comments from the diary into categories, with a tabulation of the numbers of positive and negative comments. The diary evaluation revealed that students had a generally negative view of laboratory courses. Table 1 shows significant numbers of negative comments on equipment, level of interest generated, instructions, level of challenge, relevance to the course, the time spent on the tasks and the amount of hands on activity. Students had predominantly positive comments about the support from the tutors and form working with other students. This clarified some of the staff perceptions of lack of student engagement with first year laboratory courses.

Changes for Semester 2

Some of the key issues that emerged from the semester 1 evaluation were a low level of engagement with the laboratory activities. These were seen by the students as a source of easy marks, but the level of perceived challenge was extremely low. Some students could see that they were not learning much from the laboratory activities as the outcome of the experiment was usually known in advance, making the task a simple matter of following a procedure.

Table 1: Learning issues in the laboratory in semester 1 of 2002

Issue Total no
of mentions Positive Negative

equipment and facilities 87 18 (appreciated using) 63 (faulty, not enough, hard to use)

tutors/demonstrators 70 45 (helpful) 24 (unhelpful)

engagement 53 17 (fun, interesting) 34 (boring)

instructions 51 22 (helpful, easy to follow) 26 (confusing, hard to follow)

preparation 46 23 (helped in lab) 19 (unhelpful, not done, too hard)

challenge 25 2 (challenging) 21 (easy)

peer support 25 17 (other students helped) 7 (worked alone or others unhelpful)

relevance to course 22 11 (linked to lectures, etc.) 10 (unrelated or badly timed)

time taken 17 1 (about right) 15 (too short or too long)

hands on 10 4 (learnt from or enjoyed doing) 6 (didn't do enough)

While the laboratory activities were seen as a standard and safe way to run laboratory courses, they were not really meeting the students' educational needs. The staff involved in teaching the laboratory courses decided to try an open ended project as a learning task. The students were asked to select an area of interest and to design an experiment, to carry it out and to write a report and to make a presentation to the class on their findings. This was done as a group activity, to encourage the students to develop communication, teamwork and negotiation skills as well as learning scientific research and reasoning capabilities.

To establish the level of challenge and engagement for the students, and the challenges for staff in introducing, facilitating, and managing the process, projects were introduced to one course on a trial basis in Semester 2, following the initial diary evaluation. Students did a range of practicals in the first half of the semester, and spent the second half on their group projects. The online diary was used again in second semester. This involved a different cross section of students, some of whom were in the class that had the projects introduced.

Results for semester 2

In semester 2, the students answered open questions about what helped and hindered their learning in the main parts of the course: lab experiments, tutorials, lectures and project work (introduced for some students as a pilot programme).

Although the questions asked about what helped and hindered learning in these various course components, the students frequently mentioned, unprompted, other aspects of their study. These are listed in Table 2.

Table 2: Emergent issues in semester 2 of 2002

issue total no. of
comments positive
comments negative
comments

tutors and demonstrators 179 106 73

integration of components 101 34 67

prelab work 101 58 43

textbook 99 83 16

lecturer 77 43 34

lecture notes 68 49 18

lab manual 65 34 31

problem practice 58 45 13

web 51 25 26

Over both semesters, there were a number of comments, prompted and unprompted, on the main programmed learning activities (lectures, tutorials, projects, lab experiments). These are listed in Table 3, with some typical examples.

Table 3 provides an interesting illustration of the effectiveness of the introduction of the projects in semester 2. While the comments on the 'normal' laboratory courses and tasks were split evenly between positive and negative, comments on the projects were predominantly positive. The online diaries revealed a major change in attitude among the participants in the diary project.

Table 3: Comments on the main course components

course component total
no. of
comments positive comments
[typical example] negative comments
[typical example]

lectures 125 35
[Richard is very good at explaining tricky concepts, he gives real life examples ... lots of demos (always good).] 81
[I found the lectures extremely boring. They were difficult to understand and made learning physics difficult.]

tutorials 82 36
[... tutorial problems, I knew which questions I could do or not, so in tutorial I could ask the tutor about those questions.] 49
[There needs to be way more teaching, because at the moment it is just some dude writing solutions on the board.]

projects 39 35
[I really liked the lab projects because it gave me the opportunity to pursue a topic of physics that I had particular interest in. Working in a group also helped in enhancing my organisational, group work and communication skills.] 1

lab experiments 26 13
[...felt good to know that I understood what was being taught by being able to apply it to the real world.] 13
[I was doing experiments I had no idea about many weeks before we discussed them in lectures.]

Outcomes from the diary project

The results described above resulted in many discussions amongst staff on the purpose of various course components and whether these purposes were being achieved. In particular, the findings raised concern over the apparent lack of development of generic skills such as problem solving and communication skills which it had been assumed were being developed through laboratory classes, and to a lesser extent in tutorials. This prompted a closer look at the laboratory course, and some modifications to the program on a trial basis in semester 2, 2002, and then on a more extensive basis in 2003. In addition, the result that overall the students were not finding laboratory classes challenging enough, has resulted in a further look at individual experiments.

The generally negative response to tutorials is also being addressed. However, this is a more complex problem to tackle, and the causes for the negative responses come from a variety of problems including poor preparation by both the students and the tutors, and a lack of agreed learning objectives, and hence teaching strategy, amongst the lecturers and the tutors.

The immediate outcomes from the online diary feedback from 2002 were:

The introduction of open ended research based laboratory projects for a pilot study with a small group of students in session 2, 2002.
Evidence of the value of the laboratory projects piloted in semester 2, supporting their introduction for all students in 2003. Laboratory projects are now offered to all physics students undertaking the full laboratory component of the course.
Continuation of the online diary project in semester 1 2003, with a focus on gathering more data on student responses to the laboratory project, and to other changes in the course.
Recognition of the need to undertake ongoing evaluation of student response to courses and course components. An efficient method for obtaining feedback from students is currently under development.
The initiation of a project to break down barriers between lectures, tutorials and laboratories, providing students with a more integrated learning environment. The first stage, the development of tutorials that engage students in an active learning environment is now being implemented on a trial basis, and is likely to be extended in 2004 -2005.

Discussion and conclusions

Analysis of the student diaries reveals patterns of lack of student engagement, alienation and loss of interest, similar to those reported in the literature. The diary project has significantly changed the way in which the laboratory component of UNSW first year physics courses is delivered. Dialogue with students has enabled the identification of problems commonly perceived by the first year student cohort, and the monitoring of their reaction to changes introduced as a result. The success of the changes to the laboratory programme has led to plans for a much wider reshaping of the first year physics program. An additional outcome of the diary project has been an increased focus on educational research within the School of Physics.

The diary project was valuable as a one off exercise in evaluating student experience of the existing courses and for investigating the efficacy of newly introduced activities, such as the open ended laboratory projects, which were designed to be more engaging to students. The resources and support required to run an online diary project and carry out the thematic analysis of text responses are perhaps not justifiable as an ongoing exercise for routine evaluation. However, at a time when the curriculum is being reviewed it provided useful information for redesigning some aspects of the course, and identified issues for ongoing evaluation.

References

Belcher, J. W. (2001). Studio Physics at MIT. MIT Physics Annual. [Apr 2003, verified 21 Oct 2004] http://web.mit.edu/physics/papers/Belcher_physicsannual_fall_01.pdf

Biggs, J. (1999). What the student does. Higher Education Research & Development, 18(1), 57-75.

Hanson, D. & Wolfskill, T. (2000). Process workshops - a new model for instruction. Journal of Chemical Education, 77(1), 120-130.

Heller, P. & Heller, K. (1999). Cooperative Group Problem Solving in Physics. Minnesota: University of Minnesota.

McInnis, C., James, R. & Hartley, R. (2000). Trends in the First Year Experience In Australian Universities. Canberra: Department of Education, Training and Youth Affairs.

Redish, E. F. (1999). Millikan Award Lecture (1998): Building a Science of Teaching Physics. American Journal of Physics, 67, 562-573. http://www.physics.umd.edu/perg/papers/redish/mlknpre.pdf [verified 21 Oct 2004].

Redish, E. F. (2002). Teaching Physics With the Physics Suite. Online Book. Department of Physics, University of Maryland. [Feb 2003, verified 21 Oct 2004] http://www2.physics.umd.edu/~redish/Book/

Walsh, M. (2003). Teaching qualitative analysis using QSR NVivo. The Qualitative Report, 8(2), 251-256. http://www.nova.edu/ssss/QR/QR258-252/walsh.pdf

Zuber-Skerritt, O. Y. T. (1992). Action Research in higher education: Examples and reflections. London: Kogan Page.

Please cite as: McAlpine, I., Wilson, K., Russell, C. & Cunningham, M. (2004). An online diary as a research and evaluation tool for first year physics. In R. Atkinson, C. McBeath, D. Jonas-Dwyer & R. Phillips (Eds), Beyond the comfort zone: Proceedings of the 21st ASCILITE Conference (pp. 616-622). Perth, 5-8 December. http://www.ascilite.org.au/conferences/perth04/procs/mcalpine.html

© 2004 Iain McAlpine, Kate Wilson, Carol Russell and Maria Cunningham
The authors assign to ASCILITE and educational non-profit institutions a non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full and this copyright statement is reproduced. The authors also grant a non-exclusive licence to ASCILITE to publish this document on the ASCILITE web site (including any mirror or archival sites that may be developed) and in printed form within the ASCILITE 2004 Conference Proceedings. Any other usage is prohibited without the express permission of the authors.

[ ASCILITE ] [ 2004 Proceedings Contents ]
This URL: http://www.ascilite.org.au/conferences/perth04/procs/mcalpine.html
HTML created 30 Nov 2004. Last revision: 30 Nov 2004.

Issue	Total no of mentions	Positive	Negative
equipment and facilities	87	18 (appreciated using)	63 (faulty, not enough, hard to use)
tutors/demonstrators	70	45 (helpful)	24 (unhelpful)
engagement	53	17 (fun, interesting)	34 (boring)
instructions	51	22 (helpful, easy to follow)	26 (confusing, hard to follow)
preparation	46	23 (helped in lab)	19 (unhelpful, not done, too hard)
challenge	25	2 (challenging)	21 (easy)
peer support	25	17 (other students helped)	7 (worked alone or others unhelpful)
relevance to course	22	11 (linked to lectures, etc.)	10 (unrelated or badly timed)
time taken	17	1 (about right)	15 (too short or too long)
hands on	10	4 (learnt from or enjoyed doing)	6 (didn't do enough)

issue	total no. of comments	positive comments	negative comments
tutors and demonstrators	179	106	73
integration of components	101	34	67
prelab work	101	58	43
textbook	99	83	16
lecturer	77	43	34
lecture notes	68	49	18
lab manual	65	34	31
problem practice	58	45	13
web	51	25	26

course component	total no. of comments	positive comments [typical example]	negative comments [typical example]
lectures	125	35 [Richard is very good at explaining tricky concepts, he gives real life examples ... lots of demos (always good).]	81 [I found the lectures extremely boring. They were difficult to understand and made learning physics difficult.]
tutorials	82	36 [... tutorial problems, I knew which questions I could do or not, so in tutorial I could ask the tutor about those questions.]	49 [There needs to be way more teaching, because at the moment it is just some dude writing solutions on the board.]
projects	39	35 [I really liked the lab projects because it gave me the opportunity to pursue a topic of physics that I had particular interest in. Working in a group also helped in enhancing my organisational, group work and communication skills.]	1
lab experiments	26	13 [...felt good to know that I understood what was being taught by being able to apply it to the real world.]	13 [I was doing experiments I had no idea about many weeks before we discussed them in lectures.]