ASCILITE 2004: Advice to Reviewers

[ ASCILITE 2004 ] [ Call for papers ] [ Review Form ]

ASCILITE 2004 Advice to Reviewers

Reviewers, please read the Call for Papers and Review Form on the Conference website under http://www.ascilite.org.au/conferences/perth04/prog/call-for-papers.html
The role of Reviewers	Recommendations on acceptance, Publication format and Presentation format
Double blind reviewing	Recommendations on Outstanding Paper Award
Selecting and appointing Reviewers	Formative feedback to authors ('Notes to Authors')
The review process	Additional comments ('Notes to the Program Committee')
The review criteria ('General ratings')

The role of Reviewers Reviewers or 'referees' are pivotal in ensuring the quality of the papers and thus the Conference. Reviewers provide an independent assessment of the quality of each submission for Conference presentation, in both absolute terms ('accept' or 'reject') and in 'rank order' terms. Whilst Reviewers have considerable latitude, and a considerable responsibility, for interpretation of the concept of 'quality', we hope that the notes below will help us towards a reasonable uniformity of perception of quality standards, a fair, unbiased review process, and helpful, formative feedback for authors.

Double blind reviewing ASCILITE Conferences use a 'double blind' review process. That is, reviewers are not given the names and institutional affiliations of the authors, and authors are not given the names of the reviewers assigned to their article. In practice, use of a 'double blind' process to ensure objectivity is not always effective. Reviewers may identify authors, by encountering clues within the article, or by using search engines, or by using their knowledge of 'who is doing what' in topic areas related to the article under review. Also, authors may be able to guess the identity of a reviewer, if editors do not take sufficient care when passing on reviewer advice to authors.
If you feel that your objectivity as a Reviewer has been compromised because you have identified an author, either inadvertently through routine checking of references, or other avenues, please advise the Program Committee and we will seek a replacement reviewer.
ASCILITE Conferences commission at least two 'double blind' reviews for each submission. These may be supplemented, if appropriate, with another 'double blind' review by a third Reviewer, or 'non-blind' reviews obtained from members of the Conference Committee.

Selecting and appointing Reviewers Reviewers are appointed on the basis of their expertise and experience in areas relevant for the Conference. It is an honorary role, being rewarded only by acknowledgements in the printed and online versions of the Proceedings. Each ASCILITE Conference relies to a large extent upon Reviewers commissioned for previous Conferences. This has helped us sustain a uniformly high standard of reviewing over the years, as most of our Reviewers are 'experienced'. The ranks of 'experienced' Reviewers may be supplemented from other sources, such as AJET reviewers and authors. It is not necessary for reviewers to be members of ASCILITE, or to be registrants for the Conference.
ASCILITE Conferences also have an established policy of encouraging the induction of 'novice' Reviewers, who will broaden the Reviewer pool, and be in line to become the next generation of 'experienced' Reviewers. This policy is facilitated by ensuring that a review allocated to a 'novice' Reviewer is also allocated to an 'experienced' Reviewer, and is backed up by further double blind reviews and Program Committee reviews, if appropriate.

The review process All Reviewers will be emailed a 'please stand by' reminder in mid-July, with the final version of 'Advice to Reviewers' attached. 'Advice to Reviewers' (this file) and the 'Review Form' [Review Form] are web mounted to enable author access.
We anticipate that each Reviewer will be allocated 1 to 3 papers or exceptionally 4, usually a mix of full and concise papers, despatched during the period 26 July to 3 August, with a due date three weeks after despatch. All despatches of papers for reviewing and accompanying Review Forms, and returns of Forms, will be as Word documents attached to emails. Each Review Form will have customised content and a unique file name. If you accidentally lose a Review Form or change its name, please ask the Program Committee (per rjatkinson@bigpond.com) for a replacement copy.
Whilst three weeks may seem to be a hectic deadline, it is similar to review process deadlines for the Auckland 2002 and Adelaide 2003 Conferences. Given that ASCILITE Conferences offer authors the latest possible submission dates, it is essential that Reviewers maintain good turnaround times. If you find that you must call for help and seek re-allocation of all or part of the reviewing assigned to you, it will be vitally important to inform the Program Committee sooner rather than later!
Papers under review and completed Review Forms are confidential and the contents are not to be revealed to other persons.

The review criteria ('General ratings') You will have to use your own best judgement on the four criteria listed in 'Review Form'. As an approximation, consider the four criteria to be equally weighted.
'Suitability for an ASCILITE Conference' is a criterion that places an emphasis upon the Conference as a context, an environment for promoting an immediate exchange of ideas with peers, in contrast to other contexts such as journal publication, government reports, or undergraduate teaching. Contributions must be interesting and stimulating. Going beyond the 'comfort zone' of established techniques and familiar contexts, we seek to recognise innovative exploration of the next wave of research topics and techniques, and imaginative anticipation of future contexts for computers and related technologies in teaching and learning.
'Academic merit' groups two criteria. The first part in the Review Form attempts to assess the quality of the components (literature review, methodology, etc) of the work, whilst the second part attempts to assess the originality and novelty of the work's contribution to knowledge, in accordance with the intentions underlying the DEST definition of research (see the Call for papers page on the Conference website for references to DEST definitions). In our Conference's context, we seek to go beyond the idea of research that establishes new facts or new concepts, to encompass also the idea of research that helps to establish and disseminate new procedures, new ways to teach and learn, and develops practical illustrations, applications and validations of the principles explored by previous research.
'Standard of writing' is a criterion that enables a signal to authors and to the Proceedings editors. Through this criterion, Reviewers can indicate in a summative way the amount of work that authors need to do, to obtain an acceptable, clear standard of writing, possibly with some or even considerable support from the Proceedings editors. However, we do not expect Reviewers to compile a detailed list of corrections, or a detailed list of deletions to shorten an overlength paper. A summative judgement, perhaps with a few illustrative examples, is all that we seek under this criterion. We want Reviewers to have their eyes on the 'big picture', whilst the Proceedings editors will look after the details in 'Standard of writing', in consultation with authors and subject to the time constraints of the production schedule.
Full papers and concise papers are to be reviewed using the same criteria, using your own best judgement about how well the authors have used their chosen length. The 'amount' of research represented in a concise paper may be about one third to one half the 'amount' in a full paper, but the quality is to be the same. However, with concise papers Reviewers can allow a weighting towards the statement in Call for papers that concise papers are "an avenue for work in progress, for pilot studies, small scale exploratory projects, reports on highly specialised topics, brief studies on recent new developments with evaluation in progress, and similar". Papers submitted as poster proposals will be reviewed by the Conference Committee, but will not be eligible for DEST recognition, regardless of final, published length, because of the lack of external reviewing.

Recommendations on acceptance, Publication format and Presentation format In this section, we expect the approximate, somewhat subjective relationship between 'General ratings' and recommendation to be:

acceptance 'as is' - four 'Good' ratings or better.
acceptance with 'minor revisions' - two 'Good' and two 'Fair' ratings or better
acceptance 'after major revisions' - four 'Fair' ratings or better
reject - three 'Poor' ratings or worse
(Although this is subjective, we can score 'Poor' = 1, 'Fair' = 2, etc, to obtain a summative figure)
The sections on Publication format and Presentation format are included because, in some cases, the Reviewer may feel that it is appropriate to recommend to the Program Committee that the authors be offered a format differing from the format they nominated, eg, an outstanding Concise paper may be given a 'Traditional' 25 minute presentation slot. However, please note that some changes of format cannot be offered. For example, changes from 'Concise' or 'Poster' to 'Full' in publication format cannot be offered. Also, changes from 'Full' to 'Concise' should be recommended only in exceptional cases where the paper could be substantially improved by a major revision with shortening.
Papers submitted as 'Full' or 'Concise' that are recommended by Reviewers for acceptance, but with the Reviewers or the Committee changing the Presentation format to 'Poster', will be published in full and will be eligible for DEST recognition, if the authors accept the 'Poster' offer from the Committee.

Recommendations on Outstanding Paper Award In this section, we expect that a 'Yes' recommendation would be supported by at least 3 'Very good' ratings given under 'General ratings'. The Conference Committee will combine Reviewer recommendations with ratings for author's presentation, obtained from Committee members attending the sessions for short listed papers. Both 'Full' or 'Concise' papers are eligible.

Formative feedback to authors ('Notes to Authors') With this section we hope to obtain formative feedback that may be 'copied and pasted' into emailed advice to authors. Here you should specify minor or major revisions that are to be completed and resubmitted for Program Committee approval. You could give amplifying comments and brief, illustrative examples to help authors understand the summative judgements that you have given under 'General ratings'.
Please remember that the aim here is to encourage authors to improve their work, not only for this Conference, but also for future conference and journal submissions. It is a section in which you can emphasise 'how you may progress...', in contrast to emphasising 'your work is bad because...'.
Another aim in this section is to alert the Proceedings editors to minor or major revisions that they should check, upon receiving a revised version from the authors. Owing to production time constraints, it won't be possible to send revised versions to the original or new Reviewers.

Additional comments ('Notes to the Program Committee') Under this heading you may add any special, confidential comments for 'Program Committee eyes only', that may assist the Program Committee with the selection process, and in using its discretion when providing feedback to authors. Other matters that you may raise could include alerting the Committee to instances of excessive repetition of previously published work, or inadequate acknowledgment of the work of other writers.

Please cite as: ASCILITE (2004). Advice to Reviewers. Beyond the Comfort Zone. Proceedings ASCILITE 2004 Conference. Perth, Western Australia, 5-8 December. http://www.ascilite.org.au/conferences/perth04/prog/asc04-advice-reviewers.html

© ASCILITE. We grant educational institutions and societies a non-exclusive licence to use this document for personal use and adapt it for non-profit academic purposes provided that in any adaptation the source is acknowledged. Any other reproduction is prohibited without the express permission of ASCILITE.

[ ASCILITE 2004 ] [ Call for papers ] [ Review Form ]
This URL: http://www.ascilite.org.au/conferences/perth04/prog/asc04-advice-reviewers.html
HTML created 5 June 2004. Last revision: 19 July 2004.

The role of Reviewers	Reviewers or 'referees' are pivotal in ensuring the quality of the papers and thus the Conference. Reviewers provide an independent assessment of the quality of each submission for Conference presentation, in both absolute terms ('accept' or 'reject') and in 'rank order' terms. Whilst Reviewers have considerable latitude, and a considerable responsibility, for interpretation of the concept of 'quality', we hope that the notes below will help us towards a reasonable uniformity of perception of quality standards, a fair, unbiased review process, and helpful, formative feedback for authors.
Double blind reviewing	ASCILITE Conferences use a 'double blind' review process. That is, reviewers are not given the names and institutional affiliations of the authors, and authors are not given the names of the reviewers assigned to their article. In practice, use of a 'double blind' process to ensure objectivity is not always effective. Reviewers may identify authors, by encountering clues within the article, or by using search engines, or by using their knowledge of 'who is doing what' in topic areas related to the article under review. Also, authors may be able to guess the identity of a reviewer, if editors do not take sufficient care when passing on reviewer advice to authors. If you feel that your objectivity as a Reviewer has been compromised because you have identified an author, either inadvertently through routine checking of references, or other avenues, please advise the Program Committee and we will seek a replacement reviewer. ASCILITE Conferences commission at least two 'double blind' reviews for each submission. These may be supplemented, if appropriate, with another 'double blind' review by a third Reviewer, or 'non-blind' reviews obtained from members of the Conference Committee.
Selecting and appointing Reviewers	Reviewers are appointed on the basis of their expertise and experience in areas relevant for the Conference. It is an honorary role, being rewarded only by acknowledgements in the printed and online versions of the Proceedings. Each ASCILITE Conference relies to a large extent upon Reviewers commissioned for previous Conferences. This has helped us sustain a uniformly high standard of reviewing over the years, as most of our Reviewers are 'experienced'. The ranks of 'experienced' Reviewers may be supplemented from other sources, such as AJET reviewers and authors. It is not necessary for reviewers to be members of ASCILITE, or to be registrants for the Conference. ASCILITE Conferences also have an established policy of encouraging the induction of 'novice' Reviewers, who will broaden the Reviewer pool, and be in line to become the next generation of 'experienced' Reviewers. This policy is facilitated by ensuring that a review allocated to a 'novice' Reviewer is also allocated to an 'experienced' Reviewer, and is backed up by further double blind reviews and Program Committee reviews, if appropriate.
The review process	All Reviewers will be emailed a 'please stand by' reminder in mid-July, with the final version of 'Advice to Reviewers' attached. 'Advice to Reviewers' (this file) and the 'Review Form' [Review Form] are web mounted to enable author access. We anticipate that each Reviewer will be allocated 1 to 3 papers or exceptionally 4, usually a mix of full and concise papers, despatched during the period 26 July to 3 August, with a due date three weeks after despatch. All despatches of papers for reviewing and accompanying Review Forms, and returns of Forms, will be as Word documents attached to emails. Each Review Form will have customised content and a unique file name. If you accidentally lose a Review Form or change its name, please ask the Program Committee (per rjatkinson@bigpond.com) for a replacement copy. Whilst three weeks may seem to be a hectic deadline, it is similar to review process deadlines for the Auckland 2002 and Adelaide 2003 Conferences. Given that ASCILITE Conferences offer authors the latest possible submission dates, it is essential that Reviewers maintain good turnaround times. If you find that you must call for help and seek re-allocation of all or part of the reviewing assigned to you, it will be vitally important to inform the Program Committee sooner rather than later! Papers under review and completed Review Forms are confidential and the contents are not to be revealed to other persons.
The review criteria ('General ratings')	You will have to use your own best judgement on the four criteria listed in 'Review Form'. As an approximation, consider the four criteria to be equally weighted. 'Suitability for an ASCILITE Conference' is a criterion that places an emphasis upon the Conference as a context, an environment for promoting an immediate exchange of ideas with peers, in contrast to other contexts such as journal publication, government reports, or undergraduate teaching. Contributions must be interesting and stimulating. Going beyond the 'comfort zone' of established techniques and familiar contexts, we seek to recognise innovative exploration of the next wave of research topics and techniques, and imaginative anticipation of future contexts for computers and related technologies in teaching and learning. 'Academic merit' groups two criteria. The first part in the Review Form attempts to assess the quality of the components (literature review, methodology, etc) of the work, whilst the second part attempts to assess the originality and novelty of the work's contribution to knowledge, in accordance with the intentions underlying the DEST definition of research (see the Call for papers page on the Conference website for references to DEST definitions). In our Conference's context, we seek to go beyond the idea of research that establishes new facts or new concepts, to encompass also the idea of research that helps to establish and disseminate new procedures, new ways to teach and learn, and develops practical illustrations, applications and validations of the principles explored by previous research. 'Standard of writing' is a criterion that enables a signal to authors and to the Proceedings editors. Through this criterion, Reviewers can indicate in a summative way the amount of work that authors need to do, to obtain an acceptable, clear standard of writing, possibly with some or even considerable support from the Proceedings editors. However, we do not expect Reviewers to compile a detailed list of corrections, or a detailed list of deletions to shorten an overlength paper. A summative judgement, perhaps with a few illustrative examples, is all that we seek under this criterion. We want Reviewers to have their eyes on the 'big picture', whilst the Proceedings editors will look after the details in 'Standard of writing', in consultation with authors and subject to the time constraints of the production schedule. Full papers and concise papers are to be reviewed using the same criteria, using your own best judgement about how well the authors have used their chosen length. The 'amount' of research represented in a concise paper may be about one third to one half the 'amount' in a full paper, but the quality is to be the same. However, with concise papers Reviewers can allow a weighting towards the statement in Call for papers that concise papers are "an avenue for work in progress, for pilot studies, small scale exploratory projects, reports on highly specialised topics, brief studies on recent new developments with evaluation in progress, and similar". Papers submitted as poster proposals will be reviewed by the Conference Committee, but will not be eligible for DEST recognition, regardless of final, published length, because of the lack of external reviewing.
Recommendations on acceptance, Publication format and Presentation format	In this section, we expect the approximate, somewhat subjective relationship between 'General ratings' and recommendation to be: acceptance 'as is' - four 'Good' ratings or better. acceptance with 'minor revisions' - two 'Good' and two 'Fair' ratings or better acceptance 'after major revisions' - four 'Fair' ratings or better reject - three 'Poor' ratings or worse (Although this is subjective, we can score 'Poor' = 1, 'Fair' = 2, etc, to obtain a summative figure) The sections on Publication format and Presentation format are included because, in some cases, the Reviewer may feel that it is appropriate to recommend to the Program Committee that the authors be offered a format differing from the format they nominated, eg, an outstanding Concise paper may be given a 'Traditional' 25 minute presentation slot. However, please note that some changes of format cannot be offered. For example, changes from 'Concise' or 'Poster' to 'Full' in publication format cannot be offered. Also, changes from 'Full' to 'Concise' should be recommended only in exceptional cases where the paper could be substantially improved by a major revision with shortening. Papers submitted as 'Full' or 'Concise' that are recommended by Reviewers for acceptance, but with the Reviewers or the Committee changing the Presentation format to 'Poster', will be published in full and will be eligible for DEST recognition, if the authors accept the 'Poster' offer from the Committee.
Recommendations on Outstanding Paper Award	In this section, we expect that a 'Yes' recommendation would be supported by at least 3 'Very good' ratings given under 'General ratings'. The Conference Committee will combine Reviewer recommendations with ratings for author's presentation, obtained from Committee members attending the sessions for short listed papers. Both 'Full' or 'Concise' papers are eligible.
Formative feedback to authors ('Notes to Authors')	With this section we hope to obtain formative feedback that may be 'copied and pasted' into emailed advice to authors. Here you should specify minor or major revisions that are to be completed and resubmitted for Program Committee approval. You could give amplifying comments and brief, illustrative examples to help authors understand the summative judgements that you have given under 'General ratings'. Please remember that the aim here is to encourage authors to improve their work, not only for this Conference, but also for future conference and journal submissions. It is a section in which you can emphasise 'how you may progress...', in contrast to emphasising 'your work is bad because...'. Another aim in this section is to alert the Proceedings editors to minor or major revisions that they should check, upon receiving a revised version from the authors. Owing to production time constraints, it won't be possible to send revised versions to the original or new Reviewers.
Additional comments ('Notes to the Program Committee')	Under this heading you may add any special, confidential comments for 'Program Committee eyes only', that may assist the Program Committee with the selection process, and in using its discretion when providing feedback to authors. Other matters that you may raise could include alerting the Committee to instances of excessive repetition of previously published work, or inadequate acknowledgment of the work of other writers.