Excerpted Referee Instructions

This document is an excerpt of a draft of our intended instructions to referees. It is intended to inform authors about the reviewing process.

In our experience, poster sessions are an efficient way to convey technical information. It is our expectation that most papers will be accepted as posters. If a paper is to be accepted for oral presentation, we expect that reviewers identify the special features of a paper that make it appropriate for oral presentation. These features do not include the strength of the paper, but do include potential interest in its content. It is quite possible that papers that are presented as posters will deserve and receive awards.

In general, less harm is caused by including a weak paper than by excluding a strong (but misunderstood) paper. We feel that reviewers should review papers with the intention that no paper with real merit be excluded from the conference (as opposed to ensuring that no paper without merit be included).  If a paper contains an interesting and really novel idea, but hasn't been thoroughly evaluated, a research conference is an excellent place to air it; journals are the place to archve the result when all the details have been fleshed out.

It is uncommon for a conference paper to change the field, and unreasonable to expect every conference paper to do so. Reasonable expectations of a paper are:

• that it be of reasonable length (reviewers are not required to wade through overlong papers);
• that it identify a problem and its significance;
• that it either solve a problem, or show why the problem is much harder than anyone thought (i.e. negative results can be acceptable);
• that it be substantially novel;
• that some section of the community will want to have read it;
• that it fairly --- but not necessarily comprehensively --- place the work within the body of  scholarship by allocating  credit to other authors; space limits may reasonably prevent an author referring to every paper on a subject
• that it be accurate and truthful;

Together with these general expectations, we see different kinds of papers meeting different kinds of standards. We provide the following examples, but do not intend that our remarks indicate either a comprehensive list or a checklist for authors.

• Theory papers should offer novel theoretical insights into one or more vision problem. Generally, very few papers in vision are theory papers, because most vision problems are problems of technique rather than theory. For example, a theory paper might offer a completely new view of the overall process of object recognition, and explain why that view is better than current thinking. It is unfair to criticize a theory paper for a lack of experimental results. This means that the burden on the author to show that their theory offers substantial insights is high --- does it allow us to think about a problem in a new way, that might be helpful? does it clarify why some problems are hard? An unattractive feature of theory papers is self-referential problem solving, where the paper merely resolves mathematical questions that arise if one adopts a particular framework of no clear significance.

 
• Technique papers show how to use, adapt or enhance existing techniques to solve vision problems. A substantial number of vision papers are about technique. For example, one might use linear algebra to do colour constancy. Technique papers that use a technique for the first time should show why the technique is appropriate and useful. More commonly, a technique that is used in vision is adapted or improved. In this case, the paper should identify the improvements, explain their virtues, and show some experimental examples that support the case. It is uncommon to encounter technique papers with substantial experimental verification --- often, the technique is known to work already, or the author can make the case that the broader range of problems that the technique can solve offsets poorer experimental properties. Very often readers value technique papers not for what the technique does in the paper, but what they can do with it. Technique papers that repeat a piece of math, applied math or physics without establishing a context --- what problems can be solved if one knows this? --- are unattractive.

 
• Application papers identify a problem that can be solved using vision, show that it is worth solving, and solve it. An application paper may use strategies that may be hard to justify using broad principles, but work in practice for the given application. The paper should demonstrate that there is sufficient experimental evidence to believe that the application really works, and that the application is worthwhile, in the sense that it has users. The important difference between application and experimental papers is that application papers solve other people's problems, and experimental papers address what are primarily vision issues. If there is sufficient evidence a system works on a real-world problem at a real-world scale, it is unfair to criticize the authors for using dubious or unprincipled techniques in building it. Generally a paper that identifies an application but does not solve it is unattractive, but there may be acceptable papers of this form.

 
• Experimental papers show a body of experimental evidence either to support or to discourage the use of a technique or a system. An experimental paper should indicate why the experiment is worth performing or what difficulties will be resolved by knowing the result of the experiment. It should show evidence of a substantial experiment, analysed carefully; ideally, the experimental design will be discussed as well. Careful analysis includes showing an overall statistical description, as well as explaining and illustrating special cases or non-obvious features of the experiment. There are numerous difficulties in performing experiments in vision (e.g., the scale of the experiment, difficulties in performing controls, assessing the quality of results, obtaining appropriate experimental materials, etc.), and reviewers will recognise that some experiments are difficult to do, or to analyse precisely. Running a system on several images is very seldom a meaningful experiment.