Software Evolution (COMP 762)

How to Read and Evaluate Technical Papers

Original Text by Bill Griswold

(Modified by Gail Murphy, November 18, 1998 and January 6, 2000)

Reading research papers effectively is challenging. These papers are often written in a very condensed style because of page limitations and the intended audience, which is assumed to already know the area well. Moreover, the reasons for writing the paper may be different than the reasons the paper has been assigned, meaning you have to work harder to find the content that you are interested in. Finally, your time is very limited, so you may not have time to read every word of the paper or read it several times to extract all the nuances. For all these reasons, reading a research paper can require a special approach.

The first thing to understand is that the research papers you will read have been published in different kinds of places. Some papers have been published in the proceedings of a conference. These papers have been peer-reviewed but are typically strictly limited in length to about 10-12 pages. Other papers have been published in archival journals. These papers have also been peer-reviewed but there are typically not any length restrictions, so they generally include more detail in every dimension. Some papers are technical reports. These are not peer-reviewed. In areas related to Computer Science, you may find that there is first a technical report version which is then later published in a conference or a journal. If a paper appears both in conference and journal form, the journal version is typically a later, expanded and revised version of the work.

To develop an effective reading style for research papers, it can help to know what you should get out of the paper, and where that information is located in the paper. Typically, the introduction will state not only the motivations behind the work, but also outline the solution. Often this may be all the expert requires from the paper. The body of the paper states the authors' solution to the problem in detail, and should also describe a detailed evaluation of the solution in terms of arugments or an empirical evaluation (case study, experiment, etc.). Finally, the paper will conclude with a recap, including a discussion of the primary contributions. A paper will also discuss related work to some degree. Papers are often repetitive because they present information at different levels of detail and from different perspectives. As a result, it may be desirable to read the paper ``out of order'' or to skip certain sections. More on this below.

The questions you want to have answered by reading a paper are the following:

As you read or skim a paper, you should actively attempt to answer the above questions. Presumably, the introduction should provide motivation. The introduction and conclusion may discuss the solutions and evaluation at a high level. Future work is likely in the concluding part of the paper. The details of the solution and the evaluation should be in the body of the paper. You may find it productive to try to answer each question in turn, writing your answer down. In practice, you are not done reading a paper until you can answer all the questions.