All Things Distributed

Evaluating Systems Papers

There are some issues on my list of things to write about that are still academia related. If there is one thing that academics are supposed to be better at than people from industry, it is writing about their work. If you have a good advisor, then during your formative years as a Ph.D. student he or she will teach you the rules of good paper writing. This is the general perception; reality is very, very different. If I take a look at the papers I reviewed for the program committees I was on in the past year, I estimate that 70% or more of the rejections are due to the authors are not capable of communicating their ideas properly. And I don’t mean the occasional language or grammar mistake, but basic composition of your paper. There are a number of components that each paper should have, and the absence of them make a reader wonder whether you just forgot, or whether you didn’t execute those steps in your research.

Some things that I always look for in a paper:

• User or system requirements. Most of the papers I read are about networking, operating systems, distributed systems, but being active is such a deep technical field does not exempt you from thinking about WHY you are doing this. Who or what is going to use your system? Can you write down the requirements and constraints such that it is clear to the reader why this drives your research? Do you have trace or input data that matches your requirements? Even if you did not start out with requirements (sometimes you just have a cool idea), when you write about it you must define what the criteria for success for your project are and why.
• Alternative design decisions. It cannot have been the case that there is only one path to your goal. You must have seen other roads along the way, but you decided not to follow those. Why not? It is important to motivate the choices you have made, if you want your reader to learn from your process, not just from your final success.
• Context. You are not the only person doing research and you are not the only one producing these wonderful unique solutions. You have to spend more than 2 paragraphs on placing your work in context. It is OK to say that others have done the same before, but that you are approaching it from a different angle, or with different data. For example a good engineering paper might not have fundamentally new results, but can focus on how to bring results into practice, which might be just as hard a problem or even harder. Do not discard other people’s work with just a single sentence but give them the respect you would like your work to get.
• Implementation details. If you add this to your paper, make sure you really provide details. Not just a schema of how your components fit together, but also information about why you got to this software design. It cannot have been the case that you got it right with one try. What were the other attempts that you discarded and why?
• Experiments I can talk days about how to perform good experiments in support of your research. But I consider experiments in support of a paper to be something else; here the goal of the experiments becomes communication, not just data mining. With the selection and description of the experiments you have to keep your audience in mind. What were your requirements and how can you show that your solution meets those requirements, by presenting your experimental results. Often your experiments you ran during your research are not the same as the ones you use for your paper. There is not enough room in papers for exhaustive experiment descriptions, so design tests to get the message across.
• Boundaries. It cannot be that your solution is perfect for all cases. Show us what is the window of your success, but also show what are the boundaries. This is often just as important as the describing the successful application.
• Lessons. What lessons did you learn from your work? Positive as well as negative. Your audience will often learn more from your negative experiences than from your glorious positive ones.

Another general advise I also always give is not to try to and inflate your work. If you feel it is not grandiose than don’t write about like it is. Focus on the message you want your audience to read, and that has to be about the work, solution, technology, and your love and passion for it. It is OK to be passionate about your work in your paper. If you inflate stuff, which in general is for your ego and not for your passion of the solution, you always take a hard fall. Your readers will look past your big word and realize what it is you are doing.

There was a paper in the 83 SOSP by Roy Levin and David Redell about how to write a systems paper which is worth reading in this context.