All Things Distributed

Why doesn't Academia understand Industrial Work?

In a comment on his weblog Bryan Cantrill responds to my posting yesterday about papers from industry at conferences. Bryan rejects my statement that one of the main causes that people from industry do not participate in program committees and paper submission is that not sufficient reward is given inside the enterprise culture for these activities, which then basically becomes a personal volunteer effort. According to Bryan, people from cutting-edge software industry are often encouraged to write papers. My experiences are different so we probably both have our score of examples for either side of the story.

In the same comment Bryan hints at a trend that has many people from industry discouraged to write and submit papers:

But here's the problem: the gap between industrial work and academic work has grown so large, that reviewers often don't understand the work or (more commonly) don't appreciate that the problem being solved is actually a problem. This, coupled with the naturally high rejection rate of most conferences, leads to increased odds of rejection. And this is the real problem: it's not that we don't want to do the work, it's just that we don't want to do the work and then have it amount to nothing.

The statement is too general for me to agree with, but I do recognize the general sentiment. Let me address some of the individual points:

• The gap between industry and academic work has grown to be large: Some fields have traditionally stronger ties between industry and academia and as such those academics are better aware of problems industry is facing. Some fields have a shorter period between the transition of academic results into industry. For example in databases as well as in networking the symbioses appears much stronger than for example in the operating systems world. Some smaller more targeted conferences such as Usenix File and Storage Technologies attract a lot of industry involvement, while some larger conferences such as SuperComputing are dominated by industrial folks, with the academics only playing a minor role. On the other hand some of our premier operating systems conferences such as SOSP and OSDI are almost completely dominated by academic-style research whether it comes from academia or the industrial labs. If you are lucky you will find one or two papers that address issues surrounding production systems. But the lead time between the introduction of new operating systems concepts developed by research into production systems often takes 5 years or more, while introduction of new HPC concepts in the financial industry is a matter of months. These different cycles often make it appear as if the particular research area is too far out when looked at from a industrial point of view.
• Reviewers often don't understand the work: I disagree with this statement, I have not been on a PCs where I feel that reviewers totally missed the brilliance of a particular solution, when reviewing a particular paper from industry. Especially given that in general  papers from industry get preferential treatment. We actually sometimes track these rejected papers to see whether they show up in other conferences. Also a PC meeting is not a docile gathering of like minded people. They are often very contentious, where people really fight to get papers accepted or rejected. It can be emotionally draining. As a program chair you want to reach consensus on each of the accepted papers, which is very hard. I remember that I was being a real pain at the Usenix'04 meeting with respect to one or two papers, where Remzi had to work hard to have the other members convince me. In this competitive atmosphere, the role of the pc chair is also to do 'due diligence', to make sure all the papers get sufficient attention.
  But I may understand where this experience originates: The feedback process of the reviewers to the authors is becoming a real big problem. The increasing number of submissions make it much harder to give appropriate, detailed feedback to the  authors. And to be honest not all reviewers do a real good job on this, especially if you already know that this paper will be rejected no matter what. And if you are on the receiving end of these reviews which lack any details I can understand why the idea forms that the reviewers didn't understand the work.
• Reviewers don't appreciate the problem being solved is actually a problem. In my eyes this is actually a real issue. I work very hard on keeping in touch with industrial reality, but I know that many of my colleagues do not or have never done. I am sometimes even surprised about the lack or real-world insight by researchers from industrial labs. But this is a sort of perpetual problem: industry can also be very secretive with respect to the problems they are facing, and it is very, very hard for academics to get access to relevant data. An example in my own back yard is that for years I tried to get access to information about distributed systems failures from vendors and their customers. Even though they have all of this data readily available they do not want to share it with academia. How can they expect me to do a good job in building the next generation of fault-tolerant systems that are relevant for industry if they are not willing to share with me what their problems really are? There is a lot of room for improvement at both sides here.
• These misconceptions lead to a higher reject rate for industrial papers, which has a discouraging effect. I not sure whether there is a higher reject rate for industrial papers. I have on the program committees for 5 conferences this year, which all are of interest to industry, and the number of true industry submissions is extremely low. I would be willing to work out the percentage of industrial rejects compared to other papers, but the submission numbers are so low even that you can not use a statistical measurement on it. Maybe the industrial accept rate ate Usenix'04 was 100% or 50%, which is much better than the general accept rate, but that doesn't say much if there were only one or two such papers.
  But dealing with discouragement after a rejection is an important part of paper writing life. If there are over 300 papers submitted to SIGCOMM and only 24 get accepted, does that mean that the other 92% are bad papers. Absolutely not. I know that even as a program committee member you often think that the resulting program should have been different. There are very good papers in the 92%, but maybe the PC this year may decide not to spend too much time any more on DHT based p2p protocols, and the paper about your real-life system with actual deployment gets booted out just because of that.  You curse a few times, try to learn from the feedback and move on to find a venue that is more appreciative of your work.

I believe that the problems that Bryan mentions here transcend just paper writing. In the US there is a strong history in collaboration between Academia and Industry, but in general the knowledge stream is one way: academics transfers their results either through papers or presentations to industry, but the number of researchers that is really allowed a look into industry's kitchen is very limited. I know this because I have been one of those fortunate ones. Students frequently return from internships with gag-orders preventing them to speak about the problems companies are working on.. Having more papers by industry in conferences will not solve this problem, as papers in general only describe positive, successful results. They do not give academia input into a realistic research agenda and thus contribute little to a better understanding of the problems that industry is struggling with.

If we want to, in Bryan's words, close the gap between academic and industrial work, such that a better understanding for industrial problem is the result (which could lead to improved paper acceptance rates, among other things), than industry will need to be more pro-active in making researchers aware of what the problems are that they need to solve, on short as well as long term. Industry has a chance to drive the research agenda as long as it is willing to open up and show what the real problems are.