Can we improve the journal review process in computer science?
As in all scientific disciplines, peer review plays a core role in computer science research. But one aspect that sets apart our discipline from others is that most areas of computer science are driven by conference rather than journal publications. This has been discussed numerous times, and whether our community should change its publication culture is a controversial subject; just Google "conferences vs. journals in computer science" if you are not familiar with this debate.
One key advantage of conference publications is their quick reviewing cycle, which is often assumed to be in conflict with a careful, high-quality reviewing process. While the high reviewing load and the tight deadlines imposed on program committee (PC) members of top conferences do indeed endanger the peer review process, I argue that the conference reviewing process is in many ways of higher-quality than that used by journals. I will discuss two such aspects below.
(1) Conference reviewing puts a lot of emphasis on discussions among PC members. Top CS conferences have both an online discussion stage, typically lasting a couple of weeks, as well as an in-person PC meeting taking one or two days. Given the emphasis that our community (justifiably!) places on these discussions, I find it surprising that this is completely absent from journal reviewing. When I review for a journal, I typically only receive the other reviewers' comments (if at all!), without any opportunity to further discuss the paper and clarify our respective positions. Contrast this with the conference reviewing process, where I often have lengthy discussions with the other PC members, both online and in-person, which I believe considerably improves our understanding of the paper and the overall reviewing process.
(2) Conferences have a much easier time convincing top researchers to dedicate a lot of their time reviewing submissions. This has an impact both on the quality of the reviews (with experts sometimes unwilling to review for journals) and on the length of the reviewing cycle. If there is any chance for our community to adopt journals more widely (and I'm not arguing here either way), journals need to have a faster publication cycle: having to wait more than one year until your paper is published (and in many cases longer!) is certainly not a big selling point. Whenever I ask editors of top journals in our field why things take so long, they mention the difficulty of finding reviewers. Conversely, I know many fellow junior and mid-career researchers who admit declining journal reviews as a general rule. However, these same people would rarely refuse the invitation to join the PC of a top conference, despite the latter requiring ten times more work! Why?
The answer is simple: journal reviewers are assigned a secondary role compared to conference reviewers. For instance, while they make a recommendation, they don't have any opportunity to further discuss the paper and their position with their co-reviewers, the final decision being taken solely by the editor.
More importantly, journals provide little incentives to reviewers compared to conferences. Firstly, the PC of a top conference is widely advertised, giving its members a lot of visibility in their community. Contrast this to journals, where only the editorial staff gets this benefit. Secondly, being part of a conference PC gives you the opportunity to meet and have technical discussions with top researchers in your community, both online and in person (not to mention that conference PCs sometimes organize informal workshops co-located with the PC meeting). Contrast this to journal reviewing, where you don't get the chance to exchange a single word with your co-reviewers!
These are not the only strengths of conference reviewing, and there are certainly many weaknesses as well when compared to journal reviewing. But in relation to some of the points above, I believe journals could improve their reviewing process. In particular, why not give journal reviewers the possibility to have an online discussion? Our community justifiably puts a lot of emphasis on discussions among conference reviewers, and there is no reason for not having this be a core part of the journal reviewing process as well.
One key advantage of conference publications is their quick reviewing cycle, which is often assumed to be in conflict with a careful, high-quality reviewing process. While the high reviewing load and the tight deadlines imposed on program committee (PC) members of top conferences do indeed endanger the peer review process, I argue that the conference reviewing process is in many ways of higher-quality than that used by journals. I will discuss two such aspects below.
(1) Conference reviewing puts a lot of emphasis on discussions among PC members. Top CS conferences have both an online discussion stage, typically lasting a couple of weeks, as well as an in-person PC meeting taking one or two days. Given the emphasis that our community (justifiably!) places on these discussions, I find it surprising that this is completely absent from journal reviewing. When I review for a journal, I typically only receive the other reviewers' comments (if at all!), without any opportunity to further discuss the paper and clarify our respective positions. Contrast this with the conference reviewing process, where I often have lengthy discussions with the other PC members, both online and in-person, which I believe considerably improves our understanding of the paper and the overall reviewing process.
(2) Conferences have a much easier time convincing top researchers to dedicate a lot of their time reviewing submissions. This has an impact both on the quality of the reviews (with experts sometimes unwilling to review for journals) and on the length of the reviewing cycle. If there is any chance for our community to adopt journals more widely (and I'm not arguing here either way), journals need to have a faster publication cycle: having to wait more than one year until your paper is published (and in many cases longer!) is certainly not a big selling point. Whenever I ask editors of top journals in our field why things take so long, they mention the difficulty of finding reviewers. Conversely, I know many fellow junior and mid-career researchers who admit declining journal reviews as a general rule. However, these same people would rarely refuse the invitation to join the PC of a top conference, despite the latter requiring ten times more work! Why?
The answer is simple: journal reviewers are assigned a secondary role compared to conference reviewers. For instance, while they make a recommendation, they don't have any opportunity to further discuss the paper and their position with their co-reviewers, the final decision being taken solely by the editor.
More importantly, journals provide little incentives to reviewers compared to conferences. Firstly, the PC of a top conference is widely advertised, giving its members a lot of visibility in their community. Contrast this to journals, where only the editorial staff gets this benefit. Secondly, being part of a conference PC gives you the opportunity to meet and have technical discussions with top researchers in your community, both online and in person (not to mention that conference PCs sometimes organize informal workshops co-located with the PC meeting). Contrast this to journal reviewing, where you don't get the chance to exchange a single word with your co-reviewers!
These are not the only strengths of conference reviewing, and there are certainly many weaknesses as well when compared to journal reviewing. But in relation to some of the points above, I believe journals could improve their reviewing process. In particular, why not give journal reviewers the possibility to have an online discussion? Our community justifiably puts a lot of emphasis on discussions among conference reviewers, and there is no reason for not having this be a core part of the journal reviewing process as well.
Comments
Post a Comment