{"id":21,"date":"2014-10-20T04:31:34","date_gmt":"2014-10-20T04:31:34","guid":{"rendered":"http:\/\/www.aarondefazio.com\/tangentially\/?p=21"},"modified":"2014-10-20T04:31:34","modified_gmt":"2014-10-20T04:31:34","slug":"the-nips-consistency-experiment-my-experience","status":"publish","type":"post","link":"https:\/\/www.aarondefazio.com\/tangentially\/?p=21","title":{"rendered":"The NIPS Consistency Experiment &#8212; My Experience"},"content":{"rendered":"<div id=\"globalWrapper\">\n<script type=\"math\/tex\">\n\\newcommand{\\lyxlock}{}\n<\/script><br \/>\n<noscript><\/p>\n<div class=\"warning\">\nWarning: <a href=\"http:\/\/www.mathjax.org\/\">MathJax<\/a> requires JavaScript to correctly process the mathematics on this page. Please enable JavaScript on your browser.\n<\/div>\n<hr\/>\n<p><\/noscript><\/p>\n<div class=\"Unindented\">\nThis year the NIPS (Neural Information Processing Systems) conference organisers decided to run an experiment on the consistency of paper reviews. They selected 10% of papers to be reviewed twice. Different area chairs and a different set of 3 reviewers were chosen for those papers.\n<\/div>\n<div class=\"Indented\">\nLuckily for me, my paper with Francis Bach and Simon Lacoste-Julien was one of those 10%. My paper was initially submitted as Paper ID 867. They essentially created a duplicated paper id for me, #1860, which contained the second set of reviews.\n<\/div>\n<div class=\"Indented\">\nThis duplication of reviews was particularly interesting in my case. There was a very large discrepancy between the two sets of reviews. I won\u2019t know if this is representative of the consistency of other reviews until NIPS releases the statistics from there experiment.\n<\/div>\n<div class=\"Indented\">\nFor reference, the two sets of reviews gave the following scores, before rebuttal:\n<\/div>\n<div class=\"Indented\">\nSet 1 review 1: Quality 9, impact 2 (high) , confidence 4 (confident but not certain)\n<\/div>\n<div class=\"Indented\">\nSet 1 review 2: Quality 6, impact 1 (incremental), confidence 3 (fairly confident)\n<\/div>\n<div class=\"Indented\">\nSet 1 review 3: Quality 6, impact 1 (incremental), confidence 5 (Absolutely certain)\n<\/div>\n<div class=\"Indented\">\n&#8212;\n<\/div>\n<div class=\"Indented\">\nSet 2 review 1: Quality 5, impact 1 (incremental), confidence 5 (Absolutely certain)\n<\/div>\n<div class=\"Indented\">\nSet 2 review 2: Quality 3, impact 1 (incremental), confidence 4 (Confident)\n<\/div>\n<div class=\"Indented\">\nSet 2 review 3: Quality 6, impact 1 (incremental), confidence 5 (Absolutely certain)\n<\/div>\n<div class=\"Indented\">\n&#8212;\n<\/div>\n<div class=\"Indented\">\nGenerally for NIPS a 9\/6\/6 in quality gives a high change of acceptance, where as a 5\/3\/6 is a certain non-acceptance. So one set of reviews was a clear accept and the second a clear reject! The meta reviews were as follows:\n<\/div>\n<div class=\"Indented\">\n<div class=\"smallskip\"> <\/div>\n<\/div>\n<div class=\"Indented\">\n<div class=\"Boxed\" style=\"width: 100%;\">\nThe paper introduces a new incremental gradient method that allows adaptation to the level of convexity in the input. The paper has a nice discussion of related methods, and it has a simpler proof that will be of interest to researchers. Recommendation: Accept.\n<\/div>\n<\/div>\n<div class=\"Indented\">\n<div class=\"smallskip\"> <\/div>\n<div class=\"Boxed\" style=\"width: 100%;\">\nUnfortunately, the scores are too low for acceptance to NIPS, and none of the reviewers were willing to argue for acceptance of the paper. The reviewers discussed the paper after the author rebuttal, and all reviewers ultimately felt that the paper could use some additional polish before publishing. Please do keep in mind the various criticisms of the reviewers when submitting to another venue.\n<\/div>\n<\/div>\n<div class=\"Indented\">\n<div class=\"smallskip\"> <\/div>\n<\/div>\n<div class=\"Indented\">\nThe paper we submitted was fairly rough in its initial state, and the reviewers suggested lots of improvements. Particularly the Set 2\/review 1, which was the most in depth review. I generally agree with the second meta-review in that the paper needed addition polish, which we have done for the camera ready.\n<\/div>\n<div class=\"Indented\">\nIn the end the paper was accepted. I suspect most papers with this kind of accept\/reject split would be accepted, as it would just seem unfair if it were not.\n<\/div>\n<div class=\"Indented\">\nThe issue of consistency in paper reviews is clear to any body who has ever resubmitted a rejected paper to a different venue. It feels like luck of a draw to a degree. There is no easy solutions to this, so I\u2019ll be interested to see if NIPS changes there process in future years, and what changes they make.\n<\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Warning: MathJax requires JavaScript to correctly process the mathematics on this page. Please enable JavaScript on your browser. This year the NIPS (Neural Information Processing Systems) conference organisers decided to run an experiment on the consistency of paper reviews. They selected 10% of papers to be reviewed twice. Different area chairs and a different set &hellip; <a href=\"https:\/\/www.aarondefazio.com\/tangentially\/?p=21\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">The NIPS Consistency Experiment &#8212; My Experience<\/span> <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"_links":{"self":[{"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=\/wp\/v2\/posts\/21"}],"collection":[{"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=21"}],"version-history":[{"count":1,"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=\/wp\/v2\/posts\/21\/revisions"}],"predecessor-version":[{"id":22,"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=\/wp\/v2\/posts\/21\/revisions\/22"}],"wp:attachment":[{"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=21"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=21"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aarondefazio.com\/tangentially\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=21"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}