This year the NIPS (Neural Information Processing Systems) conference organisers decided to run an experiment on the consistency of paper reviews. They selected 10% of papers to be reviewed twice. Different area chairs and a different set of 3 reviewers were chosen for those papers.
Luckily for me, my paper with Francis Bach and Simon Lacoste-Julien was one of those 10%. My paper was initially submitted as Paper ID 867. They essentially created a duplicated paper id for me, #1860, which contained the second set of reviews.
This duplication of reviews was particularly interesting in my case. There was a very large discrepancy between the two sets of reviews. I won’t know if this is representative of the consistency of other reviews until NIPS releases the statistics from there experiment.
For reference, the two sets of reviews gave the following scores, before rebuttal:
Set 1 review 1: Quality 9, impact 2 (high) , confidence 4 (confident but not certain)
Generally for NIPS a 9/6/6 in quality gives a high change of acceptance, where as a 5/3/6 is a certain non-acceptance. So one set of reviews was a clear accept and the second a clear reject! The meta reviews were as follows:
The paper introduces a new incremental gradient method that allows adaptation to the level of convexity in the input. The paper has a nice discussion of related methods, and it has a simpler proof that will be of interest to researchers. Recommendation: Accept.
Unfortunately, the scores are too low for acceptance to NIPS, and none of the reviewers were willing to argue for acceptance of the paper. The reviewers discussed the paper after the author rebuttal, and all reviewers ultimately felt that the paper could use some additional polish before publishing. Please do keep in mind the various criticisms of the reviewers when submitting to another venue.
The paper we submitted was fairly rough in its initial state, and the reviewers suggested lots of improvements. Particularly the Set 2/review 1, which was the most in depth review. I generally agree with the second meta-review in that the paper needed addition polish, which we have done for the camera ready.
In the end the paper was accepted. I suspect most papers with this kind of accept/reject split would be accepted, as it would just seem unfair if it were not.
The issue of consistency in paper reviews is clear to any body who has ever resubmitted a rejected paper to a different venue. It feels like luck of a draw to a degree. There is no easy solutions to this, so I’ll be interested to see if NIPS changes there process in future years, and what changes they make.
I am just in the process of finishing writing my PhD, which I wrote entirely in LyX. The productivity advantages of using LyX over LaTeX are too large to ignore, which is why I went with LyX, and why you should too. In this post I will go over the process I went through to get LyX producing documents conforming with my university’s thesis formatting guidelines.
If your anything like me, you have a mixture of past papers you have written in LaTeX, as well as a bunch of notes and drafts in LyX. The university provides a thesis template in LaTeX which the recommend you use. Fortunately, it is actually not too difficult to convert such a template into a working LyX document. Likewise, the papers in LaTeX can also be imported into LyX.
We will get LyX to format our document correctly using the thesis style by overriding most things using a preamble file containing TeX commands. The advantage of this approach is we can pretty much copy and paste the LaTeX from the thesis template provided by the university.
I used two preamble files. The primary file contains the usual TeX package statements, includes and the like. It points to the ANU thesis style file as well:
This is not exactly the same as the preamble in the ANU provided style files. Several packages are already automatically imported by LyX, so they don’t need to be included here.
The default style uses the traditional American indenting of the first line of each paragraph. I think that looks old-fashioned, so I change it to just put a little but more padding between paragraphs instead:
You probably want to setup your thesis so that each chapter is in a separate document. In LaTeX you would have each chapter imported into your main TeX file using \input{}. In LyX, this is done using child documents. As an example, the main document for my thesis looks like the following:
Setting up the main document
Create a new document, and go to Document->Settings->LaTeX Preamble, put in the preamble files we created above:
\input{general-preamble}
\input{main-preamble}
Each of the child documents is included via Insert->File->LyX Document ... The “include type” needs to be set to “include” for it to work correctly. The \mainmatter command signifies the switch over from page numbering using Roman numerals (for the introductory material) to Arabic numerals (for the thesis proper). It is inserted using the “TeX Code” insertion (Ctrl-L), which just directly inserts LaTeX commands into the document.
The Append is started with Document->Start Appendix Here, which should probably be in the insert menu of LyX instead. The bibliography is created with Insert->List/TOC->BibTex bibliography.., which lets you select a BibTex .bib file to use for your citations. LyX supports using multiple different bib files, which would appear useful if your combining multiple papers you have written into a thesis. A word of caution: if there are repeated entries between the bib files you will run into various hard to debug problems in LyX, especially if the entries only differ in capitalisation. I would suggest taking all the bib files and merging them using a command line tool into one file. I used the command:
bibtool -s -d *.bib > all.bib
LyX obeys most of the styling information specified by the preamble we created above. However, there are a few things that it overrides. Follow steps 3-7 from the next section to fix these up.
Setting up the child documents
In my case I put each child document in it’s own folder. This is not necessary, but it seemed like a good idea to keep the child documents in a folder together with any figures used by that document. I will walk through the creation of a single child document here. I created all the chapters initially by just copying the first one I created, but you could do it in a more organised way using LyX templates. The steps for creating the chapter are:
Create a new empty LyX document and save it to a folder with the chapter within your thesis directory.
Add the document to your main document using the include procedure described above.
Open up the settings of the chapter document, and set the document class to Book, and in the master document field, point it at your main.lyx document
On the font section, change the default family base font size to what ever your university requires. This was different than the default for me.
Tick the Two-sided document check-box under Page Layout. This sets it so that the margin is wider on the outside-edge of each page, so it looks right when bound as a book.
On the Language section, change it to English (Australian) or English (UK) if your not in the ’ol US of A.
Your university almost certain specifies a particular citation style. Specify that in the Bibliography section. My university (ANU) recommends a Author-year style:
In the LaTeX Preamble, add the command \input{../general-preamble}, which just points to the previously created preamble file. The main-preamble file is not used by the child documents.
If your using sub-folders for each chapter, place a copy of the thesis.sty and (for ANU) the anuthesis.sty files in the subfolder as well.
The above setup will mean that you can view each chapter separately in LyX using the eye button
in the toolbar. The chapter can be viewed within the whole-thesis PDF using the
button. When viewing separately, the bibliography references will display as question marks (?), whereas they will be displayed correctly in the whole-thesis PDF. There are some suggested work-arounds for this issue on on the LyX Wiki, but I couldn’t get them to work.
Other productivity enhancements
There is a few additional setup steps you should go through in LyX if you haven’t already. These are not required, but they will generally increase your productivity.
Setup Forward and Reverse Search
Setup groups of chapters as LyX branches to make sending subsets of your thesis easier
Setup math macros
Add keyboard shortcuts for citations and cross-references.
LyX is a brilliant tool for writing in the mathematical sciences. It provides a WYSIWYG style editor on top of LaTeX with just the right amount of abstraction for getting-stuff-done. With the proper setup, typing math into LyX is easier, faster and less error prone then using LaTeX directly, and it has virtually no down sides.
Part of the setup is the use of use of LyX math macros. Math macros allow you to define new commands that you can use within LyX math mode. When writing pure LaTeX, most people include a set of standard macros in the header of the LaTeX documents. For example, a common one is a shortcut for argmin:
LyX math macros allow you to do the same thing within LyX.
To create the equivalent macro in LyX, create a new math macro
This will insert the following into your document:
This won’t appear when you render out your document, so you can put it anywhere you like. To create the macro, just fill out the 3 parts. The first part is what you will type when using the macro, the second part is the LaTeX that will be output, and the third part controls what it looks like on screen within LyX. For most macros the TeX and LyX sections should be the same, for the argmin example we want the TeX code to produce a “mathop”, which LyX can’t natively display. So we put a \TeXtrm{argmin} in the LyX part. The filled in macro is thus:
Note that for this to work you have to have amsmath package imported. To set that up, go Document -> Settings..., then select the radio buttons under the math options section:
To use the macro, just type \argmin within a LyX math box, then press space. It should insert the macro seemlessly. Within LyX the subscripts will look like:
but when you render it out to a pdf it will be typeset correctly as:
A few useful macros
I have the following macros setup in a lyx-math-macros.lyx file:
These macros illustrate the parameters feature as well. To add parameters to a macro, use the macro bar that appears at the bottom of the screen when editing the macro.
I have a lyx template setup that I use for new documents. It includes a LyX sub-document include statement
which gives access to the macros.
How it’s implemented
If you convert your LyX document to a pdfLaTeX document, you’ll see the following command is generated in the preamble
in the toolbar. The chapter can be viewed within the whole-thesis PDF using the 
button. When viewing separately, the bibliography references will display as question marks (?), whereas they will be displayed correctly in the whole-thesis PDF. There are some suggested work-arounds for this issue on on the LyX Wiki, but I couldn’t get them to work.
which gives access to the macros.