I thought it would be good to share what tools and processes we used to keep things running smoothly, to help other program committees.
- We allocated NNN minutes of discussion per paper (this will vary depending on PC size and #papers to discuss). Basically, take the total amount of time and divide by #papers to discuss, subtracting 1-2 minutes per paper as slack time.
- We used the iPad app Lightning Talk to keep track of time (thanks to Jenna Date for pointing me to this app)
- On the projector, we displayed what paper we were discussing and who the conflicts were. This helped speed up conflicts getting out of the room. Here is a shortened version of the slides. (Thanks to Morley Mao and Landon Cox for this idea)
- Conveniently, if you print these slides 6 to a page, they are roughly post-it size. We taped each slide to a post-it, making it easier to see what the decisions were for each paper. See the picture below to see how we used these post-its (note that we blurred out information about most papers in the picture). We had 4 categories: Accept, Weak Accept, Maybe Accept, and Reject.
- We started the TPC by discussing the strongest papers and then the weakest papers, to help the PC with calibration of the other papers.
- We assigned a discussion lead for each paper, typically the most positive reviewer. Each lead was asked to start with 2min summary, and then we moved on to discussion from the other reviewers.
- We asked committee members to be decisive about making decisions within the NNN minute period. Earlier, several days before the meeting, we also asked TPC members to discuss online papers for which there was not a clear consensus. This approach helped with faster decision making.
- If a paper was a clear accept or reject after a few minutes, we cut off discussion to keep moving forward. This let us have more time for more papers that needed more discussion time.