From a43301823ee657a6d297ce301c6325ba143bd044 Mon Sep 17 00:00:00 2001 From: Benjamin Mako Hill Date: Mon, 15 Jan 2018 11:23:25 -0800 Subject: [PATCH] several small improvements - added a statistics to the writeup and fixed an error - changed the description in the scrapers to be more accurate --- easychair-review-scraper.py | 2 +- easychair-submissions-scraper.py | 2 +- opensym2017_postmortem.Rmd | 7 ++++--- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/easychair-review-scraper.py b/easychair-review-scraper.py index 356cb90..f057d73 100755 --- a/easychair-review-scraper.py +++ b/easychair-review-scraper.py @@ -1,6 +1,6 @@ #!/usr/bin/python3 # -*- coding: utf-8 -*- -""" Bot to scrape a list of EasyChair submissions and upload them to a wiki """ +""" script to scrape a list of EasyChair review data and save them as CSV files """ # # (C) Benjamin Mako Hill, 2018 # (C) Federico Leva, 2016 diff --git a/easychair-submissions-scraper.py b/easychair-submissions-scraper.py index e839258..8cbd4b1 100755 --- a/easychair-submissions-scraper.py +++ b/easychair-submissions-scraper.py @@ -1,6 +1,6 @@ #!/usr/bin/python3 # -*- coding: utf-8 -*- -""" Bot to scrape a list of EasyChair submissions and upload them to a wiki """ +""" script to scrape a list of EasyChair submission metadata and save them as CSV files """ # # (C) Benjamin Mako Hill, 2018 # (C) Federico Leva, 2016 diff --git a/opensym2017_postmortem.Rmd b/opensym2017_postmortem.Rmd index f5726b5..111c8ed 100644 --- a/opensym2017_postmortem.Rmd +++ b/opensym2017_postmortem.Rmd @@ -123,7 +123,6 @@ tbl.tmp <- as.data.frame(rbind( colnames(tbl.tmp) <- c("Statistics", "") kable(tbl.tmp) - ``` The program was similar in size to the last 2-3 years in terms of the number of submissions. OpenSym is a small but mature and stable venue for research on open collaboration. This year was also similar, although slightly more competitive, in terms of the conference acceptance rate (`r round(num.papers.accepted / (nrow(submissions) - 2)*100)`%—it had been slightly above 50% in previous years). @@ -161,7 +160,7 @@ The figure above shows a breakdown of papers in terms of these categories as wel # Scores and Reviews -As in previous years, review was single blind in that reviewers' identities are hidden but authors identities are not. Each papers received between 3 and 4 reviews plus a metareview by the Associate Chair assigned to the paper. Almost all papers received 3 reviews but ACs were encouraged to call in a 4th reviewer at any point in the process. In addition to the text of the reviews, we used a -3 to +3 scoring system where papers that are seen as borderline will be scored as 0. Reviewers scored papers using half-point increments. +As in previous years, review was single blind in that reviewers' identities are hidden but authors identities are not. Each papers received between 3 and 4 reviews plus a metareview by the Associate Chair assigned to the paper. Almost all papers received 3 reviews but ACs were encouraged to call in a 4th reviewer at any point in the process. In addition to the text of the reviews, we used a -3 to +3 scoring system where papers that are seen as borderline will be scored as 0. Reviewers scored papers using full-point increments. ```{r, echo=FALSE} ## generate the score graphs @@ -177,7 +176,9 @@ ggplot(data=scores) + aes(x=sub.id) + ``` -The figure above shows scores for each paper submitted. The vertical grey lines reflect the distribution of scores where the minimum and maximum scores for each paper are the ends of the lines. The colored dots show the arithmetic mean for each score (unweighted by reviewer confidence). Colors show whether the papers were accepted, rejected, or presented as a poster. It's important to keep in mind that two papers were *submitted* as posters. Although Associate Chairs made the final decisions on a case-by-case basis, every paper that had an average score of less than 0 (the horizontal orange line) was rejected and most (but not all) papers with positive average scores were accepted. We ultimately accepted `r num.papers.accepted` papers (`r paste(round(num.papers.accepted / (nrow(submissions) - 2)*100), "%", sep="")`) of those submitted. +The figure above shows scores for each paper submitted. The vertical grey lines reflect the distribution of scores where the minimum and maximum scores for each paper are the ends of the lines. The colored dots show the arithmetic mean for each score (unweighted by reviewer confidence). Colors show whether the papers were accepted, rejected, or presented as a poster. It's important to keep in mind that two papers were *submitted* as posters. + +Although Associate Chairs made the final decisions on a case-by-case basis, every paper that had an average score of less than 0 (the horizontal orange line) was rejected and most (but not all) papers with positive average scores were accepted. Although a positive average score seemed to be a required publication, negative individual scores weren't necessary showstoppers and we accepted `r table(unique(na.omit(scores.after$sub.id[scores.after$score < 0])) %in% submissions$sub.id[submissions$result == "ACCEPT" & submissions$type == "full paper"])["TRUE"]` papers with at least one negative score. We ultimately accepted `r num.papers.accepted` papers (`r paste(round(num.papers.accepted / (nrow(submissions) - 2)*100), "%", sep="")`) of those submitted. # Rebuttals -- 2.39.5