several small improvements

- added a statistics to the writeup and fixed an error
- changed the description in the scrapers to be more accurate

diff --git a/easychair-review-scraper.py b/easychair-review-scraper.py
index 356cb904b9070026e3256b03ec29929c346d5b15..f057d73743500c8932f3280c4be0145d2ba78062 100755 (executable)
--- a/easychair-review-scraper.py
+++ b/easychair-review-scraper.py
@@ -1,6 +1,6 @@
 #!/usr/bin/python3
 # -*- coding: utf-8  -*-
-""" Bot to scrape a list of EasyChair submissions and upload them to a wiki """
+""" script to scrape a list of EasyChair review data and save them as CSV files """
 #
 # (C) Benjamin Mako Hill, 2018
 # (C) Federico Leva, 2016

diff --git a/easychair-submissions-scraper.py b/easychair-submissions-scraper.py
index e839258982a07d77e5045033772b2dc098d23c1b..8cbd4b115c808e4f668f13a8969d694c9f5453ad 100755 (executable)
--- a/easychair-submissions-scraper.py
+++ b/easychair-submissions-scraper.py
@@ -1,6 +1,6 @@
 #!/usr/bin/python3
 # -*- coding: utf-8  -*-
-""" Bot to scrape a list of EasyChair submissions and upload them to a wiki """
+""" script to scrape a list of EasyChair submission metadata and save them as CSV files """
 #
 # (C) Benjamin Mako Hill, 2018
 # (C) Federico Leva, 2016

diff --git a/opensym2017_postmortem.Rmd b/opensym2017_postmortem.Rmd
index f5726b5911d31603b6bec619cb5674a6cae8019e..111c8ed402130ce910a6b894f864caf754a095a4 100644 (file)
--- a/opensym2017_postmortem.Rmd
+++ b/opensym2017_postmortem.Rmd
@@ -123,7 +123,6 @@ tbl.tmp <- as.data.frame(rbind(
 colnames(tbl.tmp) <- c("Statistics", "")
   
 kable(tbl.tmp)
-
 ```
 
 The program was similar in size to the last 2-3 years in terms of the number of submissions. OpenSym is a small but mature and stable venue for research on open collaboration. This year was also similar, although slightly more competitive, in terms of the conference acceptance rate (`r round(num.papers.accepted / (nrow(submissions) - 2)*100)`%—it had been slightly above 50% in previous years).
@@ -161,7 +160,7 @@ The figure above shows a breakdown of papers in terms of these categories as wel
 
 # Scores and Reviews
 
-As in previous years, review was single blind in that reviewers' identities are hidden but authors identities are not. Each papers received between 3 and 4 reviews plus a metareview by the Associate Chair assigned to the paper. Almost all papers received 3 reviews but ACs were encouraged to call in a 4th reviewer at any point in the process.  In addition to the text of the reviews, we used a -3 to +3 scoring system where papers that are seen as borderline will be scored as 0. Reviewers scored papers using half-point increments.
+As in previous years, review was single blind in that reviewers' identities are hidden but authors identities are not. Each papers received between 3 and 4 reviews plus a metareview by the Associate Chair assigned to the paper. Almost all papers received 3 reviews but ACs were encouraged to call in a 4th reviewer at any point in the process.  In addition to the text of the reviews, we used a -3 to +3 scoring system where papers that are seen as borderline will be scored as 0. Reviewers scored papers using full-point increments.
 
 ```{r, echo=FALSE}
 ## generate the score graphs
@@ -177,7 +176,9 @@ ggplot(data=scores) + aes(x=sub.id) +
 ```
 
 
-The figure above shows scores for each paper submitted. The vertical grey lines reflect the distribution of scores where the minimum and maximum scores for each paper are the ends of the lines. The colored dots show the arithmetic mean for each score (unweighted by reviewer confidence). Colors show whether the papers were accepted, rejected, or presented as a poster. It's important to keep in mind that two papers were *submitted* as posters. Although Associate Chairs made the final decisions on a case-by-case basis, every paper that had an average score of less than 0 (the horizontal orange line) was rejected and most (but not all) papers with positive average scores were accepted. We ultimately accepted `r num.papers.accepted` papers (`r paste(round(num.papers.accepted / (nrow(submissions) - 2)*100), "%", sep="")`) of those submitted.
+The figure above shows scores for each paper submitted. The vertical grey lines reflect the distribution of scores where the minimum and maximum scores for each paper are the ends of the lines. The colored dots show the arithmetic mean for each score (unweighted by reviewer confidence). Colors show whether the papers were accepted, rejected, or presented as a poster. It's important to keep in mind that two papers were *submitted* as posters.
+
+Although Associate Chairs made the final decisions on a case-by-case basis, every paper that had an average score of less than 0 (the horizontal orange line) was rejected and most (but not all) papers with positive average scores were accepted. Although a positive average score seemed to be a required publication, negative individual scores weren't necessary showstoppers and we accepted `r table(unique(na.omit(scores.after$sub.id[scores.after$score < 0])) %in% submissions$sub.id[submissions$result == "ACCEPT" & submissions$type == "full paper"])["TRUE"]` papers with at least one negative score. We ultimately accepted `r num.papers.accepted` papers (`r paste(round(num.papers.accepted / (nrow(submissions) - 2)*100), "%", sep="")`) of those submitted.
 
 # Rebuttals