several small improvements

[opensym2017_postmortem.git] / opensym2017_postmortem.Rmd

diff --git a/opensym2017_postmortem.Rmd b/opensym2017_postmortem.Rmd
index 62db158a9c5f08b80967c4569abe089e2c14b769..111c8ed402130ce910a6b894f864caf754a095a4 100644 (file)
--- a/opensym2017_postmortem.Rmd
+++ b/opensym2017_postmortem.Rmd
@@ -86,12 +86,12 @@ Along with [Claudia Müller-Birn](https://www.clmb.de/) from the [Freie Universt
 In OpenSym 2017, we made several changes to the way the conference has been run:
 
 * In previous years, OpenSym had tracks on topics like free/open source software, wikis, open innovation, open education, and so on. In 2017, **we used a single track model**.
 In OpenSym 2017, we made several changes to the way the conference has been run:
 
 * In previous years, OpenSym had tracks on topics like free/open source software, wikis, open innovation, open education, and so on. In 2017, **we used a single track model**.

-* Because we eliminated tracks, we also eliminated track-level chairs. Instead, **we appointed a series of Associate Chairs or ACs**.
+* Because we eliminated tracks, we also eliminated track-level chairs. Instead, **we appointed Associate Chairs or ACs**.

 * **We eliminated page limits and the distinction between full papers and notes**.
 * **We allowed authors to write rebuttals before reviews were finalized.** Reviewers and ACs were allowed to modify their reviews and decisions based on rebuttals.
 * To assist in assigning papers to ACs and to reviewers, **we made extensive use of bidding**. This means we had to recruit the pool of reviewers before papers were submitted.
 
 * **We eliminated page limits and the distinction between full papers and notes**.
 * **We allowed authors to write rebuttals before reviews were finalized.** Reviewers and ACs were allowed to modify their reviews and decisions based on rebuttals.
 * To assist in assigning papers to ACs and to reviewers, **we made extensive use of bidding**. This means we had to recruit the pool of reviewers before papers were submitted.
 

-Although each of these things have been tried in other conferences, all were new to
+Although each of these things have been tried in other conferences, all were new to OpenSym.

 
 # Overview
 
 
 # Overview
 


@@ -123,7 +123,6 @@ tbl.tmp <- as.data.frame(rbind(
 colnames(tbl.tmp) <- c("Statistics", "")
   
 kable(tbl.tmp)
 colnames(tbl.tmp) <- c("Statistics", "")
   
 kable(tbl.tmp)

-

 ```
 
 The program was similar in size to the last 2-3 years in terms of the number of submissions. OpenSym is a small but mature and stable venue for research on open collaboration. This year was also similar, although slightly more competitive, in terms of the conference acceptance rate (`r round(num.papers.accepted / (nrow(submissions) - 2)*100)`%—it had been slightly above 50% in previous years).
 ```
 
 The program was similar in size to the last 2-3 years in terms of the number of submissions. OpenSym is a small but mature and stable venue for research on open collaboration. This year was also similar, although slightly more competitive, in terms of the conference acceptance rate (`r round(num.papers.accepted / (nrow(submissions) - 2)*100)`%—it had been slightly above 50% in previous years).


@@ -157,11 +156,11 @@ ggplot(data=topics) + aes(x=topic, fill=decision) +
   theme(legend.position="bottom", legend.direction = "horizontal")
 ```
 
   theme(legend.position="bottom", legend.direction = "horizontal")
 ```
 

-The figure above shows a breakdown of papers in terms of these categories as well as indicators of how many papers in each group were accepted. Research on FLOSS and Wikimedia/Wikipedia continue to make up a sizable chunk of OpenSym's submissions and publications. That said, these now make up a minority of total submissions. Although Wikipedia and Wikimedia research made up a smaller proportion of submission pool, it was accepted at a higher rate. Also notable is the fact that 2017 saw an uptick in the number of papers on open innovation. I suspect this was due, at least in part, to work by the General Chair [Lorraine Morgan's](https://www.nuigalway.ie/our-research/people/lorrainemorgan/) involvement (she specializes in that area). Somewhat surprisingly to me, we had a number of submission about Bitcoin and blockchains. These are natural areas of growth for OpenSym but have never been a big part of work in our community in the past.
+The figure above shows a breakdown of papers in terms of these categories as well as indicators of how many papers in each group were accepted. Research on FLOSS and Wikimedia/Wikipedia continue to make up a sizable chunk of OpenSym's submissions and publications. That said, these now make up a minority of total submissions. Although Wikipedia and Wikimedia research made up a smaller proportion of the submission pool, it was accepted at a higher rate. Also notable is the fact that 2017 saw an uptick in the number of papers on open innovation. I suspect this was due, at least in part, to work by the General Chair [Lorraine Morgan's](https://www.nuigalway.ie/our-research/people/lorrainemorgan/) involvement (she specializes in that area). Somewhat surprisingly to me, we had a number of submission about Bitcoin and blockchains. These are natural areas of growth for OpenSym but have never been a big part of work in our community in the past.

 
 # Scores and Reviews
 
 
 # Scores and Reviews
 

-As in previous years, review was single blind in that reviewers' identities are hidden but authors identities are not. Each papers received between 3 and 4 reviews plus a metareview by the Associate Chair assigned to the paper. Almost all papers received 3 reviews but ACs were encouraged to call in a 4th reviewer at any point in the process.  In addition to the text of the reviews, we used a -3 to +3 scoring system where papers that are seen as borderline will be scored as 0. Reviewers scored papers using half-point increments.
+As in previous years, review was single blind in that reviewers' identities are hidden but authors identities are not. Each papers received between 3 and 4 reviews plus a metareview by the Associate Chair assigned to the paper. Almost all papers received 3 reviews but ACs were encouraged to call in a 4th reviewer at any point in the process.  In addition to the text of the reviews, we used a -3 to +3 scoring system where papers that are seen as borderline will be scored as 0. Reviewers scored papers using full-point increments.

 
 ```{r, echo=FALSE}
 ## generate the score graphs
 
 ```{r, echo=FALSE}
 ## generate the score graphs


@@ -177,11 +176,13 @@ ggplot(data=scores) + aes(x=sub.id) +
 ```
 
 
 ```
 
 

-The figure above shows scores for each paper submitted. The vertical grey lines reflect the distribution of scores where the minimum and maximum scores for each paper are the ends of the lines. The colored dots show the arithmetic mean for each score (unweighted by reviewer confidence). Colors show whether the papers were accepted, rejected, or presented as a poster. It's important to keep in mind that two papers were *submitted* as posters. Although Associate Chairs made the final decisions on a case-by-case basis, most papers that had an average score of less than 0 (the horizontal orange line) were rejected and most papers with positive average scores were accepted. We ultimately accepted `r num.papers.accepted` papers (`r paste(round(num.papers.accepted / (nrow(submissions) - 2)*100), "%", sep="")`) of those submitted.
+The figure above shows scores for each paper submitted. The vertical grey lines reflect the distribution of scores where the minimum and maximum scores for each paper are the ends of the lines. The colored dots show the arithmetic mean for each score (unweighted by reviewer confidence). Colors show whether the papers were accepted, rejected, or presented as a poster. It's important to keep in mind that two papers were *submitted* as posters.
+
+Although Associate Chairs made the final decisions on a case-by-case basis, every paper that had an average score of less than 0 (the horizontal orange line) was rejected and most (but not all) papers with positive average scores were accepted. Although a positive average score seemed to be a required publication, negative individual scores weren't necessary showstoppers and we accepted `r table(unique(na.omit(scores.after$sub.id[scores.after$score < 0])) %in% submissions$sub.id[submissions$result == "ACCEPT" & submissions$type == "full paper"])["TRUE"]` papers with at least one negative score. We ultimately accepted `r num.papers.accepted` papers (`r paste(round(num.papers.accepted / (nrow(submissions) - 2)*100), "%", sep="")`) of those submitted.

 
 # Rebuttals
 
 
 # Rebuttals
 

-This was the first time that OpenSym used a rebuttal or author response and were thrilled with how it went. Although they were entire optional, almost everybody used it! Authors of `r length(reviews[reviews$label == "Author response","sub.id"])` of our `r nrow(submissions)` submissions (`r round(length(reviews[reviews$label == "Author response","sub.id"]) / nrow(submissions)*100)`%!) submitted rebuttals.
+This was the first time that OpenSym used a rebuttal or author response and we are thrilled with how it went. Although they were entire optional, almost everybody used it! Authors of `r length(reviews[reviews$label == "Author response","sub.id"])` of our `r nrow(submissions)` submissions (`r round(length(reviews[reviews$label == "Author response","sub.id"]) / nrow(submissions)*100)`%!) submitted rebuttals.

 
 ```{r, echo=FALSE}
 # histogram of changes
 
 ```{r, echo=FALSE}
 # histogram of changes


@@ -222,7 +223,7 @@ Although, I won't post any analysis or graphs, bidding worked well. With only tw
 Given a reviewer pool whose diversity of expertise matches that in your pool of authors, bidding works fantastically. *But everybody needs to bid*. The only problems with reviewers we had were with people that had failed to bid. 
 It might be reviewers who don't bid are less committed to the conference, more overextended, more likely to drop things in general, etc. It might also be that reviewers who fail to bid get good matches become less interested, willing, or able to do their reviews well and on time.
 
 Given a reviewer pool whose diversity of expertise matches that in your pool of authors, bidding works fantastically. *But everybody needs to bid*. The only problems with reviewers we had were with people that had failed to bid. 
 It might be reviewers who don't bid are less committed to the conference, more overextended, more likely to drop things in general, etc. It might also be that reviewers who fail to bid get good matches become less interested, willing, or able to do their reviews well and on time.
 

-Having used bidding twice as chair or track-chair, my sense is that bidding is a fantastic thing to incorporate into any conference review process. The major limitations are that you need to build a PC before the conference (rather than finding the perfect reviewers for specific papers) and you have to find ways to incentive or communicate the importance of getting your PC members to bid.
+Having used bidding twice as chair or track-chair, my sense is that bidding is a fantastic thing to incorporate into any conference review process. The major limitations are that you need to build a PC before the conference (rather than finding the perfect reviewers for specific papers) and you have to find ways to incentivize or communicate the importance of getting your PC members to bid.

 
 # Conclusions
 
 
 # Conclusions
 


@@ -230,9 +231,9 @@ The final results were [a fantastic collection of published papers](https://blog
 
 Although we tried quite a lot of new things, my sense is that nothing we tried made things worse and many things made things smoother or better. Although I'm not directly involved in organizing OpenSym 2018, I am on the OpenSym steering committee.  My sense is that most of the changes we made are going to be carried over this year.
 
 
 Although we tried quite a lot of new things, my sense is that nothing we tried made things worse and many things made things smoother or better. Although I'm not directly involved in organizing OpenSym 2018, I am on the OpenSym steering committee.  My sense is that most of the changes we made are going to be carried over this year.
 

-Finally, it's also been announced that [OpenSym 2018 will be in Paris on August 22-24](http://www.opensym.org/os2018/). The call for papers should be out soon with, I suspect, a spring paper submission deadline. You should consider submitting! I hope to see you in Paris!
+Finally, it's also been announced that [OpenSym 2018 will be in Paris on August 22-24](http://www.opensym.org/os2018/). The call for papers should be out soon and **the OpenSym 2018 paper deadline has already been announced as March 15, 2018**. You should consider submitting! I hope to see you in Paris!

 
 # This Analysis
 
 
 # This Analysis
 

-OpenSym used the gratis version of [EasyChair](https://www.easychair.org/) to manage the conference which doesn't allow chairs to export data. As a result, data used in this this postmortem was scraped from EasyChair using two Python scripts. Numbers and graphs were created using a [knitr](https://yihui.name/knitr/) file that combines R visualization and analysis code and markdown. I've made all the code I used to produce this analysis available in [this git repository](FIXME). I hope someone else finds it useful. Because the data contains sensitive information on the review process, I'm not going to publish the data.
+OpenSym used the gratis version of [EasyChair](https://www.easychair.org/) to manage the conference which doesn't allow chairs to export data. As a result, data used in this this postmortem was scraped from EasyChair using two Python scripts. Numbers and graphs were created using a [knitr](https://yihui.name/knitr/) file that combines R visualization and analysis code and markdown. I've made all the code I used to produce this analysis available in [this git repository](https://code.communitydata.cc/opensym2017_postmortem.git). I hope someone else finds it useful. Because the data contains sensitive information on the review process, I'm not going to publish the data.