X-Git-Url: https://code.communitydata.science/opensym2017_postmortem.git/blobdiff_plain/564bb075377b39b9b0fae9650d47db563aa9b879..fea6b23d3390f224aadb52fef7c1904d7ab0197d:/opensym2017_postmortem.Rmd diff --git a/opensym2017_postmortem.Rmd b/opensym2017_postmortem.Rmd index 8f4a181..5f3b7c5 100644 --- a/opensym2017_postmortem.Rmd +++ b/opensym2017_postmortem.Rmd @@ -79,9 +79,9 @@ scores$sub.id <- ordered(scores$sub.id, levels=scores$sub.id[sort.list(scores$av scores.after$sub.id <- ordered(scores.after$sub.id, levels=levels(scores$sub.id)) ``` -The [International Symposium on Open Collaboration](http://www.opensym.org/) (*OpenSym*, formerly *WikiSym*) is the premier academic venue exclusively focused on scholarly research into open collaboration. OpenSym is an [ACM](http://www.acm.org/) conference which means that, like conferences in computer science, it's really more like a journal that gets published once a year than it is like most social science conferences. The "journal", in this case, is called the *Proceedings of the International Symposium on Open Collaboration* and it consists of complete archival papers which are typically also presented at the conferece. Papers that are published in the proceedings are not typically published elsewhere. +The [International Symposium on Open Collaboration](http://www.opensym.org/) (*OpenSym*, formerly *WikiSym*) is the premier academic venue exclusively focused on scholarly research into open collaboration. OpenSym is an [ACM](http://www.acm.org/) conference which means that, like conferences in computer science, it's really more like a journal that gets published once a year than it is like most social science conferences. The "journal", in this case, is called the *Proceedings of the International Symposium on Open Collaboration* and it consists of final copies of papers which are typically also presented at the conferece. Like journal articles, papers that are published in the proceedings are not typically published elsewhere. -Along with [Claudia Müller-Birn](https://www.clmb.de/) from the [Freie Universtät Berlin](http://www.fu-berlin.de/), I served as the *Program Chair* for OpenSym 2017. For the social scientists reading this, the role of program chair is similar to being an editor for a journal. My job was *not* to organize keynotes or logistics at the conference—that is the job of the General Chair. Indeed, in the end I didn't even attend the conference! Along with Claudia, my role as Program Chair was to recruit submissions, recruit reviewers, coordinate and manage the review process, make final decisions on papers, and ensure that everything makes it into the published proceedings in good shape. +Along with [Claudia Müller-Birn](https://www.clmb.de/) from the [Freie Universtät Berlin](http://www.fu-berlin.de/), I served as the *Program Chair* for OpenSym 2017. For the social scientists reading this, the role of program chair is similar to being an editor for a journal. My job was not to organize keynotes or logistics at the conference—that is the job of the General Chair. Indeed, in the end I didn't even attend the conference! Along with Claudia, my role as Program Chair was to recruit submissions, recruit reviewers, coordinate and manage the review process, make final decisions on papers, and ensure that everything makes it into the published proceedings in good shape. In OpenSym 2017, we made several changes to the way the conference has been run: @@ -89,7 +89,7 @@ In OpenSym 2017, we made several changes to the way the conference has been run: * Because we eliminated tracks, we also eliminated track-level chairs. Instead, **we appointed Associate Chairs or ACs**. * **We eliminated page limits and the distinction between full papers and notes**. * **We allowed authors to write rebuttals before reviews were finalized.** Reviewers and ACs were allowed to modify their reviews and decisions based on rebuttals. -* To assist in assigning papers to ACs and to reviewers, **we made extensive use of bidding**. This means we had to recruit the pool of reviewers before papers were submitted. +* To assist in assigning papers to ACs and reviewers, **we made extensive use of bidding**. This means we had to recruit the pool of reviewers before papers were submitted. Although each of these things have been tried in other conferences, or even piloted within individual tracks in OpenSym, all were new to OpenSym in general. @@ -127,7 +127,7 @@ kable(tbl.tmp) The program was similar in size to the last 2-3 years in terms of the number of submissions. OpenSym is a small but mature and stable venue for research on open collaboration. This year was also similar, although slightly more competitive, in terms of the conference acceptance rate (`r round(num.papers.accepted / (nrow(submissions) - 2)*100)`%—it had been slightly above 50% in previous years). -As in recent years, there were more posters accepted than submitted because the PC found that some rejected work, although not ready to be published in the proceedings, was promising and advanced enough to be presented as a poster at the conference. Authors of posters submitted 4-page extended abstracts for their projects which were published in a "*Companion to the Proceedings*."" +As in recent years, there were more posters presented than submitted because the PC found that some rejected work, although not ready to be published in the proceedings, was promising and advanced enough to be presented as a poster at the conference. Authors of posters submitted 4-page extended abstracts for their projects which were published in a "*Companion to the Proceedings*." # Topics @@ -178,7 +178,7 @@ ggplot(data=scores) + aes(x=sub.id) + The figure above shows scores for each paper submitted. The vertical grey lines reflect the distribution of scores where the minimum and maximum scores for each paper are the ends of the lines. The colored dots show the arithmetic mean for each score (unweighted by reviewer confidence). Colors show whether the papers were accepted, rejected, or presented as a poster. It's important to keep in mind that two papers were *submitted* as posters. -Although Associate Chairs made the final decisions on a case-by-case basis, every paper that had an average score of less than 0 (the horizontal orange line) was rejected and most (but not all) papers with positive average scores were accepted. Although a positive average score seemed to be a requirement for publication, negative individual scores weren't necessary showstoppers. We accepted `r table(unique(na.omit(scores.after$sub.id[scores.after$score < 0])) %in% submissions$sub.id[submissions$result == "ACCEPT" & submissions$type == "full paper"])["TRUE"]` papers with at least one negative score. We ultimately accepted `r num.papers.accepted` papers—`r round(num.papers.accepted / (nrow(submissions) - 2)*100)`% of those submitted. +Although Associate Chairs made the final decisions on a case-by-case basis, every paper that had an average score of less than 0 (the horizontal orange line) was rejected or presented as a poster and most (but not all) papers with positive average scores were accepted. Although a positive average score seemed to be a requirement for publication, negative individual scores weren't necessary showstoppers. We accepted `r table(unique(na.omit(scores.after$sub.id[scores.after$score < 0])) %in% submissions$sub.id[submissions$result == "ACCEPT" & submissions$type == "full paper"])["TRUE"]` papers with at least one negative score. We ultimately accepted `r num.papers.accepted` papers—`r round(num.papers.accepted / (nrow(submissions) - 2)*100)`% of those submitted. # Rebuttals