From: Benjamin Mako Hill Date: Tue, 16 Jan 2018 00:09:14 +0000 (-0800) Subject: final set of changes before uploading to website X-Git-Url: https://code.communitydata.science/opensym2017_postmortem.git/commitdiff_plain/HEAD?hp=fea6b23d3390f224aadb52fef7c1904d7ab0197d final set of changes before uploading to website --- diff --git a/.gitignore b/.gitignore index b7efeb5..639c192 100644 --- a/.gitignore +++ b/.gitignore @@ -7,3 +7,5 @@ opensym-reviewers-20180113.csv opensym-reviews-20180113.csv opensym-submissions-20180113.csv opensym2017_postmortem.html +figure/* +opensym2017_postmortem.md diff --git a/opensym2017_postmortem.Rmd b/opensym2017_postmortem.Rmd index 5f3b7c5..f3a7a5f 100644 --- a/opensym2017_postmortem.Rmd +++ b/opensym2017_postmortem.Rmd @@ -79,7 +79,7 @@ scores$sub.id <- ordered(scores$sub.id, levels=scores$sub.id[sort.list(scores$av scores.after$sub.id <- ordered(scores.after$sub.id, levels=levels(scores$sub.id)) ``` -The [International Symposium on Open Collaboration](http://www.opensym.org/) (*OpenSym*, formerly *WikiSym*) is the premier academic venue exclusively focused on scholarly research into open collaboration. OpenSym is an [ACM](http://www.acm.org/) conference which means that, like conferences in computer science, it's really more like a journal that gets published once a year than it is like most social science conferences. The "journal", in this case, is called the *Proceedings of the International Symposium on Open Collaboration* and it consists of final copies of papers which are typically also presented at the conferece. Like journal articles, papers that are published in the proceedings are not typically published elsewhere. +The [International Symposium on Open Collaboration](http://www.opensym.org/) (*OpenSym*, formerly *WikiSym*) is the premier academic venue exclusively focused on scholarly research into open collaboration. OpenSym is an [ACM](http://www.acm.org/) conference which means that, like conferences in computer science, it's really more like a journal that gets published once a year than it is like most social science conferences. The "journal", in this case, is called the *Proceedings of the International Symposium on Open Collaboration* and it consists of final copies of papers which are typically also presented at the conference. Like journal articles, papers that are published in the proceedings are not typically published elsewhere. Along with [Claudia Müller-Birn](https://www.clmb.de/) from the [Freie Universtät Berlin](http://www.fu-berlin.de/), I served as the *Program Chair* for OpenSym 2017. For the social scientists reading this, the role of program chair is similar to being an editor for a journal. My job was not to organize keynotes or logistics at the conference—that is the job of the General Chair. Indeed, in the end I didn't even attend the conference! Along with Claudia, my role as Program Chair was to recruit submissions, recruit reviewers, coordinate and manage the review process, make final decisions on papers, and ensure that everything makes it into the published proceedings in good shape. @@ -93,7 +93,7 @@ In OpenSym 2017, we made several changes to the way the conference has been run: Although each of these things have been tried in other conferences, or even piloted within individual tracks in OpenSym, all were new to OpenSym in general. -# Overview +### Overview ```{r, echo=FALSE, results="asis"} # two papers were originally submitted as posters @@ -129,7 +129,7 @@ The program was similar in size to the last 2-3 years in terms of the number of As in recent years, there were more posters presented than submitted because the PC found that some rejected work, although not ready to be published in the proceedings, was promising and advanced enough to be presented as a poster at the conference. Authors of posters submitted 4-page extended abstracts for their projects which were published in a "*Companion to the Proceedings*." -# Topics +### Topics Over the years, OpenSym has established a clear set of niches. Although we eliminated tracks, we asked authors to choose from a set of categories when submitting their work. These categories are similar to the tracks at OpenSym 2016. Interestingly, a number of authors selected more than one category. This would have led to difficult decisions in the old track-based system. @@ -180,7 +180,7 @@ The figure above shows scores for each paper submitted. The vertical grey lines Although Associate Chairs made the final decisions on a case-by-case basis, every paper that had an average score of less than 0 (the horizontal orange line) was rejected or presented as a poster and most (but not all) papers with positive average scores were accepted. Although a positive average score seemed to be a requirement for publication, negative individual scores weren't necessary showstoppers. We accepted `r table(unique(na.omit(scores.after$sub.id[scores.after$score < 0])) %in% submissions$sub.id[submissions$result == "ACCEPT" & submissions$type == "full paper"])["TRUE"]` papers with at least one negative score. We ultimately accepted `r num.papers.accepted` papers—`r round(num.papers.accepted / (nrow(submissions) - 2)*100)`% of those submitted. -# Rebuttals +### Rebuttals This was the first time that OpenSym used a rebuttal or author response and we are thrilled with how it went. Although they were entirely optional, almost every team of authors used it! Authors of `r length(reviews[reviews$label == "Author response","sub.id"])` of our `r nrow(submissions)` submissions (`r round(length(reviews[reviews$label == "Author response","sub.id"]) / nrow(submissions)*100)`%!) submitted rebuttals. @@ -199,7 +199,7 @@ kable(data.frame("Lower"=rebut.tbl["decrease"], The table above shows how average scores changed after authors submitted rebuttals. The table shows that rebuttals' effect was typically neutral or positive. Most average scores stayed the same but nearly two times as many increased as decreased in the post-rebuttal period. We hope that this made the process feel more fair for authors and I feel, having read them all, that it led to improvements in the quality of final papers. -# Page Lengths +### Page Lengths In previous years, OpenSym followed most other venues in computer science by allowing submission of two kinds of papers: full papers which could be up to 10 pages long and short papers which could be up to 4. Following some other conferences, we eliminated page limits altogether. This is the text we used in [the OpenSym 2017 CFP](https://perma.cc/87QY-27FS): @@ -216,7 +216,7 @@ ggplot(data=pdf.pages) + aes(x=pages) + geom_bar() + scale_x_continuous("Pages", In the end `r table(pdf.pages$pages > 10)["TRUE"]` of `r num.papers.accepted` published papers (`r round(table(pdf.pages$pages > 10)["TRUE"] / num.papers.accepted * 100, 2)`%) were over 10 pages. More surprisingly, `r table(pdf.pages$pages < 10)["TRUE"]` of the accepted papers (`r round(table(pdf.pages$pages < 10)["TRUE"] / num.papers.accepted * 100, 2)`%) were below the old 10-page limit. Fears that some have expressed that page limits are the only thing keeping OpenSym authors from submitting enormous rambling manuscripts seems to be unwarranted—at least so far. -# Bidding +### Bidding Although, I won't post any analysis or graphs, bidding worked well. With only two exceptions, every single assigned review was to someone who had bid "yes" or "maybe" for the paper in question and the vast majority went to people that had bid "yes." However, this comes with one major proviso: people that did not bid at all were marked as "maybe" for every single paper. @@ -225,7 +225,7 @@ It might be reviewers who don't bid are less committed to the conference, more o Having used bidding twice as chair or track-chair, my sense is that bidding is a fantastic thing to incorporate into any conference review process. The major limitations are that you need to build a PC before the conference (rather than finding the perfect reviewers for specific papers) and you have to find ways to incentivize or communicate the importance of getting your PC members to bid. -# Conclusions +### Conclusions The final results were [a fantastic collection of published papers](https://blog.communitydata.cc/opensym-2017-program/). Of course, it couldn't have been possible without the huge collection of [conference chairs, associate chairs, program committee members, external reviewers, and staff supporters](http://opensym.lero.ie/organisation/organization/). @@ -233,7 +233,7 @@ Although we tried quite a lot of new things, my sense is that nothing we changed Finally, it's also been announced that [OpenSym 2018 will be in Paris on August 22-24](http://www.opensym.org/os2018/). The call for papers should be out soon and **the OpenSym 2018 paper deadline has already been announced as March 15, 2018**. You should consider submitting! I hope to see you in Paris! -# This Analysis +### This Analysis OpenSym used the gratis version of [EasyChair](https://www.easychair.org/) to manage the conference which doesn't allow chairs to export data. As a result, data used in this this postmortem was scraped from EasyChair using two Python scripts. Numbers and graphs were created using a [knitr](https://yihui.name/knitr/) file that combines R visualization and analysis code with markdown to create the HTML directly from the datasets. I've made all the code I used to produce this analysis available in [this git repository](https://code.communitydata.cc/opensym2017_postmortem.git). I hope someone else finds it useful. Because the data contains sensitive information on the review process, I'm not publishing the data.