Patrick D Schloss
(Stepping up to break the ice and comment formally here instead of just on twitter)
1. I think the Ben Schwessinger experience described here (https://blushgreengrassatafrid... is worth a mention for a couple of different reasons. It's the first time that I can recall that a journal had to step up and actually deal with a situation where scooping by preprint (or because of preprint) may have occurred. As such the policy at PLoS has been refined. When things change, there are always the uneasy situations like this that force people to make difficult (and sometimes wrong) decisions
2. I think it's also worthwhile to mention sites like PubPeer. Public reviews and comments on preprints are part of overlapping discussions but aren't necessarily the same discussion. Feels like there's something to be said about that although I'm not sure what that is right now.
3. My whole take on "but it's not peer reviewed" is that those that will be reading the preprints in order to cite them are well qualified as reviewers themselves. If you don't trust the paper or don't like it, don't cite it. If you read through the paper and don't see fault with experiments, why not cite it? We all have blindspots but it's not like we don't review papers all the time and critique them anyway even if they've been through peer review.
4. I think we should make a greater effort to write positive comments on preprints and not just use this as a forum for review. Positive comments can help those who maybe aren't in the literature figure out which preprints are great and which have holes (by their lack of positive comments). I see this as important if preprints are going to be written about by the popular press and digested by those who aren't necessarily experts. We as experts need to endorse good papers just as we will trash the bad papers.
5. I had the first preprint in biorXiv under Microbiology, why are you taking this achievment away from me Schloss?
6. Looping back on number 4...if we are going to be the ones reviewing grants and papers and we see a preprint cited, we can actually review this work. Some are going to use it to get around page limits but, like you point out, we as scientists should be pretty good at snuffing shoddy and rushed work out and so that this could also theoretically backfire on the person trying an end run on page limits. Sure it may give you more space to write, but if you do a terrible job you may otherwise poison the impression of a grant reviewer that might otherwise like your grant. I'm tired of having to see (in press) or (in prep) when work is cited in a paper or grant. If it's an important enough story for the grant, I want to be able to read the story myself and preprints allow this.
7. There are different costs and benefits for preprints depending on the field you are in and the point in your career. I don't know that we've figured this out at all yet or if there is a great answer across the board. It seems as though the pop gen fields have taken to preprints more than other fields, but in my experience evolutionary biology in general tends to be less "scoopy" or "eat their young" than other fields. I'd like the world to exist where everyone can freely post preprints and get credit, but I can see this going horribly wrong in fields that are much more competitive and potentially containing more selfish PIs. I mean this not as a positive or negative commentary on different fields, but it's quite obvious to me that some fields are more cutthroat than others for a variety of reasons and the cost/benefit analysis for preprints in these fields will be different.
Couple more points that I think are worth mentioning (and sorry if they are in there already, read it yesterday):
8. Worth mentioning that preprints can be an important "minor league" for journals to scout for papers from and that the more that we can show this happening the more preprints will benefit
9. There's been discussion about bioRxiv not "allowing" methods papers (there was a twitter scuttlebutt a few weeks ago). Their stance is that (I'm putting words in their mouths) they don't just want to publish step by step protocols that have been published elsewhere (like in Sambrook). They will publish step by steps for new protocols though. Would be good to point out that ProtocolsIO is a "live" preprint server (effectively) for protocols though. I like the interface there better for describing protocols because it's constantly growing and being annotated by those doing the protocols and with nuances for other organisms.
10. Jeff Ross-Ibarra (and probably others) have started using preprints during their journal clubs and leaving comments at the various venues. I think this is a great way to help students learn to review, but also want to highlight that it's important to think about protecting the student's identities when commenting on preprints until this is a commonplace thing.
11. It would be nice if comments left on preprints would populate NCBI comment sections. I don't know if this will ever happen, but it would be nice to know that others have talked about papers before they are published.
12. There are also starting to be "review communities" (like PCI Evolutionary Biology https://evolbiol.peercommunity... which will evaluate preprints and could (could!) act as reviewers for forums like mSphere Direct or otherwise if we let them.