Why include research of variable quality?

Inevitably, this question arises from people who know what good research looks like. It is a good point and worth thinking about. It raises the question of who is going to sit in judgement of their peers. It seems that journals solved that a long time ago with the idea of peer review. Yet, even though we have peer review in the journal process, we still have weak papers being published.

The problem is that the judgment is not binary. No matter how rigorous the process of selection is, papers will differ in quality. Maybe, if we could do much better, we would have a range from excellent to good enough. But there will still be variability in quality. And it would be more difficult for those not yet at their peak to get through increasingly tough hurdles.

We benefit from seeing the field as it is, rather than as a self-selecting group of arbiters thinks it should be. There is benefit in giving individuals the space to determine for themselves what the difference is between good and bad research. And we rely on PhD examiners, journal editors and the great pool of anonymous referees to keep the worst research from being published. Some will get through this imperfect system. Judgement is involved. It is important for all of us to hone our capacity to make judgements ourselves.

A catalogue is both a discovery aid and a record of the field. It is there to help people find work, and to show what the field has produced. It is not there to tell anyone which work is worth reading. Nobody can know in advance what a given record will be needed for. A paper that looks weak today might hold the one data point a later reviewer is looking for, or a negative finding worth knowing about, or evidence of what the field believed at the time. Reviews of the literature, citation analysis and any account of how the discipline developed all need the full record rather than a selection from it. Arbitrarily removing "weak" work alters the history of the field. Deciding what is good enough belongs earlier in the process, with the people who choose what to publish. The role of this catalogue is to record what was produced, not to weigh it a second time.

So, the answer is a different question. If a small number of people decide on our behalf, what would be gained and what would be lost?