journals

NPR's Science Friday interviewed open science advocate Michael Nielsen last week: "Can science be done without secrecy?" I like the headline.

FLATOW: Why are scientists the last ones to get in on this?

NIELSEN: Well, it's kind of funny. I mean, they certainly helped bring us the Web back in the early '90s. Unfortunately, they're pretty bought into doing things in the standard way, the way they've done them for centuries, which is you do your work in the lab, you get all your results, you write them up in a paper and possibly several years later, it all appears for your colleagues to digest at that point.

And, you know, that's a great system if you're back in the 1600 or 1700s, but today we've got better tools, but people still haven't adopted them.

It's a long, thoughtful interview. I want to point to a later part also:

FLATOW: Mm-hmm. Is it - do you find that there is resistance to the way - to changing the way, the old way, of things being done? Are older scientists resistant to change and becoming part of this process?

NIELSEN: I'm not so sure I'd say resistance so much as it's just difficult to see how to cause a large-scale change. How do you get everybody simultaneously to adopt the new way of doing things? And so some people will kind of throw up their hands and say, well, it just can't be done. Probably the people actually get that the most from, scientists who are sort of in their mid-career. They're doing post docs or they're at the end of their graduate studies. And while they're subject to the system, they don't really feel like there's very much they can do to change it. When I talk to younger scientists, they're often very enthusiastic. And sometimes, when I talk to much more senior scientists, because they feel like they have some power to actually change the system, they can actually be quite enthusiastic.

Some thoughts on this later. For now, let me point again to my "Public interests in data from federally funded research."

Philip Ball: "The h-index, or the academic equivalent of the stag's antlers".

Few topics excite more controversy among scientists. When I spoke about the h-index to the German Physical Society a few years ago, the huge auditorium was packed. Some deplore it; some find it useful. Some welcome it as a defence against the subjective capriciousness of review and tenure boards.

...

No one officially endorses the h-index for evaluation, but scientists confess that they use it all the time as an informal way of, say, assessing applicants for a job. The trouble is that it's precisely for average scientists that the index works rather poorly: small differences in small h-indices don't tell you very much.

In anthropology, the h-index has almost no utility at the time it matters -- hiring and tenure. Citations have a long tail distribution -- a few papers will usually capture the majority of citations of a scholar's work, with most papers being relatively uncited. The h-index provides a measure that discounts the citations from one or two super-highly-cited papers, in an attempt to quantify more of the shape of the distribution of citations among an individual's works. The number of publications and citations for early-career scholars is just too low for the shape to differ much among scholars that have published the same number of papers. You see, just as an individual's distribution citations have a long tail, so does the distribution of citations among scholars. Publication count gives a proxy for effort, but whether that effort has translated into important effects is generally not well indicated by citations until later in the career.

Metrics are a way to deflect accountability from promotion committees. Stag antlers work, in principle, because they are honest signals of the stag's ability to survive and thrive in the face of a significant handicap. If that's true of later-career scholars with high citation counts, it's probably a sign that the handicaps should be removed for younger academics!

I always look through the table of contents of Nature Genetics, which I have delivered to my inbox. Over the last couple of years, the journal has included a high fraction of papers that are either original genome-wide association studies or meta-analyses of multiple studies. These are substantial studies that have dozens of authors, on conditions of broad interest -- for example, this month there is a meta-analysis paper about type 2 diabetes. So I have no criticism of the journal, these studies need to be published somewhere.

But others might be impatient with this course of research. The studies are formulaic: put together a large set of cases and controls, run them across a genotyping chip, and report the results. In the current issue, the journal's editorial board enters an op/ed suggesting that the current situation will not continue forever, because GWAS studies just aren't that interesting anymore [1]:

Which Mendelian variants produce results suitable for publication in the journal? Our general principles are and have always been to select papers for review by the amount of new data and new ideas and the resource value contained within. Papers must meet current field-specific standards set by our latest benchmark papers and referee advice. Finally, we consider the value of the paper as a research tool, prioritizing those that will motivate larger numbers of scientists to do their research differently as a consequence. In principle it should be possible to find a phenotype for each of the tens of thousands of genetic elements in the human genome, but not all such results will be equally informative. However, if, say, 50 other labs will drop everything and instead use the results of your work, that paper is certainly suitable for this journal!

Well, there you go. The editorial also addresses pedigree research, stating that new identifications of Mendelian disorders in single families will not be sent for review.

I think this all is appropriate, it's just interesting that research has advanced to the point that finding a genetic cause for a disorder is no longer a sufficient reason for publication. If you look through the GWAS Catalog, you find study after study published in Nature Genetics. Those days are probably numbered.

References

Jason Baird Jackson posts some insights on how traditional journals can turn to open access tools (if not become open access), and how a startup online journal can strategize archiving for permanence: "Genres Leak, Being a Reflection on Michael E. Smith’s Essay on Semi-, Quasi- and Psuedo- Journals".

Jackson's post attracted an insightful reply by commenter Barbara, that I want to post in part:

There are many pseudo-journals and pseudo-books being published in an attempt to mimic the old formats rather than experiment, hoping to pass as scholarship by virtue of looking scholarly. And so long as we weigh scholarship by the pound, so to speak, there will be incentives to populate these fake journals and books rather than create something new and insightful.

And in so doing, I emphasize that one of the historical aspects of form that online communication makes obsolete is names. How we attribute ideas must change if labels and signifiers are changed. Here I don't have commenters, and that does reduce some confusion that attends authorship.

Personally I was looking around for how to integrate digital object identifiers (DOI) into online content. Seems to me that integrating online discussion into the academic literature would be done most simply by exploiting the system most widely used for citation tracking in the literature itself -- and that many blogs (including mine) already track. However, adding DOI to content turns out to involve an expensive membership to a cartel run by publishers.