Header image

john hawks weblog

paleoanthropology, genetics and evolution

Photo Credit: Remains of Homo naledi. John Hawks CC-BY-NC-ND

Hominin species and time in peer review

In 2015, two new hominin species were published: Australopithecus deyiremeda and Homo naledi. One of the criticisms I’ve seen of both discoveries is the idea that they had not been given sufficient peer review. Since I was involved in the H. naledi discovery and analysis, this criticism took me by surprise.

So I have been taking some time to look at earlier hominin discoveries to look at the time they took to publish. This record provides a lot of evidence about how we have defined and published new species during the last 25 years.

What is the criticism?

Here is a section of Carl Zimmer’s New York Times article describing Au. deyiremeda last May: “The Human Family Tree Bristles With New Branches”.

But some hominid experts remain unconvinced that the road to Homo took so many turns. Tim D. White, a paleoanthropologist at the University of California, Berkeley, argues that most of the new studies have been rushed into publication without careful peer review.
The 3.3-million-year date for the ancient stone tools, for example, “seemed quite sketchy to me,” Dr. White said. The tools could have been made hundreds of thousands of years later, he said.
Dr. White is also skeptical that the new fossils represent a wealth of new species. He suspects that most of them, including Australopithecus deyiremeda, are just Australopithecus afarensis.
“Lucy’s species just got a few more new fossils,” he said of Wednesday’s announcement.

Many similar comments have been made about the Homo naledi publication, and White’s remarks here are not extreme. I wanted to point to this one because obviously this criticism is not specifically about H. naledi, nor is it about open access journals. Au. deyiremeda and the 3.3-million-year-old tools from Lomekwi 3 were both published in Nature.

Time from submission to acceptance

Is it true? Fortunately, we can consider some data from papers describing hominin species about the timeline of review and editorial decision. The paper describing Au. deyiremeda spent 6 months in the editorial process, while the Lomekwi 3 paper was in review from November 1, 2012 to April 13, 2015—more than 2 years and 5 months.

I’ve examined most of the research papers with new formal descriptions of hominin taxa since 1990, and have noted the date of submission and acceptance of each paper. The total list includes some with near-universal acceptance and some with relatively little acceptance from other professionals, but they all were peer-reviewed and satisfy the formal requirements for naming species under the International Code of Zoological Nomenclature.

Here are the data:

Time spent in peer review by hominin species diagnoses

The median time to acceptance for these papers is 70 days. The shortest time, 6 days, was taken by the paper describing Orrorin tugenensis in Comptes Rendus, while Homo floresiensis was 189 days, or more than 6 months, from submission to acceptance in Nature. Au. deyiremeda is the second longest time from submission to acceptance; H. naledi is in the second quartile with Au. garhi, Au. ramidus and H. antecessor.

I’ve looked at a number of ways in which these species descriptions might vary, and have found no obvious correlations yet with the time from submission to acceptance.

  1. The time does not seem to depend on whether the holotype of the taxon is being newly described, or whether it was described in a previous publication (possibly by other authors), as was the case for H. gautengensis and H. cepranensis.

  2. The time does not depend on the number of authors of the paper, which ranges from one (H. gautengensis) to 47 (H. naledi). Only 2 of the papers have more than 9 authors, and 9 of the papers have between 6 and 9 authors.

  3. Two of the taxa were later formally revised by their authors: Australopithecus ramidus to Ardipithecus ramidus and Ardipithecus ramidus kadabba to Ardipithecus kadabba. The editorial time for the initial descriptions was not unusually long or short compared to other species here.

  4. New taxa with Tim White on the author list do not have significantly different review times than those without him on the author list.

  5. One possible correlation: new genus names (Orrorin, Sahelanthropus, Kenyanthropus) are drawn from the shorter times in the distribution. Ardipithecus was named in a correction to White et al. 1994 that does not indicate the time of submission or acceptance, so I cannot test this one.

What can we learn from editorial time?

I’ve included several caveats below with the notes. Keeping these in mind, what does the editorial time tell us? The data can provide evidence concerning the question of whether a paper was “rushed into publication” as some critics may say. But this information is limited. The time from submission to acceptance is a measure only of time, and not necessarily quality or depth of review.

Nearly all of these papers are in three journals, Nature, Science, and Comptes Rendus, all known for a rapid review and editorial timeline. A referee for one of these journals typically agrees to return her review in a short time, sometimes as short as a week to 10 days. For experienced referees, returning a review in 10 days is a matter of ensuring they have time in their schedule, it is not a rush job. A careful review for a journal should not take longer than this.

If peer review shouldn’t take so long, why does it routinely take longer for most journals? I can comment on this from my own perspective as a working scientist and associate editor for PLoS ONE for many years. I schedule editorial and review time very deliberately. It is not every week that I can spare several hours to make a careful review of a manuscript checking for possible problems. As an editor, I often want the opinions of scientists who themselves have many competing demands for their time. When they graciously agree to look carefully at a manuscript, I work with them to do so on as reasonable a timeline as is possible. In my experience, the quality of reviews does not depend upon the time that a referee takes to return them. I have had work accepted in both Nature and Science, and work rejected by each of them. In cases where the work was sent for review, my experience is that these reviews are approximately the same length as reviews I’ve received from more specialized journals. I have only rarely had a referee for one of my papers for Nature or Science reveal her identity to me, but my perception is that they draw from the same pool of reviewers as more field-specific journals.

So I do not think we can draw conclusions from the data above about the quality of reviews, beyond the simple observation that the hominin species published in the past few years are not different in their editorial timeline from other species published as early as 1994. This may be the most remarkable fact. There is no trend evident in the data, and both the shortest and longest editorial processes happened in the middle of the time series. There were no “good old days” in which referees took months and months to carefully review the descriptions of new hominin species.

It’s surprising that the duration of the editorial process remains virtually unchanged despite the vast increase in the speed of communication during the last 25 years. In 1994 when the Australopithecus ramidus description was published, journal submission happened through the post, and peer review required sending papers by fax or post to referees. Ann Gibbons in her 2006 book, The First Human: The Race to Discover Our Earliest Ancestors, describes the submission and editorial process of the Au. ramidus description. A highlight of her story is her account of Nature editor Henry Gee meeting Berhane Asfaw in the lobby of a hotel in South Kensington, London, to be hand-delivered the manuscripts “like a character out of a John le Carré novel, receiving state secrets”. As Gibbons explains, the real motive for the face-to-face meeting was to save time. Today papers are submitted electronically, and instantly, but the process of peer review takes basically the same time.

It may surprise people that the more transparent referee process in eLife still yielded a time between submission and acceptance for Homo naledi that is only slightly less than the median for similar papers during the last 25 years. In my experience, the eLife process led to a faster consideration of the crucial issues with the study, but the referees spent the same time and effort that they would have done in any other journal.

Peer review in context

Peer review has become a cornerstone of the public’s trust in science. An expert can read an argument in a paper and evaluate it from the evidence it contains, but non-experts rely upon assessment by expert referees to confirm that a paper presents sufficient evidence to justify its conclusions.

Still, most professionals have at some time in their careers seen work published by colleagues with what seem to be obvious flaws. When this happens we sometimes ask, “How could that have gotten through peer review?” I’ve done it several times on the blog, focusing on specific criticisms of published papers. For example, in 2009, I called Science “a rinky-dink journal” for allowing the publication of a description of Ardipithecus ramidus material without including standard dental measurements (“Whoa, who stole the data?”). I have sometimes noticed misleading figures in a published paper and wondered how they were not caught and corrected in peer review. Improperly manipulated figures are one of the most common reasons for retraction of scientific papers, so peer reviewers should be especially on the watch for them.

Part of being a scientist is being willing to answer critics who challenge the reliability of research. Anyone who has been a referee for many papers has probably given advice to authors or an editor that was not ultimately taken, sometimes letting through what seem like obvious flaws. Peer review cannot remove all errors; and if we expect peer review to make papers error-free, we are expecting far too much.

But it’s obvious that part of being an effective critic is to obtain the best information and evidence possible. There is no value to blanket assertions that today’s papers are not receiving adequate peer review. The evidence is the opposite: the time and effort in peer review of new hominin taxa has remained basically the same across at least the past 25 years.

I have also written a related post discussing the data on time from discovery to publication: “How long does it take to publish new hominin species after discovery?”.

Notes

  1. I’m waiting on a physical copy of the description of Australopithecus bahrelghazali by Brunet and colleagues (1996), so I haven’t yet included this species.

  2. I’ve listed all taxonomic names based on their first formal description. Some of these have been revised by their original authors.

  3. The list includes both new species based on never-before-described material and new species based on previously-described material.

  4. In my experience publishing scientific papers, the dates for submission and acceptance provided by a journal are often unreliable. It is in the journal’s interest to advertise a rapid publication timeline, so that scientists will be more likely to submit important, time-sensitive work. So journals sometimes print the date only of the most recent or final submission, and do not provide the date of initial submission. They almost never indicate the date of initial editorial inquiry, which may precede formal submission by many months. As a result, the data available for these papers are probably biased to a shorter time of review and editorial guidance than the authors actually experienced.

  5. Also, some of these papers were first submitted to a different journal. The Homo naledi description, for example, was submitted in a different form to another journal before its submission to eLife. In such cases, the editorial timeline in the journal of publication does not reflect the time spent in peer review and editorial correspondence.

  6. The reference list below includes a much longer set than cited in this post; I will be continuing this inquiry with papers from earlier in the history of the field.

References

Arambourg, C. (1954). [The fossil man of Ternifine (Algeria).]. Comptes rendus hebdomadaires des seances de l'Academie des sciences, 239(15), 893-895.

Ascenzi, A., Biddittu, I., Cassoli, P. F., Segre, A. G., & Segre-Naldini, E. (1996). A calvarium of late Homo erectus from Ceprano, Italy. Journal of Human Evolution, 31(5), 409-423.

Asfaw, B., White, T., Lovejoy, O., Latimer, B., Simpson, S., & Suwa, G. (1999). Australopithecus garhi: a new species of early hominid from Ethiopia. Science, 284(5414), 629-635.

Berger, L. R., de Ruiter, D. J., Churchill, S. E., Schmid, P., Carlson, K. J., Dirks, P. H., & Kibii, J. M. (2010). Australopithecus sediba: A new species of Homo-like australopith from South Africa. Science, 328(5975), 195-204.

Bermúdez de Castro, J. M., Arsuaga, J. L., Carbonell, E., Rosas, A., Martınez, I., & Mosquera, M. (1997). A hominid from the Lower Pleistocene of Atapuerca, Spain: possible ancestor to Neandertals and modern humans. Science, 276(5317), 1392-1395.

Broom, R. (1936). New fossil anthropoid skull from South Africa. Nature, 138, 486-488.

Broom, R. (1938). The Pleistocene anthropoid apes of South Africa. Nature, 142, 377-379.

Broom, R. (1949). Another new type of fossil ape-man. Nature, 163, 57.

Broom, R., & Robinson, J. T. (1949). A new type of fossil man. Nature, 164, 322-323.

Brown, P., Sutikna, T., Morwood, M. J., Soejono, R. P., Saptomo, E. W., & Due, R. A. (2004). A new small-bodied hominin from the Late Pleistocene of Flores, Indonesia. Nature, 431(7012), 1055-1061.

Brunet, M., Beauvilain, A., Coppens, Y., Heintz, E., Moutaye, A. H., & Pilbeam, D. (1995). The first australopithecine 2,500 kilometres west of the Rift Valley (Chad). Nature 378, 273-275.

Brunet, M., Beauvilain, A., Coppens, Y., Heintz, E., Moutaye, A. H. E., & Pilbeam, D. (1996). Australopithecus bahrelghazali, une nouvelle espèce d'Hominidé ancien de la région de Koro Toro (Tchad). Comptes rendus de l'Académie des sciences. Série 2. Sciences de la terre et des planètes, 322(10), 907-913.

Brunet, M., Guy, F., Pilbeam, D., Mackaye, H. T., Likius, A., Ahounta, D., ... & De Bonis, L. (2002). A new hominid from the Upper Miocene of Chad, Central Africa. Nature, 418(6894), 145-151.

Curnoe, D. (2010) A review of early Homo in southern Africa focusing on cranial, mandibular and dental remains, with the description of a new species (Homo gautengensis sp. nov.). HOMO - Journal of Comparative Human Biology, 61, 151–177.

Dart, R. A. (1948). The Makapansgat proto‐human Australopithecus prometheus. American Journal of Physical Anthropology, 6(3), 259-284.

Dirks, P. H., Kibii, J. M., Kuhn, B. F., Steininger, C., Churchill, S. E., Kramers, J. D., ... & King, G. C. (2010). Geological setting and age of Australopithecus sediba from southern Africa. Science, 328(5975), 205-208.

Gabounia, L., de Lumley, M. A., Vekua, A., Lordkipanidze, D., & de Lumley, H. (2002). Découverte d’un nouvel hominidé à Dmanissi (Transcaucasie, Géorgie). Comptes Rendus Palevol, 1(4), 243-253.

Groves, C. P. (1999). Nomenclature of African Plio-Pleistocene hominins. Journal of Human Evolution, 37, 869-872.

Haile-Selassie, Y. (2001). Late Miocene hominids from the middle Awash, Ethiopia. Nature, 412(6843), 178-181.

Haile-Selassie, Y., Suwa, G., & White, T. D. (2004). Late Miocene teeth from Middle Awash, Ethiopia, and early hominid dental evolution. Science, 303(5663), 1503-1505.

Haile-Selassie, Y., Gibert, L., Melillo, S. M., Ryan, T. M., Alene, M., Deino, A., ... & Saylor, B. Z. (2015). New species from Ethiopia further expands Middle Pliocene hominin diversity. Nature, 521(7553), 483-488.

Hughes, A. R., & Tobias, P. V. (1977). A fossil skull probably of the genus Homo from Sterkfontein, Transvaal. Nature, 265(5592), 310-312.

Leakey, L. S. B. (1959). A new fossil skull from Olduvai. Nature, 184(4685), 491-493.

Leakey, L. S. B., Tobias, P. V., & Napier, J. (1964). A new species of Homo from Olduvai Gorge. Nature, 202, 7-9.

Leakey, M. G., Feibel, C. S., McDougall, I., & Walker, A. (1995). New four-million-year-old hominid species from Kanapoi and Allia Bay, Kenya. Nature, 376(6541), 565-571.

Leakey, M. G., Spoor, F., Brown, F. H., Gathogo, P. N., Kiarie, C., Leakey, L. N., & McDougall, I. (2001). New hominin genus from eastern Africa shows diverse middle Pliocene lineages. Nature, 410(6827), 433-440.

Leakey, R. E. F., & Wood, B. A. (1973). New evidence of the genus Homo from East Rudolf, Kenya. II. American Journal of Physical Anthropology, 39(3), 355-368.

Mallegni, F., Carnieri, E., Bisconti, M., Tartarelli, G., Ricci, S., Biddittu, I., & Segre, A. (2003). Homo cepranensis sp. nov. and the evolution of African-European Middle Pleistocene hominids. Comptes Rendus Palevol, 2(2), 153-159.

Oppenoorth, W. F. (1932) Homo (Javanthropus) soloensis, en Pleistocene mensch van Java. Wetenschapelijke Medische Dienst Mijnbouw Nederlandsch Indië 20, 49–74.

Senut, B., Pickford, M., Gommery, D., Mein, P., Cheboi, K., & Coppens, Y. (2001). First hominid from the Miocene (Lukeino formation, Kenya). Comptes Rendus de l'Académie des Sciences-Series IIA-Earth and Planetary Science, 332(2), 137-144.

White, T. D., Asfaw, B., DeGusta, D., Gilbert, H., Richards, G. D., Suwa, G., & Howell, F. C. (2003). Pleistocene Homo sapiens from Middle Awash, Ethiopia. Nature, 423(6941), 742-747.

White, T.D., WoldeGabriel, G., Asfaw, B., Ambrose, S., Beyene, Y., Bernor, R.L., Boisserie, J.R., Currie, B., Gilbert, H., Haile-Selassie, Y. and Hart, W.K., 2006. Asa Issie, Aramis and the origin of Australopithecus. Nature, 440(7086), pp.883-889.

White, T. D., Suwa, G., & Asfaw, B. (1994). Australopithecus ramidus, a new species of early hominid from Aramis, Ethiopia. Nature, 371(6495), 306.

WoldeGabriel, G., White, T. D., Suwa, G., Renne, P., de Heinzelin, J., Hart, W. K., & Heiken, G. (1994). Ecological and temporal placement of early Pliocene hominids at Aramis, Ethiopia. Nature, 371(6495), 330-333.

Woodward, A. S. (1921) A new cave man from Rhodesia, South Africa. Nature 108, 371–372.