data access

The White House this week announced a new policy on public access to results from federally funded research. The announcement has gotten

Michael Eisen comments: "No celebrations here: why the White House public access policy sucks".

The administration fell hook line and sinker for the ridiculous argument put forth by publishers that the only way for researchers and the public to get the servies they provide is to give them monopoly control over the articles for a year – the year when they are of greatest potential use.

Think about how absurd this is. Publishers, whose role should be to disseminate information as widely as possible, are now the only reason why the public will continue to not have access to research results their tax dollars paid for.

Why is Eisen so exercised? Here's an excerpt from the White House policy memo describing the policy on publication access:

In developing their public access plans, agencies shall seek to put in place policies that enhance innovation and competitiveness by maximizing the potential to create new business opportunities and are otherwise consistent with the principles articulated in section 1.

Agency plans must also describe, to the extent feasible, procedures the agency will take to help prevent the unauthorized mass redistribution of scholarly publications.

In other words, it's no longer just a matter of copyright agreements with publishers; now the federal agencies themselves must help police PDF sharing among researchers. I wonder where "mass redistribution" will kick in.

Further, the memo does not set a 12-month access embargo as a maximum, it directs agencies to adopt the 12-month embargo as a guideline. There is a lot not to like in the memo.

Most of the public attention to the decision has been directed at the effects on scientific publications. I have long been interested in a second area: the public access to data generated by federally funded research.

The White House Office of Science and Technology Policy last year requested public comment on two questions: open dissemination of federally-funded research and open access to data resulting from federally-funded research. I commented last year in response to the OSTP request ("Public interests in data from federally funded research") about the value of data to scientists and others who are not members of federally funded labs. The present announcement from the White House did not indicate how these comments from last year may have contributed to the decision, but it includes general recommendations on both publication and data access.

As it stands, the text of the memo essentially keeps in place the data access requirements established under the Bush administration. That is not a bad thing, and indeed the recommendations listed in the memo seem very reasonable. I quote them here at length:

Each agency’s public access plan shall:

a) Maximize access, by the general public and without charge, to digitally formatted scientific data created with Federal funds, while:

i) protecting confidentiality and personal privacy,

ii) recognizing proprietary interests, business confidential information,and intellectual property rights and avoiding significant negative impact on intellectual property rights, innovation, and U.S. competitiveness, and

iii) preserving the balance between the relative value of long-term preservation and access and the associated cost and administrative burden;

b) Ensure that all extramural researchers receiving Federal grants and contracts for scientific research and intramural researchers develop data management plans, as appropriate, describing how they will provide for long-term preservation of, and access to, scientific data in digital formats resulting from federally funded research, or explaining why long-term preservation and access cannot be justified;

c) Allow the inclusion of appropriate costs for data management and access in proposals for Federal funding for scientific research;

d) Ensure appropriate evaluation of the merits of submitted data management plans;

e) Include mechanisms to ensure that intramural and extramural researchers comply with data management plans and policies;

f) Promote the deposit of data in publicly accessible databases, where appropriate and available;

g) Encourage cooperation with the private sector to improve data access and compatibility, including through the formation of public-private partnerships with foundations and other research funding organizations;

h) Develop approaches for identifying and providing appropriate attribution to scientific data sets that are made available under the plan;

i) In coordination with other agencies and the private sector, support training, education, and workforce development related to scientific data management, analysis, storage, preservation, and stewardship; and

j) Provide for the assessment of long-term needs for the preservation of scientific data in fields that the agency supports and outline options for developing and sustaining repositories for scientific data in digital formats, taking into account the efforts of public and private sector entities.

These recommendations are all basically already in the NSF data access policies, meaning that the new White House memo will maintain the status quo at that level.

The problem is that the current policy is toothless. Continued data access is a very serious problem threatening the integrity of science. Self-archiving and institutional archiving have been sufficient to pass data management portions of grant applications, but have proven to be woefully insufficient to enable access to data. Meanwhile, some fields have intensive data collection but very little or no data entering the public domain as part of digital repositories. The recommendations listed above do nothing to change the current situation.

Nevertheless there is some room within the recommendations for agency directors to take bolder action on data access. Section (j) perhaps provides the best hope. If federal funding agencies actually assess the long-term needs of each field supported by funding, many (including anthropology) will clearly benefit from the establishment of standard digital repositories.

I hope that NSF will not sit on its current policy but will instead work to extend access more broadly. At the same time, I wish the White House had given clearer guidance to enable the creation of digital repositories and to require their standard use as a condition of continued funding of research projects.

I submitted the following essay in response to the Request for Information on Public Access to Digital Data Resulting from Federally Funded Research from the National Science and Technology Council's Interagency Working Group on Digital Data.

This RFI is not the same as the current bill before Congress ("Open access op/ed in NY Times"), which would restrict public access to research articles based on federally funded research. Research articles are a very important issue, but I hope that the access to digital data will not be overshadowed by the attention to published results. As a paleoanthropologist, I believe that access to digital data from federally funded research projects is a fundamentally important issue, as I remark below.

Introduction

The United States provides grant funding to scientists through many federal programs. This funding advances work of public interest that might not happen without federal assistance.

The creation of scientific knowledge may serve the public interest directly by enabling useful inventions or supplying actionable information on issues of public importance. A funded project may also serve the public interest indirectly, by (1) finding negative results that prevent wasted effort or public harm; (2) building the scientific infrastructure that enables future discoveries and advances; (3) training new and established scientists in effective research techniques; (4) enhancing international cooperation and public/private partnerships.

Congress and the Executive Branch have recognized that access to the published results of scientific research is not sufficient to advance the direct and indirect public interests served by federally funded projects. Facilitating the indirect benefits of research is a major aim of federal agencies' "Broader Impacts" and data access rules. These policies have been a qualified success since their implementation, limited mainly by the exceptions carved out by programs and agencies to avoid requiring certain kinds of data to be reported along with research reports.

I argue that open public access to digital data should be a requirement for all federally funded scientific research. Digital data can be maintained by federal agencies as a part of the reporting requirement of federal grant funding. Doing so will advance the interest of the public and ensure that today's science generates a continuing heritage of research excellence.

Data access and transparency

Transparency is essential to public trust. Scientific conclusions are formed by observation and replication, and for this process to be transparent, all data must be available for independent inspection. The possibility of such inspection should not be limited to qualified researchers, because the very existence of special access requirements blocks transparency of the scientific process.

Changing technology has shifted the public's expectations about transparency. Digital technology enables most research data to be shared rapidly and at low cost. If data are produced in digital form, and digital data can be shared at low cost, researchers and agencies cannot credibly claim that the difficulty of reproducing and disseminating data is a sufficient reason to restrict access. Where no competing interest argues for restricted access (such as human subjects protections), a lack of access to digital data itself can now be a compelling reason for public distrust.

Therefore, federally funded researchers should release digital data to the public by default. Federal agencies should facilitate this public reporting by requiring digital data to be supplied as part of final project reporting.

Data access has a well-established record of success

The recent history of human genetics demonstrates that open access to data has unforeseen benefits that can spawn innovation, support more effective education, and catalyze new discovery. In genetics, both federal and journal policies require release of data; raw data from federally funded projects are often available as they are generated, long before publication.

My own laboratory has no federal research funding to date, but is actively engaged in research using data from federally funded projects. Today my laboratory trains undergraduate students in genetics with new data from ongoing federally funded genetic projects such as the 1000 Genomes Project. We use open access data from archaic human genomes to investigate the variation of ancient people and their relationships to living humans. This kind of work would be impractical without clearly established open data access policy.

The open access to data from the Human Genome Project facilitated the rapid development of microarrays that are now used on a broad scale in human genetics to investigate the genetic correlates of human health and disease. Access to data from these studies has enabled other scientists to independently replicate many genetic associations. More important, meta-analysis of such data has shown that many associations cannot be replicated, while also showing some cases in which nonsignificant results across different samples give rise to a significant finding when pooling those samples. Access to negative results and raw data is necessary, in other words, to establish the facts in subsequent research. This goes beyond access to published research results and requires open access to unpublished digital data.

Intellectual property protections and data access

Research data are somewhat distinct from the intellectual property issues relating to research publications. Some kinds of data do not meet the standard of originality necessary for copyright protection, such as sequence data, CT or MRI data, or data from measurement instruments. For raw data from instruments, there is no intellectual property reason why federal agency should not maintain an open archive for the public.

Much research data is unquestionably subject to copyright protection, such as lab notebooks, written descriptions, photographs, and original reconstructions. Yet there is still a substantial public and scientific interest in inspecting such data. For example, photographic documentation of archaeological sites and specimens are of particular scientific value and are today routinely produced by digital technologies and stored in digital form. Some primary digital records are unique products that cannot be recreated at another time and place: for example, in situ photographs of specimens, photographs and records of sites before excavation, and digital reconstructions. The scientific record would be incomplete without such contributions, and maintaining an archive of such data over the long term is a difficult task for a single investigator, beyond the scope of a grant term.

In cases where it is impracticable to obtain Creative Commons or other open licenses to such content, a funding agency should at a minimum require that a copy of all such archival information be deposited along with the final project report and a limited-use non-commercial license permitting electronic dissemination of these materials to the public as part of the report.

Metadata and data access

Many have noted that raw data may be useless in the absence of additional information about how the data were obtained. Such information is known as "metadata". Researchers generate instrumental data using particular instrument settings and recording standards. They gather observational data under particular research protocols. These standards are may change quickly as instrumentation, technology, and scientific results themselves demand new practices.

Some scientists note the problem of incompatible metadata, using it as an argument against to delay the establishment of open public access to data. In their view, the public are likely to misunderstand or misuse scientific data where metadata are not clearly indicated. Meta-analyses combining data from multiple research projects are an important secondary use of digital data, and such meta-analyses are impossible when data cannot be reconciled into common observational or instrumental frameworks. Performing original work with data collected in heterogeneous contexts is a research speciality of its own, and is itself sometimes targeted by federal grants.

However, meta-analysis is only one purpose of data access. Transparency, replicability, and education are central public interests that do not require the reconciliation of data collection methods from multiple studies. They require only clear description of the methods under which data were obtained. At a minimum, final research reports on federally funded projects must describe the standards of data collection with sufficient detail to allow independent replication, including all unpublished results and data.

Successes of data access in paleoanthropology

I am an anthropologist, and am most familiar with the scientific data relating to human evolution. These data include genetic observations on living and skeletal samples of humans. They also include fossil and archaeological evidence such as photographs, CT scans, isotopic records, anatomical measurements and descriptions.

For many years, nearly all genetic data resulting from federally funded research have been made available for public download. Much genetic data generated by non-federally funded research programs, including foreign and domestic institutes, has also been free for public download. These data have resulted in a massive acceleration of research on recent human evolution and human origins. They have also led to unexpected discoveries and a burgeoning contribution of other disciplines to understanding our evolution.

Data from radiocarbon dating and other isotopic sampling has also been made available to the public. Human occupation sites are among the best sources of evidence about past climates. The investment of federal resources in human evolution research has generated a temporal record that is now essential to studying changes in the faunal and plant compositions of past environments. Free access to records has enabled stronger calibration of radiocarbon dates, the development of a more secure chronology, and a more highly replicable scientific record correlating different regions of the world. Our understanding of such events changes is vastly stronger when data are made public.

Institutions and data access in paleoanthropology

By contrast, CT scans and photographs pertaining to human origins are typically not made freely accessible to the public. The United States funding agencies are not the only parties with an interest in such data. In particular, museums and institutes that curate specimens often permit data collection under agreements that restrict the dissemination of the resulting data. Such agreements may be equated to "non-disclosure agreements" with respect to scientific data.

An institution has a legitimate interest in controlling the public use of images and access to curated materials. Nevertheless, the lack of access to digital data results in reduplication of effort, overapplication of destructive sampling and measurement techniques, and unnecessary handling of precious and fragile specimens. Where it is practical, the United States should facilitate agreements with institutions that allow the release of digital data produced by public funding. Where release is not possible, funding should be granted only for those activities that will result in the release of data under a limited-use non-commercial license. Non-disclosure of data from instruments such as CT scanners, electron microscopes, or mass spectrometers is incompatible with scientific replication.

Scientific careers and data access in paleoanthropology

The economy of federal funding for scientific production sometimes leads to perverse incentives for high-ranking researchers that prevent public access to research data. Some scientists believe that their own future research will require exclusive access to data. Others want to impede research achievements by their academic rivals, or to maintain prestige and future funding opportunities.

Scientific data in some areas may constitute "trade secrets" until they are protected by patents. Even in noncommercial research, federally funded scientists sometimes claim exclusive ownership over data that they plan to use in future research. In my own field of paleoanthropology, data secrecy supports a clandestine "quid pro quo" economy among researchers, in which established researchers and institutions allow furtive looks at unpublished data, to support and consolidate their power and influence.

This is a game that the United States should simply decline to play. When federal research supports scientific results that are not subject to independent replication, it betrays the public interest in science.

Established collaborations and centers of scientific research will always exert a strong influence upon the future of science, irrespective of federal data access policies. But established players should not use federal funding to construct barriers to open inquiry.

Conclusion

Open public access to data is one indication that a research project is following scientific principles. Making digital data available to the public would be good practice for any researcher, irrespective of funding source. Data access mitigates the risk that negative data will be unreported. Data access facilitates broader stewardship of research projects, in particular where collaborations create data that are distributed across many institutions. Data access and reporting standards enable other researchers to fill in for those who cannot complete scientific project due to health or other personal reasons.

Federal grant agencies already have successful repositories for many kinds of digital data. Such data are shared with the public at minimal cost relative to the overall budget for federal research grants. Supporting digital data repositories has itself been an important granting aim for several federal agencies and continues to be an active part of scientific infrastructure. Limiting such repositories for the exclusive use of a small cadre of researchers is enormously wasteful of resources, when they can be opened to an interested public for a small incremental cost.

The public has repeatedly invented surprising uses for digital data that can complement or enhance the scientific record. But much more important, open access to digital data serves the scientific values of transparency and independent replication, essential to maintaining public trust and investment in the research enterprise.

Inside Higher Ed reviews and interviews an author who argues that the scholarly monograph shackles academics to an obsolete model of communication:

So it is strategic that Kathleen Fitzpatrick, director of scholarly communication at the Modern Language Association and a professor of media studies at Pomona College, invokes the living dead early to illustrate her argument in Planned Obsolescence: Publishing, Technology, and the Future of the Academy (NYU Press). The scholarly press book, she writes, “is no longer a viable mode of communication … [yet] it is, in many fields, still required in order to get tenure. If anything, the scholarly monograph isn’t dead; it is undead."

I agree with this thesis in part. Sixty-dollar monographs are going the way of the thylacine. Locking scholarly content in the tall stacks of university libraries doesn't disseminate it. Peer review no longer improves work to the extent that it's worth locking it up in response. It is ridiculous for anyone to judge the quality of a young scholar's work by the imprint of a "prestigious" academic press. Tenure committees have simply delegated their responsibilities to editors, and the editors do a poor job.

But I disagree that the scholarly monograph is dead. Personally, I expect monographs to undergo a renaissance as more academics adopt e-publishing. Academic presses affiliated with universities should be going all-digital, and should start massively promoting their back catalogs as e-books at fire-sale prices. The smart ones will take the opportunity to change their agenda, competing to publish new books by a new generation of scholars who are building a broad readership both inside and outside academia. There's no reason why we need to constrain our scholarship to books so boring that nobody wants to read them. Tomorrow's scholars should be engaging with a much broader public than university presses have historically cultivated.

The stumbling block is that these books still must serve as a guide to the academic quality of young scholars' work. On this count, Fitzpatrick provides some useful ideas about how to build quality scholarship under a more collaborative model:

The way to make this work, Fitzpatrick says, is to change the currency of scholarly communications from paper to credit. Instead of rewarding faculty for getting a lot of paper published, universities should consider how helpful tenure candidates have been in parsing other people’s articles written and helping others refine their ideas, she says. Journals could help out with this by creating “trust metrics” that cede more weight to academics who consistently give constructive feedback. They could also encourage frequent, thoughtful reviews by making them prerequisites for publishing one’s own work — thus attracting the sort of critical mass of reviewers that Fitzpatrick argues is necessary for successful peer-to-peer review (and which some previous high-profile experiments with the model failed to get).

Under such a system, faculty members could glide to tenure on the wings of their reputations as positive contributors to the advancement of knowledge in their field — a metric the current “publish-or-perish” model does not adequately represent, Fitzpatrick says. “Little in graduate school or on the tenure track inculcates helpfulness,” she writes, “and in fact much militates against it.”

Obviously I think this model would be better than our current one. Still, I worry about the actual assignment of credit. Quite frankly, all my writing here has done wonders for my influence, but has had a substantial drawback: Many of my ideas are used by other scholars without credit or citation. We compete for research support, and in that competition I get no credit or acknowledgement whatsoever for any contributions I make. That's a cost I've been willing to pay for what I do, but if we expect more young academics to share their ideas broadly, we're going to need to change the culture of research funding to recognize their contributions appropriately.

My favorite part of the interview is the last question, which asked Fitzpatrick to give advice about new models of publication to a junior faculty member, librarian, and university provost, respectively.

Finally, to the provost: understand that scholarly communication is a core responsibility of the university – so fundamental to the university mission, in fact, that it must be thought of as part of the institution’s infrastructure, not as a revenue center. And every university must develop some kind of plan for scholarly communication. If you leave disseminating the work of your faculty exclusively to corporate publishers, corporations will profit from it at your institution’s expense. Instead, invest in the structures that will get your faculty’s work into broader circulation – not least because those structures will help you make clear to the concerned public why the university continues to matter today.

I'm going to append to this post the first link to my entry in the Anthropologies project: "What's wrong with anthropology?" where I discuss my own perspective on these problems. Needless to say, I think things need to change. I expect the change in scholarly communication to be highly specific to each academic field, as what works for cultural anthropology will not be the same as what works for genetics or English. But new approaches will be digital, and that means a university may find much more ability to support multiple approaches than is possible with print. The tools to support varied forms are already available, if universities would support and extend them, they could capture much of the need for academic communication.