Concurrent Scientific Session (Core Administration): Rigor and Reproducibility Practices in a Core Setting
Developing an action plan using results from the Committee on Core Rigor and Reproducibility (CCoRRe) Survey
The ABRF Committee on Core Rigor and Reproducibility (CCoRRe) solicited feedback from ABRF members via a survey designed to gain insight into how NIH initiatives on advancing scientific rigor and reproducibility may influence core leaders in providing services and developing new technology. The survey returned valuable feedback regarding challenges, opportunities, and new practices/resources that can help ensure the conduct of rigorous research. The survey results showed that awareness of the NIH guidelines is variable. However, a consistent picture also emerged from the survey results, highlighting the factors that support or impede the conduct of rigorous research, and the types of supportive services that research cores can offer to their users.
This talk will highlight an institutional approach to developing guidelines and best practices for research cores and shared resources to support overall scientific rigor, reproducibility and transparency (RR&T) at Vanderbilt University Medical Center (VUMC). The collaborative process used to draft new guidelines for RR&T will be described, and direct connections between VUMC's work-in-progress and the best practices identified by respondents to the CCoRRe survey will be emphasized. Attendees will see how the CCoRRe survey data might inspire their own research cores and institutions to develop similar resources and spark new initiatives.
Rigor and reproducibility in shared resources: ABRF Committee on Core Rigor and Reproducibility survey update
Shared scientific resources, also known as core facilities, support a significant portion of the research pursued in biomolecular research institutions. The Committee on Core Rigor and Reproducibility (CCoRRe) conducted a survey aimed at the scientific core community to gain information on how NIH initiatives on advancing scientific rigor and reproducibility have influenced current services and new technology development in shared scientific resources. In addition, the survey aimed to identify the challenges and opportunities scientific cores have related to implementation of new reporting requirements to identify new practices and resources needed to assure rigorous research. Presented is a rigorous re-analysis of the participant’s responses and commentaries. Results revealed mixed perspectives and levels of awareness regarding the NIH guidelines. Many of the perceived challenges to the effective implementation of scientific rigor and reproducibility practices were similarly noted as challenge areas in effectively providing support services in a core setting. Further, most cores routinely use best practices and offer services that support rigor and reproducibility. These factors include access to well-maintained instrumentation, training on experimental design and data analysis, as well as data management. Feedback from the survey will enable the ABRF to build better educational resources and share critical best practice guidelines. ABRF sponsors a central information portal that can be expanded to provide timely information on meetings, training modules, online repositories and webinars supporting research integrity. These resources will provide important tools to the core community and the researchers they serve to positively impact rigor and transparency across the range of science and technology.
Quality control of purified proteins to improve research data reproducibility
As the research community strives to make published research ever more transparent and reliable, the quality of reagents used comes into focus. One category of such reagents that requires much stricter quality controls are recombinant proteins. Examples of typical quality issues with recombinant proteins will be presented, along with some results as to how this affects the reliability of the intended downstream application. One very problematic issue to be presented is aggregation and its effect on protein-protein affinity measurements. In order to improve the reliability and reproducibility of data using purified proteins in life science research, a group of professionals involved in protein purification and protein characterization/molecular biophysics from both the ARBRE-MOBIEU (Association of Resources for Biophysical research in Europe – MOlecular BIophysics in EUrope) and P4EU (Protein Production and Purification Partnership in Europe) networks have drafted guidelines for improved quality control (QC). These guidelines, consisting of (i) minimal (but obligatory) information to be provided about the protein production process and methods used (ii) a minimal set of quality tests, i.e. purity, identity, homogeneity and lack of aggregation and (iii) some further recommendations (DNA binding, LPS contamination, ‘competent’ fraction, batch-to-batch reproducibility, storage conditions, etc.) for tests based on the intended application of the proteins will be presented. Furthermore, over a one-year period, the networks have attempted to evaluate the impact of these guidelines by correlating the levels of quality control applied to given samples with the success and reproducibility of downstream experiments. The results indicate that QC guideline implementation can facilitate both experimental reliability and protein quality optimization. It seems, therefore, that investing in protein QC is advantageous to all the stakeholders in life sciences (researchers, editors and funding agencies alike) by improving data veracity and minimizing loss of valuable time and resources.
Reproducibility as a value-add
Researchers face increasing expectations to incorporate reproducible research practices in their work. The shifting landscape of data mandates, author submission guidelines, community checklists, global IDs for reagents or shared protocols, and other new practices can be daunting for researchers to engage with, especially when so many of these practices are tied to the work done at shared research facilities where they are not the domain experts. We discuss an approach that presents greater consultation with and documentation of the work of shared research facilities in resulting publications as a valuable service that can be provided to authors to help them address these new reproducibility expectations for their work. Using papers from the Center for Open Science's "Reproducibility Project: Cancer Biology" project as models, we discuss potential workflows and tools to support this increasingly deep collaboration with researchers as well as related marketing and framing suggestions to help persuade researchers of the value of this approach.