1. Describe what is meant by the “replicability crisis” in psychology.
2. Describe some questionable research practices.
3. Identify some ways in which scientific rigor may be increased.
4. Understand the importance of openness in psychological science.
In this final section, we discuss a major ongoing historical shift in the research methods of psychology.
At the start of this book we discussed the “Many Labs Replication Project,” which failed to replicate the original finding by Simone Schnall and her colleagues that washing one’s hands leads people to view moral transgressions as less wrong (Schnall, et al., 2008)[1]. Although this project is a good illustration of the collaborative and self-correcting nature of science, it also represents one specific response to psychology’s recent “replicability crisis,” a phrase that refers to the inability of researchers to replicate earlier research findings. Consider for example the results of the Reproducibility Project, which involved over 270 psychologists around the world coordinating their efforts to test the reliability of 100 previously published psychological experiments (Aarts et al., 2015)[2]. Although 97 of the original 100 studies had found statistically significant effects, only 36 of the replications did! Moreover, even the effect sizes of the replications were, on average, half of those found in the original studies (see Figure 13.5). Of course, a failure to replicate a result by itself does not necessarily discredit the original study as differences in the statistical power, populations sampled, and procedures used, or even the effects of moderating variables could explain the different results (Yong, 2015)[3].
Figure 13.5 Summary of the Results of the Reproducibility Project Reprinted by permission from Macmillan Publishers Ltd: Nature [Baker, M. (30 April, 2015). First results from psychology’s largest reproducibility test. Nature News. Retrieved from http://www.nature.com/news/first-results-from-psychology-s-largest-reproducibility-test-1.17433\], copyright 2015. [Image description]
Although many believe that the failure to replicate research results is an expected characteristic of cumulative scientific progress, others have interpreted this situation as evidence of systematic problems with conventional scholarship in psychology, including a publication bias that favors the discovery and publication of counter-intuitive but statistically significant findings instead of the duller (but incredibly vital) process of replicating previous findings to test their robustness (Aschwanden, 2015[4]; Frank, 2015[5]; Pashler & Harris, 2012[6]; Scherer, 2015[7]). Worse still is the suggestion that the low replicability of many studies is evidence of the widespread use of questionable research practices by psychological researchers. These may include:
It is important to shed light on these questionable research practices to ensure that current and future researchers (such as yourself) understand the damage they wreak to the integrity and reputation of our discipline (see, for example, the “Replication Index,” a statistical “doping test” developed by Ulrich Schimmack in 2014 for estimating the replicability of studies, journals, and even specific researchers). However, in addition to highlighting what not to do, this so-called “crisis” has also highlighted the importance of enhancing scientific rigor by:
One particularly promising response to the replicability crisis has been the emergence of open science practices that increase the transparency and openness of the scientific enterprise. For example, Psychological Science (the flagship journal of the Association for Psychological Science) and other journals now issue digital badges to researchers who pre-registered their hypotheses and data analysis plans, openly shared their research materials with other researchers (e.g., to enable attempts at replication), or made available their raw data with other researchers (see Figure 13.6).
Figure 13.6 Digital Badges from the Center for Open Science
These initiatives, which have been spearheaded by the Center for Open Science, have led to the development of “Transparency and Openness Promotion guidelines” (see Table 13.7) that have since been formally adopted by more than 500 journals and 50 organizations, a list that grows each week. When you add to this the requirements recently imposed by federal funding agencies in Canada (the Tri-Council) and the United States (National Science Foundation) concerning the publication of publicly-funded research in open access journals, it certainly appears that the future of science and psychology will be one that embraces greater “openness” (Nosek et al., 2015)[11].
Table 13.7 Transparency and Openness Promotion (TOP) Guidelines Reproduced with permission
| Criteria | Level 0 | Level 1 | Level 2 | Level 3 |
| Citation Standards | Journal encourages citation of data, code, and materials, or says nothing | Journal describes citation of data in guidelines to authors with clear rules and examples. | Article provides appropriate citation for data and materials used consistent with journal’s author guidelines. | Article is not published until providing appropriate citation for data and materials following journal’s author guidelines. |
| Data Transparency | Journal encourages data sharing, or says nothing | Article states whether data are available, and, if so, where to access them. | Data must be posted to a trusted repository. Exceptions must be identified at article submission. | Data must be posted to a trusted repository, and reported analyses will be reproduced independently prior to publication. |
| Analytic Methods (Code) Transparency | Journal encourages code sharing, or says nothing | Article states whether code is available, and, if so, where to access them. | Code must be posted to a trusted repository. Exceptions must be identified at article submission. | Code must be posted to a trusted repository, and reported analyses will be reproduced independently prior to publication. |
| Research Materials Transparency | Journal encourages materials sharing, or says nothing | Article states whether materials are available, and, if so, where to access them. | Materials must be posted to a trusted repository. Exceptions must be identified at article submission. | Materials must be posted to a trusted repository, and reported analyses will be reproduced independently prior to publication. |
| Design and Analysis Transparency | Journal encourages design and analysis transparency, or says nothing | Journal articulates design transparency standards | Journal requires adherence to design transparency standards for review and publication | Journal requires and enforces adherence to design transparency standards for review and publication |
| Preregistration of studies | Journal says nothing | Journal encourages preregistration of studies and provides link in article to preregistration if it exists | Journal encourages preregistration of studies and provides link in article and certification of meeting preregistration badge requirements | Journal requires preregistration of studies and provides link and badge in article to meeting requirements. |
| Preregistration of analysis plans | Journal says nothing | Journal encourages preanalysis plans and provides link in article to registered analysis plan if it exists | Journal encourages preanalysis plans and provides link in article and certification of meeting registered analysis plan badge requirements | Journal requires preregistration of studies with analysis plans and provides link and badge in article to meeting requirements. |
| Replication | Journal discourages submission of replication studies, or says nothing | Journal encourages submission of replication studies | Journal encourages submission of replication studies and conducts results blind review | Journal uses Registered Reports as a submission option for replication studies with peer review prior to observing the study outcomes. |
Question
What are open science practices?
Answer the question above the continue reading. iTELL evaluation is based on AI and may not always be accurate.
Image Descriptions
Figure 13.5 long description: Infographic titled “Reliability Test.” It says, “An effort to reproduce 100 psychology findings found that only 39 held up (based on criteria set at the start of each study). But some of the 61 non-replications reported similar findings to those of their original papers.”
There is a graphic representing these 100 reproductions as squares of various shades of blue and black. The graphic answers the question, “Did replicate match original’s results?” There are 61 squares on the “No” side and 39 on the “Yes” side.
Each square’s colour is determined by how closely the findings of the experiment it represents resemble the original study. The ratings are:
For the “No” side, the results break down as such:
For the “Yes” side, the results break down as such:
Last updated at
