Volume 2, Issue 4

In recent years there has been increased attention on research methods and practices that increase reproducibility, transparency, and data sharing, as well as attempts to replicate previous research. These open science initiatives increase confidence in empirical findings, leading to improved theory development. The goal of this special issue of the Journal of Expertise is to examine and discuss open research practices and reproducibility in the area of expertise research.

Across research areas, general issues of low statistical power, publication bias, undisclosed flexibility in data analysis, and researcher degrees of freedom, can be recipes for irreproducibility. To address the problem, a reform movement known as the “credibility revolution” emphasizes the need for greater transparency in how research is conducted. In this article, we describe a general approach to creating a culture of openness—tailored for expertise researchers—and describe how and why practices such as preregistration, open notebooks, open data, open materials, and open communication, might be applied to research on experts. We argue that adopting these practices helps to connect end-users with the entire research lifecycle and helps reconnect researchers with the process of gaining knowledge. By sharing notes about our predictions and plans along the way, we are forced to confront their merits. By documenting design and data analytic decisions ahead of time, and by sharing data and materials, we make errors and insights more discoverable. And by inviting research partners, expert practitioners, and the public into the lab, we stand the best chance of successfully translating research into practice.

A number of recent reforms in psychological science have centered around following best practices to improve the robustness and reliability of empirical findings. Among these, preregistration has become a fundamental component, on the rise in the last few years, yet it remains relatively uncommon in expertise research. In this paper, I point out the numerous benefits of preregistration, drawing on specific examples from the field of expertise. I then examine some of the challenges the field of psychology is currently facing to implement systematic preregistration, including many that are particularly exacerbated in expertise research. Specifically, I discuss widespread design characteristics such as small sample sizes, the lack of consistent definitions regarding what constitutes expert performance, and inherent difficulties in conducting replication studies with rare, elite populations. Finally, I make a number of recommendations to facilitate preregistration in expertise research, including tips to handle and report deviations from original plans, and discuss future directions toward more prevalent open science practices.

Eight hundred and fifty-four researchers were signatories of the article “Retire statistical significance” published in Nature in March 2019. I was one of them, and in this article, I expand on why retiring statistical significance would improve research in general, and psychology and expertise research in particular. The proposal involves eliminating the binary statistical significance decision in a single study and adopting the collective effort of accumulating evidence over several studies. When statistical significance is removed, the file drawer problem would disappear, and practices that are ethical (but unlawful in the statistical significance regime) such as interim analyses to decide on continuing or stopping data collection, would be acceptable. This change of paradigm suits expertise research. Expertise research suffers from small sample sizes, which are problematic in a world in which individual studies are done to make grandiose claims of statistical significance, but they are perfectly acceptable in a world of slow and humble collective accumulation of evidence. I suggest that pre-registration is a good scientific practice, which should be encouraged when it could be done. However, I criticize the use of pre-registration as a gold standard for publication. In a world without statistical significance, most of the problems that pre-registration aims to solve are no longer problems. I conclude that, for those reasons, retiring statistical significance would boost expertise research.

It has been claimed by prominent authors that there is no relationship between differences in some human traits (e.g., cognitive ability, physical ability) and differences in skill among experts. We assert that the failure to detect such associations is often due to an extreme form of range restriction that particularly plagues research focused on expert samples: right-tail range restriction (RTRR). RTRR refers to a lack of representation of data from the far right segment of the normal distribution, inhibiting the observation of statistical associations. Using two example studies we demonstrate that, when RTRR is not present, relationships between differences in experts’ traits and differences in their degree of skill can be observed. Based on the characteristics of these studies we make recommendations for methodological practices that can be followed to help investigators overcome RTRR and facilitate the continued development of a robust and replicable science of expertise.