Exploratory vs. Confirmatory Research: Striking the Right Balance

This is a slightly revised and condensed version of a post that originally appeared on our channel, "Cognitive Psychometrics."

Today’s post is dedicated to the relationship between exploratory and confirmatory research in science. It so happens that students and researchers in the social sciences are more often taught to conduct confirmatory studies. This means that research hypotheses must be generated based on a theory or previously obtained results. Exploratory research, which allows you to look at data “directly” without prior hypotheses or statistical models, is less frequently conducted and rarely published. Moreover, such studies sometimes can be stigmatized as p-hacking or p-fishing.

Try submitting a study with the general idea of "we wanted to see what would happen," and you’re likely to be asked by most editors and reviewers: Where are your hypotheses? Where’s the theoretical justification? What theory underpins your assumptions?

The problem is that scientific discoveries often occur during exploratory research.

 In 2019, a special issue of The American Statistician titled “Statistical Inference in the 21st Century: A World Beyond p < 0.05” was published. Most articles in this issue focused on p-values and their proper use. However, some authors addressed the broader context of using statistical methods and inferences in the social sciences. Specifically, one article, “Statistical Inference Enables Bad Science; Statistical Thinking Enables Good Science” by Christopher Tong, discussed what statistical thinking is and the relationship and balance between confirmatory and exploratory studies.

Christopher Tong, a physicist and biostatistics expert, argues that scientific research should rely less on formal statistical inference. He suggested that formal statistical inferences are valid and appropriate only for confirmatory analyses, where the design and analysis methods are strictly predetermined. These analyses represent the final stages of the iterative scientific process.

He suggests that nearly all forms of statistical inference suffer from significant flaws, particularly due to the “optimism principle” discussed in detail by Picard and Cook in 1984. In short, this principle states that a model selected based on a specific data will fit that data well but may perform worse on other data. Since inferential statistics rely on selecting a model that best fits the data, this principle raises concerns.

The model selection process often involves multiple steps—variable transformations, producing and including interaction terms, checking residuals, and so on. This flexibility gives researchers considerable freedom. Additionally, these steps can lead to overfitting, where a model fits the data too well but lacks generalizability. Chatfield (1995) referred to this as "model selection bias". Tong emphasizes the importance of cross-validation and regularization to address this bias, but insists that these alone are insufficient an it is nessesary to confirm hypothesis on different data and conditions. This is the aim of confirmatory studies.

At the same time, the researchers should have more freedom and Tong argues that most scientific studies should adopt an exploratory approach: the design, implementation, and analysis of research must remain flexible and open to uncovering unexpected patterns that generate new questions and hypotheses. In exploratory studies, Tong advocates for abandoning statistical inferences and relying more heavily on descriptive statistics and graphical data exploration. Exploratory analysis, however, is not limited to descriptive methods; it involves a deeper, almost detective-like investigation of data to uncover unexpected or unusual patterns. The goal of exploratory analysis is to generate hypotheses, not to test them.

Thus, researchers find themselves caught between two “fires.” On one hand, they can choose exploratory research, but in this case, their statistical inferences—based on model selection—might be unreliable due to the optimism principle and overfitting. On the other hand, they can opt for confirmatory research, applying already established models derived from other datasets. In this case, the model can be validated across multiple datasets, allowing for more robust conclusions based on its parameters. However, this approach reduces flexibility and limits the researcher’s degrees of freedom.

I generally like this idea but it remains unclear how to combine confirmatory and exploratory approaches in real life where researchers are under the pressures of "publish or perish" and have limited resources. Researchers frequently face a choice: produce publishable work quickly or invest more time and resources in additional validation or explorative analysis. If you look at the proportion of exploratory vs. confirmatory studies in psychology, for example, you’ll see that confirmatory studies are more prevalent. Moreover, the confirmatory approach increases the likelihood of obtaining significant results, which in turn increases the probability of publication.

On our Telegram channel, after discussing another article on proper research methods, we conducted a lighthearted poll: “What do you do when your hypothesis testing yields statistically insignificant results?” Participants could select up to two answers. The results (28 participants total) were as follows:

• 46%: “I try to figure out what went wrong and tweak the methods to achieve significant results.”

• 39%: “I consult colleagues or supervisors for advice and reassurance.”

• 35%: “I get upset and look for other significant effects.”

• 28%: “I still try to publish the results without changes.”

• 28%: “I try to publish the results using alternative approaches.”

• 17%: “I collect more data to increase the sample size.”

• 10%: “I get upset and abandon the study.”

While the poll was informal, it’s telling that the most common reaction was frustration and a desire to adjust methods to achieve significance. This isn’t just about p-hacking; it reflects a deeply ingrained belief that non-significant results are somehow “wrong” and that many scientists aspire to confirm the hypothesis and their aspiration rather than investigate something new and unexpected.

In one paper, I encountered a possible way to implement the exploratory approach in ordinary research practice. This approach is proposed in the article "Harking, Sharking, and Tharking: Making the Case for Post Hoc Analysis of Scientific Data" (Hollenbeck & Wright, 2017). HARKing (Hypothesizing After Results Are Known) generally refers to unethical research practices. It usually manifests as follows: authors analyze data, find some significant relationships, and then justify these relationships with a theory or other results. Retrospectively, the authors explain the relationships they found and describe their results as if they had a hypothesis before obtaining results. The authors of the article refer to this type of HARKing as SHARKing (Secretly HARKing in the Introduction section), and it should certainly be discarded.

However, there is another type—THARKing (Transparently Hypothesizing After the Results Are Known). According to the authors, THARKing should become an integral part of any empirical study, and its results should be presented in the Discussion section. Following the authors’ suggestion, this section should obligatorily include a subsection titled Post Hoc Exploratory Analyses, where new hypotheses based on the obtained data will be presented. This section may include exploratory data analysis and the testing of hypotheses proposed after the results were obtained. But this should be done openly and clearly, as a supplement and extension of the main research. I like this approach, although it requires more work.

Another approach is described in one of my favorite books, Lab Girl by Hope Jahren (a geochemist and geobiologist). Although she is not a social scientist, I believe her approach is widely used by many social scientists. During regular working hours, you focus on your “official” research plan and grants to confirm well-established hypothesis and well-known theories to fulfill your commitments, while in your spare time, you work on new or interesting projects following an exploratory approach. That being said, I wouldn’t want to say that confirmatory studies cannot be interesting. However, the hypotheses for them should emerge from exploratory studies if you want to develop your own ideas and theories, rather than relying on others' ideas or ready-made models. Unfortunately, in real life, it’s often only "spare" time that remains for such exploratory research.

As you can see, any approach requires more work and time.

References

Hollenbeck, J. R., & Wright, P. M. (2017). Harking, sharking, and tharking: Making the case for post hoc analysis of scientific data. Journal of Management, 43(1), 5-18

Tong, C. (2019). Statistical inference enables bad science; statistical thinking enables good science.

Jahren, H. (2016). Lab Girl: A Memoir. Vintage.

Note: the picture was generated by AI