The Reproducibility and Robustness of Secondary Analyses in Educational Research: The Role of Publication Bias and Researcher Degrees of Freedom (META-REP-IQB)
Project management
Dr. Malte Jansen
Dr. Aleksander Kocaj
Research associate
Aishvarya Aravindan Rajagopal (from 01.04.2022)
Student assistant
Nikolas Weigt
Duration
February 01, 2022 - January 31, 2025
Funding
Deutsche Forschungsgemeinschaft (DFG)
Projekt description
The Priority Programme "META-REP: A Meta-scientific Programme to Analyse and Optimise Replicability in the Behavioural, Social, and Cognitive Sciences" (SPP 2317), funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation), will start in February 2022. The goal of this interdisciplinary project is to systematically investigate a) what replicability means and when a replication can be considered as successful, b) which factors influence replication success rates, and c) which measures can be used to increase replication success rates so that robust research results are obtained. The programme is coordinated by the LMU Munich. It is a meta-scientific research program that aims to foster networking and collaboration between different disciplines. A total of 15 projects from the fields of cognitive psychology, neuropsychology, methodological research, communication sciences, economics, and educational research will be funded for three years each.
The subproject at the IQB's Research Data Centre (FDZ, Forschungsdatenzentrum), led by Dr. Malte Jansen and Dr. Aleksander Kocaj, addresses questions concerning the reproducibility and robustness of research results based on secondary analyses in empirical educational research (Funding period: 02/2022-01/2025; total funding: 231.578€). The project is based on applications for secondary data analyses at the FDZ at IQB. In these applications, researchers describe their central questions, hypotheses and their planned analytic approach. We will systematically compare these applications with publications resulting from the applications. In addition, we aim to reproduce the results for selected publications. The project at the FDZ of the IQB has four goals. First, we examine whether data applications that result in significant or hypothesis-confirming results are also more likely to be submitted for publication and published (publication bias). Second, we plan to develop an index that quantifies the similarity between data use applications and resulting publications to provide evidence of selective reporting. Third, we will test whether the results of selected applications that resulted from data usage applications be reproduced using the information provided in the manuscripts. This will take advantage of the fact that the underlying data are available at the FDZ of the IQB. Based on this, we will investigate how robust selected research results are with regard to alternative plausible analytic approaches (multiverse analysis) in the fourth step. Overall, the project aims at identifying sources of heterogeneity that may influence the replicability and reproducibility of research findings in empirical educational research.
Homepage of the priority programme
https://www.psy.lmu.de/soz/meta-rep/index.html