Quality science from quality measurement: The role of measurement type with respect to replication and effect size magnitude in psychological research
View/ Open
Author
Kornbrot, Diana
Wiseman, Richard
Georgiou, George
Attention
2299/19826
Abstract
The quality of psychological studies is currently a major concern. The Many Labs Project (MLP) and the Open-Science-Collaboration (OSC) have collected key data on replicability and statistical effect sizes. We build on this work by investigating the role played by three measurement types: ratings, proportions and unbounded (measures without conceptual upper limits, e.g. time). Both replicability and effect sizes are dependent on the amount of variability due to extraneous factors. We predicted that the role of such extraneous factors might depend on measurement type, and would be greatest for ratings, intermediate for proportions and least for unbounded. Our results support this conjecture. OSC replication rates for unbounded, 43% and proportion 40% combined are reliably higher than those for ratings at 20% (effect size, w = .20). MLP replication rates for the original studies are: proportion = .74, ratings = .40 (effect size w = .33). Original effect sizes (Cohen’s d) are highest for: unbounded OSC cognitive = 1.45, OSC social = .90); next for proportions (OSC cognitive = 1.01, OSC social = .84, MLP = .82); and lowest for ratings (OSC social = .64, MLP = .31). These findings are of key importance to scientific methodology and design, even if the reasons for their occurrence are still at the level of conjecture.
Publication date
2018-02-12Published in
PLoS ONEPublished version
https://doi.org/10.1371/journal.pone.0192808Other links
http://hdl.handle.net/2299/19826Metadata
Show full item recordRelated items
Showing items related by title, author, creator and subject.
-
The Experience of Being a Trainee Clinical Psychologist From a Black and Minority Ethnic Group: A Qualitative Study
Shah, Snehal (2010-12-22)Aim: The existing evidence-base indicates that the experience of being a Black and Minority Ethnic (BME) trainee clinical psychologist is under-researched. The aim of the current study was to capture the broader training ... -
Decolonising Clinical Psychology Training in Singapore: Trainee and Recently Qualified Psychologist Views about Diversifying Therapeutic Models
Wong, Caleb H. Q. (2022-11-30)Despite literature suggesting that Western psychology and models might not be as culturally appropriate in other cultures (e.g. Henrich et al., 2010; Llewelyn & Shimoyama, 2012), clinical psychology courses in Singapore ... -
The Mother and The Child Clinical Psychologist: a Discursive Analysis of Professional Conversations
Puchalska, Sylvia (2012-02-01)Aim Motherhood is placed under a lens by society: mothers are expected to fit within narrowly defined characteristics which dictate who mothers should be and how they should act. Although there are numbers of articles ...