Consider a couple of different cases (all of these from my work):
1. a mechanical turk survey that can be repeated with many variations for a small amount of money
2. a sequence of studies with typically-developing children at a local preschool, easily repeated but with a month or two of RA effort
3. a longitudinal, school-based RCT in a foreign country, difficult and costly to repeat at best
In the case of (1) I don't think I'd recommend pre-registering each individual study. Once the study has been repeated, tweaked, and varied, a pre-registered version can be re-run as a confirmatory replication. But pre-registering each variation creates a system where the cost of running small-scale studies is increased, to the detriment of the productive process of "poking around" that can be very useful at the beginning of a study set.
For (2), personally I would again conduct several studies, progressing from pilots to investigations, with only the last one pre-registered, again for purposes of convincing readers and presenting a strong confirmatory test.
Finally, for (3), I would definitely advocate preregistration. The analyses can be complex and the temptation to poke around post-hoc and discover something unpredicted is just too great. I wish we had pre-registered our study of this type. Here the effort is minimal relative to the cost of the study.
Dale Barr tweeted earlier today "Tukey (1980) said we need both exploratory and confirmatory. #prereg gives us an honest way to signal the difference" - I think this is exactly the right approach.
1. a mechanical turk survey that can be repeated with many variations for a small amount of money
2. a sequence of studies with typically-developing children at a local preschool, easily repeated but with a month or two of RA effort
3. a longitudinal, school-based RCT in a foreign country, difficult and costly to repeat at best
In the case of (1) I don't think I'd recommend pre-registering each individual study. Once the study has been repeated, tweaked, and varied, a pre-registered version can be re-run as a confirmatory replication. But pre-registering each variation creates a system where the cost of running small-scale studies is increased, to the detriment of the productive process of "poking around" that can be very useful at the beginning of a study set.
For (2), personally I would again conduct several studies, progressing from pilots to investigations, with only the last one pre-registered, again for purposes of convincing readers and presenting a strong confirmatory test.
Finally, for (3), I would definitely advocate preregistration. The analyses can be complex and the temptation to poke around post-hoc and discover something unpredicted is just too great. I wish we had pre-registered our study of this type. Here the effort is minimal relative to the cost of the study.
Dale Barr tweeted earlier today "Tukey (1980) said we need both exploratory and confirmatory. #prereg gives us an honest way to signal the difference" - I think this is exactly the right approach.
(Post adapted from a message I sent to the OpenScience list.)
No comments:
Post a Comment