There's an interesting and relevant read in this month's issue of The Atlantic about a "data vigilante." Uri Simonsohn, a social psychologist, has been outing fraudulent research based on his re-inspection of the data in the published works.
I've pulled a few facts from the article that are going to sound obvious to most of you, but they're lessons worth remembering:
- If your data looks too good to be true, it probably is.
- Data that doesn't vary at all from group to group cannot be trusted.
- Omitting data to achieve significance should not be the norm.
- If the outcomes that you predicted ahead of time are eerily similar to your actual outcomes, your data might be biased.
Certainly, not all social science is bunk, but as Simonsohn says:
“When you have scientific evidence and you put that against your intuition, and you have so little trust in the scientific evidence that you side with your gut—something is broken.”
For more, read the full article in The Atlantic.
