Irreal

Yesterday, I wrote about the Pacific Standard article on problems with reproducibility in the social sciences. Today I stumbled upon a second Pacific Standard article that discusses another insidious way that distortions can creep into studies in fields like psychology and anthropology: the fact that Americans are highly atypical when compared to most other peoples. That’s important because many studies are conducted using cheap and available American undergraduates as subjects. The problem is, they aren’t really like most people in the world and trying to draw broad conclusions based on them is going to produce erroneous or at least misleading conclusions

This is a long article and a bit out of the usual Irreal fare but it’s very interesting. It turns out that the familiar notion that “most people are the same under the skin” simply isn’t true. People in different cultures really can be substantially different in the way they experience the world. That sounds like it might be a trivial remark but consider the familiar Müller-Lyer illusion where a line with arrow tips is perceived as shorter than the same line with feathered tips. You might think that everyone is deceived by the illusion but it turns out that some cultures do see the lines as being the same length and that Americans, in fact, experience the illusion more strongly than others, even others from Western cultures.

Although from last year, it is, as I said, an interesting article and well worth your time. I found it fascinating. The research it describes is, apparently, shaking up psychology and several other disciplines.

Update: those → others

Update 2: Added link to the article

As you all know, I’m a big fan of reproducible research. Most people have an idealized conception of scientific research and that, by and large, it’s all about the pursuit of truth. Sadly that’s not always the case. The problem isn’t that all problematic results are faked; sometimes human nature is enough to corrupt the results.

Over at the Pacific Standard, Jerry Adler has an interesting article that is nominally about research in Psychology but has implications for other scientific research, including, astonishingly, Mathematics.

It’s astounding how often data is massaged to support a preconceived hypothesis. The article describes a study that used real data to “prove” an preposterous result. The study was up front about the silliness of the result and was making a point about how easily data can be manipulated.

Not all of these problems would be solved by reproducible research but having the data and the specific manipulations that it was subjected to available would go a long way towards ending this problem. Read the article to get an idea of the extent of the problem. After reading it, see if you don’t agree that journals should insist that submitted articles provide access to the type of data envisioned by reproducible research. Perhaps, then, the Reproducibility of important research in fields such as bio-medicine would do a bit better than one in ten.