Today’s article is brought to you by “researchers” at NYU and UCLA. In a recent study, designed to troll the heck out of the entire American populace, neuro-science researchers “proved” that the brains of liberals and conservatives are wired differently. Among their list of conclusions were:
- “conservatives tend to be more structured and persistent in their judgments”
- conservatives tend to ignore information
- “liberals are more open to new experiences”
- liberals can be expected to accept new scientific and social ideals faster
- the results “provided an elegant demonstration that individual differences on a conservative-liberal dimension are strongly related to brain activity.”
Now I don’t know about the rest of you, but I find this fascinating! These conclusions are revolutionary, and best of all have been made using Science ™! One imagines that there are several deep experiments correlating data from MRI observations with behavioral experiments, and well designed procedures meant to isolate variables and ensure the proper statistical power to draw these conclusions. One would imagine these things… and be horribly and utterly wrong.
No, it turns out the only experiment conducted was asking subjects if they were liberal or conservative, presenting them with a series of “M”’s and “W”’s on the screen, and asking them to hit a button when they saw an “M”. They then recorded correctness, brain activity from a broad region of the brain and from this data drew their specious conclusions.
This is one of my biggest beef’s with modern research, especially from Biologists and Psychologists. People love to draw broad conclusions from results that are utterly unable to support them. Due to media buzz, increasing popular influence, and just plain old bad science, this has become an increasing problem. Whats worse is that it detracts from the credibility of scientists everywhere, and public understanding of what science is. While the research conducted by the scientists at NYU and UCLA was doubtless important, like many researchers in the field, their results are utterly bogus. There is no way they can draw the broad behavioral conclusions that they draw based on people pressing buttons when seeing a given letter.
Not that this stops them from getting the headlines and fame, however. Not until the press hires competent science writers who can see these sorts of sham conclusions for what they are.
-Angry Midwesterner
September 27, 2007 at 12:56 pm
Without defending the experiment in question or its conclusions, that particular experimental task isn’t as crazy as it might seem. One thing that often bugs lay audiences is the extent to which experimental stimuli lack “face validity,” i.e., seem to be related to the topic being studied. Unfortunately, face valid measurement often “gives away” the point of the study.
In another example, job applicants are often really bothered by the fact that they are given a test like the Wonderlic rather than a job-specific test. The reason is that the Wonderlic is way, way cheaper than any job-specific test and gives you something like 90% of the predictive power, so there is usually no point in making the specific one.
The other point I will make is about a point AM made: “Not that this stops them from getting the headlines and fame, however.” Unfortunately in today’s research climate of “How big is the grant you have today?” these kinds of publicity stunts are common, simply to raise name recognition. In science, just like anywhere else, there’s no such thing as *bad* press.
September 27, 2007 at 5:37 pm
Whenever you read a media report about a scientific study (especially when you don’t like the results, as seems to be the case here), read the original study and then compare to how it was reported.
Typically, the media report will try and make the big claims (i.e., headlines) to attract readers, whereas the study may offer some speculations, but the actual conclusions will be not that exciting.
Researchers are constantly ticked off at how magazines/newspapers twist their words to make it sound ’sexy’. This is a big problem…the popular media want people like you to read it and react…that gives them more hype and more readers.
If you don’t want to or can’t read the original study, don’t jump to conclusions that its bad science (although sometimes it certainly is, I won’t deny that).
September 27, 2007 at 5:53 pm
Trust me, I checked the paper. The authors are the ones who draw the specious conclusions, not the media. If you’d like to check the paper out, here is the Bibliography:
DM Amodio, JT Jost, SL Master, CM Yee, “Neurocognitive correlates of liberalism and conservatism” Nat Neurosci, 2007
September 28, 2007 at 12:41 am
I wouldn’t be rushing to judgement either way . . . apparently Messrs. Amodio, Jost, Master, and Yee used a grand total of 26 liberals and 7 conservatives (perhaps they were using them to change light bulbs?). Their political bent was determined by self proclamation, and all were college students (not a bad thing, but not representative of either the general liberal or the general conservative populations). No reaction times were given as was pointed out at the following interesting site: http://neurocritic.blogspot.com/2007/09/liberals-are-neurotic-and-conservatives.html
September 28, 2007 at 9:49 am
“Not until the press hires competent science writers who can see these sorts of sham conclusions for what they are.”
Sorry, the budget doesn’t allow for science experts. We spent all of this year’s budget hiring Faith and Religion reporters because, as you know, none of us have ever picked up the bible. I guess it’s a pretty new book eh? Need to look into that.
Yes, the media have no clue how to report on how science works. Yes, sometimes studies are done to publish them rather than advance science. Yes, sociological studies should be examined very closely as to methodology.
“This is one of my biggest beef’s with modern research, especially from Biologists and Psychologists.”
That’s a fairly broad statement. Not a fan of the “we can’t eat eggs this week” new reports only to get “we can eat eggs again” the following week? That’s not science. Those are individual studies that, as always, need further study. That’s called job security. You know that…
September 28, 2007 at 12:28 pm
[...] various categories of other WordPress blogs. Under the “science” category was this post: How to Draw Specious Conclusions From Research. How could I not click on [...]
September 28, 2007 at 2:24 pm
“That’s not science. Those are individual studies that, as always, need further study. That’s called job security. You know that…
”
This gets to the heart of the issue for me. Individual studies should be scientific, but conclusions can only be scientific when they’re drawn from a sizable number of studies. One single study, no matter how well done, can only tell us so much.
More than anything else, I want the media to just stop. Just stop reporting individual studies. Instead, maybe gather all the studies on the topic over the past month or year (depending on the rate of production) and discuss the results once a month or year. Then we’d get some context, we’d see which studies contradict each other, we’d see where the areas of agreement are, and we’d see what areas the researchers are focusing on.
True, we can do all that by indexing a dozen different media articles of varying accuracy and sophistication, but why should we have do this work: isn’t this why we pay the press? In other words, the value of the “science journalist” is almost never going to be in helping us to understand a given study (they’d need a lot more skill to do that). But they could help us get a layman’s picture of the current state of research from time to time.
To be fair to the media, I’ve seen these articles reviewing multiple studies fairly often, and I’ve always thought they were much more useful than the “study of the week” sort.
Sadly, too many people still have a nice 19th century absolutist view of “Science!”
September 28, 2007 at 3:56 pm
AOC wrote:
###To be fair to the media, I’ve seen these articles reviewing multiple studies fairly often, and I’ve always thought they were much more useful than the “study of the week” sort.###
Actually it’s worse. Proper aggregation of studies is very difficult—WAY more difficult than simply reading studies and “vote counting” effects. Most scientists don’t know how to do this since reasonable techniques are only at most a few decades old and, well, a lot of scientists (in all areas) are trained in out-of-date methods.
###Sadly, too many people still have a nice 19th century absolutist view of “Science!”###
Yeah, that too. It’s been said that scientists see science primarily as “practice” while laymen see it primarily as “grab-bag of facts”. It’s probably inevitable that this arises, but it’s unfortunate because the process view is much more correct (IMO) and also much less likely to be fooled.
Good science journalists (and they are around) know this and put the caveats in, but when ideology or kewl factor gets in the way, sometimes things get seriously distorted.
September 28, 2007 at 5:16 pm
Hedwig wrote:
###That’s a fairly broad statement. Not a fan of the “we can’t eat eggs this week” new reports only to get “we can eat eggs again” the following week?###
There’s been a lot of criticism of over-interpreted epidemiological studies with weak effects of late. For a summary:
http://www.nytimes.com/2007/09/16/magazine/16epidemiology-t.html?_r=1&oref=slogin
I recall seeing essentially the same criticisms coming out of the epi community in a Science magazine article from a few years back, though.
September 29, 2007 at 11:35 am
That’s a fairly broad statement. Not a fan of the “we can’t eat eggs this week” new reports only to get “we can eat eggs again” the following week?
No, I just regularly attend some joint Bio-CS-ECE-Psych seminars where I get to regularly see the typical results they expound on, and their lack of statistical/mathematical knowledge.
September 30, 2007 at 2:02 pm
Am wrote:
###No, I just regularly attend some joint Bio-CS-ECE-Psych seminars where I get to regularly see the typical results they expound on, and their lack of statistical/mathematical knowledge.###
You do realize that 90% of everything is crap, right? I used to go to a seminar run by some complex systems physicists… guess what? Most of the presentations there—many based on published papers—were crap, too.
September 30, 2007 at 2:18 pm
You do realize that 90% of everything is crap, right?
Sure, but we specifically select and review the seminal papers in the fields of interest (computational brain theory). The Bio folks are particularly bad at math and science.
October 1, 2007 at 11:58 am
AM wrote:
###You do realize that 90% of everything is crap, right?
Sure, but we specifically select and review the seminal papers in the fields of interest (computational brain theory). The Bio folks are particularly bad at math and science.###
Oh, I don’t want to leave anyone with the impression that I think a lot of people in areas like bio and psych are GOOD at math. (Science? Well I’m not sure what you mean there, as that’s an overly broad term. Many of them could chase the likes of us around a lab bench and out the door quite easily.)
There are major deficiencies in the mathematical preparation of students in a lot of areas these days, that’s for sure.
From my perspective, a lot of this has to do with the fact that many experimentalists have developed the notion that everything about statistics and modeling they need to know was finished by about 1970. This simply isn’t true. This is not helped by the fact that a lot of stat is approached bass-ackwards, but there is a deep myopia on the part of researchers as well, who consider learning about models to be a major chore they have to do.Here’s an extended quote on this issue from a book review I found in a journal. The author is Prof. Brian Everitt of Institute of Psychiatry, King’s College, London:
“So here we have two more examples of the apparently never ending stream of introductory statistical texts for psychologists and other behavioural and social scientists. They are both attractive books, and presumably the publishers know that lots of psychology students, particularly in the United States (and particularly on courses given by [the authors]), will by them. However, it would have been so refreshing (at least for a statistician) to have seen some attempt to make future psychologists aware that it is not only statistical software that has developed apace over the past three decades. Statistics itself has evolved in response to progress in computers, and there are now many new techniques that could be described or at the very least, mentioned, in even an introductory work. Graphical methods, computationally intensive procedures, exact methods, and generalized linear models are but a few of the possibilities. Because none of the “Statistics for Psychologists” texts I have come across include such topics, one begins to wonder whether their writers are actually aware of recent (and no so recent) advances in statistics? Perhaps they retain the believe that many research psychologists appear to share, namely that Fisher delivered analysis of variance on tablets of stone from Mount Rothampsted in the 1930s and that, subsequently, there has been nothing more to say?”
One can do a search-and-replace for psychologist and put in many other disciplines easily.
October 1, 2007 at 4:13 pm
Science? Well I’m not sure what you mean there, as that’s an overly broad term. Many of them could chase the likes of us around a lab bench and out the door quite easily.
Science isn’t the bench. The lab bench is data collection, and yes its hard, but too many people believe that is Science!™. Science is properly controlling your independent/dependent variables, setting up a proper hypothesis, and then showing the results of your testing in relation to the hypothesis.
Far too often, what I see these days is a collection of experimental results in a “experiment:result” format, and then some vague conclusions which are in no way tied to the data, along with grandious comments of a qualitative nature.
October 1, 2007 at 4:33 pm
AM Wrote:
###Science isn’t the bench. The lab bench is data collection, and yes its hard, but too many people believe that is Science!™.###
That’s true, but it is an important part of the endeavor, and one which many such programs favor greatly, so in no small part what you’re seeing is response to incentives.
###Science is properly controlling your independent/dependent variables, setting up a proper hypothesis, and then showing the results of your testing in relation to the hypothesis.###
I’m a little wary of the word “hypothesis” but that’s because I seriously dislike null hypothesis statistical testing…. In the broader sense of the word, I think we’re on the same page. But the word means different things to different people.
FWIW, http://dictionary.reference.com/browse/science
###Far too often, what I see these days is a collection of experimental results in a “experiment:result” format, and then some vague conclusions which are in no way tied to the data, along with grandiose comments of a qualitative nature.###
Yeah, well that’s nothing new.
You may want to take a look at a now about fifteen year old book Discovering by Robert Scott Root-Bernstein. One contention he has is that the growth of quality work is a logarithmic function of the number of scientists actually practicing. The dirty little secret is, of course, the fact that the vast majority of people working in science will not amount to all that much… they’re doing the very necessary “lab bench” stuff and churning through variations of other people’s hypotheses in the pursuit of the most normal of “normal science”. This is not unlike most other fields of human endeavor. Music or art of the past is often so good because the shite has long since been relegated to the dustbin of history.
October 1, 2007 at 4:34 pm
Here’s an old review of RSR-B’s book:
http://query.nytimes.com/gst/fullpage.html?res=9C0CE7DB153EF93BA15752C0A966958260