http://query.nytimes.com/gst/fullpage.html?res=9C0CE7DB153EF93BA15752C0A966958260

###Science isn’t the bench. The lab bench is data collection, and yes its hard, but too many people believe that is Science!™.###

That’s true, but it is an important part of the endeavor, and one which many such programs favor greatly, so in no small part what you’re seeing is response to incentives.

###Science is properly controlling your independent/dependent variables, setting up a proper hypothesis, and then showing the results of your testing in relation to the hypothesis.###

I’m a little wary of the word “hypothesis” but that’s because I seriously dislike null hypothesis statistical testing…. In the broader sense of the word, I think we’re on the same page. But the word means different things to different people.
FWIW, http://dictionary.reference.com/browse/science

###Far too often, what I see these days is a collection of experimental results in a “experiment:result” format, and then some vague conclusions which are in no way tied to the data, along with grandiose comments of a qualitative nature.###

Yeah, well that’s nothing new.

You may want to take a look at a now about fifteen year old book Discovering by Robert Scott Root-Bernstein. One contention he has is that the growth of quality work is a logarithmic function of the number of scientists actually practicing. The dirty little secret is, of course, the fact that the vast majority of people working in science will not amount to all that much… they’re doing the very necessary “lab bench” stuff and churning through variations of other people’s hypotheses in the pursuit of the most normal of “normal science”. This is not unlike most other fields of human endeavor. Music or art of the past is often so good because the shite has long since been relegated to the dustbin of history.

Science isn’t the bench. The lab bench is data collection, and yes its hard, but too many people believe that is Science!™. Science is properly controlling your independent/dependent variables, setting up a proper hypothesis, and then showing the results of your testing in relation to the hypothesis.

Far too often, what I see these days is a collection of experimental results in a “experiment:result” format, and then some vague conclusions which are in no way tied to the data, along with grandious comments of a qualitative nature.

###You do realize that 90% of everything is crap, right?

Sure, but we specifically select and review the seminal papers in the fields of interest (computational brain theory). The Bio folks are particularly bad at math and science.###

Oh, I don’t want to leave anyone with the impression that I think a lot of people in areas like bio and psych are GOOD at math. (Science? Well I’m not sure what you mean there, as that’s an overly broad term. Many of them could chase the likes of us around a lab bench and out the door quite easily.)

There are major deficiencies in the mathematical preparation of students in a lot of areas these days, that’s for sure.

From my perspective, a lot of this has to do with the fact that many experimentalists have developed the notion that everything about statistics and modeling they need to know was finished by about 1970. This simply isn’t true. This is not helped by the fact that a lot of stat is approached bass-ackwards, but there is a deep myopia on the part of researchers as well, who consider learning about models to be a major chore they have to do.Here’s an extended quote on this issue from a book review I found in a journal. The author is Prof. Brian Everitt of Institute of Psychiatry, King’s College, London:

“So here we have two more examples of the apparently never ending stream of introductory statistical texts for psychologists and other behavioural and social scientists. They are both attractive books, and presumably the publishers know that lots of psychology students, particularly in the United States (and particularly on courses given by [the authors]), will by them. However, it would have been so refreshing (at least for a statistician) to have seen some attempt to make future psychologists aware that it is not only statistical software that has developed apace over the past three decades. Statistics itself has evolved in response to progress in computers, and there are now many new techniques that could be described or at the very least, mentioned, in even an introductory work. Graphical methods, computationally intensive procedures, exact methods, and generalized linear models are but a few of the possibilities. Because none of the “Statistics for Psychologists” texts I have come across include such topics, one begins to wonder whether their writers are actually aware of recent (and no so recent) advances in statistics? Perhaps they retain the believe that many research psychologists appear to share, namely that Fisher delivered analysis of variance on tablets of stone from Mount Rothampsted in the 1930s and that, subsequently, there has been nothing more to say?”

One can do a search-and-replace for psychologist and put in many other disciplines easily.

Sure, but we specifically select and review the seminal papers in the fields of interest (computational brain theory). The Bio folks are particularly bad at math and science.

###No, I just regularly attend some joint Bio-CS-ECE-Psych seminars where I get to regularly see the typical results they expound on, and their lack of statistical/mathematical knowledge.###

You do realize that 90% of everything is crap, right? I used to go to a seminar run by some complex systems physicists… guess what? Most of the presentations there—many based on published papers—were crap, too.

No, I just regularly attend some joint Bio-CS-ECE-Psych seminars where I get to regularly see the typical results they expound on, and their lack of statistical/mathematical knowledge.

###That’s a fairly broad statement. Not a fan of the “we can’t eat eggs this week” new reports only to get “we can eat eggs again” the following week?###

There’s been a lot of criticism of over-interpreted epidemiological studies with weak effects of late. For a summary:

http://www.nytimes.com/2007/09/16/magazine/16epidemiology-t.html?_r=1&oref=slogin

I recall seeing essentially the same criticisms coming out of the epi community in a Science magazine article from a few years back, though.

###To be fair to the media, I’ve seen these articles reviewing multiple studies fairly often, and I’ve always thought they were much more useful than the “study of the week” sort.###

Actually it’s worse. Proper aggregation of studies is very difficult—WAY more difficult than simply reading studies and “vote counting” effects. Most scientists don’t know how to do this since reasonable techniques are only at most a few decades old and, well, a lot of scientists (in all areas) are trained in out-of-date methods.

###Sadly, too many people still have a nice 19th century absolutist view of “Science!”###

Yeah, that too. It’s been said that scientists see science primarily as “practice” while laymen see it primarily as “grab-bag of facts”. It’s probably inevitable that this arises, but it’s unfortunate because the process view is much more correct (IMO) and also much less likely to be fooled.

Good science journalists (and they are around) know this and put the caveats in, but when ideology or kewl factor gets in the way, sometimes things get seriously distorted.

This gets to the heart of the issue for me. Individual studies should be scientific, but conclusions can only be scientific when they’re drawn from a sizable number of studies. One single study, no matter how well done, can only tell us so much.

More than anything else, I want the media to just stop. Just stop reporting individual studies. Instead, maybe gather all the studies on the topic over the past month or year (depending on the rate of production) and discuss the results once a month or year. Then we’d get some context, we’d see which studies contradict each other, we’d see where the areas of agreement are, and we’d see what areas the researchers are focusing on.

True, we can do all that by indexing a dozen different media articles of varying accuracy and sophistication, but why should we have do this work: isn’t this why we pay the press? In other words, the value of the “science journalist” is almost never going to be in helping us to understand a given study (they’d need a lot more skill to do that). But they could help us get a layman’s picture of the current state of research from time to time.

To be fair to the media, I’ve seen these articles reviewing multiple studies fairly often, and I’ve always thought they were much more useful than the “study of the week” sort.

Sadly, too many people still have a nice 19th century absolutist view of “Science!”