Psychology v. Science: it’s a no-brainer

You know, there are times when I almost manage to convince myself that psychologists aren’t a bunch of too-stupid-to-make-a-go-of-it-in-a-real-science idiots, and that they might actually have some useful insights into the management of the symptoms of mental illness.  And then they go and publish utter twaddle like this

Some psychologists at the University of Making Stuff Up Toronto engaged in a ridiculous orgy of timewasting study to look at the effects of loneliness and social ostracisation.  In the first of two “experiments” they divided some volunteers into two groups.  One group was asked to remember a time when they had felt lonely and unpopular.  The other group were asked to remember a time when they had felt accepted and welcomed.  Both groups were then asked to estimate the temperature of the room they had been sitting in.  On average the ones feeling lonely estimated the temperature of the room to be lower than those who had remembered feeling accepted.  The scientists psychologists concluded from this that feeling lonely or isolated makes you feel physically cold.

Ok, so hands up: what’s wrong with this “experiment”?

Where to start?  Well, for one thing, there’s no control group.  I would suggest in the context of this “experiment” that a control group could have been created by asking a third group of people to remember an event that was neutral in terms of its position on the welcomed – isolated spectrum.  The scientists psychologists didn’t, it would appear, do this.  One possible reason they didn’t bother with the control group is that, given the fundamentally flawed design of the “experiment”, it wouldn’t have made any difference.

The subjective experience of temperature varies from individual to individual.  Put me and my brother into a room that’s heated to, let’s say, 17° C and my brother will go round obsessively looking for draughts and trying to find a nice warm cardigan while I’ll be trying to open the windows and complaining about how hot it is.  The “experiment” as carried out had no mechanism for establishing whether or not there was a disproportionate number of subjectively hot or subjectively cold people in either group.  Like the lack of a control group, there was actually a way round this problem.

The solution would have been to introduce a second stage to the “experiment”.  In this second stage each group of volunteers could have been asked to come back into the room, which would have been maintained at a constant temperature.  However, the volunteers would be told that the temperature had been altered, although they wouldn’t be told if it had been supposedly increased or decreased.  They would then be asked to recollect the opposite experience to that which they had recalled in the first stage of the “experiment”, and would again be asked to estimate the temperature of the room.  By tracking the difference in estimates of temperature made by the same individuals in different emotional circumstances it would have been possible to identify what effect, if any, social exclusion has on perception of temperature.

If the majority of volunteers estimated a lower temperature when recollecting a negative event, the experimenters would have had some grounds for concluding that remembering feeling isolated makes you feel cold.  If the majority of volunteers estimated a higher temperature, they’d have had some grounds for concluding the opposite.  But, unlike with the one stage experiment, there would also have been a third possible outcome – that there would be no clear majority either way.

The scientists psychologists who designed the one stage experiment appear to have created a falsifiable hypothesis – their experiment might have shown that, contrary to expectations, people felt subjectively warmer when they remembered feeling isolated.  But there is actually a deeper hypothesis underlying the experiment which they have not attempted to falsify – that recollecting a negative social experience will have an effect on the subjective experience of temperature.  The two stage experiment I have proposed here would, by allowing a potential outcome in which the data showed no clear majority either way, have enabled the deeper hypothesis to be proven or disproven.

The scientists psychologists did recognise one potential flaw in the first “experiment”.  By asking their volunteers to remember a positive or negative situation, they had introduced an uncontrollable variable into the experiment.  The second “experiment” tried to avoid the problem by attempting to directly re-create negative and positive social experiences.  The experimenters did this by designing a computer simulation of a ball throwing game.  Some volunteers had the ball thrown to them lots of times (this was supposed to make them feel included), while others didn’t (this was supposed to make them feel excluded).  All the volunteers were then asked to rate the desirability of hot and cold food and drink.  The volunteers who had been excluded showed a distinct preference for hot food and drink when compared to their more popular counterparts.

Again there would seem to be noticeable problems with this experiment.  There appears to be no mechanism for taking account of the general preferences of the volunteers.  The hot items on the menu were coffee and soup – volunteers who didn’t like either of those would have gone for the Coke and crackers no matter how excluded they felt.  The reverse is obviously also true.

Again there would seem to be a simple solution.  The volunteers could have been asked to rate the desirability of the various items both before and after they had their positive or negative experiences.  By tracking any changes in preference it would have been possible to prove not only whether or not negative social experiences stimulate a desire for hot food and drink, but also the deeper underlying hypothesis – that snack selection is based on social experience.  Again the experimenters appear to have created a falsifiable hypothesis – excluded volunteers might have, contrary to expectations, gone for the cold food – but have left a more fundamental hypothesis unexamined.

There’s something that bugs me really quite profoundly about this.  I am by no means a scientist.  My highest ‘achievement’ in the sciences was a narrow fail at A-level Biology, but even I can point out the obvious structural flaws in these experiments.  The University of Toronto paper is due to be published in the journal Psychological Science (now there’s an oxymoron for you…).  Psychological science is apparently not a formal research journal, but is nonetheless peer reviewed.  So here’s the thing.  If a paper so obviously flawed that even an unqualified amateur can point out its problems passes an academic peer review process, how can psychology even begin to claim status as a science?

Things actually get worse with the conclusions that are drawn from the data.  According to the press release, the experimenters are not satisfied with simply establishing that an experience of social exclusion makes volunteers select hot food and drink:

Their preference for warm food and drinks presumably resulted from physically feeling cold as a result of being excluded.

Even if I were to accept that the selection of hot snacks is entirely the result of experiencing social isolation (which I don’t), this is an entirely groundless presumption.  Hot foods are widely regarded as comforting.  This may result from the fact that those in need of comfort feel physically cold, but it may also result from the fact that individuals have learned to associate warmth with comfort, irrespective of their actual perception of temperature.  Nothing in the “experiment” carried out by the University of Toronto enables a conclusion to be reached on this issue.

Things get even worse when other scientists psychologists start commenting on the study:

Dr Lesley Prince, a lecturer in psychology at Birmingham University, said: “This is very interesting, and shows there are physiological correlates to emotions.”

Now, this is precisely not what the study has shown.  Even disregarding the methodological flaws in the experiments, and the dodgy conclusions they lead to, all the study has “shown” is that people who are socially isolated have a subjective sense that their environment is colder, and that this makes them feel like having a nice hot drink.  A ‘physiological correlation’ would mean that the objectively measured body temperature of socially excluded people was lower than that of socially included people.  An experiment of this nature wouldn’t have asked volunteers if they felt cold, it would have used a thermometer and proved they actually were colder.

And there we have it.  This whole thing is a ridiculous farrago of poorly designed experiments giving rise to unreliable data which is then misrepresented in order to suggest that it supports at best dubious, and at worst frankly ludicrous, hypotheses.  This is nothing more or less than a grotesque parody of the scientific method, and it’s been the same with every psychology “experiment” I’ve ever seen described. 

And yet psychologists feel qualified to call into question the efficacy of psychiatric medications which have been proven to benefit patients in rigorous double-blind clinical trials.  In their place they want to promote their own unproven and unprovable theories as having greater objective value in the treatment of mental illness.

So.  Is it just me, or are psychologists really taking the piss?

Advertisements
This entry was posted in Media commentary, Psychiatry, Psychology, Stuff I've read, Stupidity. Bookmark the permalink.

14 Responses to Psychology v. Science: it’s a no-brainer

  1. I think you are giving Ben Goldacre a run for his money there! Let’s just say that not every journal has a high ‘impact rating’ and low impact rating journals will be a lot less fussy about what they publish…

  2. Cellar_Door says:

    Hehe. You would make a very good scientist :0) Having done a psychology degree, I can confirm that the majority of psychologists appear to be taking the piss. I found some of it interesting and valid (usually the older stuff, oddly) and rather a lot of it complete twaddle…

  3. Press releases should be trusted about as far as BBC News science reporting. ie. Not very much at all.

    The thing is, we don’t know what these psychologists did, whether they saw the same problems you do and whether they took them into account in their experimental design. I haven’t been able to find a copy of the paper online (not that I was looking very hard).

    I suspect that BBC News got the quote from Lesley Prince like this:

    “Hello?”

    “Mr Prince? It’s BBC News. Just wanted to ask for your opinion on a study. Apparently some blokes in Canada have done an experiment that shows feeling lonely makes you cold.”

    “Well, it’s very cold in Canada.”

    “Quite so, Mr. Prince. So what do you reckon? It’s interesting isn’t it? People feel lonely and get cold. Makes you think.”

    “Well, uh, yes, I guess it’s an example of emotions having physiological correlates – a bit like intense fear making you shake and that kind of-”

    “So, Mr. Prince, could you make people feel better by warming them up do you think?”

    “If they were feeling really cold, I suppose being warmer would make them feel better.”

    “Thanks Mr. Prince.”

  4. Mariah says:

    Why is it that I only ever find flawed research?

    Like you, my initial assumption as to why someone would rank warm food as desirable is the comfort factor. Of course I’d probably be one of those confusing people in the survey- as coffee is disgusting. What would it mean if someone picked Coke and soup?

    Something else that bothered me was the fact that crackers aren’t (usually) cold. They’re room-tempature, whereas soda has (usually) been chilled. So if the psycologists made everyone pick just one food or beverage, would the Coke-pickers supposedly feel more included than those who chose crackers?

  5. cb says:

    Interesting stuff. I share your scepticism but doubt I could word it as eloquently. There seems to be so much that is wrong going on here. However, in defence of (some) psychologists, I honestly couldn’t speak highly enough about those that I work alongside consistently but these kinds of games in university don’t really help much with the general attitude towards the ‘science’.

  6. aethelreadtheunread says:

    Wow, thanks for all the comments – i’d kind of assumed this was going to be one of those “Aethelread’s mubling into his beard again, best to leave him to it” posts, so i’m thrilled. :o)

    DeeDee Ramona – Psychological Science apparently has an impact rating of 4.251, which, the publishers say, places it amongst the top 10 psychological journals in the world ( http://www.blackwellpublishing.com/journal.asp?ref=0956-7976&site=1 ). Of course, without knowing how many psychology journals there actually are in the world that doesn’t really tell us a lot.

    Cellar_Door – I’d make a good scientist… are you calling me a geek?! You’d be right… ;o) I’m sure quite a lot of psychology is genuinely valid and interesting, but i’m afraid research like this is like a red rag to a bull to me.

    Experimental Chimp – i also looked for the original article online – it looks as though it hasn’t actually been published yet. I take on board your point about press releases. But, while the researchers may have been aware of the problems in the research, and may have written about it at length, and that would certainly exonerate them from the charge of doing bad science, i’m not sure it answers the point about peer review.

    Part of the aim of a peer-review process (as i understand it, and, again, i’m only an amateur so i could have this all wrong) is not only to establish that the researchers have followed appropriate methods, but that the research has led to interesting or significant conclusions. With all the flaws in its methodology, this study can’t actually draw any robust conclusions whatsoever. Even if the published paper is entirely up-front about this, it’s still surprising that the peer reviewers would regard such an inconclusive study as worthy of publication.

    I like your little play script for the quote from Dr Prince – i suspect it’s very accurate. :o) But any academic worth their salt should know better than to answer questions from the media on a paper they haven’t read. My guess is s/he was excited by the thought of talking to the BBC… ;o)

    Mariah – That’s a good point about people choosing a mixture of hot and “cold” food. I wonder if they were actually offered ‘soup and coffee’ or ‘coke and crackers’ with no option of mixing and matching? If they were, then the fact that soup and coffee is a more unusual combination than coke and crackers might be significant – i.e. social rejection makes you choose weird mixtures of food, not feel cold.

    Thinking about it, there’s also the question of which order they were offered the various options in – people who felt uncertain about their position in the group might be more inclined to take the first thing offered to them rather than hold out for what they actually wanted. Again that would mean they were influenced by something other than the temperature of the food.

    cb – you are, i’m sure, right about the psychologists you work with. I actually do think psychology can be very helpful in learning to manage and cope with unpleasant mental symptoms – i use a variety of psychological techniques to try and get through panic attacks, for example. I’m still uneasy about its status as a science, though.

  7. Most of psychology seems to involve eye-tracking experiments.

    That’s a scarily high impact rating if the conduct of the research is that poor…

  8. Alex says:

    I’m not so sure it’s just psychologists – scientists of all fields can be equally prone to this, I think.

  9. Jo says:

    I will go in a different direction – do you feel like a hot or cold drink, right now?

  10. E says:

    Psychologists aren’t the only ones to do daft or poorly conceived research. These guys are “real” scientists. I particularly like the award for the alarm clock that runs off and hides. I want one of those.

    http://news.bbc.co.uk/1/hi/sci/tech/4318888.stm

  11. aethelreadtheunread says:

    Thanks for the extra comments – all gratefully received.

    E – sorry your comments didn’t appear straight away, wordpress seemed to throw a wobbler because of the links. It did the same thing to one of Alex’s comments a while back – it’s obviously feeling sensitive at the moment, poor thing…

  12. bippidee says:

    if the room is empty, is it still cold?

  13. bippidee: the answer to that is, “the correct question is: if the room is empty, do we need to leave the heating on?” :)

Comments are closed.