I’m a strong supporter of marriage equality. I believe that same-sex couples have an equal right to get married if they want to, even though I think it’s unlikely I’ll ever want to exercise that right myself (though that’s what I would say, of course, given my current and enduring relationship status: Profoundly Single…). Here in the UK, I back the campaign to open up civil partnerships to opposite-sex couples and open up marriage to same-sex couples. In the US, I back the campaign to overturn DOMA, and the various state-by-state campaigns to extend marriage and block homophobic amendments to the US Constitution. None of this means I’m prepared to blindly endorse every piece of evidence that seems to support the campaign for marriage equality.
As a good case in point, take this story, which reports on a study into the public health benefits of legalising same-sex marriage:
The number of visits by gay men to health clinics dropped significantly after same-sex unions were allowed in the state Massachusetts.
This was regardless of whether the men were in a stable relationship […]
Researchers from Columbia University Mailman School of Public Health surveyed the demand for medical and mental health care from 1,211 gay men registered with a particular health clinic in the 12 months prior to the change, and the 12 months afterwards.
They found a 13% drop in healthcare visits after the law was enacted.
There was a reduction in blood pressure problems, depression and “adjustment disorders”, which the authors claimed could be the result of reduced stress.
Lesbian women were not included in the study as there were insufficient numbers to give a statistically meaningful result.
The abstract of the article – accepted for publication by the American Journal of Public Health – is here. The article itself is behind a paywall, so in the process of researching this post I’ve made use of the BBC website story, the abstract, and the press release hosted on the Columbia University Mailman School of Public Health website. As always, I should make it very clear that I’m not a scientist (my highest scientific qualification is a D at A level biology), and I have no kind of professional experience or expertise in this or any related field. What follows are merely the thoughts of a layman – an interested, tolerably well-informed layman, hopefully, but a layman nonetheless.
I guess I should start by drawing attention to what seems, in my inexpert view, to be the biggest problem with this study – the way participants were selected. The press release describes the approach the researchers took:
In the 12 months following the 2003 legalization of same-sex marriage in Massachusetts, gay and bisexual men had a significant decrease in medical care visits, mental healthcare visits, and mental healthcare costs, compared with the 12 months before the law change. […] For the study, researchers surveyed 1,211 patients from a large, community-based health clinic in Massachusetts that focuses on serving sexual minorities. Examining the clinic’s billing records in the wake of the approval of Massachusetts’ same-sex marriage law […]
If I have understood correctly, the researchers looked at the same group of 1,211 patients both before and after the law was changed. There are really two related problems I see with this.
Firstly, wouldn’t we expect that – irrespective of external factors like law changes – a patient who had to attend the clinic frequently in the first 12-month period would need, on average, to attend less often in the second, as their health problems were either resolved or stabilised? This is, after all, the purpose of all medical treatment: to improve the health of the patient. Secondly, I’m concerned by the fact that new patients approaching the clinic for treatment in the second year were excluded from the study. If patients who have already been receiving treatment at the clinic for at least a year are included but patients coming forward to seek treatment for the first time are excluded, wouldn’t we expect to see a reduction in the consultation rates anyway? Not because of any change in the health of the overall population, but just because this particular research methodology would include those patients whose health you’d expect, on average, to improve over the second half of the period, but would exclude a proportion of those whose health worsened (i.e., those who needed treatment for the first time). It seems to me that, even if the overall health of the gay male population in Massachusetts was worsening, a study designed this way wouldn’t necessarily have found evidence of it.
I understand that it was necessary for the researchers to compare like with like, and that simply looking at the total number of consultations in each period might have produced a distorted result if, for example, the clinic recruited new patients from, or lost existing patients to, other healthcare providers. What I find harder to understand is why the researchers didn’t calculate a standard measure of consultation rates (number of consultations per hundred patients, perhaps) in both periods, and then compare them. It strikes me that this would have given a far more accurate picture of the trend in consultations. Given that the description above suggests the researchers harvested their data from the (presumably anonymised) billing records of the clinic rather than from individual medical files, such an approach would have seemed as practicable as the approach they took.
Let me move on by drawing attention to something else in the research that immediately struck me as sounding too good to be true: the finding that the same level of benefit was experienced by married and unmarried people. Or, as Hatzenbuehler et al put it in their abstract:
These effects were not modified by partnership status, indicating that the health effect of same-sex marriage laws was the same for partnered and nonpartnered men.
Without access to the full paper I can’t know the extent to which the authors considered and rejected alternative explanations for their findings, and what reasons, if any, they gave for doing so, but on its own that bald assertion doesn’t strike me as especially convincing. I’d be particularly interested to know how the authors established that the effects they observed were indeed the result of the change in marriage laws, rather than some other, coincidental, factor. On the face of it, after all, the finding that the benefit was the same for partnered and non-partnered men would seem to undermine the assumption that this specific legal change was the influencing factor.*
In this regard, it’s a shame that the researchers don’t seem – so far as I can tell from the press release and abstract – to have taken steps that might have helped to confirm that the observed effects were indeed the result of the amendment to marriage laws. They don’t appear, for example, to have conducted the same research at a similar clinic in another state where the law wasn’t changed to see if they could detect the same variation in consultation results there or, for that matter, to have compared fluctuations in consultation rates at the same clinic over a period not affected by the law change. This last seems particularly unfortunate since their method for gathering data – examining billing records – would seem to have made researching an equivalent period before the law change relatively straightforward.
There are one or two other things that strike me about this research. For example, I note that the researchers report a decrease in consultations for both physical and mental health problems, but when it comes to healthcare costs they only report a result for mental health consultations. Without access to the paper I can’t know what explanation, if any, the researchers give for this apparent omission, but I’m concerned that it might be because they were unable to demonstrate a statistically significant reduction in healthcare costs for physiological health problems. If this is the case (and, of course, it may not be) then this would be troubling, because a negative result here would serve to undermine part of the assertion made by the researchers in their abstract conclusion:
Policies that confer protections to same-sex couples may be effective in reducing health care use and costs among sexual minority men.**
I also note that the abstract results quote a Cohen’s d statistic of 0.17 for the medical care visits result. I’ll be honest, reading this abstract was the first time I had ever encountered a Cohen’s d statistic, but it would seem to be a standard measure of effect size. I’m not able to put this into proper context – I have no idea whether Cohen’s d=0.17 would be considered large or small by someone who knows what they’re talking about – but even I can see that it’s quite small relative to the same statistic for the other results in this study (0.35 for mental health care visits and 0.41 for mental health care costs). I think (though I may well be wrong – please tell me in the comments if I am) this suggests that, compared to the changes in mental health consultations and costs, the reduction in the number of consultations for physical health problems was the least substantial change observed by the researchers. It’s perhaps unfortunate, therefore, that both the press release and BBC story draw attention to the impact on physiological ill-health, when it seems that this study mainly points to positive effects in the area of mental health.
In the University of Columbia press release, Dr Hatzenbuehler is quoted as saying,
This research makes important contributions to a growing body of evidence on the social, economic and health benefits of marriage equality
I realise he and the university will have reasons for talking up the significance of his research, and I’m glad of anyone joining the campaign for marriage equality, but I’m really not sure it does.
I think the decision not to look at standardised consultation rates before and after the change in the law has curtailed the ability of the study to accurately reflect trends in health consultations among the wider gay male population in Massachusetts. I think the decision to focus exclusively on a group of individuals who had received treatment in the year before legalisation runs the risk of distorting the results, since patients who have been receiving treatment could be expected to see an improvement in their health, even if the factor being studied by the researchers had made no difference at all. And I think that, so far as I can tell from the materials I’ve had access to, the researchers have not, in fact, demonstrated that the variation in consultation rates was the result of the legalisation of same-sex marriage. It may have been, but it may also have been the result of another factor, or the simple consequence of focussing the research on patients who had been receiving treatment for some time at the expense of new patients falling sick for the first time. So far as I can tell, without a comparison with a control group of some description – people living in a state where same-sex marriage wasn’t legalised, perhaps, or patients at the same clinic in a time period not affected by the law change – it’s simply not possible to say.
* – A spokesman for the Terrence Higgins Trust, speaking to the BBC, suggested a link between laws that treat gay men as second class citizens (such as a ban on same-sex marriage) and feelings of low self esteem, and this is a plausible mechanism by which a law change directly affecting only a subset of gay men might produce a positive effect amongst all of them. My major reservation about this as an explanation of the data found in this research is that it doesn’t seem entirely plausible that a single legal measure could accomplish such a significant change in self-esteem. In 2003, when the change occurred, gay men in Massachusetts remained subject to legal discrimination in a whole range of other areas. It would seem odd if non-partnered men, who did not benefit directly from this change, experienced exactly the same boost in self-esteem as partnered men, even though many other discriminatory practices, which did directly affect their lives, remained in place.
** – I dislike the use of the term ‘sexual minority men’ in this context, since it’s unnecessarily imprecise. Men who are turned on by, let’s say, the scent of women’s shoes are in a ‘sexual minority’, but it’s unlikely Dr Hatzenbuehler and his colleagues intended to include them in their discussion. An established term – men who have sex with men, typically abbreviated to MSM – was available to describe the patients who attended the clinic without having to make reference to subjective sexual identities.