Last week there was a lot of discussion of Zack Cooper (et al)’s paper asserting that competition saves lives. The paper has been cited by the government in terms of their reforms, and is prominent on the web page of the Economic and Social Research Council. It is not, as some commentators have suggested, just more evidence – it’s take-up has meant that it is now far more than that. This is because it has become central to the debate around whether we should continue down the road of using competitive forces to reform the NHS.
Cooper has now responded to the criticisms made of his work. I’ve included the full response below, but it seems to be based on four assertions:
1. Comparing before and after 2006 is sensible because that’s when competition proper started in the NHS.
2. Competition since 2006 has become so strong in the NHS that it affects AMI, which is not even subject to competitive pressures.
3. There is no more competition in urban areas than there is in rural areas, as competition is not about how urban an area is.
4. The work is backup up by research from Propper (and others), which comes to similar findings
Now I’m trying to be sensible here, and have included Cooper’s response in full below (making this a bit of a long one), along with a link to the Guardian web-site – one of the places where it was published. Equally, I don’t want to be appear to be an ideologue – if it is the case that using competitive forces really does save lives, that would be great. It would provide a neat solution to NHS reform, and we could get on with driving improvements. But I really don’t buy it – healthcare is messy, and complicated, and presuming any one-size-fits-all model seems incredibly unlikely to me.
The key question then, is do Cooper’s assertions make sense? I
The first one, that competition started proper in 2006, is based on the implementation (as I understand it) of patient choice policies at that time. The thing is, I can’t find any consensus among NHS clinicians or managers that they started doing things differently at that point, or even later, as a result of patient choice. So either the effect Cooper has found is so profound that it works on people without them knowing it, or the 2006 date is not a valid one.
The second assertion is that competitive pressures have affected AMI (the clinical area they look at), even though AMI is not itself subject to competitive forces. The argument is that competition in NHS settings has driven improvements in areas which aren’t even subject to competition. Does this seem likely to you? Economists talk a great deal about incentives (including Cooper in the paper). What Cooper seems to be saying is that, even in areas where there are no incentives, competition drives improvements because of a knock-on effect of some kind. Again, I’d have to ask how credible this seems to you – it’s an extended version of the effect above where competition not only works on people without them knowing it, but also when the competition isn’t actually applying to them. Pretty remarkable stuff.
The third argument is that competition isn’t related to how urban provision is. That’s a big surprise. I characterised Cooper’s argument last week as being a measure of urbanity, not of competition. In the paper itself there is a great deal of time and effort discussion how the proxy to measure markets was conducted, but the end result is intriguing. Are Cooper and his collaborators really saying that the degree of urbanity is not related to how much competition a patient or GP might experience in their choice? That would suggest either that the measure is faulty, or GP referrals have some extremely unlikely patterns.
The fourth argument is that Cooper’s work is backed up by other research that comes to similar findings. There is some research that supports Cooper’s view, but there is also some that does not. As Nick May’s very balanced summary in the BMJ suggests (http://bit.ly/ojVaio ), there is work that doesn’t support Cooper’s view which sadly he doesn’t explore in any depth at all.
So I’m sorry, but I still can’t see any basis for accepting this research as offering us anything we can base policy on. If you are going to make the case that an expensive and complex reform to a health system is going to work, then you need extremely strong evidence. I don’t think this fits that bill.
Zack Cooper’s response – included on a comment at:
It has been interesting to watch the work we’ve produced in the Economic Journal be hyped up in some quarters, played down in others and just be generally mischaracterized inside and outside of academia. I thought Iâd simply attach the paper and encourage folks to read it. They may be surprised that my co-authors and I have already addressed, in the paper, almost all of the criticisms that have been raised. The paper happens to have a fairly clear-cut implication (fixed price hospital competition improves quality), which, for better or worse, dovetails with the current toxic political debate in England. Much of the response to the paper has been swept up with frustration of politics in the UK. The fact that the paper is fairly technical has not made this any easier. Long term, it would be a shame to ignore the evidence from this paper, dismiss the findings as marginal etc when the results we find are non-trivial (a 7% relative reduction in AMI over three years) and the results have been confirmed by further work by Marty Gaynor and Carol Propper.
Per our work, the paper uses a difference-in-difference style estimator. Difference-in-difference is as close as you can get in a policy setting to a randomized control trial. Is it perfect? No, but the timing of the reforms in England were a real asset to our estimation strategy. The general diff-in-diff idea is to compare two groups, one exposed to a policy, another not, before and after the policy was introduced.
In our paper, we compare patient outcomes in hospital markets with varying levels of hospital concentration before and after patient choice and provider competition was introduced in England by the Labour government in 2006. Crucially, contrary to what Evan Harris and others have suggested, this is not a ‘London market thing’. I would encourage him to examine page 249 of the paper, where we write, Table 7 presents robustness checks to illustrate that the effect we identify in our interactions between our post-2006 time trend and our measure of market structure are not simply spurious associations with urban density’. The paper uses approximately 20 measures of market structure. Further, we go out of our way to show that this is not an issue related to urban/rural splits or population density. The effort we put in to show this has somehow gotten lost in the analysis of our paper.
First, we measure hospital market structure in the paper almost 20 different ways, and then illustrate that our results are robust across each measure. Further, we show that our preferred measure of market structure is only correlated by 3% with population density, suggesting that this isn’t an urban thing. Then, we test whether urban areas in England did better after the reforms; they did not. What’s more, we test whether London did better after the 2006 reforms; it did not. Finally, we run a placebo test to determine whether school market structure was associated with improved performance; it was not. This collection of robustness checks is pretty strong evidence that it was indeed the degree of hospital competition in England that was associated with improvements in hospital performance.
There has also been criticism for our strategy of using AMI as a quality indicator. This was a deliberate choice, not an oversight. In the paper, we used an emergency outcome as our quality indicator because there’s a real concern about endogeneity if you’re examining the relationship between elective competition and elective outcomes. Our rationale for using AMI mortality (which we discuss in detail of pg 237 of the paper) is that AMI mortality is correlated with overall hospital performance (which we demonstrate empirically). However, it is unrelated to the market for elective care, since AMI patients don’t really have choice. The explanation then is that competition leads to improvements in hospital performance, which we capture using AMI mortality. AMI mortality is appealing because there’s a substantial mortality rate and good care is generally associated with good outcomes.
There’s also been concern about whether there has been any meaningful choices being made. First, Anna Dixon’s work at the King’s Fund illustrates that about fifty percent of patients in the NHS are aware of their ability to choose. Second, the work that just came out from the Cooperation and Competition Panel finds that patients are readily making choices and going non-local (when PCTs oblige). Finally, Carol Propper is finishing up work with Stephen Seiler and Marty Gaynor which has found that after the reforms, there were changes in patient flows and that sicker patients tended to be more sensitive to hospital quality. Given that during this period in the NHS, hospitals were under substantial pressure to maintain surpluses, these observable swings in market share are certainly enough to create incentives (when the IO literature suggests that a 5-10% change in market share is generally enough to do so).
Finally, important to note that this paper is co-authored with Ali McGuire, Stephen Gibbons and Simon Jones. They all also put in a great deal of work, and should be acknowledged too.