Evidence, argument and NHS reform

Something that has been a theme in the present NHS reforms is arguments about whether or not evidence supports them. I’m not going to focus here on the claims the government has made in relation to how the NHS is a clinical laggard – you can find refutations about most of the points raised in relation to that at http://abetternhs.wordpress.com/2011/06/29/competition/ and http://justanotherbleedingblog.blogspot.com/2011/04/nhs-reform-from-liberating-to.html.

What I do want to talk about here is research that claims to provide evidence that competition is already up and working in the NHS, and why this is so disputed. I’m doing this in the spirit of trying to understand difference (as well, of course, to explain why I think I’m right!). And bear in mind that this research is being used by the government to justify the NHS reforms, and so needs to be scrutinised.

There are two (at least) key papers that claim to support competition, one by Gaynor and Propper (google ‘Death by market power’ and one by Cooper et al (‘Does hospital competition save lives’). I’ve written at length about the second paper in this blog, but it is important to understand not only what this research shows or doesn’t show, but also how it was done. It is important to understand the differences between their view (that the research shows competition is showing an effect) and mine (and others, that we can’t see how this can be true) in order to assess which is more likely to be the case.

This research depends on you accepting at least two points that I find difficult to square. First, on there being an ‘on-date’ for competition of 1st January 2006, when choice-based reforms were formally introduced. If you don’t buy that competition kicked into gear then, there it is hard to sustain that argument that it is what is making the difference in health outcomes that they authors claim to have found.

I’m a bit baffled by this claim because we have the NHS Collaboration and Competition panel laying into commissioners because they aren’t referring to new providers and working with market incentives, and a Civitas report from the end of 2010 reported that ‘There  is  a  strong  case  to  be  made  that  such  policies  have  been ineffective  because  to  date  there  has  not  been  a  functioning ‘market’  in  the  NHS’ – (Refusing Treatment available at  http://www.civitas.org.uk/nhs/.)

Equally, it seems odd to me that NHS managers and clinicians were, by this account, all not trying as hard as they could before 2006, then suddenly, when the choice policy kicked in, got their games together to secure clinical improvements. And it’s not as if they were doing this to get richer, which is the standard motivation you get in economics textbooks.

It also seems to me that these papers confuse choice policies with competition. Choice policies only mean competition in an environment of scarce demand – where a patient choosing a private provider means that public providers lose out. But in the areas I know best in the NHS, there is frankly enough demand to go around for all the providers in the locality. It could be that these aren’t representative – I really don’t know. But neither, it seems, does any one else.

So how do we reconcile the research findings that competition saves lives and the evidence and argument about that we’ve seen very little in the way of competition?

This brings me to the second point I want to raise in relation to this research. What troubles me is that the research doesn’t seem interested in showing that competition is actually happening, especially given the work above that suggests it isn’t. The econometric models (which take quite a bit of getting your head around) infer that competition is causing outcome improvements from a range of variables they hold to be representative even if they can’t explain exactly how exactly how competition itself is working. It seems good enough that they are using the best statistical tests (difference in difference) which indicate the effect is down to competition. But there is a difference between statistical and substantive significance – the first is a numeric relationship, and the second is based on an explanation that the numbers attest to.

The problem with this is that the actual workings of the NHS become a ‘black box’ where competition is assumed and inferred rather than being shown. No clear mechanism is specified and shown to be working.  I think, in contrast, it’s really important to be clear about how a mechanism works. If it is the case that it is possible to save lives through competition, that would be the most important finding in the history of NHS organisation. But the reticence of researchers in engaging with this really is a problem. We are asked to take their findings on trust, without them saying how they are happening.

The trouble is no-one has yet explained how, if competition is working, patients or GPs are making the choices that send patients to the best providers, or why hospitals are competing if choices can’t be made (as hospitals would then be wasting their time by doing so). We know that GPs are most likely to refer on the basis of experience or informal knowledge rather than engaging with performance data, about which they are often suspicious. Patients, in turn, are most likely to ask their GP where they should go for secondary treatment. In the National Patient Survey only 10% of patients reported having used the NHS choices website – it would be interesting to know how many of that 10% actually found it any use in making their decision. There is a lack of up-to-date, comparative outcome data on which choices that could improve outcomes could actually be made, and even if it were, GPs would have to routinely use it, which they don’t. There’s lots of work that explores these topics, but a good starting point is http://www.bmj.com/content/342/bmj.d408.full?sid=4090c6d4-7d48-4762-b644-064cb218e5ac

Instead we are asked to assume that improvements in coronary care, the main example in the papers, are down to the effects of competition being so pervasive that they even affect AMI when that, in itself, isn’t subject to competition or choice. Now again, if that were true, that would be a huge breakthrough, but no-one can explain to me (or anyone else) exactly how this is happening without vague appeals to market mechanisms.

The NHS is not a black box. We need to understand how changes are occurring, especially where it is not clear exactly how the present structures can be made to improve outcomes. If researchers are going to claim that the competition reforms they advocate are the answer to the NHS’s problems, they need to explain exactly how they to work – the onus is on them to make their case.


2 Responses to “Evidence, argument and NHS reform”

  1. Carl May Says:

    Given that competition is coming – whether anyone likes it or not (and I don’t) – I’m amazed that no-one is thinking about competition in the one domain where it would really make a difference to patients. That is, primary care. Instead, the assumption is that GPs will offer (competitive) choices to patients rather than be chosen (competitively) themselves. Through PCTs and their successors they will be able to sustain completely consumer-proof local monopolies.

    • Ian Greener Says:

      Thanks Carl. I seem to remember this coming up in a BMJ piece ‘Government has “systematically underestimated” cultural shift needed to implement reforms’ – which made a similar point – 80% of all encounters take place in primary care, yet we are intent on disrupting that area to reform the other 20%. I think the problem is that, if it’s difficult to choose secondary care, how do we get patients to choose primary care where they won’t have any support for their choices at all? We urgently need good comparative websites (compare the care?) where we can plug in what we want and get some straightforward answers about which providers might be best for us.
      Also alarming that more than 1 in 5 GPs have a financial interest in a private provider in their local area, suggesting that local monopolies with conflicts of interest are a serious problem.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: