One of the common refrains coming from a range of very sensible people concerned with NHS reform, as well as in other public policy areas, is that it should be based on evidence. ‘Evidence’ is often implicitly or explicitly counterposed with ideology. This all sounds very sensible – don’t we want our governments to base their policies on evidence, and wouldn’t that be better than if they were based on ideology, anecdote or opinion?
The problem here for me is that having evidence about something policy-related is not enough in itself when it comes to policymaking. In medicine it’s often possible to come up a degree of confidence about what the best treatments are (at least on average), but even then we need trained professions to help us work out how that research applies to us individually, or how it can be used to better organise clinical care. The trick is in interpreting the evidence to make a diagnosis of what might be best for the patient, who will themselves have opinions and biases they want incorporating into the decision.
The situation is more complicated when it comes to policy. Economists like large data sets from which they can derive models. Social policy academics often like surveys. Sociologists like to go and talk to people to find out what is going on. The findings from different kinds of research can often end up in open contradiction, with researchers claiming that the ‘evidence’ leads to entirely different conclusions.
What this suggests for me is that it’s not enough to ask for policy to be based on evidence. In the present NHS reforms, for example, the government do have evidence for at least some of the elements of their reforms – take the Cooper research on competition for example (its published form is here http://onlinelibrary.wiley.com/doi/10.1111/j.1468-0297.2011.02449.x/abstract, but you can find working papers if you google the title terms). Now this research, using what economists say is cutting edge methods, claims that the use of competition in the NHS has ‘saved lives’ and an obvious policy response is to say there should be more of it. However, researchers from other traditions (including me) have argued this work is deeply flawed (at http://www.lancet.com/journals/lancet/article/PIIS0140-6736(11)61553-5/fulltext). On the Lancet website you can also find a response from the original researchers, as well as from the critics.
What this kind of dispute points to, is that it isn’t possible simply to say that policy should be evidence-based – research evidence can often come to very different conclusions.
So what do we do? Throw up our hands and give up? No – I think the situation is retrievable.
A way out of this is to use evidence in a different way, and try and aim for what we might call ‘argument-based policy’. Argument-based policy would ask researchers and perhaps more importantly, policymakers, not what their evidence is (that comes later), but instead to go through the argument for what they advocate step by step, and only when we they have made that clear should they present the evidence they have for each of those steps.
This may sound like a tedious process for you, but could act as a clarifying device to find out what policymakers are simply asserting, and what they claim they can actually demonstrate using evidence. Equally, it requires researchers to be very clear not only about what they are claiming, but also how they went about making those claims, and on what data and assumptions they rest.
To give an example. In the present NHS reforms, the government are claiming that CCGs will liberate the NHS by reducing bureaucracy and putting more power in the hands of responsive local GPs. The argument seems to be:
- At present PCTs commission local services
- PCTs commissioning local services means that local people and local clinicians are not much involved
- Putting GPs (and others) in control of commissioning local services would engage them in commissioning local services
- Putting GPs (and others) in control of commissioning local services moves decisions closer to patients and make GPs more aware of budgets, so making decisions more efficient
- Therefore. Putting GPs (and others) in control of commissioning local services would liberate the NHS, make it less bureaucratic (so more efficient) and move care closer to patients.
Now expressed this way we can break down the claims. (1) was fairly uncontroversial at the beginning of the reform process (although we might question how much discretion PCTs often had about many of the services they commissioned). (2) is less obvious – there certainly were problems with PCT commissioning, but whether it was a big problem that clinicians were not involved requires research evidence that we could go through in terms of its claims in a similar way. The government hasn’t made much effort to show this though.
(3) is a kind of truism – if we get GPs to commission, then they will be more engaged in commissioning. However, this doesn’t mean that GPs are best placed to do the job, or that CCGs will actually represent their views any better than PCTs did (it depends which GPs end up commissioning care, who is advising them, how decisions will be made etc). All of this requires careful engagement with evidence, as well as further work explaining exactly how this is meant to work. In terms of (4) I co-published a piece a few years ago that reviewed evidence from GP fundholding, for example, that suggested that it led to a fall in patients satisfaction and only a one-off fall in referrals or prescribing (http://www.bmj.com/content/333/7579/1168.full). That would seem to suggest there the GP/patient relationship is rather more complex than (4) suggests, although there might be other evidence showing otherwise. We’d have to work through both and try and work through the assumptions and differences of the work to come to a judgement. In any case, it seems fair to say that the conclusion at (5) is being made prematurely.
What I’m asking for here is that we become more transparent about our claims, and that they are structured into an argument form so they can be tested and debated. Policy prescriptions are messy and complex, but can be made more straightforward if we can look at them as arguments, and then to look for the evidence that supports the premises and conclusions of those arguments in a more systematic manner. Then, at least, we can agree where we differ.