It’s not enough just to ask for evidence-based policy

One of the common refrains coming from a range of very sensible people concerned with NHS reform, as well as in other public policy areas, is that it should be based on evidence. ‘Evidence’ is often implicitly or explicitly counterposed with ideology. This all sounds very sensible – don’t we want our governments to base their policies on evidence, and wouldn’t that be better than if they were based on ideology, anecdote or opinion?

Mmmm.

The problem here for me is that having evidence about something policy-related is not enough in itself when it comes to policymaking. In medicine it’s often possible to come up a degree of confidence about what the best treatments are (at least on average), but even then we need trained professions to help us work out how that research applies to us individually, or how it can be used to better organise clinical care. The trick is in interpreting the evidence to make a diagnosis of what might be best for the patient, who will themselves have opinions and biases they want incorporating into the decision.

The situation is more complicated when it comes to policy. Economists like large data sets from which they can derive models. Social policy academics often like surveys. Sociologists like to go and talk to people to find out what is going on. The findings from different kinds of research can often end up in open contradiction, with researchers claiming that the ‘evidence’ leads to entirely different conclusions.

What this suggests for me is that it’s not enough to ask for policy to be based on evidence. In the present NHS reforms, for example, the government do have evidence for at least some of the elements of their reforms – take the Cooper research on competition for example (its published form is here http://onlinelibrary.wiley.com/doi/10.1111/j.1468-0297.2011.02449.x/abstract, but you can find working papers if you google the title terms). Now this research, using what economists say is cutting edge methods, claims that the use of competition in the NHS has ‘saved lives’ and an obvious policy response is to say there should be more of it. However, researchers from other traditions (including me) have argued this work is deeply flawed (at http://www.lancet.com/journals/lancet/article/PIIS0140-6736(11)61553-5/fulltext). On the Lancet website you can also find a response from the original researchers, as well as from the critics.

What this kind of dispute points to, is that it isn’t possible simply to say that policy should be evidence-based – research evidence can often come to very different conclusions.

So what do we do? Throw up our hands and give up? No – I think the situation is retrievable.

A way out of this is to use evidence in a different way, and try and aim for what we might call ‘argument-based policy’. Argument-based policy would ask researchers and perhaps more importantly, policymakers, not what their evidence is (that comes later), but instead to go through the argument for what they advocate step by step, and only when we they have made that clear should they present the evidence they have for each of those steps.

This may sound like a tedious process for you, but could act as a clarifying device to find out what policymakers are simply asserting, and what they claim they can actually demonstrate using evidence. Equally, it requires researchers to be very clear not only about what they are claiming, but also how they went about making those claims, and on what data and assumptions they rest.

To give an example. In the present NHS reforms, the government are claiming that CCGs will liberate the NHS by reducing bureaucracy and putting more power in the hands of responsive local GPs. The argument seems to be:

  1. At present PCTs commission local services
  2. PCTs commissioning local services means that local people and local clinicians are not much involved
  3. Putting GPs (and others) in control of commissioning local services would engage them in commissioning local services
  4. Putting GPs (and others) in control of commissioning local services moves decisions closer to patients and make GPs more aware of budgets, so making decisions more efficient
  5. Therefore. Putting GPs (and others) in control of commissioning local services would liberate the NHS, make it less bureaucratic (so more efficient) and move care closer to patients.

Now expressed this way we can break down the claims. (1) was fairly uncontroversial at the beginning of the reform process (although we might question how much discretion PCTs often had about many of the services they commissioned). (2) is less obvious – there certainly were problems with PCT commissioning, but whether it was a big problem that clinicians were not involved requires research evidence that we could go through in terms of its claims in a similar way. The government hasn’t made much effort to show this though.

(3) is a kind of truism – if we get GPs to commission, then they will be more engaged in commissioning. However, this doesn’t mean that GPs are best placed to do the job, or that CCGs will actually represent their views any better than PCTs did (it depends which GPs end up commissioning care, who is advising them, how decisions will be made etc). All of this requires careful engagement with evidence, as well as further work explaining exactly how this is meant to work. In terms of (4) I co-published a piece a few years ago that reviewed evidence from GP fundholding, for example, that suggested that it led to a fall in patients satisfaction and only a one-off fall in referrals or prescribing (http://www.bmj.com/content/333/7579/1168.full). That would seem to suggest there the GP/patient relationship is rather more complex than (4) suggests, although there might be other evidence showing otherwise. We’d have to work through both and try and work through the assumptions and differences of the work to come to a judgement. In any case, it seems fair to say that the conclusion at (5) is being made prematurely.

What I’m asking for here is that we become more transparent about our claims, and that they are structured into an argument form so they can be tested and debated. Policy prescriptions are messy and complex, but can be made more straightforward if we can look at them as arguments, and then to look for the evidence that supports the premises and conclusions of those arguments in a more systematic manner. Then, at least, we can agree where we differ.

Advertisements

5 Responses to “It’s not enough just to ask for evidence-based policy”

  1. Evidence Matters (@EvidenceMatters) Says:

    The process that you outline seems appropriate and remarkably similar to the ‘challenge and account’ exchanges that are favoured in some interviews conducted under legal caution.

    Too often ‘evidence’ is presented as a set of facts when those facts are dependent on assumptions that are not readily apparent. Interpretations of data are presented as evidence without making the workings to arrive at those data available, nor the models that underpin them.

    only when we they have made that clear should they present the evidence they have for each of those steps.

    Agreed that it’s sometimes essential to have on overview of someone’s arguments just to have a feel for different views of the problem space and possible contributions/solutions before beginning to look any deeper.

    It seems facile to mention this but I wonder if some sources might benefit from guidance as to appropriate standards of evidence. I’ve been surprised at some very expensive, very glossy reports that rely upon the most insubstantial of evidence. One only needs to look at the too frequent corrections by FactCheck or Straight Statistics to understand that some very senior people repose a surprising amount of confidence in the accuracy of newspaper accounts that coincide with their current emphases or ideology. Even after being corrected numerous times, it seems that some of those people continue to cite this evidence (your recent Frankfurt post comes to mind here).

    if we can look at them as arguments, and then to look for the evidence that supports the premises and conclusions of those arguments in a more systematic manner

    I’m interpreting this as a compressed way of saying that evidence that supports or undermines the argument is scrutinised for its quality and appropriateness (in itself a complex issue). I’m sufficiently ignorant that I have to ask if social policy has something akin to the systematised approach that is adopted in Cochrane Reviews when gathering and reviewing data.

    • Ian Greener Says:

      Thanks for your reply. I’d like to argue that there is no single best way to systematise research. It depends on the problem you are trying to address. Cochrane’s produce a particular understanding of clinical interventions, but of course have their detractors, some of which I respect a great deal. I think whether that style of review works depends on the question you are asking. When you start looking at interventions which are more social than science (as in reorganisations) it is almost impossible to put in controls, and context becomes hugely important. In those circumstances I think we need a theory-driven evaluation model like the one suggested by Ray Pawson, and which seems to me to have huge potential to build matches between theory and evidence towards understanding policy interventions.

      So for me, quality and appropriateness tend to depend on the question being asked, and the particular policy context being researched. There are likely to be no single ‘right’ answers, so we probably need a range of different approaches. And as Pawson himself has argued, even poor research can provide interesting results sometimes – we’d be poorer for excluding it from reviews by having selection criteria which is about alleged quality of methods rather than the importance or potential contribution of the findings.

      • Evidence Matters (@EvidenceMatters) Says:

        I would agree that Cochrane Reviews aren’t the only way to systematise research: nonetheless, I’m typically charmed and slightly horrified by the effort that they put into a rigorous literature search for relevant work and wondered if social policy had anything similar.

        The search space for relevant literature seems to be vast if one is adopting a Pawson approach for some questions: this isn’t an argument against it at all, more a recognition of just how very vigorous some of those argument/challenge discussions might be if the relevance of the evidence is to be challenged.

        “poor research can provide interesting results sometimes”: this is true and yet much depends on how those results are interpreted and what further edifices are built on them. It seems, too often, as if poor methodology leads to an unrealistic evaluation of results and yet those results are used to justified continued exploration that perpetuates a poor design etc.

        A timely publication might be From Efficacy to Effectiveness in the Face of Uncertainty which addresses comparative-effectiveness research and mapping results from ‘ideal’ groups to individuals that might have portmanteau co-morbidities (that bears some analogy to applying evidence gleaned from one group and applying it to another in social policy?)?

        An urban geography chum has pointed me towards pp 700-2 as a useful discussion of research and methodology so I probably need to take a thorough look at that: Urban geography, relevance and resistance to “policy turn” (pdf).

      • Ian Greener Says:

        Thanks for the AMA and urban geography pieces – I enjoyed them both. There are attempts to apply Cochrane-style criteria to social science more generally, but for me they tend to end up reproducing the ideas of systematic reviews in non-appropriate ways by being dominated by quantitative data. I’m happy to send you an example if you email me.

        I think the way out of endless reviews in the Pawson approach is to treat research as either reinforcing an emerging view, challenging it, or amending it. Once you have mapped the field you can begin to work out where the disputes lie, whether they are methodological or empirical of theoretical (or something else), but also avoid having to read and re-read research that ends up coming to much the same argument (even though you might note this helps make other word credible).

  2. beyond evidence-based policy « theputneydebates Says:

    […] beyond evidence-based policy 16 February, 2012 “What this kind of dispute points to, is that it isn’t possible simply to say that policy should be evidence-based – research evidence can often come to very different conclusions.” (Ian Greener) […]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


%d bloggers like this: