Something that is very apparent if you spend much time hanging around on Twitter, or with policy people, clinicians or health managers, is that people from different backgrounds have very different understandings of what good evidence is, and what good research looks like.
I think the central problem here is that we expect all research to conform to the methods of the natural sciences. That means there is a tendency to favour experiments or experimental methods, and results which produce produce clear answers in relation to hypotheses. Where we can’t do experiments directly we try and mimic those methods as closely as possible.
There’s lots of good reasons for this. Scientific methodology has brought us a great deal from the laptop I’m typing this on, through to many of the medical treatments that can literally be life-saving. So it would seem it provides a high benchmark to which we must aspire.
However, I think that it’s a mistake to uncritically use natural science methods in the social sciences. The problem is that science presumes that we can close off the system we are investigating and control for individual variables. It also presumes that we treat the system as if its made up of objects rather than subjects, and that we can take large samples to check our results.
In social research none of these things is the case. Social research deals with open systems in which there are no clear boundaries between what we are investigating and their environment, where it is almost impossible to control for variables one at a time because of the complex interactions between them, where people can confound our results by behaving differently to the same stimulus on the same day, and where large samples may not be possible because we are dealing with whole social systems rather than discrete experimental settings.
The problem is that a great deal of social research still uses methods as if these differences don’t matter. So it concludes that particular interventions work, when it is extraordinarily unlikely that this is the case – they may work, but only in certain circumstances, for some people, some of the time. Yelling at me to do work might work today, but I may just ignore you tomorrow – in fact I might ignore you tomorrow because you yelled at me today, because the relationship between us is complex and not mechanical.
I think the way to get around this problem is try and be a bit more humble and a bit more theoretical. We need to be humble because nothing will work all the time. We need to try and find out when things have the best chance of working, and for whom, and to let those commissioning and asking for research, be they policymakers or managers, know that demanding magic solutions is going to lead to disappointment. We need to be theoretical in the sense of ‘there’s nothing so practical as a good theory’, coming up with pragmatic ideas that are sensitive to contexts, individuals and organisations, and to support those who have to try and make change happen. It isn’t just policymakers that need to demonstrate a more grown-up notion of what research can achieve, but also managers. Managers need to understand that they can generate a great deal of insight from social research, but there really are no magic answers we can provide in a series of bullet points.
We see a number of generic answers to making the NHS better, from the use of competition to making use of social enterprises or employee ownership, to greater local democracy and strong performance management. None of these things work universally, but many of them may work in some circumstances, some of the time. The trick is finding out where, when and for whom. But it is also crucial to be able to explain why and how. Anyone claims to have an answer to reforming the NHS but can’t explain exactly why their ideas should work, and exactly how, should be a cause of concern. And when I say ‘exactly how’, I don’t mean ‘by competition’ or ‘through incentives’ or any of the other vague answers academics are wont to give. I mean exactly how.
What all this adds up to is a different understanding of evidence for social research. The best we can do is to say that good evidence is that which explains what has the best chance of working, when, where, why and for whom. We also need to explain situations where it hasn’t worked, and to try and work out what we can learn from them. Both policymakers and managers need to be sceptical of researchers bearing theories and ideas that are more simple than this. We don’t live in simple worlds, and anyone suggesting that the answers we need to navigate through it are any simpler is very likely to be wrong.