Good Evidence, Bad Decisions
Evidence and decisions
This paper sets out to challenge the uncritical claim, presented as a mantra implying a virtual truism, that we should aim for ‘evidence-based decisions’. It is often claimed, for example, that most medicine is now ‘evidence-based’, with the implication that that is a ‘good thing’. On the face of it, it looks obvious that the best way of going about things is surely to base decisions, or judgements that lead to decisions, on the evidence. This will if accepted without nuance and unquestioningly, it is argued here, lead to poor decisions, even catastrophic decisions, or at the very least decisions being poorer than they need to be.
Evidence may not be appropriate
Of course, the mantra is not applied to all decisions, and it’s important to get that out of the way at the start, as otherwise it will be said that the claim addresses a straw man. If someone close to you – a partner or offspring – asks you if you love them, it’s going to go all wrong if you ask them to hang on a minute while you check or gather the evidence. So we set aside, for the sake of argument, those areas where no-one has ever seriously held that decisions should be evidence-based.
Assumptions about evidence
There is however a large class of judgements leading to decisions that are assumed to be best made on the basis of evidence. This implies that not to do so, perhaps as it was done in the past, was a flawed and dubious method that we have now advanced upon, putting those old dodgy ways behind us.
Large swathes of decisions taken by governments and other institutions are now said to be at their best if we apply the mantra that they should be evidence-based. Whether they always are or not is beside the point here. What is at issue is the normative claim that they should be, regardless of whether it is put into practice effectively, but rather, perhaps, appears as a rhetorical accolade for the decisions that are made. There are plenty of other areas where the claim that decisions should be evidence-based is trumpeted loudly. Medicine, as has been mentioned, is a prime example. But there are many other areas, such as environmental issues and strategies in education, where the claim is also made. It is not hard to add to the list. Indeed, so powerful is the unblinking acceptance of the mantra that the push is to extend the method as widely as is feasible, that the only constraints are perhaps ones of practicality, cost and time. So some might hope that perhaps one day all decisions – surely the best way of making judgements – will be evidence-based, barring those odd cases of emotional reaction – but you never know, perhaps even there one day it would be a good idea.
On the face of it, it just looks blindingly obvious that judgements should be evidence-based, and thus so should the decisions that follow from those judgements to act or not act in certain ways.
The usual claim is that good decisions, or judgements that lead to decisions, are best based on good evidence. How could it not be otherwise? There is as has been suggested much fashionable talk of ‘evidence-based decision making’. Of course, it is not supposed that basing what one thinks true, and decides to do on the basis of that idea, leads to infallibly good, and not poor, decisions. But the claim thrown up is that by doing this, going through that process, one did the best one could. Any other way would obviously be worse. If there is a shortfall between the evidence and the quality of the decision, then it is usually ascribed to either the insufficiency or quality of the evidence, or the inadequacy of the theory connecting the bare evidence to some inferred truth.
Countering the evidence claim
This essay shows how this can be a mistake, and that basing one’s decisions on good evidence can lead to poor outcomes in determining the truth, and poor, perhaps sometimes catastrophic, decisions that flow from the acceptance or assertion of such truths.
One way in which poor decisions lead to a poor grasp of the truth, and decisions following it, will be set aside, namely that the move from evidence to conclusion becomes clouded with normative, non-factual matters, that are not amenable to being determined by any piling up of the evidence.
This certainly is a way in which basing decisions on good evidence can nevertheless lead to poor decisions, especially if normative assumptions go unquestioned, and are indeed mistakenly thought to be determinable by the, say, empirical evidence alone. For example, one might determine that following some medical result a certain medical treatment might be beneficial – but here, once words like, ‘beneficial’ are introduced, normative considerations come into play which go beyond what any factual evidence may determine, namely, value judgements about what kind of life someone wants to live. Many of these get assumed, when they may be opened up reasonably or even unreasonably, to disagreement. In which case the decision that a certain treatment is beneficial in a normative sense cannot follow from the factual evidence alone, no matter how strongly it may appear to.
The objection here to truth and decisions being determined by evidence is that it leads one to make the decision based only on what one can gather evidence for, usually in a precise and measurable way. But that evidence, that information, might not be what would be the best thing to pay attention to in determining the truth and the decisions based on it. If one is determined to base truth, judgement, and decisions based on evidence, then one limits these to all to only that for which evidence may be gathered – leaving those features which fail to qualify as evidence neglected – and it leaves open the question of what evidence is, what counts as evidence, that should be used. Note here it is not just that decisions will only be made on the basis of putative good evidence, but that decisions will only be made about things for which such evidence can be gathered. Other things will be left out of even being ones about which proper decisions can be made.
The dangerous tendency then is that a tight or rigorous notion of evidence will put out of consideration as not-really-evidence factors that cannot be rigorously articulated, but which might be more relevant to determining well truth, judgement, and decisions. This might be someone’s extensive experience in a practice which cannot be tightly codified and thus meet the standard of good evidence. What works well when considering a lecturer or teacher may be hard to pin down with exactitude. Those features, perhaps a heuristic gestalt view, will be excluded by not being evidence because they cannot be expressed in a certain way. There is no articulable algorithm that a conclusion or set of truths and decisions could follow from.
It should be said that there is, in case it may be supposed, nothing mystical about this claim. It may be that there are certain things about the world that are beyond the determination of measurable factual evidence. But that need not be claimed for this argument to go through. All that needs to be claimed is that any attempt to gather evidence tending to be determined by pursuing mostly or only those features of the world for which the gathering of precise evidence can be accomplished will exclude features that cannot be tightly gathered in that way. Other features of the world may not be beyond the possibility of precise measurement or determination, but undoubtedly some features are harder than others. The tendency will be – if the mantra of ‘evidence-based decision making’ is accepted uncritically and becomes a habit of mind – that the evidence will shift the focus on decisions being based on truths that may be determined by evidence crowding out other considerations which are not part of the assembled evidence one has, or are indeed considered unsound for truth and decision determination, because they are not gatherable as evidence in a precise and determinable form. In short, evidence-based decision making leads one to make decisions, following the putative establishment of certain truths, ignoring or neglecting all those relevant features to determining truth and decisions that one happens not to have such evidence for. And thus the decision, the truth determination, will be poorer for based on evidence, not better, and not the best way to make whatever decision one is trying to make.
At this point it will be said that in this case, the response will be, if it is found that the truth is not as the evidence suggests, and thus whatever decision one has made is similarly impaired, that more evidence is need. We often hear the catchphrase ‘more work is needed’. But this is to overlook the central point here that there will always be factors relevant to determining truth and the decisions one makes on the basis of such truth that are beyond the evidence one has, moreover some of these features are hard to determine in such a way that they can ever be counted as usable evidence. In addition, in going out and gathering evidence one circumscribes before starting what one thinks is relevant evidence to making the decision one is seeking to make. This can blind one to other features of the whole situation that might make the decision a better one. To base the decision on features beyond what one has evidence for, features for which evidence is gatherable, it will said is to lead one to base the decision not on the evidence, which the current milieu regards as a culpable mistake. If one skews decisions so that they are based only on what counts as measurable evidence, the decision will be the poorer, as relevant features that do not count as measurable evidence will not be included in the decision making process, thus the decision, one’s judgement, will be worse than it could be and by no means the best it could be.
There is nothing relativistic in its implication about the above argument. Only the view that adherence to evidence in making decision can lead one to make poor decisions by one not considering features one has not given the honorific term evidence to. The power of the evidence outweighs matters one might otherwise consider, but which strictly speaking do not count as evidence as what they are has not been pinned down in a way that is transparent in its factual precision.
Final thoughts
There is a further a bolder claim one may make: that over consideration of gatherable features of a situation that may count as ‘evidence’ leads to the atrophy and perhaps moribundity of those faculties and habits of mind that lead one to understand a situation in a heuristic gestalt way. That is using a practical sense often based on features that have not been precisely determined or measured, but which one understands well nevertheless, and which may lead one to a better assessment of the truth and thus a better decision.
◊ ◊ ◊
Dr John Shand is a Visiting Fellow in Philosophy at the Open University. He studied philosophy at the University of Manchester and King’s College, University of Cambridge. He has taught at Cambridge, Manchester and the Open University. The author of numerous articles, reviews, and edited books, his own books include, Arguing Well (London: Routledge, 2000) and Philosophy and Philosophers: An Introduction to Western Philosophy, 2nd edition (London: Routledge, 2014).
Contact information:
- Dr John Shand, The Open University, Walton Hall, Milton Keynes, Buckinghamshire, MK7 6AA, United Kingdom.
- https://open.academia.edu/JohnShand
- http://fass.open.ac.uk/philosophy/people
John Shand on Daily Philosophy:
