How can doctors get better at predicting outcomes?
Looking into the future is a science. It’s an uncertain one but it’s not just a matter of random guesswork. Some years ago the CIA (Central Intelligence Agency in the US) decided to approach the question of forecasting in a new way and they asked how they could get better predictions. The challenge was taken up by Philip Tetlock, an academic, who started the “Good Judgement Project” (GJP). Tetlock’s hunch was that some people were significantly better at predicting the future than others, and that the rest of us might be able to learn from them. By asking questions to which a definite answer would emerge within a fairly short timeframe the GJP was able to actually measure how good different forecasters are and also how the job is best done – such as individually or in groups. The sorts of questions they would ask would be, “who will win the the US Presidential election?” or “will the UK vote for Brexit?”
The results were surprising and illuminating. The first shock was that there were quite a few members of the public who did consistently better than the “experts”. This was despite the experts having been picked by hand and having access to private sources of information, whereas the public had only the use of widely available information. There were clear patterns in how the successful forecasters worked: they tended to be good at listening to a range of views from their team (about a dozen people) yet they were also independent-minded and willing to go against the crowd. These winners were also good at doing internet searching and were dedicated to the task and they put the time into researching the questions often over several weeks, but they also didn’t fit the usual preconceptions – instead of coming to a decision and sticking to it they were more willing than the average to change their minds when new facts came to light.
So Philip Tetlock is sure that we can learn from these techniques and become better forecasters – which should also give us a better sense of proportion when thinking about the news that the media serves up. He also points out that people become better with practice. In some ways this isn’t surprising – he’s proved that forecasting is like most other skills – we can learn from others what works best and we can improve by practicing the art (or science?).
Doctors have an important role in forecasting the future. Within the medical system doctors need to predict the levels of demand for services, the pressure on A&E, how effective new processes will be, whether a patient is likely to respond to a treatment regime and even which applicants will make good doctors. So what is the NHS doing about training its people in forecasting? The experience of the Good Judgement Project is that for many questions within the NHS there would be a great benefit for managers and doctors learning the techniques of good forecasting and to be given exercises to get better at predicting the future.
