This weekend, Ed pointed to the paper “Retrospective Prediction of the Global Warming Slowdown in the Past Decade”, in Nature SomethingOrOther*.
Colour me slightly underwhelmed by the actual paper. The topic is important, and I’m sure they’re on to something. These things are often much more impressive if you have a real feel for the model. I’d like to see much more detail about the dynamics that the models are simulating, including some spatial patterns (are they doing the right thing for the right reasons?), and I’d like a bigger pair of ensembles for comparison (are we really sure that the models are doing the right thing?)**.
Also, how hard is it to directly label your graphs? What did blue mean again?
My biggest complaint is the language. I’m going to go right ahead and suggest that we reserve the word “prediction” for situations where you don’t actually know the answer. I have no problems at all with using the word “prediction” for past events; a natural stance in a subjective Bayesian framework. I always thought the great (Bohr?) quote – “prediction is very difficult, especially about the future” – probably makes more sense straight up, rather than in the jokey way that it is often used.
There is a fundamental difference between a prediction and a hindcast – which is the word that I would use for what they do in the paper. I would say that the team has demonstrated that the model more successfully simulates the evolving trajectory of temperatures, when initialised with the known state of the world. Hindcasts are a really good thing, and give us lots of information, but not as much information as predictions. Adding the modifier “retrospective” to “prediction” is not good enough – in fact I would say it was a contradiction in terms.
You could argue that this version of the climate model doesn’t see the “right” answer a priori, but any prediction system is a combination of model and a human components. The opportunities for subtle biases to creep in if information about the “right” answer is available are numerous. Would the paper have been published if the model didn’t get the “right” answer?
I think that the team that wrote this paper are actually calibrating the prediction system. Great, but let’s preserve special status of true predictions by reserving the word.
* ht @micefearboggis for the journal name
** UPDATE To be fair, some of this is in the supplementary information
Doug McNeall's blog 
Could we also make “experiment” a reserved word while we’re at it? Reserving it for actual experiments?
Sure, good idea. Now what do you mean by ‘experiments’? 😉
Physical experiments constrained by physics even if those are not know. Not model runs.
I’d be find with model runs being called “thought experiments” though.
Using the word experiment for a model run is an honest statement that you have no idea what such a complex piece of code will produce.
Hey some good points there. I also worry about inadvertently fitting models to the satellite era.
I do like ‘retrospective prediction’ though, if only because it’s more easily understood by folks outside the field than ‘hindcast’ (which is just a play on ‘forecast’ and also a contradiction in terms.)
[…] to the Kennan Kerfuffle, I doodled around Doug McNeal’s blog. I found this great […]
A word I would like modellers to stop using would be “observations”.
Often they actually mean “reanalysis data”, which is a model-observation twitter.
And even if they really mean observations, to me (as someone who knows more about measurements as about modelling) it would be valuable to know which observational dataset is used. That would allow me to estimate whether the deviations could also be due to observational limitations.