“AFAIK the models are not able to reliably predict future events.”

It’s self-contradictory to say you know the predictions are not reliable if you also argue we haven’t waited long enough for data that post-dates the model to see if they’re reliable (“…we need to test the function against future data.”)

**Which is it**? It can’t be both. Answering this question would be fruitful, otherwise you’re putting the models in a Catch-22.

And when answering the question, it’s useful to specify HOW you assess whether a prediction has succeeded or failed – and whether that’s even a binary question or not, and whether predictions are single-valued or multi-valued.

I’m still not following. I could be missing something obvious (very possible– my background is not in physics simulations). Please bear with me.

Taminos models produce results with varying levels of error. He selects the best model using a greedy evaluation criteria– the model that gives the lowest error against the entire series. I would assume the same is (generally) true of the more complex models.

This does not guarantee (or even suggest) that the selected model is the best one, only that it is the best one for the data series provided.

It is very possible that one of the other models is a more accurate description of the system as whole, even though it ignores aspects that cause variations in the training series. The only way to determine this is to run the model against future data.

In the example you provided, about the independant sets: From the perspective of function approximation, how is this different than adding noise to the original data and re-running the model? This would help determinine the mean error of the interpolation but I do not see how it would help mitigate overfitting. Presumably the same factors affect temperature no matter who collects the measurements.

As I understand it the models are not constructed exclusively using data from 1900 – 1980 (for example), and then tested against 1981 – 2010 (for example). From what I have read it appears that the models are constructed using all available data. If this is the case, isolating 1985 – 1987 (T[85...87]) to use as a test sample will give you a measure of how well the data has been interpolated over that given period, but not how well the model predicts (or ‘understands’) the system.

“If it’s seriously wrong it certainly will! And there are so many ways to get the modelling wrong, and few parameters that you can really tweak, and a wealth of data to test against”

I do not agree. There are many tweakable parameters. The models use many approximations that could radically affect their results due to the chaotic nature of the system. Without even getting in to the sciency bits, we know that horizontal grid size, vertical grid size, and the time step used for calculations are highly tuneable.

“when you get a reasonably good fit across a good range of tests you have reasonable confidence that you’ve captured something about reality with reasonable fidelity.”

Are we interpolating values or approximating a function? If we’re interpolating then you are correct in everything you have said. If we are approximating a function then we need to test the function against future data.

Would you agree that if the models are not able to predict future events then the model is either a) missing parameters, b) missing data, or c) both?

AFAIK the models are not able to reliably predict future events. It is my (possibly poor) understanding that they demonstrate negative skill.

No, that does not follow. You aren’t considering the difference between dependent and independent variables.

Separately measuring independent variables (such as forcings) and (believed-to-be-for-basic-physical-reasons) dependent variables (such as global temperatures) does not mean you end up with an inappropriate model or overfitting or whatever, even if the same umbrella organisation performs both types of measurements.

The very difficult job is to produce a model that does quite well at explaining how the believed-dependent observations are explained in large part by the independent variables. And if you’re still paranoid, use (say) the GISS forcings (independent) in a model that attempts to explain HadCRUT surface temperature reconstructions (dependent, from a different organisation) – or even atmospheric records such as UAH and RSS (from other organisations).

Note that even in the webpage I linked to there are a heap of demonstrated ways for an attempted fit to be very poor.

“The kicker is that in both cases the model will do poorly against unseen data.”

If it’s seriously wrong it certainly will! And there are so many ways to get the modelling wrong, and few parameters that you can really tweak, and a wealth of data to test against, that when you get a reasonably good fit across a good range of tests you have reasonable confidence that you’ve captured something about reality with reasonable fidelity.

So to get to that point, you test it against some of the known independent data without using the related dependent data to see how well it predicts the dependent variables. In other words you blind the model to some of the dependent data you know about (thus making it “unseen” in your terms) and see how well it reproduces it. More formally you “backtest” or “hindcast” by running the model from a given point T in the past using the measured initial climate state at time T and the forcings (independent variables) from time T to T+N, and see how well it predicts the measured values of dependent variables (e.g. various types of global temperatures) over the period T..T+N. You do this for different values of T to gain more confidence that you haven’t just got very lucky at overfitting.

(This is quite a simplified description – the real scientists run all sorts of tests and look at all sorts of dependent data, and have to deal with the fact that climate is a boundary within which chaotic weather trajectories occur so at a minimum multiple executions and statistical tests are necessary to figure out how well the models reproduce dependent observations…but you hopefully get the picture.)

“So in the absence of ‘model evidence’, how do you know that it’s not water vapour, volcanoes, microbes, coal fires, tectonic activity, etc….?”

A combination of physical measurements, studies in physics (and other sciences) and attribution studies can place bounds on many of these effects, showing that they do not have a big enough energy budget impact to explain the warming. It’s always possible there’s something we don’t know about these effects – or some other effect – that makes them bigger than we believe, but it’s not very likely given the amount of study undertaken (and given constraints on climate sensitivity imposed by observations of past climatic conditions). The biggest uncertainty is the cloud feedback, but this (roughly speaking) affects the climate sensitivity, not the conclusion that CO2 is by far the biggest anthropogenic climate influence.

And look at the page I sent you to earlier. What do you think the chances are that you can find an explanation for the warming based on all of those factors, but NOT based on anthropogenic influences in particular CO2)? That example should give you an idea how unlikely that is given the observational data we have.

But in the end ruling out models as evidence is not optimal either – ultimately models are attempting to capture what we know about physics and the interconnection between various effects. It’s just a more sophisticated mechanism to assess the physics.

I was using water vapour as an example, although the explanation you provided does a good job at illustrating my point.

In the example, Tamino is using (derived/estimated?) forcing data from GISS. It would stand to reason that his conclusion, derived from GISS data, will ultimately fit to the GISS data. Am I way off on this?

As you said, climate models are also created in a simlar, though more sophisticated, manner: A series of approximations is used to derive a function that maps X to Y. Is this a fairly accurate statement?

The kicker is that in both cases the model will do poorly against unseen data. If they did then nobody would have to say that the models aren’t for forcasting.

From my perspective this is a strong indication of overfitting. Essentially the modeller has done an amazingly good job at describing what’s in front of them, but missed the underlying nuances of the system as a whole.

The issue arises when the model is used as evidence for, or against, a particular influence. In the presence of overfitting it’s more likely that the model has not captured all of the relevant factors– either through absence of data or absence of functionality.

To put it another way: one could accurately describe the same temperature plot using a DCT and a Fourier transform. This doesn’t mean that JPEG is the answer to whole-earth simulations.

I’m not at all claiming the models are useless (in fact, I’d like to see a lot more work done in this area)– I’m claiming that they cannot, at this point, be used as evidence.

So in the absence of ‘model evidence’, how do you know that it’s not water vapour, volcanoes, microbes, coal fires, tectonic activity, etc….?

That’s the sort of question “attribution” studies address. The IPCC has some reporting on them. Water vapour is a very important agent, but we have a pretty good idea of how strong that effect is – and it’s not enough on its own. We also have a pretty good idea of how strong the effect of added CO2 is, and we can’t explain the observed warming without it – especially seeing recently some of the other factors (e.g. solar radiation levels) are currently acting to make the climate cooler than they otherwise would, but it’s still much warmer than normal.

Here’s a greatly simplified analysis, not meant to be particularly accurate or exhaustive – but useful because it gives readers some idea of the most simplistic level of analysis you can bring to bear on your question about water vapour – http://tamino.wordpress.com/2009/08/17/not-computer-models/. Climate scientists tend to do more complex analyses taking into account more factors, variations and interactions…

Another way of looking at it could be like this: A house is on fire. A match inside the house ignites due to the heat. How much heat did the match add, and is it directly responsible for melting the candles in the room next door?

I realize that there are people trying to sort it out, but where is the smoking-gun of proof? Why couldn’t water-vapor from the meltic ice be causing global warming? Isn’t that just as plausible? Maybe a poor example on my part, but I would think there could be any number of factors.

Re trailing indicator…

Anyone who’s studied (say) automatic control theory in engineering, or some aspects of science, will know that one can externally drive a feedback mechanism.

Here’s a simplified analogy, not meant to be exact, but merely to illuminate the key insight:

If I ignite a match by striking it (cause) I get heat (effect). Or as some might put it, “heat is a trailing indicator of ignition”.

But if I heat a match enough (cause), I get ignition (effect) (which leads to more heat). Or as some might put it, in this case “heat is a leading indicator of ignition (which leads to a feedback producing even more heat)”.

As I said, don’t try to map this exactly onto climate change. It’s an analogy. But the lightbulb should go on – a trailing indicator in one circumstance can be forced to lead in another.

Re trailing indicator:
That is a very poor example. The theory of evolution does not make any predictions about what the structure of a species will look like after a given period of time. It would neither confirm nor deny that a dog could give birth to a cat (fyi: there are organisms which give ‘birth’ to other species through parasitic relationships). AGW via the IPCC makes plenty of predictions on how it’s subject will look after a given period of time.

Yes, actually, it is — it’s what “statistically significant” means in regard to trends. The trend since 1995 just misses — it’s at about the 93% confidence level rather than 95%; but the trend since 1994 is over the 95% confidence level.

Now you are in the same boat as me.

What boat is that? Jinchi knows what he’s talking about and isn’t throwing up erroneous denialist talking points one after the other.

Regarding the trailing indicator: it certainley disputes the current version of AGW, and that is what is at issue here.

It does no such thing; that’s like saying that, if evolution were true, we would expect to see dogs giving birth to cats. It reflects ignorance about the theory and makes claims about what the theory predicts that are simply false.

I’m not so sure that the ‘basic warming influence of greenhouse gases’ is well understood.

So what if you aren’t? It is well understood, and if you weren’t intellectually lazy and committed to only listening to professional obfuscators like Lindzen you would learn about it.