Statistical homogenisation for dummies

The self-proclaimed climate sceptics keep on spreading fairy tales that homogenisation is smoothing climate data and leads to adjustments of good stations to make them into bad stations. Quite some controversy for such an innocent method to reduce non-climatic influences from the climate record.

In this post, I will explain how homogenisation really works using a simple example with only three stations. Figure 1 shows these three nearby stations. Statistical homogenisation exploits the fact that these three time series are very similar (are highly correlated) as they measure almost the same regional climate. Changes that happen at only one of the stations are assumed to be non-climatic. The aim of homogenisation is to remove such non-climatic changes in the data.

Figure 1. The annual mean temperature data of three hypothetical stations in one climate region.

(In case colleagues of mine are reading this and are wondering about my craftsmanship: I do know who to operate scientific plotting software, but some “sceptics” make fun of people who have no experience with Excel. I just wanted to show off with being able to use a spreadsheet.)

For the example, I have added a break inhomogeneity in the middle with a typical size of 0.8 °C (1.5 °F) to the data for station A; see Figure 2.

Do you want to help with data discovery?

Reposted from the blog of the International Surface Temperature Initiative

As was alluded to in an earlier posting here, NOAA's National Climatic Data Center has recently endeavored on an effort to discover and rescue a plethora of international holdings in hard copy in its basement and make them usable by the international science community. The resulting images of the records from the first chunk of these efforts have just been made available online. Sadly, it is not realistic at the present time to key these data so they remain stuck in a half-way house, available, tantalizingly so, but not yet truly usable.

So, if you want to undertake some climate sleuthing now is your moment to shine ...! The data have all been placed at ftp://ftp.ncdc.noaa.gov/pub/data/globaldatabank/daily/stage0/FDL/ . These consist of images at both daily and monthly resolution - don't be fooled by the daily in the ftp site address. If you find a monthly resolution data source you could digitize years worth of records in an evening.

Whether you wish to start with Angola ...

A short introduction to the time of observation bias and its correction

Figure 1. A thermo-hygrograph, measures and records temperature and humidity.

Due to recent events, the time of observation bias in climatological temperature measurements has become a hot topic. What is it, why is it important, why should we and how can we correct for it? A short introduction.

Mean temperature

The mean daily temperature can be determined in multiple ways. Nowadays, it is easy to measure the temperature frequently, store it in a digital memory and compute the daily average. Also in the past something similar was possible using a thermograph; see Figure 1. However, such an instrument was expensive and fragile.

Thus normally other ways were used for standard measurements, using minimum and maximum thermometers and by computing a weighted average over observations at 3 or 4 fixed times. Another good approximation for many climate regions is to average over the minimum and maximum temperature. Special minimum and maximum thermometers were invented in 1782 for this task.

Blog review of the Watts et al. (2012) manuscript on surface temperature trends

[UPDATE: Skeptical Science has written an extensive review of the Watts et al. manuscript: "As it currently stands, the issues we discuss below appear to entirely compromise the conclusions of the paper." They mention all the important issues, except maybe for the selection bias mentioned below. Thus my fast preliminary review below can now be considered outdated. Have fun.]

Anthony Watts put his blog on hold for two days because he had to work on an urgent project.

Something’s happened. From now until Sunday July 29th, around Noon PST, WUWT will be suspending publishing. At that time, there will be a major announcement that I’m sure will attract a broad global interest due to its controversial and unprecedented nature.

What has happened? Anthony Watts, President of IntelliWeather has co-written a manuscript and a press release! As Mr. Watts is a fan of review by bloggers, here is my first reaction after looking through the figures and the abstract.

Investigation of methods for hydroclimatic data homogenization

The self-proclaimed climate sceptics have found an interesting presentation held at the General meeting of the European Geophysical Union.

In the words of Anthony Watts, the "sceptic" with one of the most read blogs, this abstract is a ”new peer reviewed paper recently presented at the European Geosciences Union meeting.” A bit closer to the truth is that this is a conference contribution by Steirou and Koutsoyiannis, based on a graduation thesis (Greek), which was submitted to the EGU session "Climate, Hydrology and Water Infrastructure". An EGU abstract is typically half a page, it is not possible to do a real review of a scientific study based on such a short text. The purpose of an EGU abstract is in practice to decide who gets a talk and who gets a poster, nothing more, everyone is welcome to come to EGU.

What is a change in extreme weather?

What is a change in extreme weather?

The reason for changes in extremes can be divided up into two categories: changes in the mean (see panel a of the figure below) and other changes in the distribution (simplified as a change in the variance in panel b). Mixtures are of course also possible (panel c).

If you are interested in the impacts of climate change, you do not care why the the extremes are changing. If the dikes need to be made stronger or the sewage system needs larger sewers and larger reservoirs, all you need to know is how likely it is that a certain threshold is reached. Much research into changes in extreme weather is climate change impact research and thus does not care much about this distinction.

If you are interested in understanding the climate system, it does matter why the extremes are changing. Changes in the mean state of the climate are relatively well studied. Interesting questions are, for instance, whether a change in the mean changes the distribution via feedback processes or whether the reduced temperature contrasts between the poles and the equator or between day and night cause changes in the distribution.

If you are interested in understanding the climate system also the spatial and temporal averaging scales matter. If rain fronts move slower, they may locally produce more extreme daily precipitation sums, while on a global scale or instantaneously there is no change in the distribution of precipitation.

I hope scientists will distinguish between these two different ways in which extremes may change in future publications and, for example, not only compute the increase in the number of tropical days, but also how many of these days are due to the change in the mean and how many are due to changes in the distribution. I think this would contribute to a better understanding of the climate system.

Figure is taken from Real Climate, which took it from IPCC (2001).

Paleo and fruitarian lifestyles have a lot in common

My new fitness trainer eats a lot of fruit. And she looks darn healthy. Now I know, you should not take weight-training advice from a professional body builder or risk serious overtraining, but still I was intrigued and did some research. The vegan and paleo communities are often not on friendly terms. Thus what struck me most researching fruitarian blogs, was how similar many of the ideas were.

A very strict fruitarian only eats fruits in the common meaning, sweet and juicy fruits from trees or bushes. Others also include vegetable fruits such as avocados, tomatoes and cucumbers, still others also include nuts, many regularly eat salad. To get sufficient calories from fruits, a fruitarian has to eat several kilograms of fruit. Some people calling themselves fruitarians actually get most calories from nuts and avocados. In this post fruitarians are people getting most calories from simple carbohydrates, that is from sweet fruits.

The paleolithic lifestyle is inspired by the way people lived before agriculture. As the information from the Paleolithic Age is scarce, in praxis this often means, that existing hunter gatherers and their diets and lifestyles are studied. Such bands often trade with nearby agriculturalists and thus no longer live a true stone-age life. Still as long as they are free from the deceases of civilisation, they provide good role models in my view. Similarly, many paleos also look at other existing cultures that are in good health. In this respect the paleo community is close to the Weston A Price Foundation, who seek guidance with how people lived a few generations ago. The paleo diet is best defined by what it not eaten: processed foods, grains, sugar and refined seed oils.

HUME: Homogenisation, Uncertainty Measures and Extreme weather

Proposal for future research in homogenisation

To keep this post short, a background in homogenisation is assumed and not every argument is fully rigorous.

Aim

This document wants to start a discussion on the research priorities in homogenisation of historical climate data from surface networks. It will argue that with the increased scientific work on changes in extreme weather, the homogenisation community should work more on daily data and especially on quantifying the uncertainties remaining in homogenized data. Comments on these ideas are welcome as well as further thoughts. Hopefully we can reach a consensus on research priorities for the coming years. A common voice will strengthen our voice with research funding agencies.

State-of-the-art

From homogenisation of monthly and yearly data, we have learned that the size of breaks is typically on the order of the climatic changes observed in the 20th century and that period between two detected breaks is around 15 to 20 years. Thus these inhomogeneities are a significant source of error and need to be removed. The benchmark of the Cost Action HOME has shown that these breaks can be removed reliably, that homogenisation improves the usefulness of the temperature and precipitation data to study decadal variability and secular trends. Not all problems are already optimally solved, for instance the solutions for the inhomogeneous reference problem are still quite ad hoc. The HOME benchmark found mixed results for precipitation and the handling of missing data can probably be improved. Furthermore, homogenisation of other climate elements and from different, for example dry, regions should be studied. However, in general, annual and monthly homogenisation can be seen as a mature field. The homogenisation of daily data is still in its infancy. Daily datasets are essential for studying extremes of weather and climate. Here the focus is not on the mean values, but on what happens in the tails of the distributions. Looking at the physical causes of inhomogeneities, one would expect that many of them especially affect the tails of the distributions. Likewise the IPCC AR4 report warns that changes in extremes are often more sensitive to inhomogeneous climate monitoring practices than changes in the mean.