Temperature trend over last 15 years is twice as large as previously thought

UPDATED: Now with my response to Juddith Curry's comments and an interesting comment by Peter Thorne.

Yesterday a study appeared in the Quarterly Journal of the Royal Meteorological Society that suggests that the temperature trend over the last 15 years is about twice a large as previously thought. This study [UPDATE: Now Open Access] is by Kevin Cowtan and Robert G. Way and is called: "Coverage bias in the HadCRUT4 temperature series and its impact on recent temperature trends".

The reason for the bias is that in the HadCRUT dataset, there is a gap in the Arctic and the study shows that it is likely that there was strong warming in this missing data region (h/t Stefan Rahmstorf at Klimalounge in German; the comments and answers by Rahmstorf there are also interesting and refreshingly civilized; might be worth reading the "translation"). In the HadCRUT4 dataset the temperature trend over the period 1997-2012 is only 0.05°C per decade. After filling the gap in the Arctic, the trend is 0.12 °C per decade.

The study starts with the observation that over the period 1997 to 2012 "GISTEMP, UAH and NCEP/NCAR [which have (nearly) complete global coverage and no large gap at the Arctic, VV] all show faster warming in the Arctic than over the planet as a whole, and GISTEMP and NCEP/NCAR also show faster warming in the Antarctic. Both of these regions are largely missing in the HadCRUT4 data. If the other datasets are right, this should lead to a cool bias due to coverage in the HadCRUT4 temperature series.".

Datasets

All datasets have their own strengths and weaknesses. The nice thing about this paper is how they combine the datasets and use the strengths and mitigate their weaknesses.

Surface data. Direct (in-situ) measurements of temperature (used in HadCRU and GISTEMP) are very important. Because they lend themselves well to homogenization, station data is temporal consistent and its trend are thus most reliable. Problems are that most observations were not performed with climate change in mind and the spatial gaps that are so important for this study.

Satellite data. Satellites perform indirect measurements of the temperature (UAH and RSS). Their main strengths are the global coverage and spatial detail. A problem for satellite datasets are that the computation of physical parameters (retrievals) needs simplified assumptions and that other (partially unknown) factors can influence the result. The temperature retrieval needs information on the surface, which is especially important in the Arctic. Another satellite temperature dataset by RSS therefore omits the Arctic from their dataset. UAH is also expected to have biases in the Arctic, but does provide data.

A further problem of satellite data it is not homogeneous due to drifts of the satellites, changes in calibration constants due to deterioration of the instruments and changes in the number and type satellites. Thus computing reliable trends from satellite data is difficult.

Finally, its results cannot be directly compared to surface measurements as they measure the temperature of the lower troposphere, which is well above the 2-meter height at which station data measures temperature. As the study writes:

The satellite record is of particular interest because it provides a uniform sampling with near-global coverage. However the satellite microwave sounding units measure lower troposphere rather than surface temperatures and so are not directly comparable to the in situ temperature record. Furthermore there are temporal uncertainties in the satellite record arising from satellite failure and replacement and the numerous corrections required to construct a homogeneous record (Karl et al. 2006). Contamination of the microwave signal from different surface types is also an issue, particularly over ice and at high altitude (Mears et al. 2003).

Reanalysis data. Already in 2008, RealClimate suggested that the Arctic data gap in the HadCRUT 3V dataset may produce a too small trend because reanalysis data and the GISS dataset show strong warming in the missing region.

An analysis dataset combines all observations into a field that can be used to initialise a weather prediction. Typically, this also uses the information of a previous short term weather prediction. In a reanalysis dataset this process is repeated produce a long climate dataset. A reanalysis is better than analysis data because more data is available as in real-time weather predictions and because a reanalysis is computed using one atmospheric model whereas an operational weather prediction model is regularly improved. However reanalysis data is still not very reliable when it comes to trends because the amount and type of observations change considerably in time. For this reason the current Cowtan and Way study is much more reliable.

Putting it all together

The study uses two interpolation methods to fill the Arctic gap. The most interesting one is the hybrid method, it uses the strength of the UAH dataset (spatial coverage) and the HadCRUT dataset (temporal consistency). It does so by interpolating the difference between the two datasets. In this way the biases in the UAH dataset are basically computed at the Arctic edges of the HadCRUT dataset and thus reduced. The bias corrected satellite data is then used to fill the gap, especially in the middle, nearer to the edges also the HadCRUT data itself is still directly important in the interpolation.

The method is carefully validated by increasing the gap at the Arctic and comparing how well the data fits in the artificial part of this gap. Very elegant, even if this may not be the final word, because the behaviour of the UAH biases may be different in the true gap as in the artificial one, which is further from the pole.


Figure 5b from Cowtan and Way (2013) that shows the temperature for the satellite period smoothed by a 60 month moving average. The Null reconstruction fills the missing data with the global average values, the kriging reconstruction performs an optimal interpolation to fill all gaps, the hybrid reconstructions is briefly explained above and the HadCRUT4 is the original dataset with gaps.

Amateur science

Another interesting aspect of this study is that the authors are scientists, but no climate scientists and did the work in their free time. The authors are part of the Skeptical Science crew and the acknowledgement and the posts by Dana Nuccitelli and Stefan Rahmstorf on this study suggest that they did get some help from professionals.

This shows that amateur scientists can make valuable contributions, as we have also seen for Zeke Hausfather and his study on the influence of urbanization. And it also suggests that collaboration with experienced people is important. Personally, I have changed my research topic a few times, I always make sure to collaborate with experts to avoid making rooky mistakes.

[UPDATE: On twitter someone complained about my use of the word amateur.

Re this via @MagWes http://t.co/ADa4QluTLT -> would like to point out Kevin Cowtan is dedicated modest hard-working sci, hardly an "amateur"
— Ruth Mottram (@ruth_mottram) November 15, 2013

Yes he is a scientist and that surely helped him a lot. But he is no climate scientist and did the work in his free time. I would also call myself an amateur at the moment I wrote my first paper on homogenization, which was also mainly written in my free time, for the love of the beautiful topic. The word is also not intended as an insult, in fact a bit more than century ago scientists being professionals was regarded a problem by many amateur scientists that were the main group then.]

Also refreshing, after previous game changing studies by amateur ostriches, is the modesty of the authors, as Dr. Cowtan states in The Guardian:

"No difficult scientific problem is ever solved in a single paper. I don't expect our paper to be the last word on this, but I hope we have advanced the discussion."

minor deviation

In fact, we are looking at a minor deviation. I am impressed that climate science seems to be able to come to conclusions on it. The change in warming of the atmosphere is only 2 percent of the warming of the full climate system and the deviation for the last 15 years is also just a few percent of the warming since 1900. Thus we are looking at a deviation of less than one in a thousand and if the current study holds even much less.

We now have several explanations for the reduced trend in the temperature record for the last 15 year, the Arctic gap, El Nino and more warming of the oceans, maybe smaller contributions by volcanic eruptions and less solar activity.

My personally hunch would be improvements in the climatic observations. I could imagine that people pay more attention to making accurate observations now that climate change is an issue (for meteorology less accuracy is sufficient) and maybe also due to the introduction of ventilated automatic weather stations, which reduces radiation errors. But that is just a hunch and still needs to be investigated. Let's see in a few years, when the dust is settled, what the main factors are.

Update 1

This update is now a new post. Skip this update.

Judith Curry makes some good comments on the paper.

Judith Curry: The[y] state that most of the difference in their reconstructed global average comes from the Arctic, so I focus on the Arctic (which is where I have special expertise in any event).

First, Kriging. Kriging across land/ocean/sea ice boundaries makes no physical sense. While the paper cites Rigor et al. (2000) that shows ‘some’ correlation in winter between land and sea ice temps at up to 1000 km, I would expect no correlation in other seasons.

I see no reason not to use kriging. It is an optimal interpolation technique and also delivers the uncertainties of the interpolated values. Given that we expect that the temperate trend is larger near the poles as around the equator, one would expect that interpolation would underestimate the trend. As scientists are conservative, that is the preferred direction of erring. The GISS dataset also uses interpolation to fill the gaps. Taking the land, ocean and sea ice into account as covariables will likely make the interpolated estimates more accurate.

Lotte Bruch makes an interesting comment at the Klimalounge (in German), that she mainly expects this study to change the temperatures in winter, because in summer the melting prevents strong temperature deviations. In the light of the Curry's remark above, it would be interesting whether this is the case. Then it would be no problem and if there are not much variations in other seasons that would also explain why the correlations are low in these seasons and then that would not be a problem for the interpolation. Definitely worth a study.

The first author, Kevin Cowtan of this study has a detailed response to Curry's comment below her post: the difference between the hybrid and kriging reconstructions of Antarctica is only really significant around 1998, so it doesn’t greatly affect our conclusions.

And also the second author, Robert Way, has written two clear, but not very friendly comments: The cross-validation steps taken in this paper are very important and the paper shows rather clearly that the Hybrid method in particular appears to be fairly robust even at long distances from adjacent cells.

Judith Curry: Second, UAH satellite analyses. Not useful at high latitudes in the presence of temperature inversions and not useful over sea ice (which has a very complex spatially varying microwave emission signature). Hopefully John Christy will chime in on this.

Because satellite data is not very reliable the authors use a combination of satellite and surface data and correct for (some of the) errors in the satellite dataset. How well this works will have to be studied in subsequent papers. I see no a priority reason why it should not work and especially not why it should be biased. If it is a problem it would just add to the uncertainties.

I really wonder why John Cristy of the UAH dataset should chime in. He is the one that delivers a dataset with values in the Arctic. Maybe it would be better to ask the mainstream scientists behind the RSS dataset. They are the ones that did not trust the data sufficiently and rather leave a gap in the Arctic.

Judith Curry: Third, re reanalyses in the Arctic. See Fig 1 from this paper, which gives you a sense of the magnitude of grid point errors for one point over an annual cycle. Some potential utility here, but reanalyses are not useful for trends owing to temporal inhomogeneities in the datasets that are assimilated.

The paper is not based on the reanalysis datasets. As I wrote above, it only presents the information that GISTEMP, UAH and NCEP/NCAR show stronger as average warming in the Arctic. The analysis in the paper itself is based on the HadCRUT and UAH datasets.

Judith Curry: So I don’t think Cowtan and Wray’s analysis adds anything to our understanding of the global surface temperature field and the ‘pause.’

That is not very generous. At least it adds the information that the gap in the Arctic is potentially important and that it is thus worthwhile to study it in more detail, including improvements of the interesting new analysis method used by the authors. This is valuable information for people interested in this minor deviation in the temperature trend. At least for people who would like to understand it.

Another valuable point is that the study illustrates how minor the deviation in the temperature is. If you have to worry about temperature inversions in the Arctic and hope that the very complex spatially varying microwave emission signature will save your point of view, then maybe it is time to broaden your view and have another look at the 0.8 °C temperature increase over the last century.

Judith Curry: The bottom line remains Ed Hawkins’ figure that compares climate model simulations for regions where the surface observations exist. This is the appropriate way to compare climate models to surface observations, and the outstanding issue is that the climate models and observations disagree.

Interesting that Dr Hawkins made the comparison that way, sounds like the best way to compare the models with the data. For the claim of the climate ostriches that the warming of the atmosphere has stopped, this is, however, not very relevant. The regional distribution of the warming is interesting for climate change impact studies, but not for the political "debate".

Update 2

The comment below by Peter Thorne deserves more attention. He is an experienced climatologist, was lead author of the IPCC report, worked a lot on the quality of the climate record at the Hadley Centre and NOAA and is now professor in Norway.

A few brief observations:

1. This issue of sampling isn't entirely new or uninvestigated. See here for example.

2. GISS and NCDC MLOST do interpolate over some distance from real observations. Even HadCRUT gridding to 5 degree is arguably a form of limited interpolation. But interpolation is a vexed issue. We certainly need to do better on producing globally complete estimates and their uncertainties. And it certainly impacts on trends, particularly shorter-term trends.

3. HadCRUT does account for spatial incompleteness. It does this through its error model rather than attempting to interpolate. So, if you use and propagate the uncertainty estimates appropriately you will find that HadCRUT's estimates are consistent with a higher warming rate than its median estimator. Probably higher than even this estimate. See some pretty pictures at here or read the whole paper here.

[UPDATE, 2015. Some new papers on this topic.
Comiso, J. C., & Hall, D. K., 2014: Climate trends in the Arctic as observed from space. Wiley Interdisciplinary Reviews: Climate Change, 5, no. 3, pp. 389-409.

Dodd, E. M., Merchant, C. J., Rayner, N. A., & Morice, C. P., 2014: An Investigation into the Impact of using Various Techniques to Estimate Arctic Surface Air Temperature Anomalies. Journal of Climate, 28, pp. 1743–1763, doi: 10.1175/JCLI-D-14-00250.1.]

Other opinions on this study

Coverage bias in the HadCRUT4 temperature record by Kevin Cowtan and Robert Way at Cowtan's homepage.

The authors have produced an informative homepage with lot's of background information on their article. It includes a nice discussion on previous work on the influence of the Arctic gap on the global mean temperature. They also wrote an article for Skeptical Science.

The Disappearing Hiatus by Michael Tobis at Planet3.0 Beyond Sustainability.

Tobis seems to have come to the same main conclusion independently: "This demonstrates is how very un-robust the “slowdown” is. It did not take much of a correction to eliminate the trend.".

Global Warming Since 1997 Underestimated by Half by Stefan Rahmstorf at RealClimate.

A post on the same paper.

Global warming since 1997 more than twice as fast as previously estimated, new study shows by Dana Nuccitelli at the Guardian.

Another post published today on the same paper.

Uncertainty in SST measurements and data sets by Judith Curry at Climate Etc.

Curry points to some other papers on uncertainties in the temperature record and then discusses the study of Cowan and Way as mentioned in the above update. In the comments Steve Mosher writes: "I know robert [Way] does first rate work because we’ve been comparing notes and methods and code for well over a year. At one point we spent about 3 months looking at labrador data from enviroment canada and BEST. ... Of course, folks should double and triple check, but he’s pretty damn solid."

About That Global Warming Hiatus… #Fauxpause by Greg Laden at his blog.

Laden explains the importance of sampling by comparing it to trying got estimate how much people bought at a mall.

Cotwan and Way: Have they killed the pause? by Lucia at The Blackboard.

Lucia did not read the paper yet, but describes it and asks some questions based on the information in the internet. Lucia: "Right now, I’m mostly liking the paper. The issues I note above are questions, but they do do quite a bit of checking".

Molecule painter and icy pal double the warming trend since 1997 by Luboš Motl at The Reference Frame.

Motl discredits himself with Ad Hominems and empty babble. Makes WUWT look like a quality blog. This link really needs a NoFollow tag, to make sure that Google does not interpret this link as a recommendation.

The 'hiatus' and the Arctic by Neven at the Arctic Sea Ice Blog.

If you want to know about the Arctic, you go to Neven and indeed he has a post on this study from the Arctic perspective.

Related information

Atmospheric warming hiatus: The peculiar debate about the 2% of the 2% by me at variable variability.

This post explains how minor the temperature trend deviation is that the climate ostriches and their friends in the media are getting wild about.

Global warming has not stopped by Ari Jokimäki at AGW Observer

Ari Jokimäki presents the latest research papers on the warming since 1998 and its variability.

About the Lack of Warming… by John Nielsen-Gammon at the Climate Abyss.

John Nielsen-Gammon created a beautiful plot showing the relationship between global mean surface temperature and El Nino. By giving El Nino and La Nina years a different color and symbol you can see the influence of SOI, without having to "fudge" the data.

Tropical ocean key to global warming ‘hiatus’ by Jeff Tollefson at Nature magazin.

An editorial article in Nature on the Kosaka and Xie paper, which studied the influence of El Nino on the temperature, which could largely explain the atmospheric warming hiatus. By taking into account historical forcings (greenhouse gases, sun, etc.) and El Nino they are able to reproduce the global surface temperature with a of correlation 0.97 (!) since 1970. Impressive!

Does the global warming 'pause' mean what you think it means? by Dana Nuccitelli at The Guardian.

The original article on the 2 percent.

Climate Science: Connecting the Hiatuses by Bill Chameides at The Scientific American

Discusses possible causes for the current hiatus on the basis of previous ones.