Falsifiable and falsification in science

"You keep using that word. I do not think it means what you think it means."

In a recent post, Interesting what the interesting Judith Curry finds interesting, I stated that "it is very easy to falsify the theory of global warming by greenhouse gasses." The ensuing discussions suggest that it could be interesting to write a little more about the role of falsifiable hypotheses and falsification in science. The main problem is that people confuse falsifiable and falsification, often do not even seem to notice there is a difference, whereas they have very different roles in science.

The power of science and falsification are beautifully illustrated in this video by asking normal people on the street to discover the rule behind a number sequence (h/t U Know I Speak Sense).

Falsifiable

Karl Popper only asked himself what distinguishes a scientific hypothesis from an ordinary idea.
Popper's beautiful thesis was that you can distinguish between a scientific and a non-scientific statement by asking oneself if it can be falsified. If it cannot, it is not science. Thus the worst one can say about an idea that is supposed to be scientific is that it is not even wrong.

Important side remark: Please, note that also non-scientific ideas can be valuable, Popper's philosophy itself is not science, just like most philosophy, political ideas, literature and religion.

And please note that wrong hypotheses are also scientific statements; that they are wrong automatically shows that they can be falsified. Even falsified hypothesis are still scientific hypothesis and can even still be useful. An good example would be classical mechanics. This illustrates that Popper did not think about whether hypothesis were right or wrong (falsified), useful or not, but whether a statement is scientific or not scientific.

To be falsifiable, falsification is only needed to be possible in principle. It does not matter whether falsification would be hard or easy for the question whether it is science. This is because the main value of the criterion is that it forces you to write up very clearly, very precisely what you are thinking. That allows other scientists to repeat your work, test the idea and build upon it. It is not about falsification, but about clarity.

That also implies that the daily job of a scientist is not to falsify hypothesis, especially not solid and well-validated ones. Scientists are also not writing down new falsifiable hypothesis most of the time, in fact they rarely do so. Those are the rare Eukeka moments.

The terms scientist and science are clearly much broader and also much harder to capture. The ambitious William M. Connolley set out to define science and what a scientist does in a recent post. Definitely worth reading, especially if you are not that familiar with science. Disclaimer: not surprisingly, the aim was not completely achieved.

Psycho analysis

A classical example for Popper of a non-scientific hypothesis would be Freud's psycho-analysis. The relationship between the current psychological problems of a patient and what happened long ago in the patients childhood is too flexible and not sufficiently well defined to be science. That does not mean that what happens to a child is not important, there are many modern findings that point into that direction (Joachim Bauer, 2010). If someone else would succeed in making Freud's ideas more specific and falsifiable, it would even be a valuable contribution to science. It also does not mean that psycho-analysis does not help patients. Finally, it also does not mean that it is wrong, rather it means that it is not even wrong. It is too vague.

Morphic fields

Another example is the idea of Rupert Sheldrake about morphic fields. Sheldrake claims that when an idea has been invented before, it becomes easier to reinvent it. He has a large number of suggestive examples where this seems to be the case. Thus there is a lot of information to validate his idea.

The problem is, it is impossible to falsify the idea. This idea is, again, too vague and if you do not find the effect in an experiment, you can always claim that the effect is smaller, that the experiment was not sensitive enough or not well executed.

When I was studying physics in at Groningen University, Sheldrake gave a talk and afterwards naturally got the question whether his ideas were falsifiable. He dogged the question and started about the science philosophy of Thomas Kuhn on paradigm changes that shows that in practice it can be hard to determine whether an idea is falsified. However, whether an idea is falsifiable is clearly another question as how falsification works, which will be discussed below. Then Sheldrake started fueling tribal sentiments, by complaining that only physicists would be allowed to have hypotheses with fields, why not biologists? Discrimination! As the climate "debate" illustrates, adding some tribal conflict is an effective way to reduce critical thinking.

This does not mean that the ideas of Sheldrake may not turn out to be valuable. The list of examples that validate his ideas is intriguing. This may well be a first step towards making a scientific hypothesis. That is also part of the work of a scientist, to translate a creative, fresh idea you got during a hike into a solid, testable scientific idea. Morphic fields are, however, not yet science.

Anthropogenic global warming

The hypothesis that the man-made increases in the concentration on greenhouse gasses leads to an increase in the global mean temperature can be falsified and is thus a scientific hypothesis. There is no need to go into details here, because Hans Custers just wrote an interesting post, "Is climate science falsifiable?", which lists ten ways to falsify the "AGW hypothesis". One would have been sufficient.

A clear example is that if the average world temperature drops one degree, back to values before 1900 and stays there for a long time without there being other reasons for the temperature decrease (e.g. volcanoes, sun, aerosols) the theory would be falsified. To get to ten ways, Custers has to come up with rather adventurous problems that are extremely unlikely because so many basic science and experiments would need to be wrong.

Seen in this light, the climate ostriches are almost right, it is highly unlikely that the theory of man-made global warming will be refuted, that would require highly surprising new findings and in most cases it would require basic physics, used in many sciences, to be wrong. However, just because it is highly unlikely in practice that the hypothesis will be falsified because there are so many independent lines of evidence and the hypothesis is well nested into a network of scientific ideas, that does not make it theoretically impossible, thus AGW is falsifiable.

Falsification

It doesn't matter how beautiful your theory is, it doesn't matter how smart you are. If it doesn't agree with experiment, it's wrong. Richard P. Feynman

This quote is a favorite one of the climate ostriches. Unfortunately, falsification is a little more complex in practice.

While two reasonable persons can likely agree upon the question whether a hypothesis is falsifiable, falsification is a much more complicated matter. The basic problem is that you never test just one hypothesis, but always a cluster of them. Such a cluster is called a paradigm by Thomas Kuhn in his book, The structure of scientific revolutions. Even if you would come up with a smart experiment that only tested one hypothesis, you would still have the definitions of the terms in the hypothesis, you have traditions of how to measure the variables in question, and so on.

Such a cluster can go into a lot of detail. For example the researchers at CERN who thought they might have measured that neutrinos can go a little faster than light, were also testing the hypothesis that they had connected the coaxial cable right. This detail turned out to be the problem and not Einstein's relativity theory. That is a good example in Richard Feynman's own field that falsification is not that trivial, as I am sure Feynman himself realized.

Flappy bird

In sciences dealing with the world outside of the laboratory even larger sets of hypothesis are tested simultaneously. Bart Verheggen gave a clear example in his post: A quick ‘n dirty guide to falsifying AGW. He wonders whether a flying bird can be used to falsify the theory of gravity. Doesn't the theory state that objects with mass fall down?

However, a bird in the sky is not just a point mass x meters above the Earth in vacuum. There are more forces at play. You can see that as an ad-hoc fix of the theory of gravity and a reason to develop a better theory. Most people seem to prefer to see the bird as an exception, to study how the exception can be explained and add an additional theory about aerodynamic forces to the explanation. More generally, if you notice something that seems to falsify your hypothesis, that is a reason to study why and improve your understanding of the problem, it is not a reason to immediately reject the hypothesis.

This is an example we know very well, which makes it strange to see a bird as falsifying gravity, but in practice it can be hard to judge whether something is an ad-hoc fix or a legitimate additional hypothesis. This is especially hard during so-called paradigm changes, periods in which important hypotheses are called into question and sometimes replaced by better ones.

Climate "debate"

What the climate ostriches see as falsification is typically similar to a flying bird. Sometimes it is an indication that reality is a bit more complicated. Sometimes not even that and the ostriches feel that something is a contradiction, when it is not. An example of the last case is the feeling that the CO2 concentration is too small to matter.



Whereas 280 parts per million can have quite an influence.


An example of of reality being a bit more complicated are claims that increases in Antarctic sea refute the AWG hypothesis. Maybe they think so because it suggests that the temperature in the Antarctic is not increasing. However, the AGW hypothesis only claims that the average global temperature is increasing, local variations are not excluded. Furthermore, the temperature is actually increasing, according the Berkeley Earth Surface Temperature project; see below.

Trying to understand the increase in sea ice, we may learn more about the climate system and our observations. It may be related to the water becoming more fresh due to more melt water from the land ice; fresh water freezes easier as salty water. It may be related to changes in the circulation. It may also be due to inaccurate observations. Recently a non-climatic change was found that changed the trend considerably. This was due to a change in the satellites used. Whatever the reason will turn out to be, given the observed Antarctic temperature increase, I do not expect that resolving this issue will refute the AGW hypothesis.



Another favorite "falsification" is the apparent slowdown in trend of the global mean temperature since the strong El Nino year 1998. I must say, I do not even know whether it would be right to talk about a slowdown. The period is so short that the uncertainties in the estimated trend is large. That there was no temperature increase cannot be excluded statistically, but also a continuation of the previous trend can not be excluded.

And the "slowdown" in air temperature would only be a sign the the AGW hypothesis is wrong, if it were a sign that heating of the climate system had stopped as well. However, the heating of the ocean is continuing, so is the sea level rise and the melting of the Arctic and total sea ice. In fact, if you do a back of the envelop computation, you can show that this "slowdown" of the air temperature maximally represents a deviation 1 in a thousand of the total anthropogenic heating of the climate system. Not the stuff refutations are made of.

It is an interesting example to better understand the fluctuations of the global temperature on decadal time scales. It seems that the slowdown can be explained by more heat going into the ocean, especially in the Pacific due to a special wind pattern. And there are many other smaller contributions (volcanoes, sun, lack of Arctic observations). So much actually, that Matthew England is wondering why the temperature did not decrease more (also here in German).

Paradigm changes

A famous example of a paradigm change is the transition from a geocentric (Earth center of universe) to a heliocentric worldview (Copernicus). The Copernican model was simpler, but, if I remember correctly, initially also less accurate as the geocentric model. That made the choice of the optimal model subjective. Adding up simplicity and accuracy is like adding apples and oranges.

With the observations of the moons around Jupiter and the phases of Venus by Galileo Galilei, the advantages of the heliocentric world view became clearer. Also the computations became more accurate, especially when the circular orbits of Copernicus were replaced by the elliptical ones of Keppler. And with classical mechanics and gravity we can now also understand the orbits. Thus by now it is clear which theory is best.

This route is probably typical. In the beginning, during the paradigm change, there is real reason for debate. What is the best hypothesis is partially comparing apples and oranges. However, after some time the evidence accumulates and it becomes clear what the best hypothesis is, to the point that a normal person would say the idea is right. A scientist should avoid such formulations and we now know that that sun in not the center of the universe and that classical mechanics has its limitations.

The structure of scientific revolutions gives many more such examples. Thomas Kuhn talks about paradigm changes in the realm of some large revolutions in science. And he calls the work in the period between the revolutions: normal science and puzzle solving. There is some truth to that, a lot of scientific work is figuring out what the consequences of the existing hypothesis are, which you can call puzzle solving. Although in case of a puzzle you know there is a solution and in science part of the job is finding interesting, solvable puzzles.

Furthermore, I would argue that "paradigm changes" also happen at smaller "disruptions" of the scientific network of ideas, down to single articles that make an interesting contribution, a little above “run-of-the-mill”.

For example, when I started using the surrogate data approach to generate 3D cloud fields, I met with quite some opposition in the beginning. People wondered why I did not use the traditional ways: fractal clouds and clouds from dynamical cloud models (LES). In the end, people realized that surrogate clouds were very well suited for empirical studies because you can easily make clouds that were similar to the ones observed. And it likely helped that I found an easier algorithm to generate the surrogate clouds. My clouds are not something that will make the history books, but within the field of 3D radiative transfer and clouds it was a minor revolution.

I am thus not sure whether Kuhn's distinction between normal science (puzzle solving within the paradigm) and paradigm changes really exists. With sufficient domain expertise one probably also sees small "paradigm changes" in normal interesting scientific articles. That would also explain why scientist have often shown quite good intuition for which theory would in the end prove to be best.

During a paradigm change, it may not be clear whether a theory is falsified. Scientists consequently have more criteria to guide them through such rough times. They may have a preference for theories that are easy to falsify, in the sense that with little assumptions they make bold and broad predictions. Theories that are very specific and easy to falsify are more likely right if they are not yet falsified. Scientists also have a preference for elegant, beautiful theories, even if that is poorly defined and subjective. And later, when the dust settles, falsification is not that important, because it is typically quite clear which theory help most in understanding the world.

The truth and the falsity of assertions

Many people and surprisingly many scientists do not like the idea of falsification. It sounds negative to claim that we can only prove an idea wrong. Once, I wrote a research proposal with falsification in the title. A colleague advised me to change this to validation. That was probably good advise, even if the proposal was still rejected.

I feel the issue is not so much one of proving ideas wrong, but as I have repeated so often in this post of making precise statements.

Another reason is that people like the idea of something being right, of something being solid and eternal, that gives some hold in a complex world. Never Ending Audit calls it a PR problem that science can only lose. That is an aspect of science that is hard to explain; long before Popper is was clear that we can never be sure and that science progresses by continually trying to find errors in our current understanding. On the other hand, one should also not exaggerate, as a rule of the thumb, the longer a hypothesis is the general understanding of a scientific field (consensus), the less likely it is that you will find an error in it, especially if you are not a Feynman.

For all science' talk about uncertainties, one should also not forget that science still provides an understanding of the world that is more solid as anything else. How can we explain this paradox?

First of all, I would argue that right and wrong are not very useful categories in science. It is important that scientific hypothesis and methods are precise (falsifiable), appropriate and produce new ideas. A flat Earth is wrong, but often a good assumption (study how a car drives), a spherical Earth is wrong (but used in climate models*), even an elliptical Earth is wrong and the true shape will be measured with more and more accuracy in future. When tackling a problem, it is thus not so much a question whether a hypothesis, method or a dataset is right, but whether it is fit for a specific purpose.

Another good example is the 1-dimensional model of the greenhouse effect. This model is wrong, it should take the geographical variation in surface temperature, humidity and clouds into account, it should model the radiative transfer (frequency) line by line, it even assumes in the solar part that the Earth is flat. However, it is because of these simplifications that it is useful, that it helps us to understand the problem better. Theoretically, you could also make a model that is just as complex as the Earth, but that would not help much in understanding the greenhouse effect. The idea of a model is that it only models the key processes needed to understand a certain question.

Thus I would argue we should reduce the importance we put on being right or wrong and emphasize being useful, interesting, precise and such characteristics.

Secondly, even if a hypothesis is found wrong, this typically does not change our understanding of everyday phenomena much. When classical mechanics was found wrong, no building or bridge collapsed and no artillery shell landed less precise. When quantum mechanics will be found wrong, your smart phone will still work and the internet will keep on buzzing. If we find out that radiative transfer of heat radiation works differently, heat seeking missiles will still work and the greenhouse effect will still exist (maybe it would change some details and values).

If a hypothesis is found wrong, this will normally expand our understanding, make it more general and more precise. Things which are well understood today and which have been studied from a large range of angels, will not suddenly change drastically even if something big like a falsification happens. Fresh snow will still be white.

philosophy of science

I hope that this post makes it clearer to outsiders how science works. For myself, as a scientist, I have the feeling that thinking a little about science in general is helpful when converting an idea into a work of science, especially when you do something that is relatively new or radical. And those are the best things in science, those are the moments for which one becomes a scientist.

And when you do something new, you can typically not copy the methods and article structure of a previous similar study and slightly modify it, you will often have to do something which is very different from what exists. In that case knowing a bit a of philosophy of science is very helpful, it helps you navigate in the dark. But stop before a philosopher starts talking about right and wrong.

* The Earth is elliptical because it spins and the rotational forces are strongest at the equator. If climate models would assume an elliptical Earth, it would also have to model these centrifugal foces. It turns out that these two factors compensate each other and it is a good approximation to assume that the Earth is a sphere and ignore these centrifugal forces.

Related reading

William M. Connolley tries to explain what science is. One of his interesting attempts is seeing science as tinkering to improve the scientific literature, putting the literature in the center and not the scientist (my rather liberal translation of his post). The post is a nice contrast to WUWT "science" that shows plots with obvious truths and does not embed these "new findings" in what is already known. And a contrast to climate ostriches that link directly to plots without the text that explains how the plot was computed and could explain how it changes the understanding in the scientific literature.

A quick ‘n dirty guide to falsifying AGW, where Bart Verheggen argues that not any deviation from common sense is a falsification.

On mismatches between models and observations. "Discrepancies between models and observations [are] .. more subtle than most people realise. Indeed, such discrepancies are the classic way we learn something new."

For the funsies: Newtongate: the final nail in the coffin of Renaissance and Enlightenment ‘thinking’.

Another post by Bart Verheggen fits well to this post. It gives some hints on how to determine which ideas are credible: Who to believe.

A Richard Feynman Primer For Deniers by Ingenious Pursuits. Recommended. And fitting to that a video of Richard Feynman lecturing on PseudoScience.

Recommended books

Joachim Bauer. Das Gedächtnis des Körpers. Wie Beziehungen und Lebensstile unsere gene steuern. (The memory of the body. How relationships and life style influence our genes), ISBN: 987-3-492-24179-3, Piper, Muenchen, Germany, 2010.

Paul Feyerabend. Against method: Outline of an Anarchistic Theory of Knowledge (1975), ISBN 0-391-00381-X,

Thomas Kuhn. The structure of scientific revolutions. Chicago: University of Chicago Press, 1962. ISBN 0-226-45808-3

Bruno Latour. Science in Action. How to Follow Scientists and Engineers Through Society, Harvard University Press, Cambridge Mass., USA, 1987.

Karl Popper. Conjectures and Refutations: The Growth of Scientific Knowledge, 1963, ISBN 0-415-04318-2.