Testing Your Theories Is Not a Matter of "Envy"

By Sean Carroll | April 2, 2012 2:11 pm

Via JenLuc Piquant’s twitter feed, here’s one time I’m not going to stick up for my colleagues in the social sciences: a misguided attempt to cast the search for empirical support as “physics envy.” It’s a New York Times Op-Ed by Kevin Clarke and David Primo, political scientists at the University of Rochester.

There is something rightly labeled “physics envy,” and it is a temptation justly to be resisted: the preference for reducing everything to simple and clean quantitative models whether or not they provide accurate representations of the phenomena under study. The great thing about physics is that we study systems that are so simple that it’s quite useful to invoke highly idealized models, from which fairly accurate quantitative predictions can be extracted. The messy real world of the social sciences doesn’t always give us that luxury. The envy becomes pernicious when we attack a social-science problem by picking a few simple assumptions, and then acting like those assumptions are reality just because the model is so pretty.

However, that’s not what Clarke and Primo are warning against. Their aim is at something altogether different: the idea that theories should be tested empirically! They write,

Many social scientists contend that science has a method, and if you want to be scientific, you should adopt it. The method requires you to devise a theoretical model, deduce a testable hypothesis from the model and then test the hypothesis against the world…

But we believe that this way of thinking is badly mistaken and detrimental to social research. For the sake of everyone who stands to gain from a better knowledge of politics, economics and society, the social sciences need to overcome their inferiority complex, reject hypothetico-deductivism and embrace the fact that they are mature disciplines with no need to emulate other sciences…

Unfortunately, the belief that every theory must have its empirical support (and vice versa) now constrains the kinds of social science projects that are undertaken, alters the trajectory of academic careers and drives graduate training. Rather than attempt to imitate the hard sciences, social scientists would be better off doing what they do best: thinking deeply about what prompts human beings to behave the way they do.

Sorry, but “thinking deeply” doesn’t cut it. People are not especially logical creatures, and we’re just not smart enough to gain true knowledge about the world by the power of reason alone. That’s why empiricism was invented in the first place, and why it’s been so spectacularly successful over the last few centuries.

Clarke and Primo seem to confuse “the need for empirical testing” with “the need for every model proposed to be backed up by data before it gets published.” If they had stuck to rejecting the latter narrow idea, they would have had a decent case. Certainly we physicists don’t require that every model be supported by data before it is published — otherwise my CV (and those of most of my friends) would be a lot shorter! But we all agree that the ultimate test of an idea is a confrontation with data, even if a theory might be too immature for that confrontation to take place just yet.

  • http://drvitelli.typepad.com/providentia Romeo Vitelli

    One of the main reasons psychology gets rapped as not being a “real” science is the active resistance on the part of a lot of psychologists at pushing evidence-based approaches to learning about human behaviour. The actual scientist/practitioners are fighting back but it’s an uphill battle…


  • Pieter

    Time for another Sokal hoax?

  • http://www.factsandotherstubbornthings.blogspot.com Daniel Kuehn

    First – please don’t associate this garbage with social science. This is not how I and my colleagues see the task of studying human social behavior. You seem to be dissociating them – I just want to make sure you do!

    Second – on the point of oversimplifying mathematical models in social science, I hear a lot of natural sciences level this accusation against my own discipline of economics, and more often than not they really don’t know what they’re talking about. Of course, as Einstein noted, everything should be made as simple as possible – but no simpler. I take the “but no simpler” part very seriously. But in my experience, natural scientists and even social scientists in other disciplines are a very, very poor judge of exactly what that entails.

    I think of social science as a sub-field of biology, and a very important sub-field. We are studying the social behavior of the most highly evolved species we know of. The social behavior of this species is so intricate that we further sub-divide into multiple disciplines just to get a grasp of it. And I also reject the hard/soft distinction. The data we have on human social behavior is far more extensive and detailed than the data that other biologists have on the social behavior of other animals, and I’d guess our models are much better at explaining and predicting observed phenomena. Who is the hard science and who is the soft science here? In my mind – the distinction doesn’t make much sense. It’s either science or its not science.

  • Chris

    People are not especially logical creatures, and we’re just not smart enough to gain true knowledge about the world by the power of reason alone.

    However since science is logical, wouldn’t that mean to study illogical beings scientifically would be fruitless? Might as well just make some stuff up.

    I was being sarcastic there. I actually agree with you that social sciences need to be studied scientifically. People are a lot like quantum particles. It’s difficult to predict what one will do, but on the average, there is a well defined expectation value. However unlike quantum mechanics there is an ethics board which makes these studies more difficult. Just think how hard it would be PETH (People for the Ethical Treatment of Hadrons) would be demanding you turn off particle accelerators.

    Huh, there actually is a PETH group.

  • Arun

    A model can be useful without being experimentally verifiable. Aren’t there all kinds of toy quantum field theories that help illustrate some feature of quantum field theory, without any chance of such a toy theory existing in our universe? If they are legitimate in physics, they are legitimate in the social sciences. I took that NYT op-ed to be calling for such simple, clarificatory models to be part of the social sciences repertoire. The examples in the essay – Anthony Downs’ model, or Kenneth Arrow’s theorem on voting – certainly suggest that interpretation.

  • Tyle Stelzig

    I think Clarke and Primo are confused about what it means for a theoretical model to be empirically tested. They suggest two examples of models which they claim are empirically untested but valuable nonetheless. However, the first of these models IS empirically tested, while the second of them is not a theoretical model at all, but rather a theorem.

    First example: Downs’ model predicts that political parties will move towards the center. They do, so his model is worth taking seriously. If it turns out that parties don’t move toward the center, his model isn’t valuable.

    Second example: Arrow uses logic to demonstrate that no voting system can satisfy all of a certain set of criteria. Since this is a conclusion deduced by logic, it need not be empirically tested. But this is not an example of a model in the social sciences which doesn’t need to be tested, because it isn’t a *model* at all! Rather, it’s merely a theorem. We have those in physics, too (e.g. Noether’s theorem).

    Clark and Primo also say the following, in the course of arguing that theories need not make predictions: “the test of a map lies not in arbitrarily checking random points but in whether people find it useful to get somewhere”. But use of a map for successful navigation is a form of empirical confirmation! As with the Downs example, they seem to be confused about what empirical testing means.

    So, if the goal is to argue that models in the social sciences need not be empirically testable to have value, they have failed.

    Now, it’s true that simplified models might have value in helping us understand how our models work, even if they are too simplified to line up with reality at all. If this is all that they were intending to argue, the best that I can say is that the article was very unclear (misleading framing of issue, misleading discussion, misleading examples, misleading title, and failure to mention that this was their point, for starters).

    For completeness, I’ll mention that they also might intend to endorse two other propositions. First, that social scientists can do useful things other than create models (e.g. make theorems, like Arrow). Second, that data can be useful even without a theory. Both of these propositions seem quite uncontroversial, of course, and would not in any way be called into question by a healthy envy of physicists.

  • Pingback: Testing Your Theories Is Not a Matter of “Envy” | Cosmic Variance | Discover Magazine « Secularity()

  • Low Math, Meekly Interacting

    There’s an easy fix for this: doff the “science” conceit and call it philosophy. No one accuses a philosopher of being a charlatan if he or she spends all his or her time thinking in illuminatingly deep ways, unfettered by the constraints of empiricism. And they needn’t envy anyone, because no one requires them to be practical, just clever.

  • http://brianfies.blogspot.com Brian Fies

    Right. Clark and Primo’s approach might produce some interesting insights and results, and they could productively develop and apply it. Good for them; more power to ’em, sincerely. Just don’t call it “science.”

  • DMcK

    “That’s why empiricism was invented in the first place, and why it’s been so spectacularly successful over the last few centuries.”

    David Deutsch’s recent book — annoying, challenging, and thought-provoking — calls this “spectacular success” into question. I believe his claim is that empiricism in its strict sense (going back to the empiricist British philosophers) fails, and that the “power of reason” is at least an equal partner, and sometimes is primary. Maybe I’m not disagreeing with you (Sean).

  • Arun

    Here is some Krugman, talking about the value of simple ideas in economics.

    You could argue that modern economics really began with David Hume’s Of the Balance of Trade, whose core is a gloriously clear thought experiment:……And Karl Smith is right: no way could Hume have published such a thing in a modern journal.
    End quote.

    Primo and Clarke:
    Many social scientists contend that science has a method, and if you want to be scientific, you should adopt it. The method requires you to devise a theoretical model, deduce a testable hypothesis from the model and then test the hypothesis against the world. If the hypothesis is confirmed, the theoretical model holds; if the hypothesis is not confirmed, the theoretical model does not hold. If your discipline does not operate by this method — known as hypothetico-deductivism — then in the minds of many, it’s not scientific……For the sake of everyone who stands to gain from a better knowledge of politics, economics and society, the social sciences need to overcome their inferiority complex, reject hypothetico-deductivism and embrace the fact that they are mature disciplines with no need to emulate other sciences.

    End quote.

    I see them as arguing the very same thing.

  • Arun

    More Krugman:

    “Simple conceptual models can also help convince you NOT to believe in economic ideas. Real business cycle theory says that economic fluctuations are the result of technological shocks, amplified by intertemporal labor substitution. My version: think of a farmer who faces sunny and rainy days. On rainy days his labor won’t be as productive as on sunny days; this effect on his output is amplified by his rational decision to stay in bed on rainy days and work extra hard when the sun shines. I think this gets at the essence of the concept; it also makes you wonder, is this really, really what you think happens in recessions?”
    End quote.

    It would seem that empirical data is not sufficient to dismiss Real Business Cycle Theory, otherwise why would Krugman propose his analogy?

  • Arun

    I also find it ironic that people who insist superstring theory is mainstream physics insist that theory must meet the test of empirical data. Forty years and counting and that has not yet happened…..

  • Mark Weitzman

    If a discipline has the word science in it is usually is not a science – computer science, social science, information science – that does not mean it is not worthwhile it simply means it isn’t science.

  • BBBShrewHarpy

    I read that editorial, but assumed it was an April’s Fool prank. Do you think it was not?

  • http://backreaction.blogspot.com/ Bee

    I read that too, thought the same thing, but then wasn’t sure if it wasn’t a joke.

  • Benish

    The gist of Sean’s post is admirable, as it reflects the standard lofty ideals of science. It is probably impossible to be reminded too often that,

    “…’thinking deeply’ doesn’t cut it. People are not especially logical creatures, and we’re just not smart enough to gain true knowledge about the world by the power of reason alone.”

    It is therefore pertinent to point out a huge domain of gravitational physics where deep thinking holds sway and empirical evidence is absent. The cliche is that General Relativity has been quite thoroughly tested throughout the Solar System and that, to quote Stephen Hawking: “We already know the laws that govern the behavior of matter under all but the most extreme conditions.” By extreme is meant extreme velocities and extremely strong gravitational fields.

    The domain I have alluded to is one where even Newtonian gravity has not been tested. Given a uniformly dense sphere with a hole through the center, a test object is dropped into the hole. What happens? By “thinking deeply” textbooks and professors answer: simple harmonic motion. But no observational data is ever cited—because we have none.

    The media is barraged with science shows and articles about black holes and wormholes. All the while we have no empirical evidence pertaining to an extremely simple (in principle) gravitational experiment involving an ordinary hole. Aside from healthy curiosity and a humble desire to live up to the ideals of science, interested readers may like to consider other reasons why it would be a good idea to fill in the gap in our empirical knowledge of gravity:


  • jonathan hopkin

    Daniel, that’s all very well, but there has got to be a fundamental difference between the social sciences and biology. After all, when did a cat’s behaviour ever change because of a research paper about cats’ behaviour? The same ability to generate knowledge and change the way we see the world is precisely what makes human behaviour so complex. This isn’t to see we shouldn’t study it rigorously – it is to say we shouldn’t expect to get the kind of replicable results you get in biology or physics. Ever.

  • Tyle Stelzig

    Hi Arun,

    I agree with you that simple models might sometimes be very useful in the social sciences. Indeed, it would be silly to undertake construction of detailed quantitative models before possessing a good qualitative understanding. I think Krugman’s articles make this case well.

    However, this is NOT in conflict with the scientific method. Quite the contrary. The models that Krugman talks about are of interest only because they help us understand the empirical facts.

    If all that Clarke and Primo mean to argue is that simple qualitative models can help economists get on the right track, and that journals are too reluctant to publish such models, they could very well be right. But in that case, they would not be arguing against “hypothetico-deductivism” – such models are of interest because they help us understand the facts – so it would be misleading of them to claim that they are.

  • Orsi Selymes

    there is difference bw understanding how things may occur (a general theoretical framework) and how they have occurred (using that framework for analysis)
    a grounded theoretical framework builds from real data, and it thus has roots in empirical “evidence”
    empirical testing comes down to a testing of fit, and relevance , when it comes to testing a framework empirically
    it is not about qualitative and quantitative. neither deductivity or inductivity..
    empirical research can be generating a hypothesis or it can be testing one
    for hypothesis generating, you need to use and compare (test) empirical data
    for hypothesis testing, you need to generate empirical data
    in theoretical physics, or biology, or archiology, or geology, hypothesis generating has been the regular approach
    in social sciences, the trend is a little different
    and then, in physics, mathematics, that’s different, too

  • Pingback: Social scientists argue that social scientists shouldn’t do science | The Finch and Pea()

  • http://www.pipeline.com/~lenornst/index.html Len Ornstein

    The basic issue here is what makes the ‘explanations’ provided by various disciplines of use for human survival.

    This thread reveals that there’s lots of confusion on this matter among relatively sophisticated people. No wonder policy makers fail to make anywhere near optimal use of science and engineering in running governments and industries.

    To try to address just this problem, I’ve drafted an essay:

    “The Skeptical Scientific Mind-Set in the Spectrum of Belief: It’s about models of ‘reality’ – and the unavoidable incompleteness of evidence, for – or against – any model or fact.”


  • Benish

    In Len Ornstein’s essay, it is written:

    “no scientific model can be either absolutely falsified – or absolutely proven.”

    I think most science-oriented persons, including myself, regard this approach as increasing the kind of confusion that Ornstein professes to diminish.

    The example that I mentioned in my previous post is a case in point. Both Newton and Einstein predict that a test object dropped into a hole through a uniformly dense sphere will oscillate back and forth in the hole. Since this prediction has never been tested, for the sake of argument we may suppose that nothing like this happens at all. Suppose instead that the test object does not even pass the center. If this is what actually happens, then it would absolutely falsify and disprove Newton’s and Einstein’s theories; they would not survive what Faraday called the “Ithuriel spear of experiment.” I don’t think any physicist would disagree with this. If the test object does not pass the center, it means we need a new theory of gravity.

    This brings us back to the question as to why physicists see no need to test this particular prediction. Previously I suggested that it was due to the kind of “deep thinking” used by Clarke and Primo that Carroll is critical of. But physicists sometimes draw conclusions lacking empirical support without even troubling to think deeply. They also have a kind of “reflex reaction” for deciding whether something is true or not.

    For example, if a prediction violated the energy conservation law, the model it is based on is likely to be rejected without further thought. The track record of this law tends to support this judgment. A more cautious, strictly empirical approach, however, would be to regard the new prediction as one more opportunity to test the law.

    With regard to the experiment I’ve proposed, physicists are neither cautious nor empirical. They fail to see the need to carry it out because the very idea does not get past their reflexes. This characteristic of physical scientists may ultimately be even more detrimental to science than the avoidance of empirical facts via deep thinking.

  • Arun

    Mona Chalabi, in the New York Times, is worth a read:



    “An economist must be a “mathematician, historian, statesman, philosopher, in some degree.” So, promisingly, begins the sixth edition of “Macroeconomics” by Greg Mankiw, a key reading for my introductory economics class at the University of Edinburgh (and still a key reading six years on). Unfortunately, the book, like the course that prescribed it, delivered on only one of those claims: to be a mathematician.

    What began as eloquent and logical graphs and formulas quickly spiraled out of control and I soon found myself reading that “economics is not only a social science, it is a genuine science. Like the physical sciences” and that financial crises can be predicted by using the following formula:…”

  • Sesh

    You could equally interpret those comments as meaning “we can think about ideas in the social sciences that cannot conceivably be verified or falsified by empirical data in the foreseeable future; such topics are valid avenues of research, even though ‘thinking deeply’ is the only way to distinguish between ideas in such fields.”

    Phrased like this it does not sound very different to a description of some research in string theory, or indeed issues of why the universe consists of something rather than nothing (a topic you have been championing recently). Why is one ‘science’ and the other ‘science envy’?

  • Arun

    In reply to Tyle Stelzig (#19):

    I read Clarke and Primo to mean that anything that doesn’t fit the “hypothetico-deductivism” approach will not be published, funded or even thought about. Without Clarke and Primo here, I suppose we cannot get clarification on that.

  • Low Math, Meekly Interacting

    I’m not sure I understand Chalabi’s point entirely. I don’t think economics suffers from an over-reliance on math necessarily (though that may be a symptom of the disease). I think it suffers from a pervasive ethos of prescription instead of description, and if the world doesn’t conform to some-or-other beautiful model, well, so much worse for the world, then.

  • Arun

    Clarke and Primo have a book “A Model Discipline: Political Science and the Logic of Representations”.


    The publisher’s blurb reads:

    “In A Model Discipline, Kevin A. Clarke and David M. Primo turn a critical eye to the methodological approach that dominates modern political science. Clarke and Primo contend that the field’s emphasis on model testing has led to a distortion of both the modeling process and the art of data analysis and cannot be logically justified. The authors argue that models should be seen as “objects” and thus regarded as neither true nor false. Models should instead be evaluated for their usefulness for a particular purpose. Divided into two parts, the book first establishes that current practice is not philosophy-free and rests on a number of questionable assumptions. The second part focuses on the different ways that theoretical and statistical models can be useful, and closes with a defensible justification for integrating theoretical and statistical models. A novel work of methodology, A Model Discipline offers a new perspective on long-held assumptions about the way research in the social sciences should be conducted.”

    There is a review by a reader, worth reading:

    A quote from that review: “Clarke and Primo tackle head-on a number of thorny philosophy-of-science questions that are usually ignored by political scientists and persuasively argue that the community of political scientists has committed itself to an untenable understanding of models and their purposes. Rather than merely criticizing, they offer a viable alternative to the hypothesis-testing that now dominates the field: namely, we should think of models as objects like maps, and evaluate them not as to their truth (since objects are neither true nor false) but as to their usefulness.”

    “On their discussion of usefulness, Primo and Clarke earn two cheers. In Chapter Four, they explain how theoretical models could play a variety of constructive roles: as foundations for more models, as a way of organizing existing empirical generalizations, as a means of investigating causal mechanisms, or as devices for forecasting (which they note are relatively rare in political science outside of election studies). All of this is helpful–but I was somewhat disappointed that in their exploration of the purposes these models can play, the authors almost entirely eschew the question: “useful to whom?””

    End quote.

    The first few pages of the book are also available, and yes, it sounds like Primo and Clarke argue that the analog in political science of the Chern-Simons model in physics would be mishandled in current political science practice.

  • Arun

    Hypothetico-Deductivism is Hopeless
    Clark Glymour
    Philosophy of Science, Vol 47, No 2 (Jun 1980) 322-325

  • Arun

    Ouch, and that leads further into the morass:
    The Raven Paradox

  • http://www.pipeline.com/~lenornst/index.html Len Ornstein

    Objects are dealt with through their definitions. If arbitrary, as part of an axiomatic deductive system, they are assumed to be ‘true’.

    More typically, definitions of objects represent shared descriptions that fit fairly well with the empirical observations made within a discipline. In this case, they are used with a grain of salt, because of the inevitable residual uncertainty related to all inductive reasoning about observables, and this is just another face of “no scientific model can be either absolutely falsified – or absolutely proven.”

    The Statement “objects are neither true nor false”, therefore is thoroughly misleading with respect to this discussion.

    The source of much of the confusion among some scientists, and most others, in deciding where scientific ‘facts’ and models belong within the spectrum of beliefs, follows from too common, glib ASSERTIONS about ’empirical truths’ associated with the most well established ‘theories’ and ‘laws’ of science. Such ‘truths’ are always subject to possible (even probable) revision.

    Some economists and politicians – without understanding that even these scientific gems are at best, provisional – may ‘envy’ science because they wish they could make such assertions within their disciplines with equal confidence. They don’t appreciate that unqualified assertions of scientific ‘truths’ are hardly distinguishable from some assertions of religious dogma.

  • vel

    “Rather than attempt to imitate the hard sciences, social scientists would be better off doing what they do best: thinking deeply about what prompts human beings to behave the way they do.”

    aka religion and we know how well that works considering that one religion Christianity, has some 10’s of thousands of sects, each which claims to have thought “deeply” and who claim to have the only ‘right’ answer.

  • Sesh

    Re my post number 25: I apologise for apparently somehow completely missing Sean’s final paragraph. If I’d seen it before I posted I wouldn’t have bothered.

  • http://noahpinionblog.blogspot.com Noah Smith

    I absolutely love this post.

    Also…does this mean Sean thinks string theory is not a real scientific theory?

  • Dan Kärreman

    I don’t share Noah Smith’s love, but I share his question. And also, of course, dark matter. And all the other dark stuff. And anti-materia. We can go on.

    I’m not sure which of the many versions of empiricism Sean subscribes to, but is clear that the model criticized in the op-ed, hypothetico-deductivism, has an abysmal track record in social science. I challenge Sean, or anyone, to mention one important discovery about the social world created by this method. And no, important discoveries about the physical world don’t count.

    That is not to say that empirical matters don’t matter. But it is to say that social science needs to get rid of the physics envy.

  • http://cornercases.wordpress.com/ W.P. McNeill

    Among empirically-minded linguists seeking to criticize less empirically-minded linguists, the phrase “physics envy” has a different meaning than the way Clarke and Primo use it. For us, “physics envy” is not taking a rigorous empirical model an applying it inappropriately to fuzzy human data. Instead it borrowing unrigorous mathematical ideas to cover up for the fact that you don’t want to build an empirical model. This cartoon sums it up nicely.

    Highly empirical computational linguistics people like myself end up having more in common with the “soft science” branches of linguistics–the parts that bleed over into psychology and sociology–than the apparently “hard science” branch of syntax (i.e. the stuff Chomsky does). The working style for the former is often to perform experiments about human language (e.g. reaction time psychological experiments with linguistic stimuli, or sociological surveys of who has what accent). These are often empirically rigorous. (Where part of rigorous means adding the appropriate caveats about human behavior being a particularly difficult thing to quantify.) Many theories of syntax, on the other hand, look at first glance like mathematics papers–there are theorems and Greek letters all over the place, and the only data is anecdotal–but lack the axiomatic rigor to be called math.

    So we all agree that “physics envy” is bad, but different practitioners may have different ideas of what makes it so.

  • Benish

    Since the seven maintainers of this blog are all physicists and astrophysicists, and the subject of this thread is the importance of “Testing Your Theories,” I will try once more to evoke a response concerning a large area where physicists feign knowledge instead of testing their theories.

    Ironically, but not surprisingly, psychological and sociological influences bear heavily on the persistence of this practice. It is not surprising because physicists are fallible human beings, not enviable gods. As such, physicists are proud of their profession and desire to cast it in a positive light. An expression of this “soft” underbelly of physics was conveyed by the physicist Daniel Kennefick in response to his experience and interactions with the community of gravitational wave theorists. Kennefick wrote:

    “There is a preference not to remember or not to overstress the significance of something which may be seen as vaguely disreputable to the field. It is a characteristic aspect of physics that to pose a problem or a question may, in itself, be taken as a sign of bad character.”

    Recall that the question that I’ve posed here and elsewhere (posts #17 and #23) concerns the result of testing the commonly discussed gravitational interior solution (Newtonian or relativistic). What happens when a test object is dropped into a hole through the center of a larger massive body?

    We don’t know the answer because empirical evidence is completely lacking. “…‘Thinking deeply’ doesn’t cut it.” Nor does the “reflexive” response of deferring to the energy conservation law, because this is one area where energy conservation has not been tested. Carroll’s ostensible adherence to empiricism implies that he should be eager to resolve this matter with an experimental test.

    So how do we explain the fact that he and other physicists choose to ignore this area of their ignorance? I think the answer resides in a parallel between Kennefick’s above characterization of the sociology of physics and the case of Galileo, when he was up against the Church. One of the latter’s key advocates was Roberto Bellarmino, the heretic hunter, who referred to those who would question scriptures as “dangerous and in some cases embarrassing to the Company.”

    The standard Newtonian and relativistic interior solutions have never been tested, but they are as Gospel. To question them is a “sign of bad character.” It would be embarrassing (to the Company) for a physicist to admit the negligence of leaving this stone empirically unturned. So they carry on, only pretending to know what they would find if they would bother to look.

    As “people,” Sean Carroll and his colleagues are not—by his own admission—“especially logical creatures.” Of course this is also true of myself. But in the present case, I am the one urging that we test our logic and our theories by inquiring of Nature. Whereas Carroll and his colleagues, by ignoring my posts, seem to be relinquishing their ideals, i.e., the need to test their theories.

    Drop a test object into the hole. What happens? Theories and logic won’t help here. Professor Carroll, what happens? Why don’t we find out?

  • Tyle Stelzig

    Benish: Physicists have access only to finite resources, and therefore have to choose which experiments are most likely to lead to surprises. The experiment you describe simply isn’t one of these – physicists are fairly confident it would go the way the expect, having done similar (but yes, not identical) experiments many times. So it isn’t worth the opportunity cost. Better to use the money looking at things we know less about.

    But no one is stopping *you* from doing this experiment. If you don’t share physicists’ confidence that existing models of gravity would correctly describe your ball-in-hole scenario, then by all means, try it out! If you stumble upon a surprise, physicists will be grateful and excited.

  • Pingback: The Problem of Instructions | Cosmic Variance | Discover Magazine()

  • http://juanrga.com Juan Ramón González Álvarez

    Sean Carroll wrote:

    There is something rightly labeled “physics envy,” and it is a temptation justly to be resisted: the preference for reducing everything to simple and clean quantitative models whether or not they provide accurate representations of the phenomena under study. The great thing about physics is that we study systems that are so simple that it’s quite useful to invoke highly idealized models, from which fairly accurate quantitative predictions can be extracted. The messy real world of the social sciences doesn’t always give us that luxury. The envy becomes pernicious when we attack a social-science problem by picking a few simple assumptions, and then acting like those assumptions are reality just because the model is so pretty.

    Excellent point! But it is not needed to go to social sciences to discover that real world is not so simple as physicists believe. Precisely, the new encyclopedic article The hierarchical structure of matter contains a discussion of Feynman’s incorrect approach to the molecular structure of water.

    As many physicists, Feynman believed that water is only composed of molecules of H2O. This is a common mistake that he shared among many of his readers. As any general chemistry textbook explains –in the chapter dealing with atomic and nuclear issues– molecules are, in a strict sense, not of the same sort. Instead they are built up of different isotopes of hydrogen and oxygen in various combinations (e.g. water also contains molecules of D2O), and such a difference is important to some chemical problems. A few chapters later, chemists learn that the H2O molecules are not inert, but dissociate into H2O+ and OH− ions, and that the concentration of all three components in water varies with temperature and pressure. The concentration of those ions governs the chemical properties of water. Another chapter that deals with electrical properties of the same pure water will explain to students that molecules are not isolated blocks but dynamically forming various entities through aggregation and disaggregation, governed by various forms of changing interactions, such as covalent bonding, hydrogen bonding, van der Waals interactions, or electrostatic and spin-spin interaction, depending on the level of molecular description. The atomic-molecular picture is still more complex, but the level of detail presented here is enough to understand that, contrary to physicists’ belief, water is not made only of H2O molecules.

  • http://juanrga.com Juan Ramón González Álvarez

    @40. Juan Ramón González Álvarez:

    A typo: “A few chapters later, chemists learn that the H2O molecules are not inert, but dissociate into H3O+ and OH− ions”.

    @17. Benish:

    Completely agree on “The media is barraged with science shows and articles about black holes and wormholes.” In my opinion, the major problem is that the academic literature on black holes is open to many fundamental objections. See last posts on



    The section 8 of the paper “Non-redundant and natural variables definition of heat valid for open systems” (cited therein) corrects many nonsensical claims done in a recent review on black hole ‘thermodynamics’ by Wald.

  • Benish

    Tyle Stelzig:

    Physicists’ confidence in the standard prediction for the interior solution experiment can be seen as misplaced in various ways. Before stating a few of them I would urge that the ideals of science are sufficient reason to insist that the experiment be done. Confidence is no replacement for empirical evidence. Furthermore, the lack of resources argument is hard to swallow in light of the seven, eight, and nine digit budgets of gravity experiments like LIGO, LISA and others. The experiment I’ve proposed would cost a small fraction of these.

    What we know about gravity is almost entirely based on observations over the surfaces of large gravitating bodies. On a graph that plots the maximum velocity due to gravity as between two bodies, the curve is reasonably complete from the surface to infinity. But from near the surface inward, we know virtually nothing. There is no “similar” experiment to the one I’ve proposed because we’ve never seen objects fall from near the surface of any large body to near its center.

    If we think of the Earth as our primary source of empirical gravitational knowledge, then what we know derives essentially from the horizon upward. Failing to extend our knowledge inward, means in effect, that we have failed to explore the most ponderous half of the gravitational universe. Physicists’ confidence is based on mathematical extrapolation from the explored exterior to the unexplored interior. The late Herman Bondi poignantly expressed the serious inadequacy of such partial knowledge:

    “It is a dangerous habit of the human mind to generalize and to extrapolate without noticing that it is doing so. The physicist should therefore attempt to counter this habit by unceasing vigilance in order to detect any such extrapolation. Most of the great advances in physics have been concerned with showing up the fallacy of such extrapolations, which were supposed to be so self-evident that they were not considered hypotheses. These extrapolations constitute a far greater danger to the progress of physics than so-called speculation.”

    This of course echoes Carroll’s remarks about the need for empirical evidence to replace all-too-often fallible human logic. Lip service to the ideals of science is extremely common. The idea is simply to test what we think we already know and to look where we have not yet looked. The need to do this even for simple problems whose results are unthinkingly taken for granted is amplified in the present case because of our gross ignorance of the ultimate nature of gravity. Nobody has ever figured out what matter supposedly does to produce gravitational attraction. What does matter do to warp spacetime? In general relativity the standard prediction for the interior experiment is based on the assumption (Schwarzschild interior solution) that the rate of a clock at the center of the body is a minimum. What does matter do to make the central clock tick slow? Intuitively it makes more sense (by symmetry) that its rate should be a maximum (in which case the falling body would not pass the center).

    Nobody has figured out how gravity relates to the other forces. Some physicists have the impression that after many decades of effort with virtually no progress, we are badly stuck. Given this situation, isn’t it obvious that the most ponderous half of the gravitational universe ought not to be presumed known before being checked?

    Deeply ingrained acceptance of standard idea is the source of your confidence. Please consider the possibility that many of these ideas are quite wrong. Why not generate interest in doing this experiment if only to finally confirm that its result is not a big surprise?

  • Anonymous

    Anybody notice that this was published on April 1st?

  • mete

    This Blog is indeed great!

    For economics, people have been discussing these issues—that is whether economics is science, or not, plus some comparisons with physics, or other disciplines— for years I can say.

    The econ literature has great discussions on that too, for example this article by economist Hal Varian from Berkeley:


    or by Greg Mankiw from Harvard-econ:


    also here


    and the blog of S. Williamson from Washington University – econ


    with a particular focus on financial markets or asset price dynamics I think I should say, here is a debate, a manifest, on “Physics vs Finance”:


    The list goes on..

    Frankly speaking, I do not know the answer, yet, whether economics is science or not. I definitely disagree though by only looking at the dictionary, we can tell if a discipline is science. Dictionaries may give the definition of what science is but it cannot answer e.g. how do we obtain scientific knowledge? Why? What are the limits of the scientific knowledge, Should we take an empiricist way or positivist way? Subjective knowledge or objective knowledge? etc..

    I think economics is a quite interesting discipline to study though: A part of it is applied math (e.g. game theory, e.g. Nash), a part of it is engineering (operations research problems such as kidney exchange, mechanism design, industrial organization, production planning), a part of is sociology, history, psychology (neuroeconomics, behavioral finance), etc…

    Just for curiosity, Dr. Carroll, as a physicist, do you think economics is science? I took liberty to directly as you because in the economics research community I can say, there are physicists who think that economics is a science, and there are some others who think otherwise. I was just wondering what you think.


  • Mete

    For those who might be interested in, another great discussion on “economics vs. physics is here:



  • Torbjörn Larsson, OM


    With the advent of standard cosmology after WMAP, general relativity has been successfully tested with precision (I think) and out to distances (I am certain) as never before. So no one entertains the notion that it can be wrong.

    However, the theory may be incorrect; it is certainly known to have a domain of validity as it is an effective theory. The means to show it so is for you to do the experiment you envision. Especially if it would “would cost a small fraction” of other experiments that scientists deem important.

  • Matt

    I think Clarke and Primo are mostly arguing against a trend in their own field (political science) where the major journals refuse to publish a theoretical model without some sort of empirical “test” of the model *in the same paper.* Never mind whether the test actually captures key aspects of the model or not or that the data may not be yet available to do a proper analysis.

  • Misha

    I can’t believe that came out of Rochester. Theirs is one of the most famously empricist political science departments.

  • http://www.zinemin.wordpress.com zinemin

    “People are not especially logical creatures, and we’re just not smart enough to gain true knowledge about the world by the power of reason alone.”

    What about topology? Mathematics in general? String theory? Multiverses?

    I think it is totally normal for all fields of sciences to sometimes have theories without being able to test them. Such theories can be extremely useful and stimulating for the field.

    In social sciences, which partially concerns the workings of society as a whole, experiments are often impossible.
    For example, someone might have the hypothesis that it would be good for society women should be allowed to vote. Others will argue that this will destabilize the society and cause problems.

    Isn’t it completely fine to have this hypothesis, think deeply about it, and argue for and against it, without being able to test it at first? And I don’t think we have actually tested it. We decided to let women vote. The society is now what it is. We can’t compare to a case where we women are not voting.

    I also think the hypothesis that they actually give in the article is very interesting, namely that competing political candidates will converge to similar views. I think this is a very useful theory to have when thinking about politics, but it will be quite hard to test.

  • Benish

    Re: #46—Torbjorn Larsson,

    Notice that this domain of validity is only for the region from near the surfaces of gravitating bodies, outward. The motion of bodies due to gravity in the domain from near the surfaces, inward is completely unexplored. If claims of the validity of general relativity—and even Newton’s theory—were more explicit (honest) about this, perhaps someone would be inspired to fill the gap.

    It is the job of physicists to fill this domain with empirical evidence, not mine. Filling it with “deep thinking,” “confidence,” and exclusion of “the notion that [Einstein] could be wrong” is the same kind of sin that this thread so meekly holds against social scientists.

    Physicists: Please test your theories. Back up your words with actions. (And don’t quibble about or make excuses regarding money. I’ve spent plenty trying to do the experiment myself in a makeshift laboratory. An institution grade laboratory is needed. But the cost would be in the 5-6 digit range, not the 7-9 digit range of big science.)

    Finally, following my post #42, for a short while there was another post that briefly echoed a liking for what I wrote. Why was that post removed?


Discover's Newsletter

Sign up to get the latest science news delivered weekly right to your inbox!

Cosmic Variance

Random samplings from a universe of ideas.

About Sean Carroll

Sean Carroll is a Senior Research Associate in the Department of Physics at the California Institute of Technology. His research interests include theoretical aspects of cosmology, field theory, and gravitation. His most recent book is The Particle at the End of the Universe, about the Large Hadron Collider and the search for the Higgs boson. Here are some of his favorite blog posts, home page, and email: carroll [at] cosmicvariance.com .


See More

Collapse bottom bar