Tuesday, April 16, 2013

What if all those times really were different?


Hopefully you've been following the tussle between the famed Reinhart and Rogoff and the researchers who tried to replicate their results (Hendon, Ash, and Pollin). This post isn't really about that, but here's a quick summary: RR claim that after a country's debt hits 90% of GDP  or more, it grows much more slowly. HAP take RR's data and find that RR have done some dubious things with it, including A) randomly excluding countries from the sample, B) weighting outcomes by "growth period" instead of by year, and C) making an Excel error. The Excel error is not a big deal, just a "gotcha", but the other two things are pretty important, especially the weird weighting. When the data are weighted by year and the excluded countries are allowed back in the sample, HAP find that:

1) There is no "sudden deceleration" or "structural break" in growth when debt hits 90% of GDP, i.e. there is nothing special about 90% as RR often claim, and

2) Countries with more debt may tend to grow more slowly, but the effect is considerably smaller than RR claim, and not statistically significant.

RR respond by email and defend themselves, focusing mainly on the first part of the latter result (i.e. that higher-debt countries grow more slowly), largely ignoring the other findings, and not really defending the oddities of their approach. In particular, the notion that 90% is some kind of special threshold - which Reinhart and Rogoff have repeated time and again when making the public case for austerity -  appears to lack material support, but RR's email response repeats the 90% level but doesn't really address its disappearance in HAP's results.

Hopefully a more comprehensive response to HAP's findings is forthcoming. (Update: Here is that response. Still pretty unsatisfying.)

Also, many smart people point out that even the weak result that slower-growing countries may tend to have (slightly) higher debt levels is not necessarily a causal relationship - it may be that slow growth makes countries borrow more in relation to GDP, not vice versa. Reinhart and Rogoff claim, in their email response, that they never presented this relationship as causal in their book, This Time Is Different, or in their papers. However, in their public op-eds, they clearly and unambiguously assert that debt causes slow growth rather than vice versa.

(In fact, as an aside, I don't even think that RR's usual critics go far enough in challenging the debt-growth result. RR's cross-country sample does not control for a country's level of development. So if debt/GDP levels tend to grow over time, and if countries converge a la the Solow growth model, then slower growth and higher debt could have a purely benign non-causal relationship.)

In short, RR appear to be doing everything they can to imply that correlation = causation, while never seriously addressing the possibility that it might not.

But actually, this post is not about that dispute. This post is about Reinhart and Rogoff's book, This Time Is Different.

The book basically presents a whole lot of features of financial crises across time and space, and shows how these features are similar. The authors thus try to draw some general conclusions about the "anatomy" of a crisis. The book has been hailed as an alternative to the sort of formal modeling done by most academic macroeconomists, and I agree that the naturalistic approach adds value. I just think its limitations are too rarely recognized. Naturalistic observations can be all wrong.

Reinhart and Rogoff carefully document a number of apparent similarities between financial crises. The implication is that financial crises are all inherently similar - that even if we don't know exactly what causes them, they must have a common cause because they look so similar. In other words, this time isn't different.

But that's not really valid! Similar-looking things can have a bunch of different underlying causes. Wars might be fought in similar fashion, but the causes of the wars might vary wildly - religious fervor for one war, desire for grazing land for another. Communicable diseases may all involve fever and weakness, but some are bacterial and some are viral.

Similarly, financial crises might be different animals that simply look the same. Some might be caused by domestic asset bubbles, others by currency pegs and foreign capital flows. Some might involve borrowing in external currencies, others mostly domestic borrowing. Some might be triggered by wars or political instability, others by the collapse of unsustainable growth models, others by overly complex financial systems. 

Bottom line: If you just pick out the similar features, you will bias yourself toward concluding in favor of structural underlying similarity, where no such conclusion is warranted!

And here's my second problem with the methodology of This Time Is Different: Documenting similarities automatically biases oneself (and one's readers) toward drawing the kind of inappropriate causal conclusions that many draw from RR's finding on debt and growth. If you focus mainly on the fact that all deadly communicable diseases involve fevers, you will probably try to treat them by dipping sick people's heads in buckets of ice. That's not going to work!

"Similar symptoms, disparate causes." When evaluating the history of financial crises, we should constantly keep this phrase in mind. This is the alternative hypothesis; maybe each time is different. Proving a common cause, or even a common structure, requires more than simply tabulating lists of similarities.

Now a disclaimer: I strongly suspect that RR are onto something, and that certain causal features of financial crises really do crop up again and again across time and space. But I think that books like This Time Is Different are merely jumping-off points for an investigation of that hypothesis; they do not constitute any kind of proof. Naturalism is where understanding of the world begins, but not where it ends.

36 comments:

  1. I agree that the Reinhardt/Rogoff argument is only marginally convincing, although certainly interesting. Much more persuasive (to me) is Gary Gorton's Misunderstanding Financial Crises.
    http://www.amazon.com/Misunderstanding-Financial-Crises-Dont-Coming/dp/019992290X

    Gorton argues that all financial crises do have a unifying cause which is loss of confidence in bank-created money of one form or another, triggering a bank run (though the run itself may be prevented by government action). Through history, the kind of money changes forms, from bank deposits to things like repo agreements in the latest crisis, but the dynamic is similar. There's a large increase in credit before a crisis, as governments or banks or someone creates money through credit devices. Sometimes there may be a real estate boom, sometimes not, but a sharp rise in created money is always key.

    The argument seems quite strong to me, although I haven't yet quite finished the book.

    ReplyDelete
    Replies
    1. I think this is indeed the unifying factor of financial crisis but I would have phrased it differently. I think a 'better' (imho) phrasing could be 'financial crisis are preceded by an increase in leverage within the financial or para-financial system and, from that, all kind of ills always result.

      I don't know if this is deep or tautological (running more risk is risky) but I think this reflect things better than the idea of a 'run', which indeed can be prevented. Or not occur, as in our present crisis, where people's trust in FDIC and the EU equivalent remains fairly high, despite all... That said, you could argue a run did occur in the shadow banking system... Still, leverage seems a better descriptor.

      Delete
    2. Mostly true.

      But it is possible to have a crash that is the result of some real shock (harvest failure), or a massive terms of trade shock (e.g. oil crisis).

      Delete
    3. Hi, reason

      We were talking financial crises. There are indeed all kind of other shocks, notably on the supply side, as you mentioned.

      I kind of touch on that in the following blogpost: http://theredbanker.blogspot.com/2013/03/inflation-or-deflation-refreshing-macro.html

      Delete
  2. Economics = the art of mendacity, especially the art of cherry picking data so as to support a pre-selected outcome

    ReplyDelete
  3. Noah, this is a bit of a straw man.

    I assume you're referring mostly to chapter 14 from TTID, "The Aftermath of Financial Crises"?

    This is the major part of the book where they come out with broad statements about what the effects of financial crises are, on average. But in this chapter they are actually only dealing with "systemic banking crises" (p. 223) and not with the whole range of crises: currency, inflation, default, etc.

    It's unsatisfying that they don't put out more data on the average aftermaths of all these other financial crises too, but oh well.

    - On the point of your post though, I completely agree that we should be thinking about the effects and aftermaths of different _kinds_ of financial crises. They may not all be different, but the effects of a 'run of the mill recession' (~2001) are certainly different from a major banking crisis (ie, 2008). RR's stats on the aftermath of banking crises (p. 224) are a pretty sobering starting point for thinking about the post-2008 recovery though...

    ReplyDelete
    Replies
    1. sam, I am going to be blunt. your post shows the mendacity and arrogance lying at the root core of all the problems of economics.

      the "data" shows nothing because it is too limited to show anything. to say "look at the data" is mendacity.

      let's back up to what little we actually know about economics.

      We know that growth comes from technology. We also know that this growth is irregular. Last, we know that certain technologies have network effects, leading to tremendous spurts in grow (railroads). Janeway had a good discussion of these propositions about a week ago on Project Syndicate.

      Now, beyond knowing this general principles, we don't know much at all. We don't know how to measure the network effects, for example. Take railroads. When, from an economic POV, can we say they really kick in and start to matter? It is nothing but bias and guess work to say that the moment that matters is when the Golden Spike was nailed in Utah.

      To suggest that I know when the network effects of railroads really kicked in would be the height of dishonesty.

      So, when people write this kind of crap it is the ultimate in intellectual dishonesty. As Noah explains, it is nothing but data selected to confirm an existing bias or prejudice. Correlation does not equal causation, especially cherry picked correlation.

      The level of this tolerated mendacity is appalling. I just stopped by John Taylor's site who has up a report about a paper he did that he says shows that we should balance the Federal Budget in 10 years. Talk about a selective memory; where was John Taylor when Bush and Cheney cut taxes, because "Deficits don't matter." Taylor represents the same kind of mendacity as the clowns about whom Noah is writing. He wrote what he wrote because it suits his current political bias. He wants the unemployed and poor to remain unemployed and poor and his rich friends to pay lower taxes and become richer.

      Everyone reach to your keyboard and repeat after me. Hockey Stick! Hockey Stick! Hockey Stick!

      It is appalling that I have to teach smart people both honesty and how to do economics. Economics is hard and it is most assuredly not the Taylor Rule. The Uncertainty Principle assures us that is the case.

      We know very little more than we knew when Alexander Hamilton laid out the framework for a strong national economy over 200 years ago. You need a strong central/national bank that is at all times a source of strength and liquidity and lender of last resort and you have to have the American system. When you have a current account surplus---a factor never considered by R&R---then you can manage most any amount of debt. That is why the Japanese are doing what they are doing. They have to get back to a current account surplus as quickly as possible (and keep such positive).

      Brad DeLong just wrote the same thing, writing "And why--given that one country's exports are another's imports--does the adoption of policies in deficit countries to reduce their imports and boost their export not automatically trigger the adoption of policies in surplus countries to boost their imports and reduce their exports?," after earlier writing, "In order for the world economy to be prosperous, adjustment to macroeconomic disequilibrium needs to be undertaken by both "surplus" and "deficit" economies--not by "deficit" economies alone."

      Now, of course, no one wants to go tell Obama the truth, for moving to a trade surplus is going to hurt and lot of powerful voting groups. It is going to cost more to shop at Wal-Marts. That will be the cost of the tariffs and currency war we need to wage. But, over time, if we move over 1/2 a trillion in mfg. back to the US, things will get a whole lot better here.

      Delete
    2. Alexander,
      Why do you first say all economics is mendacity, then use public policy advocacy as an example of this deceit, then assert that we know something about economics and cap it off citing an economist as an expert along with an offering of economic policy advice?

      It is a curious trail to follow, truthfully, its not clear where you want the reader to believe the lies stop and the truth starts.

      I am also not fond of the use of pseudonyms. I am not sure why this is such wide practice.

      I am delighted to see R&R come under scrutiny.

      dan

      Delete
    3. 1) I stated certain absolute know truths. In physics a similar statement would be there is a force of gravity and little more. We know that technology is a big part of economic growth, in the same way. We don't know much more than that. Similarly, we know there are network effects of information. We have shown such with experiments. But, we have no economic models which include information, even though it is the fourth factor of production.

      2) Taylor was all for RR's tax cuts and deficits. Taylor was all for Bush's tax cuts and deficits.

      He has suddenly discovered "deficits" only now that Obama is President. This is mendacity, simply put.

      3) DeLong is cited as one of the very few honest economists around and is being recognized for his honesty. That Hamilton's American system works has been shown again and again by actual application. First here and now in China and Germany and also in Japan, GB, France, Portugal, Spain, Italy and Greece, who are all desperate to establish current account surpluses.

      As I have written elsewhere, what economists object to about the American system is not that it works, but that it means countries are in competition and we face a choice: hungry children in the United States or in China.

      If I post anon., then you have to confront my ideas, only.

      Delete
    4. You're idea seems to be that we know almost nothing about economics - that Taylor is a liar, DeLong is not, and that Portugal Spain and Greece are examples of American economic success.

      I still a curious link to arrive at economic facts when the starting premise is we know almost nothing about economics.

      Is it even possible to lie or be honest about something that we don't know.

      Its also possible that these ideas would be more relevant to a blog post that was not about RR.

      Delete
    5. "Is it even possible to lie or be honest about something that we don't know."

      Sure -- we might assert that we know that deficits exceeding 90% are bad for economic growth, when we know nothing of the sort, and honest mathematical attempts to check this fact show nothing but a pile of gently tilted fuzz. Not that this necessarily has anything to do with RR.

      Delete
  4. at least the Excel error could have been avoided
    when finally data .. of economic papers would be shared ...




    Open Access to Data: An Ideal Professed but Not Practised

    Out of the sample, 435 researchers (89.14%) neither have a data&code section nor indicate whether and where their data is available. We find that 8.81% of researchers share some of their data whereas only 2.05% fully share.

    http://papers.ssrn.com/sol3/papers.cfm?abstract_id=2224146

    ReplyDelete
  5. Hi Noah,

    I think I agree with you on your judgment of RR's book (and on the paper itself, of course). RR's use of averages can be misleading as it can draw one's attention away from a lot the heterogeneity that is going on (a statistical lesson?). This raises a lot of questions as to how historical economic research sould be conducted and how (and even whether) one may "draw lessons from the past." But at least it's a good step forward, as you say in your last comment.

    But just one thing: regarding RR's use of the expression "This Time is Different," I may be wrong, but if I remember well I don't think it's (only?) used to mean that all crises have the same roots. They also (mainly?) use it to refer to a psychological phenomenon whereby during a boom economic actors seem to justify trust of and reliance on newly high values (of housing, stocks etc...) with the idea that "we've reached a new, high plateau of values, that we never could've reached before since we've never reached such a level of prosperity before anyway." Thus minimizing the risk that values might one day actually fall. I think this is an interesting psychological insight (not revolutionary, but interesting), and I've seen this behaviour a lot in 1920s Chicago regarding house prices as well.
    Anyway, thanks for posting, once again! Natacha

    ReplyDelete
  6. Anonymous1:26 PM

    I think the most telling thing is it is always phrased "Countries with high debt-to-GDP tend to have slow growth" or "there is a relation between high debt-to-GDP and slow growth", etc. Technically, these terms to do not claim a causal link. But by always using them in that order, they strongly imply a direction of causality. If they were always reversed, you'd get the impression they thought slow growth causes high debt-to-GDP ratios. If they actually wanted to present the evidence fairly, they would alternate phrasing.

    ReplyDelete
  7. Wonderful post, Noah. Two reflections.
    1) I have seen papers of Physics and Biology (and authors) been crucified in "Bad Science" style blogs for much less. If the thesis are not based on the true data, they are not even wrong.
    2) Some guys tried to replicate data for the UK and found no correlation for the debt with the UK after the WWII: Shock research finding: high public debt-to-GDP ratio leads to faster increase in GDP! (er..) « PRIME ECONOMICS - http://kcy.me/j0zz
    Regards

    ReplyDelete
  8. Noah,

    Long-time reader, first time commenter here. I really enjoyed the post, having read TTID and felt it touched on very interesting arguments, but didn't really elucidate the mechanisms of operation. And, more frustratingly, they use history to make broad claims I don't feel are supported very well even when you rate their argument on a qualitative historical basis (full disclosure: I'm a historian).

    The results of these kinds of arguments can be disastrous, as the Euro 'experiment' in austerity proves. There's nothing as dangerous as when a popular idea "those big spending Greeks ought to pay their debts" merges with an academic justification for that popular belief. This phenomenon begins to have a life of its own, even long after it is debunked. Just look at the vaccine scare.

    Those very big issues aside, I've found that the most lasting part of TTID for me was section 13, on the animal spirits that led to the belief of the housing market to be somehow uniquely liberated from its past (and those past growth rates). On that point I feel that they were really on to something that deserved far more attention than it had been receiving (both from historical and economic scholars).

    ReplyDelete
  9. What surprises me the most is that the big difference between the median and average growth rate for countries over 90 didn't raise any flags. 1.6 versus -0.1? I would be all over my data trying to figure out what's driving this!

    ReplyDelete
    Replies
    1. I would be all over my data trying to figure out what's driving this!

      And when you found a regression specification that restored your faith in your priors, would you stop? ;-)

      If not, you're a better scientist than most, I think.

      Delete
    2. "And when you found a regression specification that restored your faith in your priors, would you stop?"

      No, I wouldn't stop, especially if something looked suspicious! In fact, I have been working on my current paper for two years because I have triple-checked the data, even though the results were good the first time. I wanted to make sure it wasn't due to an error. But I did it not because I am a better scientist, but because I am not Rogoff. Which means that anything I put out there is going to be scrutinized way more than anything he puts out. Or maybe he just has more self-confidence than I have. :-)

      Delete
    3. Now realize that there's a publication bias in favor of people who are not as honest as that, and who simply stop looking once they have something solid-looking enough that our weak-ass peer review system isn't going to catch it...

      ...and despair.

      Delete
    4. I am. I have been pulling my virtual hair out since I read about this.

      Delete
    5. derrida derider5:41 AM

      As a micro rather than macro economist, I reckon CA is dead right.

      Such extreme skewness would not just lead you to dream up alternative specifications, but would cause you to look at AND THINK ABOUT every individual datapoint, particularly the outliers (and no, you don't just discard them unless you can explain exactly WHY they should be discarded. The simple fact of them being outliers doesn't count).

      Most micro guys certainly would - but then microeconomists have long complained about the macro peoples' cavalier ways with data. This is just an extreme case.

      Delete
  10. Noah: " randomly excluding countries from the sample"

    The whole problem with their exclusions were that they were *not* random;
    they excluded early years with high debt ratios and high growth.

    Shame on you, Noah - 'random' is not a word that an economist gets to misuse :)

    ReplyDelete
  11. It's important because at this point it's pretty much three strikes for them:

    1) Deleting data points which worked against their hypothesis (and not telling people).
    2) Using an extreme weighting scheme which worked for their hypothesis, not telling people that they were in fact using 'episodes', and not years, and (as far as we can tell) not running an unweighted or otherwise-weighted analysis for sensitivity - and not telling people.
    3) No releasing calculations for three years, while publicly using the results, and declaring a causal link, even when none is justified.

    I'd like to see sanctions against them.

    ReplyDelete
    Replies
    1. Well, wait a minute. I am not sure their weighting scheme is that extreme. And, at least in their published work, they never claimed a causal link. Should they have been more careful? Absolutely, especially because they know people take them very seriously! But does all this change the general idea that quirky things happen when debt/GDP exceeds a certain threshold? I don't think so.

      Delete
    2. ut does all this change the general idea that quirky things happen when debt/GDP exceeds a certain threshold? I don't think so.

      Yes. Zero evidence for threshold effect. Even in their original methodology it was still "testing hypotheses suggested by the data"...there was no out-of-sample verification of the trend break point, so basically they just eyeballed what they had and picked out 90%. Now with the new HAP methodology it looks like the threshold level is 50%. BUT YOU STILL NEED OUT OF SAMPLE VERIFICATION to believe in a trend break. And you know this!

      Delete
    3. But you are framing things way stronger than I am. What I take out of this exercise is that once debt gets too high we should start asking what the endpoint should be. Now whether we should start worrying when we reach 90% or 110% I cannot tell you. I suspect it depends on the economy. The physicist demands certainty, but all the economist can deliver is a general idea and an advice: "use your judgement". Alas, economics is not physics, nor can it ever be! :-)

      Delete
    4. But even that is not supported by the data and you're still implying some kind of causality "too much debt tend to lead to bad things". On which I kinda agree but with a large large helping of "it really does depend on circumstances"...

      The data itself certainly doesn't tell us very much. Except for that initial threshold of 50% Noah mentioned (I saw a graph elsewhere and eye-balled the initial break at 30% actually but what's 20% of GDP between friends?)

      Delete
  12. Noah

    You mentioned war above. You might like to look at Geoffrey Blainey's Causes of War, which tries to get at a common cause. It's quite convincing, even if the conclusion is modest, but it's also uses historical data in a professional way - something R&R have a problem with.

    ReplyDelete
  13. Do you think it's overly simple to suggest that when debt reaches levels around 90% of GDP, the political pressure to reduce debt growth increases sufficiently to actually be effective? If that were the case, then an argument that debt growth will "naturally" taper off at 90%, would actually act as a counterweight to the underlying cause supporting its own conclusion.

    In other words, if I tell an ice-cream eater not to worry about their ice cream consumption because most people's weight gain tapers off when they gain 20 pounds (but ignore the fact that most people become aware of their weight gain at that point and eat less ice cream as a result), I am giving very very bad advice.

    ReplyDelete
    Replies
    1. Another way to state the example more mathematically:

      In other words, if I tell an ice-cream eater who weights X - 1 pounds not to worry about their ice cream consumption because most people's weight gain tapers off when they gain X pounds (but ignore the fact that most people become aware of their weight gain at that point and eat less ice cream as a result), I am giving very very bad advice.

      Delete
  14. What stands out in this (and some other) instances is how very EASY it is to get a reputation in economics. No other languages, familiarity with the detail, research in archives, difficult empirical tests required. No years of fieldwork learning what the subjects actually do in various circumstances needed. Just some data and a spreadsheet. Historians, sociologists, anthropologists can rightly sneer, but they do so with twisted mouths - they don't earn as much or get to play with people's lives.

    Okay - you need medium level maths as well as the spreadsheet. But it's the spreadsheet that gets you the invites.

    ReplyDelete
    Replies
    1. This seriously misunderstands the nature of the dynamics here. Reinhart and Rogoff developed reputations based on earlier research. They traded on those reputations and their position at a top university to push a political agenda. They got away with it for 3 years, and then they were caught. The damage to their reputations will persist for a long time to come.

      Delete
    2. I think you guys are both partially right. I think it's becoming harder to get a good reputation without serious data skills. Not sure about theory though.

      Delete