Monday, May 27, 2013

What can you do with a DSGE model?



When the Bank of England invited me to give a talk at their workshop on macroeconomics, I wasn't sure if they wanted me to provoke (i.e. troll) them with the kind of skeptical stuff I usually write on this blog, or to talk about my own research on artificial markets and expectations. So I did both. Now, this is a central bank event, which means secrecy prevails - so I can't tell you what the reaction was to my talk, or what other people said in theirs. But I thought I'd reproduce part of my talk in a blog post - the part where I talked about DSGE models. (In other words, the provocative part.)

"DSGE" is a loose term. It usually implies much more than dynamics, stochastics, and general equilibrium; colloquially, to be "DSGE" your model probably has to have things like infinitely far-sighted rational expectations, rapid clearing of goods markets, certain simple types of agent aggregation, etc. So when I talk about "DSGE models", I'm loosely referring to ones whose form is based on the 1982 Kydland & Prescott "RBC" model.

In recent times, of course, RBC models themselves have fallen out of favor somewhat in the mainstream business-cycle-modeling community, and have gone on to colonize other fields like asset pricing, international finance, and labor econ. As of 2013, the most "mainstream" DSGE models of the business cycle are "New Keynesian" models. The most important of these is the Smets-Wouters model, which has gained a huge amount of attention, especially from central banks, for seeming to be able to forecast the macroeconomy better than certain popular alternative approaches. If you know only one DSGE model, Smets-Wouters is the one you should know.

Anyway, my talk asked the question: "What can you do with a DSGE model?" Most people who evaluate the DSGE paradigm don't focus on this question; they either trace the historical reasons for the adoption of DSGE (the Lucas Critique, etc.), or they discuss the ways DSGE models might be improved. Instead, in my talk, I wanted to take the perspective of an alien econ prof who showed up on Earth in 2013 and tried to evaluate what human macroeconomic theorists were doing.

A DSGE model is just a tool. It's a gizmo, like a fork lift or a lithium-ion battery. The U.S. and Europe have invested an enormous amount of intellectual capital - thousands of person-years of our best and brightest minds - in creating, testing, and using these tools.

So what can you do with these tools?


1. Forecast the economy?

One thing you might want to do with a business cycle model is to forecast the business cycle. DSGE models have improved enormously in this regard. Though early RBC models were notoriously bad at forecasting, more recent, complex DSGE models have proven much better, and are now considered slightly better than vector autoregressions, and about as good as the Fed's own forecasts.

But as Rochelle Edge and Refet Gurkaynak show in their seminal 2010 paper, even the best DSGE models have very low forecasting power. Check out these tables from that paper:
















These tables show the forecasting performance for the Smets-Wouters model (which, remember, is the "best in class") from 1992 through 2006. The first table is for inflation forecasts, the second is for growth forecasts. Look at the R-squared values. These numbers loosely describe the amount of the actual macroeconomic aggregate (inflation or growth) that the model was able to predict. An R-squared of 1 would mean that the forecasts were perfect. You'll notice that most of the numbers are very, very low. The Smets-Wouters model was able to predict a bit of inflation one quarter out (though the Fed's internal forecasts were much better at that horizon), and not at all after one quarter. As for growth, the DSGE model had very low forecasting power even one quarter ahead.

Now, this doesn't necessarily mean that DSGE models are sub-optimal forecasters. These things might just be very very hard to predict! Humanity may simply not have any good tools (yet) for predicting macroeconomies, just like we aren't yet able to predict earthquakes.

But there's also some evidence that we could be doing better than we are. In this 2013 paper, Gurkaynak et al. test the "forecast efficiency" of DSGE models, and find that their forecasts are not optimal forecasts. Also, they find that simple univariate AR models are often significantly better at forecasting things like inflation and GDP growth than the best available DSGE models! This is not an encouraging finding for the DSGE paradigm, since AR models are just about the simplest thing you can use.

Also, in this discussion of forecasting, remember that the deck has already been stacked in favor of DSGE models. Why? Because of publicity bias and overfitting. If DSGE models don't do well at forecasting, researchers will add features until they do better. As soon as they do well enough to look good, researchers will publicize the success. This is a perfectly appropriate thing to do, of course - it's like improving any machine until it's good enough to sell. But it means that the publicized models will have a tendency to overfit the data, meaning that their out-of-sample performance will usually be worse than their in-sample and pseudo-out-of-sample performance.

(Update: Via a commenter, here's a good survey of DSGE models' forecasting ability, including how they did in the Great Recession. See my new post for more...)

In other words, DSGE models are probably not very good as forecasting tools...yet. But they're about as good as anything else we have. And they have improved considerably compared to their early incarnations.


2. Give policy advice?

This is what DSGE models are "supposed to do" - in other words, most academics will tell you that this is the purpose of the models. Actually, a model can be perfectly good for policy advice even if it's bad at forecasting. This is because forecasts have to deal with lots of different effects and noise and stuff that's all happening simultaneously, while policy advice only requires you to understand one phenomenon in isolation.

But here's the problem: To get good policy advice, you need to know which model to use, and when. So how do you choose between the various DSGE models? After all, there's a million and one of them out there. And they're usually mutually contradictory; since they're fitted using many of the same macroeconomic time-series (e.g. U.S. post-WW2 GDP, employment, and inflation), one of them being a good model (even just in one specific situation) means the others must then not be good models.

So how do you choose which model to use to give you advice? Old methods like "moment matching", which were used to "validate" the original RBC models, are, simply put, not very helpful at all.

What about hypothesis testing? Again, not very helpful. If you make the model itself the null, then of course you'll reject it, because any model will be too simplified to explain everything that's going on in the economy. If you make the null the hypothesis that the DSGE model parameters equal zero, you'll almost always reject that null, even if the model is grossly misspecified.

In principle, I think you should use some kind of goodness-of-fit criterion, like an R-squared, using out-of-sample data and adjusted to favor parsimonious models. At the macro conferences and seminars I've attended, I haven't see people saying "Look at the out-of-sample adjusted R-squared of this model! We should use this one for policy!" Maybe they do say this, though, and I just haven't seen it. (Update: Here, some people, including Smets and Wouters, do evaluate the fit! Definitely check out this paper if you're into macro modeling.)

But anyway, there's a few more problems here. One is the lack of clearly defined scope conditions; macro theorists rarely work on the difficult problem of when to stop using one model and start using another (see next section). Another is the nonlinearity problem; most DSGE models are linearized, which makes them easier (i.e. possible) to work with, but means that their policy recommendations often don't even match the model.

(As an aside, many people say "OK, we don't know which DSGE model is right, so just combine a bunch of models, with some weights." Fine...but the weights aren't structural parameters, so by doing this you give up the supposed "structural-ness" of DSGE models, which is the main reason people use DSGE models instead of a spreadsheet in the first place.)

So to sum up, DSGE models could offer policy advice if you used an appropriate model selection criterion, and dealt carefully with a bunch of other thorny issues, AND happened to find a model that seemed to fit the data decently well under some clearly defined set of observable conditions. But I don't think we seem to be there yet.


3. Map from DSGE models to policy advice?

OK, so it's really hard to give definitive policy advice with DSGE models. Maybe you could instead use DSGE models as maps from policymakers' assumptions to policy advice? I.e., you could say "Hey, policymaker, if you believe A and B and C, then here are the implications for policies X and Y and Z." In other words, since DSGE models are internally consistent, maybe they can help tell policymakers what they themselves think can be done with regards to the macroeconomy. (Another way of saying this is that maybe we can leave model selection up to the priors of the policymaker.)

There's just one problem with this. DSGE models are highly stylized, meaning that it's often not possible even to figure out whether you buy an assumption or not.

Let me demonstrate this. Let's take a look at a DSGE model - say, Christiano, Eichenbaum, and Evans (2005). This New Keynesian model is very similar to the Smets-Wouters model mentioned above. Here is a VERY truncated list of the assumptions necessary for this model to work:

  • Production consists of many intermediate goods, produced by monopolists, and one single consumption good" that is a CES combination of all the intermediate goods.
  • Firms who produce the consumption good make no profits.
  • Firms rent their capital in a perfectly competitive market.
  • Firms hire labor in a perfectly competitive market.
  • New firms cannot enter into, or exit from, markets.
  • All capital is owned by households, and firms act to maximize profits (no agency problems).
  • Firms can only change their prices at random times. These times are all independent of each other, and independent of anything about the firm, and independent of anything in the wider economy. (This is "Calvo pricing". The magic entity that allows some firms to change their prices is called the "Calvo Fairy").
  • The wage demanded by households is also subject to Calvo pricing (i.e. it can only be changed at random times).
  • Households purchase financial securities whose payoffs depend on whether the household is able to reoptimize its wage decision or not. Because they purchase these odd financial assets, all households have the same amount of of consumption and asset holdings.
  • Households derive utility from the change in their consumption, not from its level ("habit formation"). Households also don't like to work.
  • Households are rational, forward-looking, and utility-maximizing.

OK, I'll stop. Like I said, this is a VERY truncated list; the full list is maybe two or three times this long.

How many of these assumptions do you believe? I'm not sure that's even possible to answer. Formally, most of these are false. Some are very obviously false. The question is how good an approximation of reality they are. But how do we know that either?? Is it a good approximation of reality to say that households purchase financial securities whose payoffs depend on whether the household is able to reoptimize its wage decision or not? How would I even know? 

In principle, you could look at the micro evidence and see which of these assumptions looks kinda-sorta like real micro behavior. Some people have tried to do that with a few of the assumptions of the Smets-Wouters model; their results are not exactly encouraging. But if you tried to go ask a policymaker "Which of these things do you believe?", you'd get a blank stare.

So DSGE models don't make a clear map from assumptions to conclusions. But how about using them just to explore the robustness of models to variations in assumptions? A central bank (or the academic macro community) could make a bunch of DSGE models and compare their results, just to see how different modeling assumptions affect conclusions. In fact, that's probably what the academic macro community has been doing for the past 30 years. This seems somewhat useful to me, but there's a problem. DSGE models are not very tractable, so it's probably the case that nearly all of the modeling assumptions usable in DSGE models are poor approximations of reality. In that case, we'll be stuck searching next to the lamppost.


4. Communicate ideas?

DSGE models can definitely be used as a language in which to communicate ideas about how the economy works. But they are probably not the best such language. Simpler econ models, like OLG models, or even partial-equilibrium models, are much more flexible, and can be understood much more quickly by an interlocutor. DSGE models have a ton of moving parts, and it's generally very hard to see which assumptions end up causing which results. The better a model matches data or forecasts future data, the more moving parts it will generally have. This is called the "realism-tractability tradeoff". 

So if you only work with DSGE models, and if you try to understand everything in terms of DSGE models, you'll have a hard time communicating with other economists. I can see this being a problem in a central bank, where people need to communicate ideas very quickly in times of crisis.


So, what else would you have us do?

There are a number of alternatives that have been proposed to DSGE models. Different alternatives are generally proposed for the different purposes listed above.

For communicating ideas, the most popular alternatives are simpler, OLG-type models (which are, technically, DSGE, though not what we typically call "DSGE"!), and partial-equilibrium models (suggested by Robert Solow). I've seen some people use these at seminars, especially the OLG type, so I think this alternative may be catching on.

For forecasting, the common alternatives are "spreadsheet" type models (Chris Sims' dismissive term) that don't assume structural-ness. This is the kind of model used by the Fed (the FRB/US) and by some private forecasting firms like Macroadvisers.

Policy advice is the thorniest question, since you need your model to be structural. For this, the main alternative that has been put forth is called "agent-based modeling". I don't know too much about this, and the name is weird, because DSGE models are also agent-based. But basically what it seems to mean is to specify a set of microfoundations (behavioral rules for agents), and then do a big simulation. The big difference between this and DSGE is that with DSGE you can write down a set of equations that supposedly govern the macroeconomy, and with ABM you can't.


So are we wasting our time making all these DSGE models, or not?

My answer is: I'm not sure. So far, we don't seem to have gotten a heck of a lot of a return from the massive amount of intellectual capital that we have invested in making, exploring, and applying these models. In principle, though, there's no reason why they can't be useful. They have flaws, but not any clear "fatal flaw". They're not the only game in town, and realization of that fact seems to be slowly spreading, though cultural momentum may mean that the more recently invented alternatives (ABM) will take decades to catch up in popularity, if they ever do.

65 comments:

  1. as larry summers said: stick to bagehot, minsky and kindleberger

    ReplyDelete
  2. Anonymous4:54 PM

    This is a very good post!

    My only memory of DGSE models, dates dating back to my master degree courses a few years ago, is that they are very elegant objects through which I learned how to compute using MatLab!

    As for your first point (forecasting) I, generally, think that:

    (i) if the variable is inherently difficult to forecast (for example inflation)than you will not gain much from complex models (including DGSE). So, the best strategy would be to keep it, as Zellenr once said, "sophistically simple".

    (ii) now, if the variable is relatively simple to forecast, than adding some complexity probably will lead to some forecast improvement (but not that much!).


    I don't have much to say about 2), 3) etc.

    ReplyDelete
  3. Firms can only change their prices at random times. These times are all independent of each other, and independent of anything about the firm, and independent of anything in the wider economy. (This is "Calvo pricing". The magic entity that allows some firms to change their prices is called the "Calvo Fairy"). The wage demanded by households is also subject to Calvo pricing (i.e. it can only be changed at random times).

    Uh...whaa...I had no idea.

    ReplyDelete
  4. Anonymous5:16 PM

    "Another is the nonlinearity problem; most DSGE models are linearized, which makes them easier (i.e. possible) to work with, but means that their policy recommendations often don't even match the model."

    OK, I know next to nothing about DSGE, but if you're looking for policy recommendations, why even bother with making them analytically tractable? What's the drawback of just doing the simulations?

    ReplyDelete
    Replies
    1. Nathanael5:02 AM

      "if you're looking for policy recommendations, why even bother with making them analytically tractable? What's the drawback of just doing the simulations?"

      No drawback. It's just "pretty math worship" which causes the idiotic linearizations.

      Delete
  5. Hi Noah, I was wondering if when you dimiss the forecasting success of DSGE models, you ignore Nick Rowe's thermostat problem. Specifically, wouldn't a perfect DSGE model perfectly eliminate all forecastable shocks, so that predicted variables and realised variables would be perfectly uncorrelated? In this case any R-squared type test is worse than useless, and says nothing about the performance of DSGE models in general. Gurkaynak tends to bring this up quite a lot, but always worth mentioning

    ReplyDelete
    Replies
    1. I don't ignore it, I know it. It's just not relevant to the "forecasting" section of my piece.

      Note that I do not claim that models that are bad at forecasting must be bad at giving policy advice.

      If a model can't forecast, it can't forecast. Period. Whether it can give good policy advice is a separate and also important question.

      Delete
    2. This is what Gurkaynak and Edge say:

      "But what does that say about the use of DSGE models in central banks; both as a tool for policy analysis as well as a tool for forecast generation? We argue that in both cases the answer is “nothing”. The finding that inflation is not forecastable over the Great Moderation period is consistent with the predictions of the DSGE model given the strong monetary policy rule estimated for this period. Specifically, since under this rule the policymaker will alter the interest rate to counter forecastable deviations of inflation from the target, the rule will eliminate forecastable movements in inflation and leave only unforecastable shocks to drive fluctuations. Thus, our finding of low forecast performance is not necessarily evidence against the validity of the model."

      Delete
    3. Let's put it another way: if a central bank is targeting inflation, the only model that could forecast inflation would be a model the central bank didn't know about, or was too stupid to use.

      Since inflation targeting central banks know about DSGE models, and use them, we can conclude that DSGE models won't be able to forecast inflation.

      Delete
    4. RAstudent11:24 AM

      According to this logic, VARs shouldnt work either, yet they do.

      Delete
    5. RA student: good point. I don't know why VARS should work.

      Maybe it's easier to datamine and overfit VARS than DSGEs?? Is it just like those violations of the EMH that disappear as soon as they are published??

      Of course, if a central bank is trying to target inflation at a 2-year horizon, it may still be possible to forecast inflation at a less than 2 year horizon.

      Delete
    6. Since inflation targeting central banks know about DSGE models, and use them, we can conclude that DSGE models won't be able to forecast inflation.

      Not necessarily; that conclusion also requires that we believe that inflation targeting is feasible. Some don't believe this.

      Regardless, this point is really completely beside the point. If a model can't forecast, it can't forecast. That doesn't mean the model is bad; for instance, it could be good for policy advice. Which is why I considered "forecasting" and "policy advice" in two separate sections of this blog post.

      Delete
    7. Also, Nick:

      1. EVERYONE GETS YOUR POINT about the "thermostat". It's a cute point, probably not applicable to the real Universe in which we live, but cute nonetheless. But it's very easy to grasp, and we all grasp it already!

      2. Overfitting won't give you better out-of-sample forecast performance...

      Delete
    8. Nick, you're aware of the fact that there are "violations of the EMH" that have persisted for like 40 years, right? (I put in scare quotes, since they may turn out to have rational explanations.) I swear to God, I get the impression that the last paper in finance anyone in economics has read was in 1970 or so.

      Delete
    9. Wouldn't the 'thermostat problem' apply only if the model was actually being used to adjust the economy?

      Delete
  6. I actually think they do have a fatal flaw, which is that they all rely on microfoundations. There are many good reasons why we should not have microfoundations in macro models, and to that end, I'd encourage everyone to read, "Does Macroeconomics Need Microfoundations," by Kevin Hoover.

    ReplyDelete
  7. Noah, sounds like you gave a useful talk. Questions ... rather than lectures are an excellent frame. A few comments ... purely my opinion and understanding of some related stuff.

    At the Board, there are several forecasting / policy evaluation tools that the staff use. I am a firm believer in many models, many tools (none dominate across all settings). In the near term about a three quarter window, the staff now/forecast is data translation/signal extraction and super weedy stuff. The economists are specialists in their areas. (I have done consumption and now am learning inventories). That same group does conditional partial equilibrium forecasts out for 2-3 years. Judgment also gets layered on, stuff not in the models or quirks of the models corrected. The conditional part is that many of the exogenous inputs stock prices, exchange rates, multipliers come from elsewhere, including FRB/US, the staff's workhorse macro model. A coordinator economist pulls all the PE forecasts together and makes sure it is sensible from the top down. And FRB/US with some adds takes over for the long term. So it's complicated, but we can explain where it is coming from.

    Of course, the Board has DSGE models and along with FRB/US gets used for altsims and policy analysis. PE models or spreadsheets are not ideal for sensitivity analyses in equilibrium ... especially if you want risks quantified. And even though DSGE model output might not be a easy way to do communication policy, they are a tool for macroeconomists to talk and explore structure.

    As I understand it, no CB has a pure model-based staff forecast. Another interesting question is where there judgment comes from and how do they evaluate that piece over time.

    ReplyDelete
  8. Curious why central bank meetings need to be "secret". Seems like these institutions need to be as transparent as possible.

    ReplyDelete
  9. For this, the main alternative that has been put forth is called "agent-based modeling" ... But basically what it seems to mean is to specify a set of microfoundations (behavioral rules for agents), and then do a big simulation. The big difference between this and DSGE is that with DSGE you can write down a set of equations that supposedly govern the macroeconomy, and with ABM you can't.

    From what little I understand, an alternate way to say this is that ABM has the potential to model relationships and behavior that are not mathematical in nature. For example, phenomena with positive feedback loops and/or those that do not tend towards equilibrium and/or those that are nondeterministic.

    Also, under rational expectations, people's preferences and expectations cannot affect each other's preferences and expectations, which is particularly problematic 1) for a social science and 2) in modeling bubbles and herd-behavior.

    I would ask Barkley Rosser, though. I've only read about it.

    ReplyDelete
    Replies
    1. RAstudent5:26 PM

      I don't think this is true. Mathmatical rules just govern the behavior of agents. Abm has a lot of game thery type math. Problem is calibration just like in dgse. Both approaches if they could be made to work, have many advantages. I am just skeptical they can ever forecast very well. Small changes in the rules can lead to huge changes in global behavior. I would say that in the world of sports sports, ABM has proven to produce pretty good predictions. Think of EA sports for the NFL or Accuscore. Still seems like econmetic models are better but ABM has come a long way

      Delete
    2. As I understand ABM, you can model emergent behaviors where the system behaves very differently from its parts, rather like how the behavior of molecules does not follow the same rules of behavior governing that of atoms, which is why chemistry is a separate field from physics. So that the agents behave according to mathematical rules does not mean the system has to.

      Delete
  10. William Allen3:47 AM

    Noah, in your scheme of classification, where would you place the stock flow consistent Post Keynesian models of Wynne Godley and Stafford Cripps (New Cambridge school)? I find these far more flexible than DSGE type models.

    ReplyDelete
    Replies
    1. What does "stock flow consistent" mean? Are DSGE models "stock flow inconsistent"? Can you show me how this is the case? I am curious...

      Delete
    2. DSGE models are stock-flow consistent. An example of a non stock-flow consistent model is IS/LM.

      Delete
    3. You are right, what distinguishes so-called post-keynesian stock-flow consistent models from DSGE models is not stock-flow consistency.... but basically two general features: 1. the treatment of micro behavior by means of aggregative behavioral equations based on certain priors (which can be empirically estimated) rather than dynamic optimization -- say, an aggregate consumption function, an aggregate investment function, a loan supply function, a tobinian-type asset demand function, etc;
      2. the adjusting variables (or model closures) are also different -- output adjusts (instead of prices) to aggregate demand; asset prices (including the interest rate) adjust to supply and demand in market of stocks of assets, rather than flows (as in a loanable funds theory of the interest rate for instance,among other things; the financial system creates money endogenously... SFC are also being combined with ABM models by a few people.

      Delete
  11. Jonathan3:54 AM

    Hi,
    we/you have to better understand the main purpose of DSGE modeling: computing, simulating and estimating a microfounded model (and thus micro variables/shocks) with interlinked (structural) shocks and variables in a stochastic framework. This is the main goal and we know that DSGE modeling is unique for that purpose.

    So I have some comments:
    1. Forecast the economy?
    Comparing to other methods, DSGE forecasting is the youngest (about 8 or 10 years old) and even if it is not "the best", it is mainly better than what I call "old methods". Let us time !

    2. Give policy advice(s)?
    It is working very well, but in a qualitative manner (in a sense that it give us useful information about trends, shocks and dynamic roles of variables) rather than for a quantitative purpose. It is still hard to predict the (exact) value of one point (after two or more periods), but it is now easy, with a structural microfounded model, to predict a trend with respect to other structural (or not) shocks.

    The problem of the huge amount of models is not specific to DSGE models. It is more related to the research processes and it is available for all modeling methods...

    I am agree with the fact that we need more goodness-of-fit statistics but we have already other great tools: mean of estimated shocks, convergence statistics, student tests, IRFs, long run and short run variance decompositions, shared growth path, model based detrending, etc... the list is maybe as long as your list in Section 3 ;-)

    3. Map from DSGE models to policy advice?
    You should ask the following question: what is (economic) modeling ? Assuming several and acceptable hypotheses in order to obtain a light version of the (economic) fact you want to model. Again, assuming hypothesis is not specific to DSGE modeling. Contrariwise, assuming fatal flaw(s) is common in several other modeling/estimation methods...
    4. Communicate ideas?
    Totally agree, we need to improve the communication channels. Specifically, we need to extract more information from a DSGE model and explain this more with words, and less with numbers/equations/functional forms...
    So are we wasting our time making all these DSGE models, or not?
    Of course not and we need to mobilize more "human capital" in order to improve DSGE modeling.

    Other remarks for all previous points:
    - Of course we need to focus on second and third order DSGE models, because simple linearization provide materials to Noah ;-)
    - What is research ? From my opinion, it is to build little advances in order to find one big finding. Of course my remark is not specific to DSGE modeling, but it includes it !

    ReplyDelete
  12. Compared to some of the commentators above, I am a barbarian. I believe that macro-economics ought to be simple, like an ax to the head...

    http://theredbanker.blogspot.com/2012/03/refreshing-macro-economics-part-1.html

    http://theredbanker.blogspot.com/2013/02/refreshing-macro-economics-part-2.html

    http://theredbanker.blogspot.com/2013/03/inflation-or-deflation-refreshing-macro.html

    http://theredbanker.blogspot.com/2013/03/refreshing-macro-part-4-profit-matters.html

    That being said, I think an interesting macro-question right NOW is why is the US economy recovering (apparently)? I would find that subject worthy of macro-economic analysis! Is it the "loose" (less tight) monetary policy? If so, which channels worked? Is it fiscal expansion? How, with sequester and stuff?

    What is going on? And if macro models cannot answer that, what good are they?

    ReplyDelete
  13. Jorge Bielsa4:25 AM

    The hiatus has made wonders on You. This post is the best I have read in this blog. Thank You very much.
    I have only a little "but", in the way you conclude your excellent post:
    "invented alternatives (ABM) will take decades to catch up in popularity, if they ever do."
    Looking at the speed the times are changing, I never had used the term "decades" for better or worse. Anyway, I think you made a risky bet, specially after showing the DSGE cards in such a clear and enlightening way.

    ReplyDelete
    Replies
    1. Jorge Bielsa4:57 AM

      I have just read your previous post and I can not help to ask you:
      Have you some no-monetary pay-offs for your last bet?

      Delete
  14. "They have flaws, but not any clear "fatal flaw"."

    Not sure why you believe models that neither contain defaults nor banks are not fatally flawed. Suggest you look at Goodhart and Borio's work.

    ReplyDelete
  15. Nathanael5:01 AM

    Your conclusion is weak, and frankly you need to have the courage of your intellectual convictions.

    Your evidence shows conclusively that DGSE models are a dead line of research and a waste of time, at least for this generation.

    (If some other analytical tool is developed in 20 years which allows us to make DSGE models which work, then we can revisit them in 20 years.)

    ReplyDelete
  16. "So are we wasting our time making all these DSGE models, or not? My answer is: I'm not sure. So far, we don't seem "


    30 years of macro research. This is nothing but failure. Compare with the 30 years of research before. DSGE is a sunk cost.

    ReplyDelete
  17. Is it fair to repeat what you say by saying you told a group of central bankers that it is not possible to forecast the future?
    I trust they responded by saying, 'but no one can predict the future'.

    I am going to make a forecast, in fact I would even make a bet. My prediction is that in no time prior to July 2015 will economists learn how to predict the future.

    Strangely this is true even though we are told that History repeats itself which would suggest an AR model should work, even though you can't predict the future which is strange because...

    ReplyDelete
  18. Rothosen9:33 AM

    This is beyond me even though I'm sure it's good stuff. Will you be thinking of doing something in the future on macro-economic effects of a further 20% weakening of the Yen?

    ReplyDelete
    Replies
    1. Anonymous5:32 PM

      Don't reason from a price change, or in other words, it depends why the yen depreciates as to the consequences. In general, it would be good for Japanese exports and increase their inflation/aggregate demand.

      Delete
    2. I am not sure it'll be so good for their exporters: First, it relies on none of their competitors reacting. Two, what about input prices? But, yes, if someone should benefit, it should be exporters.

      I am pretty sure it will also boost their inflation (if only via higher imported prices).

      How's that going to do anything FOR AD, though, I wonder?

      Delete
  19. RAstudent11:03 AM

    1.) I do not think its true any type of DGSE has outperformed Bayesian variants of VARs.
    2.) How is it that DSGEs can perform better than VARs but worse than AR(1)s when ARs are simply special cases of VARs?
    3.) Looking at R^2s is a terrible way to evaluate forecast performance, so why focus on that?

    ReplyDelete
    Replies
    1. 1.) See http://www.voxeu.org/article/failed-forecasts-and-financial-crisis-how-resurrect-economic-modelling

      2.) Overfitting.

      Delete
    2. RAstudent12:19 PM

      1. Unless I am missing something this does not say how the BVAR did vis a vis the others. It simply says model based forecasts weren't that great at predicting the recession.

      2. Good point but wouldn't a good practitioner then collapse to the AR model rather than rely on an overfit model?

      Delete
    3. Unless I am missing something this does not say how the BVAR did vis a vis the others. It simply says model based forecasts weren't that great at predicting the recession.

      See here:
      http://econ.upf.edu/~brossi/GKR2013_April20.pdf

      Delete
    4. RAstudent9:17 AM

      Touche Mr Smith. Consider my mind officially blown. In all honesty though, these results make no sense to me. I really cannot understand how its possible that an AR could out perform a VAR, BVAR, and RVAR (what they call RW).

      Delete
    5. Overfitting, is my guess.

      http://en.wikipedia.org/wiki/File:Overfit.png

      Delete
  20. Anonymous12:07 PM

    the ultimate yardstick for any model in my opinion, is its out of sample forecasting performance

    I am not an economist, but my guess is, the way to go is write down assumption and do a giant simulation - sort of what they do to predict weather nowadays

    ReplyDelete
  21. Really nice post. I learned a lot. But I'm confused by what appear to be contradictory statements about forecasting:

    A. "more recent, complex DSGE models have proven much better, and are now considered slightly better than vector autoregressions, and about as good as the Fed's own forecasts."

    B. "simple univariate AR models are often significantly better at forecasting things like inflation and GDP growth than the best available DSGE models!"

    C. "DSGE models are probably not very good as forecasting tools...yet. But they're about as good as anything else we have."

    Doesn't B contradict A and C? Or did I misunderstand something?

    Also, I'd like to know your real answer to William Allen's question: regardless of the phrase 'stock-flow consistent', do you have a view on the Wynne Godley and Stafford Cripps-type models? Apparently Goldman Sachs use their approach as part of their tool box. (If you're just not familiar enough with the model, that's ok too.)

    ReplyDelete
    Replies
    1. Doesn't B contradict A and C? Or did I misunderstand something?

      Check out this paper:
      http://econ.upf.edu/~brossi/GKR2013_April20.pdf

      do you have a view on the Wynne Godley and Stafford Cripps-type models?

      Nope

      Delete
  22. Anonymous5:46 PM

    It's not a good sign that I've never heard a DSGE model mentioned in policy debates, even amoung highly technically skilled economists.
    As to forecasting, nothing can beat a futures market. If you want to make money in the futures market and use a DSGE model to do so, fine, but for the purposes of a central bank, any model is strictly second best. Furthermore, it makes no sense for a central bank to forcast a targeted variable. If the forecast is different from the target, change your actions until they match.
    You might be interested in this post by Adam Gurri which notes the mismatch between central bankers' academic work and their actual policy decisions. Technical macroeconomics has had very little impact on policy, DSGE or not.

    ReplyDelete
    Replies
    1. It's not a good sign that I've never heard a DSGE model mentioned in policy debates, even amoung highly technically skilled economists.

      Agreed.

      As to forecasting, nothing can beat a futures market.

      Not necessarily true, since the Fed has inside info (note that according to Edge and Gurkaynak, the Fed's short-term inflation forecasts are MUCH better than any model's, and in fact have an R-squared of 0.48, which is huge).

      Delete
  23. - point forecast accuracy is far from everything even in forecasting. For decision making it may be important to accurately characterise probability of various intervals or more generally multivariate regions for the variables we care about. The best model for that purpose may tell you there's a lot of unpredictable stuff going given the small number of observable variables you have. But it may be better at quantifying the uncertainty you face than a model that tries to maximise the accuracy of the point forecast at the cost of distorting probability bands.
    - Old style large macro models (what you call spreadsheet models) have tons of hidden assumptions hidden in all the zero coefficient restrictions they impose in their behavioural equations. They're just hidden and harder to uncover. And to boot, it's not even clear they respect the resource constraints and budget constraints of anyone in the economy.
    - Many of the assumptions you mention are approximations of the kind you make in any econometric specification (e.g loglinearity or constant elasticity of substitution, ignoring distributional effects , the monopolistic competition assumption that prices are independent of the number of competitors in the market etc...). We couldn't possibly ever build a model as a simplifying framework for organising our thoughts in such a messy reality if we didn't have to make these assumptions. As we get better, we build more complex models with more realistic assumptions (e.g heterogeneous households and firms, more realistic wage bargaining etc...)
    - If you end up seriously working with DSGE models and see many models you start to get a feel for which model is appropriate in which situation, and how to mentally combine different models each of which may be missing important bits of reality to get a more comprehensive understanding of the economy (but isn't this the same with micro models?). You don't this just from the core macro sequence or ocasionally reading about them.
    - There are ways to classify different clusters of models into a smaller number of more general types (e.g models of firm level credit constraints often map into models of investment taxes/wedges, other models are basicaly about taxes/wedges on employment, others microfound changes in total factor productivity etc...). Some people at the BoE should have a lot of interesting stuff to say about this. Look up business cycle accounting and an extension by Richard Harrison and co...also the BoE's new forecast methodology combining a core DSGE model with a suite of other models (DSGE or otherwise) is well documented in a recent working paper on their website.
    - To say DSGE models are not useful in policy analysis begs the question: not useful in comparison to what? Most of the criticisms that have been made could be made just as well about the large scale macro models that are supposed to be useful in policy analysis by practioners. Lucas and Sargent made some good points in the 1970's. Looking at the current specification of e.g the Moody's macro model or the Oxford models most of these points have not been addressed.
    - A significant component of a model becoming useful for policy analysis is experience in using it. There's a reluctance I think on the part of people trained in the old style macroeconomics to switch from simultaneous equations systems (often estimated by OLS despite the endogeneity bias) to the more modern state space/latent variables perspective that underlies the DSGE approach. As more central banks and other agencies like the European Comission work with DSGE models, they should become better at communicating their insights and using the extra structure that allows you to see people, firms and markets in these models (as you said, they're in that sense similar to agent base models) to satisfy your objective 2),3) and 4) better.

    ReplyDelete
  24. Mate, I love that you are an honest outlet for how economists reason and model the economy. But it is a very poor sign about the state of the discipline that you, a freshly minted star PhD, knows all these limitations to the main modelling techniques (since you are obviously well trained in them) yet are basically ignorant of other approaches that seem to offer quite useful insights.

    I am not accusing you of bias. You clearly think things through very clearly and are honest about what economists are up to. But it seems like economists in general won't try anything else except tweaking their current tool till it works a little better.

    I guess my analogy is that economists are trained in using a hammer to work on a problem. The hammer doesn't seem to work, but they won't even try a spanner or a screwdriver. They must modify the hammer until it works a tiny bit better. Meanwhile a whole field of people is out there testing other tools and getting really interesting results.

    From your writing it seems you are well aware of this. For example you refer to agent based models, but don't really know enough to say whether they might be useful. Then a commenter asked about Wynne Godley, and again you haven't had much exposure to this approach.

    If you are interested in learning about Wynne Godley's approach, his toolbox if you will, I recommend this book
    http://www.amazon.com/Monetary-Economics-Integrated-Approach-Production/dp/0230301843

    I am also studying an economic PhD and find the rather inward looking nature of the field quite disturbing. Especially when it comes to macro (there are many microeconomists happy to learn from psychology, sociology etc).

    In any case, when you do read up on some alternative macro approaches I would really like to read your opinion (in my view many have similar problems, some have different problems).

    ReplyDelete
    Replies
    1. Actually, I know a bit about agent-based models in finance...not in macroeconomics (remember, I'm a finance prof, I just took macro in grad school). There are a LOT of different things that agent-based macro models could be doing. I can think of a lot of these things. But I don't know which ones people are actually doing. However, the word on the street is that ABM is getting more and more attention. So I'll look into it at some point.

      As for Wynne Godley and his approach, I'm not going to order a book before reading some free PDFs...got any good links?

      I agree that the macro profession is way, way, way too concentrated on the DSGE approach.

      Delete
    2. So, are you talking about this kind of model?
      http://www.levyinstitute.org/pubs/wp_494.pdf

      This is just a system of linear equations for macro aggregates, with the coefficients assumed to be structural parameters.

      Delete
    3. RAstudent9:25 AM

      Must say I am unfamiliar with Wynne Godley but as for ABM, Rob Axtell's (and Joshua Epstein's) work to be the most fascinating. In particular:

      http://www.brookings.edu/ES/dynamics/papers/firms/firms.pdf

      http://www.creativeclass.com/rfcgdb/articles/Emergent_Cities.pdf

      and the seminal work

      http://www.amazon.com/Growing-Artificial-Societies-Science-Adaptive/dp/0262550253

      Delete
    4. Yep. Though note that the equations include some derivates of macro aggregates and at all time periods the accounting identities hold.

      I thought you might just get the book from the library to get a feel for why there exists this group who want to reject DSGE and use other tools. And of course, maybe outline a bit of a pros/cons list of DSGE vs monetary models vs ABM based on your reading (you seem to be able to provide good clear unbiased views on such things).

      I guess my personal view is that there won't be an approach that forecasts well. The economy is a complex system and we know that the path it takes is sensitive to initial (and all time period) conditions. Add a sprinkling of politics and you don't have much to work with!

      Delete
  25. In recent times, of course, RBC models themselves have fallen out of favor somewhat in the mainstream business-cycle-modeling community, and have gone on to colonize other fields like asset pricing, international finance, and labor econ

    Can you point me to some of the examples in labor? I have an idea I want to explore and it would be useful for me to have some idea of what's gone before.

    Thanks

    ReplyDelete
    Replies
    1. http://press.princeton.edu/titles/9217.html

      When Shimer says "neoclassical growth model" he means "RBC".

      Delete
  26. - I've looked into many examples of old style macro models and ABM's. The old simultaneous equations models just strike me as documenting a set of correlations that were found in the data set they used without much structure. This doesn't matter if you just want the best point forecast (though it's better than to just use a statistical model like a VAR or even univariate ARMA equations). But it does matter if the goal is to answer things like what's the effect of policy X or structural shock Y. As for ABM's I constantly get the impression that the rationality in them is much too bounded indeed in terms of reflecting motivations for human actions, and the market processes they model also strike me as too primitive. I appreciate the value in trying to build things from the bottom up, but I feel you may end up with models describing a market process from hundreds of years ago instead of the sophisticated markets of the 21st century. There's probably a middle way where ABM's contain more dynamic behaviour considerations and use more sophisticated matching and search protocols. I can see how such a hybrid would be both micro founded and contain more realistic heterogeneity than most DSGE models. But I don't see anyone working on this from the ABM papers I've seen, and a model with millions of agent types is bound to offer more complicated and hard to understand insights than a DSGE model with 2-4 agent types.
    - RBC models have been abandoned in the central bank monetary policy community. They're still used a lot in economic growth,development and taxation analysis. If it seems they must be trivially wrong because the real world has nominal prices that don't move all the time, you should look up the Barro critique. The basic intuition is that sticky prices don't necessarily have a big macroeconomic effect if they're attached to sticky quantities (and when quantities move prices are flexible). As for the comparison of fit: sticky price and wage models have more complicated equations with dubious assumptions about the frequency of wage adjustment. RBC models have fewer equations,so if you want better forecsting performance it's easier to specify them with more complicated (realistic?) shock processes to improve fit. It's not always clear which route is more realistic. But it's impossible to discuss this seriously in blog comments...
    -

    ReplyDelete
  27. " I don't know too much about this, and the name is weird, because DSGE models are also agent-based."

    Not in the computer science sense they aren't.

    ReplyDelete
  28. I don't understand why you say:

    i) We know the assumptions are not even closed to correctly outlining how people behave.
    ii) Their predictive power is fairly low.

    But then you conclude "I don't know". Really? I know you're physicist, and I can't imagine you saying the same about an atomic theory such that:

    i) The theory postulates that atoms behave in certain way, we put it at test, and we found that's not the case.
    ii) The theory predicts certain results in thermodynamics, and the correlation between prediction and reality is low.

    and that you don't know whether is true or false.

    I do understand that you're judging it as a "tool", but it's a theoretical model and if it's mostly false why should we expect it to have correct predictions?

    ReplyDelete
    Replies
    1. See my more recent post:
      http://noahpinionblog.blogspot.com/2013/05/dsge-financial-frictions-macro-that.html

      Delete
  29. Regarding agent-based modeling (ABM)....

    I, too, am delighted to see the open and honest discussion occurring in this blog regarding the current state of macroeconomic theory, and the advantages and disadvantages of alternative modeling approaches for macroeconomics such as DSGE and agent-based modeling (ABM).

    For a comprehensive set of introductory materials on agent-based macroeconomics, agent-based finance, and agent-based computational economics (ACE) more generally, please check out the following highly active sites:

    ACE Homepage: Growing Economies from the Bottom Up
    http://www2.econ.iastate.edu/tesfatsi/ace.htm

    Agent-Based Macroeconomics
    http://www2.econ.iastate.edu/tesfatsi/amulmark.htm

    Agent-Based Financial Economics
    http://www2.econ.iastate.edu/tesfatsi/afinance.htm

    ACE Introductory Materials
    http://www2.econ.iastate.edu/tesfatsi/aintro.htm

    Many annotated pointers to additional sites are linked at the above ACE homepage, including pointers to ACE/ABM introductory slide presentations, teaching resources (ACE/ABM courses/programs), software resources (ACE/ABM computational labs, software, and toolkits), and ongoing ACE research focusing on: verification & empirical validation; agricultural & natural resources, automated markets, business and management, economic policy, electricity markets, evolution of institutions and social norms; industrial organization; labor markets; market design; network formation; organizations; parallel experiments with real and computational agents; path dependence; political economy; technological change and economic growth.

    In short, the amount of ongoing activity related to ACE is rapidly growing, and the above sites attempt to give newcomers a broad overview of this activity. I hope that the readers of this blog who are not currently familiar with this modeling approach will be interested enough to read at least some of the introductory materials at these sites.

    Constructive comments for improving the presentation of materials at these sites would be most welcome.

    ReplyDelete
    Replies
    1. Thanks, Leigh! I'll definitely take a look. I've been meaning to read a lot more about ABM.

      Delete
  30. Anonymous10:09 AM

    "In principle, though, there's no reason why they can't be useful."

    How about this principle: Any model that is successful in predicting macro economic behavior, will be useful only if it allows policy makers to change policy decisions based on the prediction. Such predictable policy changes will allow some market participants to make money by predicting the policy changes and results. The success of the successful predictors will result in so much behavior change that it invalidates the model.

    I'd say that "success leads to behavior change and then failure" of a model is the kind of principle that DOES provide a reason that such models won't be long term useful for prediction.

    ReplyDelete
  31. My brother does something with business forecasting models. What exactly is that?

    ReplyDelete
  32. You mention that DSGE models come in huge variety which raises the question which one to use for policy advice and forecasting. The more serious question though is what does the (indiscriminate) existence of many competing models mean for each model's microfoundation (Lucas-proofness)? It simply means that an individual cannot be assumed to form its expectations only on the particular model at hand because that would be utterly irrational. It would have to consider all other (now available and all future) models instead. Therefore, Lucas-proof DSGE models (and their friends) are not Lucas-proof at all :-). See: https://www.researchgate.net/publication/308675193_Fukuyama_models_-_a_re-appraisal_of_the_Lucas_critique

    Best,
    Christian
    www.s-e-i.ch

    ReplyDelete
  33. I’d like to start with the falsifiability principle. We know that it’s based on the idea that given a theory (a conceptual representation of reality) it’s possible to think an experiment that could demonstrate that the same theory could be (or not) wrong. This idea is based on the following conventions:
    a) It exists a theory that identifies the existence of a set of variables that characterize a certain system and influence each other;
    b) Changing the state of one variable, we can forecast the change of one or more of the other variables;
    c) It exists a conventional procedure to give measure of these variables;
    At the end, given a certain state of the system, we will fit the actual values of the variable with these forecasted by the theory. If the model doesn’t fit reality, it’s plausible to reject the theory (it doesn’t pass the falsification process) even if we know that the philosophical debate is not so linear and subjected to criticism. In any case, this procedure is logically based on modus tollens: If A -> A v B is false (doesn’t fit “reality”) then A is false.
    Well, now we must separate normative and positive economics. The data you showed in your post said us, beyond a reasonable doubt, that, if you base our judgment on falsifiability principle, we must reject DSGE models. If we look at positive aspects of economics (economic policy), we should judge a theory used by policy makers to take decisions looking at its real utility. A theory rejected is like a musician who play a bass without strings...

    ReplyDelete