Forecastability And Economic Modelling

When most people think about macroeconomics, what they want is the ability to forecast economic outcomes. However, economists' (of all stripes) reputation as forecasters is not particularly high. My view is that this is not too surprising: what we want forecasters to accomplish is probably impossible. (I am hardly the first person to note this, as variants of this idea go back at least to Keynes; I could not hope to offer a history of this idea.) However, I think if we want to approach macro theory formally, we need to formalise the notion that outcomes cannot be forecast, which means we need to define non-forecastability formally.

This article gives one potential definition of forecastability, and then applies the concept to a simple stock-flow consistent (SFC) model. It should be noted that these are my preliminary thoughts, and I believe that the definition will need to be refined.

(Note: I am slowly going through the editing of my breakeven inflation book; still have a few other projects going on. Since there is no new content to be added to the breakeven inflation book, I can now start thinking about my next writing projects. It is possible I might do another "greatest hits" compilation of blog articles on banking, but otherwise it would be on business cycle analysis. One possible angle for the business cycle analysis is to organise the discussion around the notion of forecastability. If the analysis holds up, the title of the book might be something like "Forecastability of the Business Cycle." The analysis would be using stock-flow consistent models, and that will be the trigger for me to bulk out my sfc_models Python framework.)

Analogy: Controllability

The idea of forecastability is by analogy to controllability (and observability) from control systems theory. Once we write down the mathematical specification of a state space model of the system we wish to develop a control system for, we can then use a criterion (which depends upon the class of system) to test whether the system is controllable. If it is controllable, there exists a control law that can push the state of the system from any initial state to a target state in finite time.

If it is not controllable, we have the situation where there are target states which cannot be achieved under any possible control law. As a simple example, take any standard (non-flying!) car. We generally can drive the car to any point on a flat stretch of terrain, but there is no mechanism for us to put the car at any arbitrary altitude. If we define the state space as including the altitude, the car is uncontrollable.

Controllability is a useful feature for dealing with pointy-haired bosses. One can imagine such a person insisting that since the control engineer has a Ph.D. from a fancy-pants university, they should be able to develop a control system that allows the car to fly at any altitude desired by the driver. The suffering control engineer can just point to the mathematical definition of controllability, and show that such a task is theoretically impossible.

Many economists are in the career position of the junior control engineer: they have superiors that want them to do the impossible (generate accurate point forecasts). This is extremely awkward, but mainstream economist profession managed to find a way to dodge the question. Instead of giving a raw forecast, they prefer to express the effects of policies as causing deviations from an (unknown) baseline forecast: "if x happens, GDP will grow y% faster over one year." This sounds scientific, and keeps everyone happy. However, such a formulation is by its nature non-falsifiable, and it does not dispel cynicism about economist forecasts among non-economists.*

One final point about the analogy. In control systems theory (particularly linear systems theory), it is possible to find a number of conditions that are mathematically equivalent to controllability. That is, there are a set of properties (of the system in question) x, y, z,... such that the system is controllable if and only if ("iff" in blackboard-speak) property x (or y, z) holds. This means that we could use any of the properties x,y,z,... instead our original definition of controllability. This is despite the fact that the definitions appear different. When extending the analogy to forecastability, we could run into the problem that there are a number of potential definitions one could use, and although they might be equivalent for certain classes of models, they might differ for other classes.

Models Versus Reality

My discussion of controllability above deliberately skated over one issue: we need to draw a line between mathematical models and reality. The example I gave appears intuitively obvious, but there are technical details to consider.

The definition of controllability, like all well posed applied mathematics, a statement about sets. Once we write down a formal system that covers the model of interest, controllability is a statement about the existence of a control input u given an initial state x and a terminal state y. It can only be tested against a specific mathematical model.

If we look at the question of driving a car in the real world, there are an infinite number of models one could use. The simplest (and which ignores a good deal of physics) is to assume that any given time, we can set the direction of travel and speed of travel (both forward and backward) independently. This is equivalent to us being able to set the x and y velocity to arbitrary levels on a 2-D surface. (The state vector is the x and y position, as well as the x and y velocities.) This would probably be entirely adequate if we were discussing a path-planning problem where we need to find the optimal trajectory to follow given some routing constraints (including speed and acceleration limits). We can plug this model into the controllability definition, and see that the system is indeed controllable on a 2-dimensional surface. However, if we augment the state space to include the altitude (z axis) as well as the z velocity, the model fails the controllability test.

This model breaks down if our desired trajectory pushes limits. We cannot change the direction of travel and speed arbitrarily quickly, which is very apparent if we are driving on ice or a gravel road. We would need a more complex model to capture more realistic driving dynamics, which takes into account angular momentum, tire facing and traction, etc. That said, we can use intuition from physics to be able to see that even with these added dynamics, we cannot move the car to arbitrary altitudes (outside of Hollywood physics...).

This means that in the driving example we gave, we are somewhat lucky. The mathematical concept of controllability is robust across a variety of models for the same physical system. However, this will not always be the case. I have not kept up with the developments in robust control theory since 1998, but as of that era, the notion of controllability was still tied to particular models. This means that it was not tied to the notion of model robustness.**

This limits our ability to make assertions about the forecastability of economies (like I did earlier in this article...). Technically, we can only characterise the non-forecastability of particular models (or classes of models). One could then find "similar" models that are in fact forecastable. If we want to make statements about the real world, we would need to contrast the two models, and see which better fits observed data.

A Potential Definition of Forecastability

Although I am emphasising the formal mathematical nature of the definition, I will instead discuss the qualities that we hope to capture with a formal definition. In order to give a formal definition, we need to come up with a very general definition of what is an economic model, and that exercise could easily turn into a self-indulgent wave of definitions. Until this concept is better pinned down, that exercise would be a waste of time.

One unusual feature of the notion of forecastability is that we need some means to specify the idea of limitations of our knowledge of the state of the economy, or even the model dynamics. It makes little sense to discuss forecastability from the perspective of an omniscient observer.

To illustrate the need for information limitations, let us imagine a video game which is a conflict between two computer players ("artificial intelligence") in which the game features "fog of war": not all information for the game state is available to the players. (A game like chess has no fog of war.) The fog of war could result from limited observations of enemy troops in a war game, or it could be lack of knowledge of the opponent's cards (and the deck(s)) in a card game. Each computer player attempts to win, using its available information set. Unless the game is rigged (or reached a "no win" state), neither player could predict the outcome of the game. However, from the perspective of the person who has set up the computer game, all of the information is presumably available. The game just needs to be run to determine the outcome, and so the outcome could be "forecast." If we allow for randomness in behaviour or within the game rules, we can just run the game a large number of times to get an empirical estimate of the probability distribution of outcomes.

The need to specify a limited information set creates a contrast between a definition of forecastability and the existing concepts of controllability and observability.

We will now turn to steps needed to define forecastability.

The first step is defining what we mean by an economic model. Rather than coming up with an abstract definition, we just assume that it captures standard classes of economic models, such as stock-flow consistent (SFC) models as well as DSGE models. Within this definition, there will have to be a notion of all the variables within the model. I will refer to this here as the state vector out of habit, but technically the standard definition of the state vector in dynamic systems theory is a subset of all the variables.

We then need to decompose the state vector into public and private information, which is a step that presumably does not appear in most treatments of economic models.

• Public information includes data like economic aggregates and prices, which in principle can be determined by statistical agencies.
• The private information would be mainly the information that describes behaviour of economic agents (sectors), which includes fixed behavioural parameters. (This means that we need to augment the traditional state concept from system theory with these behavioural parameters; in engineering systems, those parameters would be part of the system dynamics.)

There is a lot of hidden details in the division between public and private (or hidden) information.

• The dynamics of the model could either be treated as public or private. For example, a production functions could be either private or public information. For example, one could imagine models where competing firms use different technologies, and they are not entirely sure what the technical capabilities of their competitors are. It is not obvious how to incorporate uncertainty around the model structure into the definition.
• Things like accounting identities are public information, which sensible agents would take into account when making decisions. 
• Government policy could be either public or hidden. We could have models in which the central bank commits to a monetary policy rule, and the parameters of this rule are assumed to be known.
• The treatment of not directly measured variables could be complicated. We can imagine models in which all actors believe that there is some notion of an output gap that is an important driver of inflation trends, and that an output gap model is actually correct. However, based on real-world experience, it is unreasonable to believe that all agents agree what the level of the output gap actually is. As a result, we could have a model in which there is a hidden "true" output gap, as well as variables that are output gap estimates -- which is what forecasts would have to use. This is very different from an omniscient observer, which would have access to the model's true output gap as a mathematical entity.

Finally, we then define exact forecastability for a deterministic system as: can an observer "predict the future trajectory" of the public state variables on some horizon, when it only has access to public data?  The definition would be with reference for some time t, and so the only information to be used in the forecast is the history before time t, along with the future information of public exogenous variables. (That is, we assume that we have access to the future path of those exogenous variables. We can interpret the forecast provided as being conditional on the path of the exogenous variable; if the exogenous variable does not evolve as specified, the forecast is obviously incorrect.)

The "predict the future trajectory" is non-mathematical. I think that we could convert this to a mathematical statement as follows. It is equivalent to the existence of an operator that maps the past history to forecasts, but the private/public information distinction might create some difficulties.

My suspicion is that it will be necessary to inject some notion of measurement error into the definition, and then we switch to a new version where the resulting forecast errors are "contained" relative to the measurement errors. I think it will be necessary to look at more interesting examples, and see how we can come up with a generalisation of exact forecastability to capture the desired behaviour.

I return to the question of models with random variables at the end of this article.

Example: Model SIM

I will use Model SIM -- the simplest non-trivial stock-flow consistent (SFC) model. I have written about this model extensively in the past, as it is easiest to explain concepts with the simplest model.

For more details, the reader is directed to:

• My online articles that refer to the model: http://www.bondeconomics.com/search?q=model+SIM; or
• my book on SFC models.

The model is a straightforward three sector model, with a household sector, business sector, and government. The business sector does not do very much; it is assumed to always break even, and so it has no holdings of financial assets (nor are there inventories). Government spending is specified as an exogenous time series, and essentially determines the other variables (once behavioural parameters are fixed).

This example shows some of the awkwardness of the definition of information availability. It would presumably be defined as having access to the time series of historical data. I describe the method of the solution determination in the article "Finding the Solution in a Simple SFC Model." In summary, each period, the supply and demand functions are lined up and simultaneously solved. In particular, in order for the business sector to always break even, it has to be able to predict demand exactly in the current period (so that it can match production to demand). So the solution method implies that the business sector has "perfect information" about the current period supply/demand situation. This is shoved under the carpet if we define the information availability in terms of historic time series; we can view the solution determination to be some kind of equilibrating procedure that is buried within the macro specification. (If we had a pure agent-based model, this problem would disappear.) In other words, the notion of "information availability" might not match one's intuition.

For now, we assume that all behavioural parameters (including the tax rate) are constant. Furthermore, we assume that all entities have access to the full model specification.

We can now turn to the question of private and public information.

• The household consumption function is defined in terms of a pair of propensity to consume parameters (out of income, out of wealth). Only the household sector knows what those parameters are for sure. If there is no source of measurement error, my guess is that it would be generally possible to determine exactly those parameters with access to a finite history of public variables.*** 
• The business sector is constrained to break even, and presumably all entities know this, and so it has no hidden variables associated with its behaviour. More complex models would allow for business sector behavioural variables.
• Government policy is specified in terms of government consumption and a fixed tax rate. The tax rate has to be public information (as otherwise collection is somewhat difficult...); the status of government consumption will be discussed below.
• All other time series are public.

From the perspective of the government, model SIM is (generally) forecastable (after a period of observation). It knows what it is planning to do with respect to government consumption, and it can back out the household sector behavioural parameters from observed data. It can then infer what the response to its consumption time series by plugging in the household propensity to consume parameters into its copy of the model.

For the household, forecastability depends upon how we treat government consumption. If it knows the future trajectory of government consumption, it knows its own consumption function, and can infer the path of all variables. If the household sector has no idea what government consumption will be, it cannot predict what will happen. (Finding a solution in a period requires knowing the current value of government consumption.)

From any other perspective, the forecastability status is similar to the cases for the government and household. It is possible to estimate the household consumption function, but forecastability depends on knowing the future path of government consumption.

The fact that model SIM is forecastable under the proposed definition is desired. Qualitatively, if we know the path for government spending, we can forecast what will happen, as the dynamics are almost entirely driven by a multiplier on government spending (with lagging dynamics). We need more interesting business cycle models for them to become non-forecastable.

If we allow for non-public structural changes in the behavioural parameters (consumption function alpha parameters, tax rates), forecastability then depends on time. That is, imagine that we allow the possibility that some parameter changes from one value to another at some time t. Forecasts generated before time t will be invalidated, but after observing the data after time t, it will be possible to pin down the new value for the parameter. Once again, it would be possible to make accurate forecasts.

The Awkwardness of Randomness

If we introduce randomness, we need to relax the definition. I currently do not have a good feeling how this could be done. One might be tempted to follow in the lines of neoclassical economics and finance and use a definition along this line: the expected value of forecasts will equal the expected value of the variables in the true model.

The problem is that this is a very weak condition (although it will be very congenial to neoclassical academics). It turns out to be the same thing as market efficiency -- on average, current market pricing is an unbiased forecast of future pricing (modulo financing costs). The implication is that it is difficult to make super-normal returns in the market, which is a statement that appears to fit observed behaviour. That is, one could argue that it holds (in the context of financial markets).

In fixed income terms, the use of the previous definition using expectations would be equivalent to saying that forward rates are identical to forecast future rates. This is an unreasonable use of the term "forecast": sure, it may be difficult to make money trading forward 10-year rates, but it is abundantly clear that realised 10-year rates only hit previous forwards on exceedingly rare occasions.

Instead, we would need to have a notion of a confidence interval: there is a p% chance that realised future variables are within some tolerance of the forecast. So long as we impose somewhat strict conditions on the tolerance, 10-year U.S. Treasury forwards would not meet this test. However, it might be true for short-rate forecasts: outside of crises, forwards are pretty good predictor of short rates.

(It is extremely likely that I will return to this topic at greater length. If one were to take the DSGE modellers' description of DSGE models at face value, the models would appear to imply forecastability using the expectations definition. However, I think it is possible to raise reasonable objections to that characterisation. It is clear that the status of DSGE models would be the greatest challenge to coming up with a formal definition for forecastability.)

Concluding Remarks

This will likely be the beginning of a sequence of articles discussing the forecastability of business cycle models. As model SIM shows, it is not correct to say that all macro models will be non-forecastable. My conjecture is that if we confine ourselves to "interesting" business cycle models, non-forecastability will be a standard feature. If the conjecture is correct, and if one accepts that these business cycle models capture reality, we should expect real world forecasting exercises to be doomed to failure.

Footnotes:

* The sensible approach is to give conditional forecasts: under a certain assumption, this is what we think will happen. Many mainstream academic economists (as well as heterodox academics) correctly argue that this is what we are supposed to be doing. However, I spent my entire career in industry interacting with economists, and it is clear to me that this is not what industrial employers of economists want them to do. I do not give forecasts in my writings since I view expectations about what forecasting can achieve as ridiculous. One could try to adapt the definition I outline here to handle the notion of conditional forecasts, but at the time of writing, I see some pitfalls with a formal definition of condition forecastability. It would be very easy for a prospective definition of conditional forecastability ending up being mathematically equivalent to forecastability, whereas in my view, it needs to be a much weaker property.

** For linear time-invariant systems, model robustness was largely a frequency domain concept. In the frequency domain, non-controllable (and non-observable) subspaces of the state space disappear from the mathematical specification. As a result, almost all linear robust control theory only works with systems that are controllable and observable by definition. For nonlinear systems, this may no longer be the case, but the inability to characterise the solutions to nonlinear systems in a clean fashion made the issue moot.

*** There will be trajectories for which this is impossible. For example, if wealth in the historical data is always zero. we have no way to calibrate the propensity to consume out of wealth. (More generally, the matrix used for estimation would not be invertible.) However, the set of histories for which this is true is a set of measure zero relative to the space of all possible histories. This is a topic that will probably be revisited in later articles.

(c) Brian Romanchuk 2018