Why You Should Never Use The Hodrick-Prescott Filter

One very common task in finance and economics is to calculate the underlying trend of a time series. This is a well-known problem in communication systems, and it is accomplished by designing a low-pass filter: a filter that eliminates high-frequency components of an input. For hard-to-understand reasons, some economists use the Hodrick-Prescott Filter (the "HP Filter") as a low-pass filter. Unfortunately, the HP Filter violates several principles of filter design, and generates misleading output. As a result, it should never be used. Although this topic sounds fairly technical, problems can be easily illustrated graphically. Even if you are not interested in filtering series yourself, these problems must be kept in mind when looking at economists' research if it is based upon the use of this filter. The conclusions may be based on defects created by the filtering technique.

Background

The implementation of the HP Filter is quite complex, and my guess is that this complexity gives the technique an aura of sophistication. Since you should never use the filter, there is no reason to look at the mathematics behind it. But if you wish, the HP filter is described in this R language documentation (the mFilter package, I used this package to generate the results herein).

The chart above shows a typical use of the HP Filter - using it to calculate "trend" (real) GDP. The top panel shows the original series and the filter output, which is referred to as the trend within the economics literature. The trend is a nice smooth line that fits through the GDP series. (Please note that I am using this as an example that provides a simple intuition for to the problems with the HP filter. As I note below, although the HP filter was used for this purpose historically, most serious attempts to calculate potential GDP now use different techniques.)

(On a technical note, I calculated the series as follows. I first transformed GDP by taking the logarithm of its value. This converts steady exponential growth to a straight line. I then calculate the trend of the log GDP, either using the HP filter or an alternative technique. Finally, I calculate the trend for GDP by taking the exponential of that trend series. All of my calculations use the smoothing parameter value of 1600, which is the same value that everyone else uses for quarterly data. Why 1600, and not 100? There is a long explanation (which may or may not be convincing); Professor Karl Whelan has lecture notes with a summary explanation. [UPDATE: Please note that the original explanation of the "1600" parameter choice was sarcastic and completely unfair. Apparently, I am in a more generous mood today, and updated the text. I would like to thank the commenter 'elsurexiste' for providing the reference and rapping my knuckles about my remark.]

The bottom panel shows the deviation of GDP from the trend calculated by the HP filter. It is shown as a percentage of GDP.  Interestingly enough, it shows that U.S. real GDP is 1% above trend, which should be scary if you are a bond bull - and if the HP filter was reliable (I discuss this further below).

Since the filtered series I calculate are quite close to each other in level terms, I will now demonstrate filter outputs by the deviation in trend that they generate.

As an aside, many people use a deviation of GDP from some trend as a measure of the "output gap". I would point out that the two concepts should be separated, as we need to make strong model assumptions to justify the belief that the output gap should be close to trend GDP. As an example, the relatively low rate of growth of GDP since the end of the Financial Crisis implies that trend GDP growth is lower, but that does not necessarily mean that potential GDP growth is low.

Why The HP Filter Fails For "Real-Time" Usage

The most important problem with the HP filter can be demonstrated by how it behaves under different scenarios. In the top panel of the chart above, I show three time series of the growth rate: historical data (until 2014q4, in black), a scenario of "strong growth" (more accurately, "stronger growth" -  a 3% annual rate, in red), and a scenario of a mild recession (2% contraction over 2 quarters, and then returning to 1.5% annualised growth, in blue).

The bottom panel shows what happen to the HP filter output, depending upon the scenario. The deviation from trend at the end of the data set is revised. If we have stronger growth, the estimate of real GDP being above trend at the end of 2014 is revised away, and it appears that the economy was only near the trend value. Conversely, if the economy drops into recession, the economy looks like it was as far above trend as it was in 2007.

This happens because the HP filter is non-causal, the value at a point in time depends upon future values of the time series. We can calculate a non-causal filter output if we have access to the series' data over all time, but we cannot calculate the output values in real time. For this reason, engineers do not look at non-causal filters, other than in the initial lectures of a course on systems theory.

Most statistical agencies figured this out some time ago, and they no longer use the HP filter for things like estimation of trend GDP. (I have not really looked at the new techniques being used, which are varied.) The economists that continue to use the HP filter appear to be theoreticians who are not too worried about econometrics as well market economists (and bloggers) who throw together analyses fairly quickly.

Within finance, practitioners often want to use smoothed series as inputs to trading rules. Non-causality immediately eliminates the HP filter from consideration, as any trading rule that is based upon future prices should generate strong returns in backtesting.

Historical Analysis

One could then argue that perhaps the HP filter can be used for historical analysis. Since the data that we are looking at is fixed (barring revisions), the non-causality might not be an issue.

The problem with this line of argument is that the HP filter output is unreliable near the endpoints of the data set. Unfortunately, we do not know what "near" means.

The top panel of the chart above shows how the estimation of the deviation from trend depends upon the sample you choose. The black line shows the estimated deviation from trend when we use a data set that runs from 1990q1 to 2007q4, and the red line uses the data set that ends in 2014q4. The endpoint of the shorter sample (end of 2007) shows GDP near trend, which is the inherent tendency of the HP filter. This is completely different from the picture that is given by using the larger sample.

The bottom panel shows the analysis using a sensible filter - the centered* moving average (I used 21 periods, or 5 years and 1 quarter). The centered moving average is a moving average where the output is time shifted by a "lead" of half of the moving average. In this case, the 21-period moving average is the average of a point in time, plus the time points that are within 10 periods on either side. (Since there are 10 periods on each side, plus the middle point, there are 21 points to be averaged.) Like the HP filter, the centered moving average is non-causal.

The centered moving average is not defined over the entire interval, as we lose the first and last 10 points in the data set. But where it is defined, it is very close to the HP filter value. As I would phrase it, the HP filter is almost as good as the centered moving average. (I have made no effort to adjust the filter parameters so that the outputs are closer together.)

Additionally, the centered moving average tells us where to throw out the data points at the end of the data - there is no output defined for where the results would be sketchier. The HP filter methodology offers no insight into where we need to lop off the unreliable endpoint data.

A communication or control system engineer would object that moving averages (including centered moving averages) are generally not used within engineering. They have some unfortunate tendencies to magnify some high frequency noise. This can be interpreted as the "base effect" that you see in the output. (A shock hits a moving average both when it enters the average, and when it drops out.) Typically, low order transfer functions are used; the most basic of which is known as the "exponential moving average" in economics. If I were building a trading rule, I would use these filters instead of a moving average, but for my purposes here, I use the simplest to explain filter - the moving average. My feeling is that almost all my target audience with backgrounds in finance or economics knows what a moving average is; the exponential moving average is unfortunately exotic. (For electrical engineers, the situation is reversed.)  See my primer on "adaptive expectations" for more discussion of the exponential moving average filter.

What Do We Do In Real Time?

A centered moving average is reasonable solution for historical data analysis. For estimates of current values, solutions appear less satisfying. The reason is that if we confine ourselves to causal filters, we have an unavoidable time lag built into our filter outputs.

The situation for the centered moving average is the easiest to explain. Systems theory texts typically start with the non-causal centered moving average, as it has the cleanest frequency domain representation. We then add a time lag to make it into a standard moving average, which is causal. In other words, a moving average is best thought of as a centered moving average plus a time lag.

For other filters, the proof of why a time delay is introduced requires the use of frequency domain analysis. There is the equivalent of a conservation law that explains why you have a trade-off between filtering high frequency signal components and the time delay in the output. The proof of this can be found in most digital signal processing textbooks.

One apparent exception that is well known within economics is the concept of seasonal adjustment. When they seasonally adjust data, statisticians hope to cancel out seasonal noise using a model-based approach. Unfortunately, these models will only work if the seasonal pattern is stable, which is not always the case. In order to avoid a bias, seasonal adjustment has to be done in a non-causal fashion.  Seasonal adjustment is more of an art than a science.

A More Formal List Of The Defects Of The HP Filter

I will now give a more formal short critique of problems with the HP filter.

1. Non-causality. As discussed above.
2. Unknown amount of invalid points at ends of interval. This problem is extremely critical; we do not know where the data are garbage.
3. Computationally intensive. Wastes resources.
4. Black box implementation.  It is difficult to see what the properties are of this solution, unlike alternatives that have clean frequency domain representations.
5. "Magic" input parameter. It is very difficult to see what the "lambda" parameter represents. (Yes, there is a "story" behind it, but it is still not particularly helpful when compared to sensibly designed digital filters. The story relies on arbitrarily long inputs, which is exactly not a characteristic of ral-world economic series.)

There are no advantages of the HP filter over a centered moving average, and it has the disadvantages listed above. One can use a more general non-causal Finite Impulse Response (FIR) filter to get a smoother frequency domain response than the centered moving average if desired, but that would require looking at an undergraduate text in digital communications system in order to understand how to shape the filter response.

Footnote:

* "Centered" versus "centred"? The first spelling would be the American preference, whereas the second English. I follow Canadian spelling patterns, which veer erratically between American and English usage, which probably causes people on both sides of the Atlantic to believe that I cannot spell. As for "centre", the usage in Canada seems to be drifting towards a view that "centre" refers to a building, while "center" is the point in the middle of something. Therefore, you could possibly see a sentence like "The shopping centre is at the center of the map." Since "centred" looks too much "cent-red", I'll stick with "centered".

(c) Brian Romanchuk 2015