Frisch-Waugh-Lovell Theorem: Animated

Update: The code for these animations is available here.

Another Update: I think some of the explanations on this page may be helped with more colors. I have some updated visuals here that include colors.

The Frisch-Waugh-Lovell theorem states that within a multivariate regression on and , the coefficient for , which is , will be the exact same as if you had instead run a regression on the residuals of and after regressing each one on separately.

The point of this post is not to explain the FWL theorem in linear algebraic detail, or explain why it’s useful (basically, it’s a fundamental intuition about what multivariate regression does and what it means to “partial” out the effects of two regressors). If you want to learn more about that, there’s some great stuff already on Google.

The point of this post is to simply provide an animation of this theorem. I find that the explanations of this theorem are often couched in lots of linear algebra, and it may be hard for some people to understand what’s going on exactly. I hope this animation can help with that.

Our Data

import numpy as np
import pandas as pd
import statsmodels.api as sm

np.random.seed(42069)

df = pd.DataFrame({'x1': np.random.uniform(0, 10, size=50)})
df['x2'] = 4.9 + df['x1'] * 0.983 + 2.104 * np.random.normal(0, 1.35, size=50)
df['y'] = 8.643 - 2.34 * df['x1'] + 3.35 * df['x2'] + np.random.normal(0, 1.65, size=50)
df['const'] = 1

model = sm.OLS(
    endog=df['y'],
    exog=df[['const', 'x1', 'x2']]
).fit()

model.summary()

The output of the above:

OLS Regression Results

The Animation

Here is what would happen if we actually ran a univariate regression on the residuals after factoring out .

(The animation takes a few seconds, so you might need to wait for it to restart to get the full effect.)

Notice that the slope in the final block ends up equaling 3.1931, which is the coefficient for in the multivariate regression.

Getting the coefficient for is more interesting; one thing that happens in the multivariate regression is the coefficient is negative despite the fact that is positively correlated with . What gives? Well, the following animation helps to show where that comes from:

You can mostly see here what’s happening: After we take out the effect of on , what we’re left over with is a negative relationship between and . Put another way: there is a negative correlation between and the residuals from the regression .