Turning On the Mixing Board

Module 3 · AR(p), MA(q), and the Fingerprints They Leave

Gary Cornwall

Econ 6376 · The George Washington University

Previously · the verdict

What Module 2 settled

A whole session on one question — is this series stationary? — and no one-number answer, because there isn’t one.

  • Stationarity is a property of the process, not of the one realization you were handed.
  • No single exhibit is conclusive. You build a case.
  • When the exhibits conflict, the cost asymmetry breaks the tie. Spurious regression is the wrongful conviction you can’t undo.

Everything today assumes that verdict is already in.

Stationary, precisely

Weak (covariance) stationarity — the three conditions from Module 1:

𝔼[yt]=μ,Var(yt)=σy2<,Cov(yt,ytk)=γk\mathbb{E}[y_t] = \mu, \qquad \operatorname{Var}(y_t) = \sigma_y^2 < \infty, \qquad \operatorname{Cov}(y_t,\, y_{t-k}) = \gamma_k

The third is important: the covariance depends on the separation kk, never on the date tt.

Strict stationarity is the stronger version — the entire joint distribution is shift-invariant:

(yt1,,ytm)=d(yt1+h,,ytm+h)for every h(y_{t_1}, \ldots, y_{t_m}) \;\overset{d}{=}\; (y_{t_1+h}, \ldots, y_{t_m+h}) \quad \text{for every } h

Strict \Rightarrow weak, given finite second moments. Not the reverse.

We never need it. Every instrument in this course — the correlogram, the DF regression, both fingerprints we build today — is assembled out of means, variances, and covariances. “Stationary” here always means weakly stationary.

Four exhibits

  1. The plot. Does it wander off without coming back? Does the spread change? Is there a trend?
  2. The correlogram. Fast decay \Rightarrow stationary. Slow, near-linear decay \Rightarrow unit root.
  3. The tests, in pairs. ADF (H0H_0: unit root) and KPSS (H0H_0: stationary) — opposite nulls, so agreement carries information and both failing to reject means the data cannot discriminate.
  4. The ratio. σΔy/σy<0.5\sigma_{\Delta y}/\sigma_y < 0.5 \Rightarrow — a sanity check, not a test.

Hold on to exhibit 2. In Module 2 the correlogram was a yes/no instrument: does this decay fast enough? Today it becomes an identification instrument — and it gets a second barrel.

One case, start to finish

Pick a case, read the four exhibits, then press Difference it until the two tests agree. The ladder on the right records what you found at each level.

I(d)I(d) is a marker or record, not a definition

ytI(d)d is the 𝐬𝐦𝐚𝐥𝐥𝐞𝐬𝐭 number of differences that makes yt stationaryy_t \sim I(d) \quad\Longleftrightarrow\quad d \text{ is the } \textbf{smallest} \text{ number of differences that makes } y_t \text{ stationary}

  • I(0)I(0) — already stationary. Nothing to do.
  • I(1)I(1) — one difference does it. Most macro series in levels: GDP, employment, prices.
  • I(2)I(2) — two. Occasionally price levels. Rare, and worth double-checking.

You did not look dd up in a table. dd is the number of times you had to press the button.

And one press too many is its own diagnosable mistake — that ρ10.5\rho_1 \approx -0.5 in firebrick. Remember what it looked like. It comes back this later with a name.

Today: the other two letters

ARIMA(p,dModule 2,q)\text{ARIMA}(\,p,\; \underbrace{d}_{\text{Module 2}},\; q\,)

Getting dd right is what makes everything downstream “legal”. But dd tells you nothing about what kind of series you have once you get there.

Difference a random walk and you get white noise — nothing left to model. Difference a real macro series and you get something stationary that still has structure in it: this quarter still knows something about last quarter.

That leftover structure is pp and qq.

Act I · Turning on the board

“Every contact leaves a trace.” Act I — memory of yy, memory of shocks

The board we’ve been ignoring

Module 1 wrote down the master equation for the whole course:

yt=α+δt+j=1pϕjytj+l=1qθlϵtl+ϵty_t = \alpha + \delta t + \sum_{j=1}^{p} \phi_j y_{t-j} + \sum_{l=1}^{q} \theta_l \epsilon_{t-l} + \epsilon_t

and then turned almost all of it off. We kept α\alpha and ϕ1\phi_1 and studied the AR(1) for two modules. The Dickey–Fuller regression is a rearranged AR(1); even the trend specification only added δt\delta t.

Term After Module 2 After today
α\alpha on on
δt\delta t on (trend ADF) on
ϕ1yt1\phi_1 y_{t-1} on on
ϕ2yt2,,ϕpytp\phi_2 y_{t-2}, \ldots, \phi_p y_{t-p} off on
θ1ϵt1,,θqϵtq\theta_1 \epsilon_{t-1}, \ldots, \theta_q \epsilon_{t-q} off on
ϵt\epsilon_t on on

After today the only block you haven’t seen is the seasonal one. That’s Module 5.

Two kinds of memory

Both halves of the board are memory devices. They remember different things.

  • AR memory is memory of yy. How many days of my own history matter for today? An AR(pp) remembers the last pp values of the series; every older shock still reaches the present, but only indirectly, through those yy’s.
  • MA memory is memory of shocks. How many recent surprises are still rattling around in the system? An MA(qq) carries the last qq innovations explicitly — and then forgets them.

Your bank balance at the end of day tt:

  • AR(pp) — today’s balance depends on the last pp days’ balances. Rent, payday, standing orders. The rhythm repeats because your schedule does.
  • MA(qq) — today’s balance depends on the last qq surprises: the tax refund, the car repair, the check from your aunt. Once they wash through, they’re gone.
  • Most real series are both.

pp counts how far back your own history matters. qq counts how long a surprise survives.

AR(pp): turning up the rest of the faders

yt=α+ϕ1yt1+ϕ2yt2++ϕpytp+ϵt,ϵtWN(0,σ2)y_t = \alpha + \phi_1 y_{t-1} + \phi_2 y_{t-2} + \ldots + \phi_p y_{t-p} + \epsilon_t, \qquad \epsilon_t \sim \text{WN}(0,\sigma^2)

Collect the lags on the left, exactly as we did for the AR(1) in Module 1:

ytϕ1yt1ϕpytp=α+ϵty_t - \phi_1 y_{t-1} - \ldots - \phi_p y_{t-p} = \alpha + \epsilon_t

(1ϕ1Lϕ2L2ϕpLp)Φ(L)yt=α+ϵtΦ(L)yt=α+ϵt\underbrace{(1 - \phi_1 L - \phi_2 L^2 - \ldots - \phi_p L^p)}_{\textstyle \Phi(L)}\, y_t = \alpha + \epsilon_t \qquad\Longrightarrow\qquad \Phi(L)\, y_t = \alpha + \epsilon_t

The minus signs in Φ(L)\Phi(L) are not a convention chosen to annoy you. They come for free the moment the ϕjytj\phi_j y_{t-j} terms cross the equals sign.

Still ϵt\epsilon_t for the DGP innovation, still ete_t for a residual from something we estimated.

When does an AR(pp) sit still?

The condition. A stationary AR(pp) exists if and only if every root of Φ(L)=0\Phi(L) = 0 lies outside the unit circle in the complex plane.

Check it against what you already know. For the AR(1), Φ(L)=1ϕ1L\Phi(L) = 1 - \phi_1 L, one root at L*=1/ϕ1L^\ast = 1/\phi_1. “Outside” means |L*|>1|L^\ast| > 1, which is exactly |ϕ1|<1|\phi_1| < 1 — the Module 1 condition, unchanged.

The trap. Applying |ϕj|<1|\phi_j| < 1 to the coefficients one at a time is not the generalization. Every coefficient can be comfortably less than one and the process still non-stationary — you’ll watch it happen two slides from now.

The condition is on the roots of the joint polynomial. Roots are not coefficients.

AR(2): two roots, one quadratic

yt=ϕ1yt1+ϕ2yt2+ϵt,Φ(L)=1ϕ1Lϕ2L2y_t = \phi_1 y_{t-1} + \phi_2 y_{t-2} + \epsilon_t, \qquad \Phi(L) = 1 - \phi_1 L - \phi_2 L^2

Substituting r=1/Lr = 1/L turns the lag polynomial into the characteristic equation, where the arithmetic is cleaner and the condition turns inside out:

r2ϕ1rϕ2=0λ1,2=ϕ1±ϕ12+4ϕ22r^2 - \phi_1 r - \phi_2 = 0 \qquad\Longrightarrow\qquad \lambda_{1,2} = \frac{\phi_1 \pm \sqrt{\phi_1^2 + 4\phi_2}}{2}

Stationary |λ1|<1\iff |\lambda_1| < 1 and |λ2|<1|\lambda_2| < 1. Same statement as “roots of Φ(L)\Phi(L) outside the circle” — the two sets are reciprocals, so the picture just flips. Both conventions are in print; the widget below draws them both.

The discriminant is a one-glance classifier.

ϕ12+4ϕ2\phi_1^2 + 4\phi_2 Roots What the ACF does
>0> 0 a real pair decays smoothly — a mixture of two geometrics
<0< 0 complex conjugates a damped sine wave — crosses zero, repeatedly

Stationarity is a separate question, settled by the modulus, not by the sign of the discriminant.

Three examples, on the board

Do the arithmetic with me before anyone touches a preset.

(ϕ1,ϕ2)(\phi_1, \phi_2) ϕ12+4ϕ2\phi_1^2+4\phi_2 λ1,2\lambda_{1,2} Verdict
(0.5,0.3)(0.5,\ 0.3) 1.45>01.45 > 0 0.852,0.3520.852,\ -0.352 stationary, smooth decay
(0.8,0.3)(0.8,\ 0.3) 1.84>01.84 > 0 1.078,0.278\mathbf{1.078},\ -0.278 non-stationary
(0.6,0.5)(0.6,\ -0.5) 1.64<0-1.64 < 0 0.3±0.640i0.3 \pm 0.640i, modulus 0.7070.707 stationary, oscillating

Row 1 to row 2 changes one number: ϕ1\phi_1 from 0.50.5 to 0.80.8. Two coefficients, both smaller than one, and a root still walks out of the circle. That is why you check the roots.

The AR fingerprint: it tails off

We won’t derive the ACF of a general AR(pp) — that’s Yule–Walker, and it’s in Enders ch. 2. The pattern is what we need.

  • AR(1): ρk=ϕk\rho_k = \phi^k. Pure geometric decay. You’ve known this since Module 1.
  • AR(2), real roots: a mixture of two geometrics — still smooth.
  • AR(2), complex roots: a damped sinusoid — it crosses zero and comes back.
  • AR(pp): one geometric or damped-sine term per root, all added together.

The decay rate belongs to the largest-modulus root. The shape belongs to whether the roots are real or complex.

The ACF of an AR process tails off. It never cuts off.

Half the identification rules in Act III are that one sentence. Here comes the other half.

The other kind of memory

A hurricane closes a refinery on the Gulf coast. Within a week gas prices are up forty cents everywhere east of Texas. Six weeks later the refinery is back online, and so is the price.

That shock didn’t decay. It expired. It had a length — however long it takes to bring capacity back — and then it was over.

Ask an AR(1) the same question. Does a shock ever actually leave?

yt=ϕyt1+ϵtyt+hϵt=ϕhy_t = \phi y_{t-1} + \epsilon_t \qquad\Longrightarrow\qquad \frac{\partial y_{t+h}}{\partial \epsilon_t} = \phi^h

ϕh\phi^h gets small. It gets unmeasurably small. It is never zero.

AR forgets asymptotically. MA forgets absolutely.

Same shock, two systems

Identical innovations top and bottom, one hurricane at t0t_0. The ledger beside each series breaks today’s value into its pieces; the firebrick segment is the part still owed to the shock. Before we step it: at which period does the hurricane leave each system?

MA(qq) has a horizon — at t0+q+1t_0 + q + 1 the firebrick bar slides off the register and the series has no memory of the hurricane at all. AR(1) has a half-life — the bar shrinks by a factor of ϕ\phi every period and never reaches the edge.

MA(qq): the DGP

yt=α+ϵt+θ1ϵt1+θ2ϵt2++θqϵtq=α+ϵt+l=1qθlϵtly_t = \alpha + \epsilon_t + \theta_1 \epsilon_{t-1} + \theta_2 \epsilon_{t-2} + \ldots + \theta_q \epsilon_{t-q} \;=\; \alpha + \epsilon_t + \sum_{l=1}^{q} \theta_l \epsilon_{t-l}

In lag-operator form, with Θ(L)=1+θ1L+θ2L2++θqLq\Theta(L) = 1 + \theta_1 L + \theta_2 L^2 + \ldots + \theta_q L^q:

yt=α+Θ(L)ϵty_t = \alpha + \Theta(L)\, \epsilon_t

Look hard at the right-hand side and notice what isn’t there.

There is no yy on the right. An MA(qq) is built out of nothing but the last q+1q+1 shocks.

Every strange and useful property on the next four slides falls out of that one structural fact. The series is a weighted sum of a finite window of noise — so it cannot wander off, and it cannot remember anything older than the window.

Why Θ(L)\Theta(L) has plus signs

Polynomial Course convention Where the signs came from
Φ(L)\Phi(L) — AR 1ϕ1Lϕ2L21 - \phi_1 L - \phi_2 L^2 - \ldots the ϕjytj\phi_j y_{t-j} crossed the equals sign
Θ(L)\Theta(L) — MA 1+θ1L+θ2L2+1 + \theta_1 L + \theta_2 L^2 + \ldots the θlϵtl\theta_l \epsilon_{t-l} never moved

Enders and Hamilton agree with us. Box, Jenkins and Reinsel write the MA polynomial with minus signs, so their θ\theta is our θ-\theta. Check the convention before you copy a formula out of a book. notation_dictionary.md is the final word for this course.

Put both polynomials on one line and the whole master equation collapses to:

Φ(L)yt=α+Θ(L)ϵt\Phi(L)\, y_t = \alpha + \Theta(L)\, \epsilon_t

That is ARMA(p,qp,q) — and it is the object Module 4 estimates.

MA(qq) is always stationary

No conditions on the θ\theta’s. None. Check the three requirements directly:

𝔼[yt]=α,Var(yt)=σ2(1+θ12++θq2),γk=σ2l=0qkθlθl+k(θ01)\mathbb{E}[y_t] = \alpha, \qquad \operatorname{Var}(y_t) = \sigma^2\big(1 + \theta_1^2 + \ldots + \theta_q^2\big), \qquad \gamma_k = \sigma^2 \sum_{l=0}^{q-k} \theta_l \theta_{l+k}\ \ (\theta_0 \equiv 1)

The mean is α\alpha because every innovation has mean zero. The variance moves with the θ\theta’s but not with tt. And γk\gamma_k counts how many ϵ\epsilon’s yty_t and ytky_{t-k} share — which depends on the separation kk and never on the date.

AR(pp) MA(qq)
Stationarity a condition — roots of Φ(L)\Phi(L) outside the circle. You can fail it. automatic, free, no strings
The condition it carries instead invertibility

Stationarity and invertibility are not the same property. They get conflated every single year. Three slides from now you’ll see why they can’t be the same thing.

Where the cutoff comes from

Two copies of the series, shifted by kk. Shared columns light up, the products print underneath, and they sum to γk/σ2\gamma_k/\sigma^2. Walk kk up one step at a time and watch the shared columns run out: at k=q+1k = q+1 the two rows have nothing in common and the sum is empty.

Then flip to Φ(L)\Phi(L) — the same diagram built from ψ\psi-weights that never end, so the rows never come apart. One picture has an edge. The other doesn’t. That’s the whole difference.

The algebra, once, by hand

yt=ϵt+θϵt1,𝔼[ϵsϵt]=0(st),𝔼[ϵt2]=σ2y_t = \epsilon_t + \theta \epsilon_{t-1}, \qquad \mathbb{E}[\epsilon_s \epsilon_t] = 0 \ (s \neq t), \qquad \mathbb{E}[\epsilon_t^2] = \sigma^2

γ0=Var(ϵt+θϵt1)=σ2+θ2σ2=σ2(1+θ2)\gamma_0 = \operatorname{Var}(\epsilon_t + \theta\epsilon_{t-1}) = \sigma^2 + \theta^2\sigma^2 = \sigma^2(1+\theta^2)

γ1=𝔼[(ϵt+θϵt1)(ϵt1+θϵt2)]=𝔼[ϵtϵt1]0+θ𝔼[ϵtϵt2]0+θ𝔼[ϵt12]σ2+θ2𝔼[ϵt1ϵt2]0=θσ2\gamma_1 = \mathbb{E}\big[(\epsilon_t + \theta\epsilon_{t-1})(\epsilon_{t-1} + \theta\epsilon_{t-2})\big] = \underbrace{\mathbb{E}[\epsilon_t\epsilon_{t-1}]}_{0} + \underbrace{\theta\,\mathbb{E}[\epsilon_t\epsilon_{t-2}]}_{0} + \theta\underbrace{\mathbb{E}[\epsilon_{t-1}^2]}_{\sigma^2} + \underbrace{\theta^2\,\mathbb{E}[\epsilon_{t-1}\epsilon_{t-2}]}_{0} = \theta\sigma^2

γ2=𝔼[(ϵt+θϵt1)(ϵt2+θϵt3)]=0\gamma_2 = \mathbb{E}\big[(\epsilon_t + \theta\epsilon_{t-1})(\epsilon_{t-2} + \theta\epsilon_{t-3})\big] = 0

Every term is a product of two different innovations. Nothing is shared, so there is nothing to average. The same argument kills γk\gamma_k for every k2k \ge 2.

ρ1=θ1+θ2,ρk=0 for all k2\boxed{\;\rho_1 = \frac{\theta}{1+\theta^2}, \qquad \rho_k = 0 \ \text{ for all } k \ge 2\;}

A free diagnostic — and a problem

Treat ρ1=θ/(1+θ2)\rho_1 = \theta/(1+\theta^2) as a function of θ\theta and find its extremes:

maxθθ1+θ2=12 at θ=1,minθθ1+θ2=12 at θ=1|ρ1|0.5\max_\theta \frac{\theta}{1+\theta^2} = \tfrac{1}{2} \ \text{ at } \theta = 1, \qquad \min_\theta \frac{\theta}{1+\theta^2} = -\tfrac{1}{2} \ \text{ at } \theta = -1 \qquad\Longrightarrow\qquad |\rho_1| \le 0.5

For every MA(1), whatever θ\theta is. So if the sample |ρ̂1|>0.5|\hat\rho_1| > 0.5, the series cannot be a pure MA(1). No test, no critical value, no software — just algebra you did on the last slide.

Now the problem. Feed θ\theta and 1/θ1/\theta into the same formula:

ρ1(θ)=θ1+θ2,ρ1(1/θ)=1/θ1+1/θ2=θθ2+1\rho_1(\theta) = \frac{\theta}{1+\theta^2}, \qquad \rho_1(1/\theta) = \frac{1/\theta}{1 + 1/\theta^{2}} = \frac{\theta}{\theta^{2}+1}

Identical. θ=0.7\theta = 0.7 and θ1.43\theta \approx 1.43 produce the same autocorrelation at every lag, so they produce the same correlogram, the same second moments, the same everything we can measure. The data cannot tell them apart.

Invertibility: which θ\theta do you get to keep?

Two parameter values, one correlogram. Something has to break the tie, and it isn’t the data.

The question invertibility asks. Given the observed history yt,yt1,y_t, y_{t-1}, \ldots, can we recover the innovations ϵt\epsilon_t that produced it?

  • Yes — the representation is unique, the θ\theta’s are identified, and MLE has one peak to climb.
  • No — two configurations carry the same likelihood, and MLE either refuses to converge or pins the estimate to the boundary.

The condition. An MA(qq) is invertible \iff every root of Θ(L)=0\Theta(L) = 0 lies outside the unit circle.

Side Property Polynomial Roots
AR stationarity Φ(L)\Phi(L) outside the circle
MA invertibility Θ(L)\Theta(L) outside the circle

Same language, same geometry, different polynomial — and different meaning. The MA is stationary either way; invertibility is about whether you can identify it. Of the reciprocal pair, exactly one is invertible. That’s the one you’re allowed to report.

The mirror, on the board

Same widget, same algebra, one sign flipped: Φ(L)\Phi(L) mode asks is it stationary, Θ(L)\Theta(L) mode asks is it invertible, and the region turns upside down.

At θ1=0.70\theta_1 = 0.70 the readout says L=1.43L = -1.43: the root sits outside, invertible. Note the ACF — one spike at lag 1, nothing after. Now drag the θ1\theta_1 slider to 1.431.43: the reciprocal 1/0.71/0.7, the partner from two slides ago.

The ACF panel doesn’t move. But the root becomes L=0.70L = -0.70 and crosses inside: not invertible. The coefficient and the root have traded places — that is what “reciprocal pair” looks like, and it is the tie being broken in front of you.

The over-differencing trap, finally named

Module 2, differencing a series that only had a trend. Start from yt=α+δt+ϵty_t = \alpha + \delta t + \epsilon_t:

Δyt=δ+ϵtϵt1\Delta y_t = \delta + \epsilon_t - \epsilon_{t-1}

An MA(1) with θ=1\theta = -1. Its polynomial is Θ(L)=1L\Theta(L) = 1 - L, and the root of 1L=01 - L = 0 is

L=1— sitting 𝐨𝐧 the unit circle.L = 1 \qquad \text{— sitting } \textbf{on} \text{ the unit circle.}

Non-invertible. Not “nearly” — on the boundary, by exactly zero margin.

And look where θ=1\theta = -1 landed two slides ago. It is the precise value that attains ρ1=0.5\rho_1 = -0.5: the floor of the bound.

The firebrick spike you were told to remember is the bound being hit — which is also the edge of invertibility.

So θ̂\hat\theta pinned at 1-1 after differencing — with a standard error too good to be true — is the warning light. Two modules ago you took that on faith.

Two ways to write the same dynamics

A finite stationary AR(pp) and a finite invertible MA(qq) are two spellings of the same kind of memory. Each is the infinite-order version of the other.

One direction you already did. Module 1, recursive substitution on the AR(1):

yt=α1ϕ1+i=0ϕ1iϵtiy_t = \frac{\alpha}{1-\phi_1} + \sum_{i=0}^{\infty} \phi_1^{\,i} \epsilon_{t-i}

An AR(1) is an MA(\infty). Wold’s theorem says every covariance-stationary process has such a representation.

The other direction is what invertibility buys. Take θ=0.7\theta = 0.7, so Θ(L)=1+0.7L\Theta(L) = 1 + 0.7L:

Θ(L)1=11+0.7L=10.7L+0.49L20.343L3+\Theta(L)^{-1} = \frac{1}{1+0.7L} = 1 - 0.7L + 0.49L^2 - 0.343L^3 + \ldots

yt=0.7yt10.49yt2+0.343yt30.2401yt4++ϵty_t = 0.7\,y_{t-1} - 0.49\,y_{t-2} + 0.343\,y_{t-3} - 0.2401\,y_{t-4} + \ldots + \epsilon_t

πk=(θ)k\pi_k = (-\theta)^k, so π1=0.7\pi_1 = -0.7, π2=+0.49\pi_2 = +0.49 — alternating signs, decaying at rate |θ||\theta|. (The equation above is the rearrangement onto the yy side, so its coefficients are πk-\pi_k.) That series converges because the root is outside the circle. That is what invertibility does, operationally.

Check it in R

An MA(1) has one parameter. Fit it a high-order AR anyway and see what comes back.

set.seed(8675309)
y <- arima.sim(n = 5000, list(ma = 0.7))
ar(y, order.max = 8, aic = FALSE, method = "ols")$ar
lag kk 1 2 3 4 5
fitted π̂k-\hat\pi_k (what ar() returns) 0.6800.680 0.481-0.481 0.3400.340 0.224-0.224 0.1780.178
theory πk=(1)k+1(0.7)k-\pi_k = (-1)^{k+1}(0.7)^k 0.7000.700 0.490-0.490 0.3430.343 0.240-0.240 0.1680.168

It won’t be exact — 5000 draws of noise, and the far tail generally sags below theory (lag 5 here runs a little hot) because a fit truncated at 8 lags has to absorb everything past lag 8. But the sign alternation and the decay at 0.70.7 are unmistakable, and nobody put them there by hand.

One parameter — θ=0.7\theta = 0.7 — generated that entire row.

Small pp, small qq

Why this matters past the algebra:

  • A small AR captures dynamics that would need infinitely many MA weights.
  • A small invertible MA captures dynamics that would need infinitely many AR weights.
  • An ARMA(1,1) — two dynamic parameters — describes behavior a pure AR would need six or eight lags to imitate.

A finite ARMA is a compact code for an infinite-order process.

Three places we cash this in:

  1. Module 4 model selection is largely the hunt for the most parsimonious of several equivalent representations. AR(5) or ARMA(1,1)? Take the ARMA(1,1) — fewer parameters, less overfitting.
  2. Forecasting from an MA runs on the AR(\infty) form underneath, because forecasts use past observations, not unobserved shocks. Invertibility is what licenses that substitution.
  3. The ARMA trap in Act III: both representations are effectively infinite-order, so neither correlogram cuts off cleanly. You’ll see it, and it’s why Module 4 needs information criteria.

Act II · The second fingerprint

“Two prints, or it isn’t an identification.” Act II — the PACF

One fingerprint isn’t enough

Run the ACF across the whole family and look at what comes back:

Process ACF Can you read the order off it?
AR(1) tails off no
AR(2) tails off no
AR(5) tails off no
MA(1) cuts off at 1 yes
MA(2) cuts off at 2 yes

The ACF is a complete instrument for MA order and a useless one for AR order. It will tell you “this is an AR” and then refuse to say which one.

Look at the shape of that problem. We don’t need a better ACF — we need an instrument that does for AR what the ACF already does for MA: cut off at the order.

That instrument exists, and you already own every piece of it. It’s OLS.

The partial autocorrelation function

The partial autocorrelation at lag kk, written ϕkk\phi_{kk}, is the correlation between yty_t and ytky_{t-k} after removing the linear effect of everything in betweenyt1,yt2,,ytk+1y_{t-1}, y_{t-2}, \ldots, y_{t-k+1}.

How much does ytky_{t-k} tell me about yty_t that the lags in between haven’t already told me?

The ACF never asks that. ρk\rho_k is the raw correlation between yty_t and ytky_{t-k}, and it happily counts influence that arrived indirectly, passed down through the lags in between.

That is exactly why an AR(1) has a non-zero ρ2=ϕ2\rho_2 = \phi^2. There is no yt2y_{t-2} term anywhere in the AR(1) equation — but yt2y_{t-2} moved yt1y_{t-1}, and yt1y_{t-1} moved yty_t. The correlation is real, and it is entirely second-hand.

The PACF refuses to count second-hand information. That refusal is the whole idea.

The partial-effect interview

You’re predicting whether someone is late to work today.

“Were they late yesterday?” Useful — lateness is persistent. That’s the lag-1 correlation.

“Were they late two days ago?” This also looks useful. But yesterday and two-days-ago are themselves correlated: someone late yesterday was probably late the day before as well. So the honest question is

Does two-days-ago tell me anything beyond what yesterday already told me?

The answer to that question is ϕ22\phi_{22}.

Process Does yt2y_{t-2} add anything past yt1y_{t-1}? ϕ22\phi_{22}
AR(1) No — yt2y_{t-2} only ever reached yty_t through yt1y_{t-1} 00
AR(2) Yes — yt2y_{t-2} sits in the DGP with its own coefficient ϕ2\phi_2 0\neq 0

Push it out to pp: an AR(pp) has non-zero partial autocorrelation at lags 1,,p1, \ldots, p and exactly zero from p+1p+1 onward. There is the cutoff we went looking for.

It really is just OLS

“Remove the linear effect of the intermediate lags” is a phrase you already know how to execute. Regress on them. For lag kk, run

yt=β̂0+β̂1yt1+β̂2yt2++β̂kytk+ety_t = \hat\beta_0 + \hat\beta_1 y_{t-1} + \hat\beta_2 y_{t-2} + \ldots + \hat\beta_k y_{t-k} + e_t

and keep the coefficient on the deepest lag:

ϕ̂kk=β̂k\boxed{\ \hat\phi_{kk} = \hat\beta_k\ }

The sample PACF out to lag KK is KK regressions, each one lag deeper than the last, reading the bottom coefficient off each.

Why software doesn’t do it that way. KK separate regressions is wasteful, so pacf() pushes the Yule–Walker equations through the Durbin–Levinson recursion and gets the same numbers far faster. We won’t need that machinery — Enders ch. 2 p. 65 has it if you want it.

Keep the regression definition in your head anyway. It’s the one that explains what the number means.

Build it by hand

set.seed(8675309)
y <- arima.sim(n = 500, list(ar = c(0.6, -0.3)))   # an AR(2)

y_t <- y[5:500]; y_lag1 <- y[4:499]; y_lag2 <- y[3:498]
y_lag3 <- y[2:497]; y_lag4 <- y[1:496]

phi_11 <- coef(lm(y_t ~ y_lag1))[2]
phi_22 <- coef(lm(y_t ~ y_lag1 + y_lag2))[3]
phi_33 <- coef(lm(y_t ~ y_lag1 + y_lag2 + y_lag3))[4]
phi_44 <- coef(lm(y_t ~ y_lag1 + y_lag2 + y_lag3 + y_lag4))[5]

c(phi_11, phi_22, phi_33, phi_44)
pacf(y, lag.max = 4, plot = FALSE)          # should match
ϕ11\phi_{11} ϕ22\phi_{22} ϕ33\phi_{33} ϕ44\phi_{44}
four lm() calls 0.4500.450 0.259-0.259 0.0380.038 0.051-0.051
pacf() 0.4460.446 0.257-0.257 0.0400.040 0.052-0.052

Agreement to the third decimal; the small gap is pacf() lining up the start of the sample a little differently. And lags 3 and 4 sit inside the ±1.96/500=±0.088\pm 1.96/\sqrt{500} = \pm 0.088 band. The cutoff isn’t a metaphor — it’s two numbers that stopped mattering.

What those numbers actually are

The DGP was ϕ1=0.6\phi_1 = 0.6, ϕ2=0.3\phi_2 = -0.3. Look at what came back.

ϕ̂22=0.259ϕ2=0.3\hat\phi_{22} = -0.259 \qquad\longleftrightarrow\qquad \phi_2 = -0.3

The PACF at the final lag estimates the AR coefficient at that lag. Not a coincidence: at k=pk = p the regression you are running is the AR(pp) model, so its deepest coefficient is ϕp\phi_p.

Now the one that catches people every year:

ϕ̂11=0.450butϕ1=0.6(!)\hat\phi_{11} = 0.450 \qquad \text{but} \qquad \phi_1 = 0.6\ (!)

ϕ11\phi_{11} is the lag-1 coefficient with nothing controlled for, so it is just ρ1\rho_1 — and for an AR(2),

ρ1=ϕ11ϕ2=0.61.3=0.462\rho_1 = \frac{\phi_1}{1-\phi_2} = \frac{0.6}{1.3} = 0.462

— and the sample came back at 0.4460.446. ϕ̂11\hat\phi_{11} is ρ̂1\hat\rho_1 exactly, by construction; the 0.4620.462 is the population value it estimates. ϕ11=ϕ1\phi_{11} = \phi_1 only when the process really is an AR(1). The PACF hands you the model’s deepest coefficient, not its first.

Both fingerprints, side by side

AR(1): the ACF tails away, the PACF has one spike. AR(2) real: the ACF still tails, the PACF has two. The PACF is counting pp for you. Now press MA(1) and watch the two panels trade jobs.

Why the MA side tails off

The PACF of an MA(qq) does not cut off. It decays — geometrically, sometimes oscillating.

You already know why. We inverted an MA(1) a few slides ago:

yt=0.7yt10.49yt2+0.343yt30.2401yt4++ϵty_t = 0.7\,y_{t-1} - 0.49\,y_{t-2} + 0.343\,y_{t-3} - 0.2401\,y_{t-4} + \ldots + \epsilon_t

An invertible MA(qq) is an AR(\infty). The PACF asks “does lag kk add anything new?”, and the AR(\infty) representation keeps answering yes, a little — at every lag, forever, the amount shrinking geometrically.

There is no finite kk at which the answer becomes exactly no, because there is no finite kk at which πk\pi_k becomes exactly zero.

The duality isn’t a curiosity. It’s the reason the second fingerprint works at all.

The mirror

ACF PACF
AR(pp) tails off cuts off at pp
MA(qq) cuts off at qq tails off

Read it in either direction and it’s the same sentence. An AR is finite in its own lags and infinite in shocks, so it cuts in the PACF and tails in the ACF. An MA is finite in shocks and infinite in its own lags, so it does exactly the reverse.

Two instruments, and between them they pin down both orders — as long as the process is pure.

Which raises the obvious question. What does an ARMA(p,qp,q) — finite in neither direction — do to this table?

That’s Act III.

Act III · The lineup

“No print is readable until the surface holds still.” Act III — identification

Before you read a print

You asked what an ARMA does to the table. Before we answer that — a prerequisite.

Every fingerprint in this module was computed on a stationary series. That was not a convenience. γk\gamma_k is only well defined if the covariance depends on the gap kk and nothing else, so on a non-stationary series the ACF is not a memory measurement at all.

The order of operations, for the rest of the course:

  1. Verdict first. Plot it, correlogram it, ADF and KPSS it — Module 2’s whole apparatus.
  2. Difference until stationary. That count is dd. Not one more — Act I showed you what over-differencing costs (θ̂1\hat\theta \to -1).
  3. Only now read the ACF and the PACF, on the stationary series.
  4. Where the two prints disagree or both smear, Module 4 arbitrates.

Identify dd first, then pp and qq. Never the other way round.

Step 1, on the board

Before you press anything: barely any decay. Enormous memory — or a series that isn’t ready to be read? Press Difference it once and watch the verdict flip.

A near-1 ACF at every lag is a question you haven’t answered yet, not a long-memory AR.

The lineup

Three suspects. Two instruments. One table for the rest of the course.

Process ACF PACF
AR(pp) tails off cuts off at lag pp
MA(qq) cuts off at lag qq tails off
ARMA(p,qp,q) tails off tails off

The pure cases each announce themselves with a sharp cutoff in exactly one panel. That cutoff is the order.

The third row is the honest one. An ARMA is finite in neither direction, so both prints smear and neither reads out a number. You can tell that it is an ARMA — that is genuine information — but you cannot read pp and qq off the picture.

Row three is why Module 4 exists.

Where the cutoff stops being sharp

The table is a statement about the population. You never have the population.

Go back to the gallery and drag TT from 500 down toward 60. Two things happen at once:

  • The Bartlett bands widen as ±1.96/T\pm 1.96/\sqrt{T} — at T=60T = 60 that is ±0.25\pm 0.25.
  • The sample autocorrelations get noisier, so lags past the true cutoff start poking out.

A “cutoff” is a spike outside the band followed by spikes inside it. Widen the band and blur the spikes and an MA(2) can read as an MA(1), or as an ARMA, or as nothing at all.

The fingerprint is evidence, not a confession.

Pushing it until it breaks

Everything so far shows what the theory promises. Now push it to the edge and watch where it gives.

Take an MA(1) at θ=0.97\theta = -0.97. On paper this is still entirely legal:

yt=ϵt0.97ϵt1,Θ(L)=10.97L,L*=10.97=1.031.y_t = \epsilon_t - 0.97\,\epsilon_{t-1}, \qquad \Theta(L) = 1 - 0.97L, \qquad L^\ast = \tfrac{1}{0.97} = 1.031.

The root sits outside the unit circle, so the process is invertible and every result from Act I applies. But 1.0311.031 is barely outside — and L*=1L^\ast = 1 is the over-differencing root.

Meanwhile ρ1=θ/(1+θ2)=0.4998\rho_1 = \theta/(1+\theta^2) = -0.4998, sitting right on the |ρ1|0.5|\rho_1| \le 0.5 ceiling. Legal, and one nudge from not being.

Still legal, on the board

The θ1\theta_1 slider stops at 0.95-0.95; this instance starts past the end of it on purpose.

Read the ACF. That is a clean MA(1) cutoff — one big negative spike, then nothing.

The correlogram doesn’t warn you

And it reads fine on the numbers too: ρ̂1=0.475\hat\rho_1 = -0.475, lags 2–6 all inside the Bartlett band. Identification succeeds. Then you try to estimate it:

set.seed(1999)
y_bad <- arma_simulator(n = 500, theta = -0.97)
arima(y_bad, order = c(0, 0, 1))
θ̂\hat\theta s.e. true θ\theta
MA(1), θ=0.7\theta = 0.7 0.7320.732 0.0310.031 recovered
MA(1), θ=0.97\theta = -0.97 1.000-1.000 0.0090.009 pegged

MLE slid θ̂\hat\theta onto the invertibility boundary and stopped — and then reported a standard error of 0.0090.009, which is confidence, not accuracy. θ̂=1\hat\theta = -1 is the over-differencing signature from Act I, arriving here from a completely different direction.

The ACF said MA(1) and was right. The estimator failed anyway.

Blind draw

Press Blind draw. Commit out loud before anyone reveals: which panel cuts off — ACF, PACF, neither — and at what lag?

That answer is the row, and the lag is the order. Again until the room is faster than I am.

Act IV · Back to the case file

“Real evidence is never as clean as the training slides.” Act IV — UNRATE, continued

The series we left in Module 2

Module 2 ruled UNRATE I(1)I(1), so we difference once and read the print. Call it before I do: AR, MA, ARMA — what order?

Nothing.

Not one autocorrelation out of twenty-four clears the band. ρ̂1=0.036\hat\rho_1 = 0.036, and the Bartlett band is ±0.064\pm 0.064.

Read literally, Δ\DeltaUNRATE is white noise — no AR, no MA, nothing to model. That should bother you. Unemployment does not turn on a coin flip.

So before believing a blank correlogram, look back at the series, not the bars. There is a single vertical spike in April 2020 — a +10.4 point move in one month.

One month ate the correlogram

That one observation nearly doubles the sample standard deviation, 0.2370.4150.237 \to 0.415. Every ρ̂k\hat\rho_k is a covariance over that variance, so inflating the denominator crushes the whole correlogram toward zero.

The honest verdict

Drop the 2020–21 window and the print comes back: 8 of 24 ACF lags clear the band.

reads
ACF positive and significant through lag 6, peak +0.28+0.28 at lag 2 — tails off
PACF significant through lag 5 with scattered hits out to 24, peak +0.27+0.27 at lag 2 — tails off
both a clean negative dip at lag 12

Neither panel cuts off, so by the lineup table this is an ARMA — plus something annual that our table has no row for. First guess: a low-order ARMA with a seasonal term.

The table got us to a defensible guess, not to an answer.

What you can do now

  1. AR(pp) is memory of yy; stationarity needs every root of Φ(L)\Phi(L) outside the unit circle. ACF tails, PACF cuts at pp.
  2. MA(qq) is memory of shocks; always stationary, invertible when Θ(L)\Theta(L)’s roots are outside. ACF cuts at qq, PACF tails.
  3. They are the same object from two directions — a stationary AR is an MA(\infty), an invertible MA is an AR(\infty). That duality is why the second fingerprint works.
  4. The PACF is the OLS coefficient on the deepest lag; it counts pp for you.
  5. The lineup: tails/cuts, cuts/tails, tails/tails.
  6. A fingerprint is evidence, not a confession — small TT blurs it, one outlier can erase it, and θ̂1\hat\theta \to -1 means the estimator is in trouble.

Next time

Row three of the table is unfinished business, and so is everything we just guessed at:

  • Information criteria. When both correlograms tail off, AIC and BIC arbitrate — stop eyeballing, start scoring candidate (p,q)(p,q) pairs.
  • Estimation, properly. What arima() was actually doing when it pegged θ̂\hat\theta at 1-1.
  • Then Module 5 takes the lag-12 dip seriously: seasonality and SARIMA.

You can now write down the models. Next: how to pick one.

Machinery

Defined on a visible slide — a visibility="hidden" section is dropped from the DOM and the factory ends up undefined. OJS is reactive, so the calls above resolve regardless of source order.