Econ 6376 — Applied Time Series Econometrics
This chapter has one main reading path and three optional layers.
If you missed lecture, read the main text, run the Core code, and complete the Core Practice problems. Then return to the optional sections that address your questions or interests.
By the end of this module, you should be able to:
In Module 1 we wrote down the master equation for the whole course:
\[y_t = \alpha + \delta t + \sum_{j=1}^{p} \phi_j y_{t-j} + \sum_{l=1}^{q} \theta_l \epsilon_{t-l} + \epsilon_t\]
and then immediately turned off most of it. We kept \(\alpha\) and \(\phi_1\), left every other lag silent, and studied the AR(1) for the rest of Module 1. Module 2 kept us in the same neighborhood — the Dickey-Fuller test rearranges the AR(1), and even the trend-augmented specification only turned on \(\delta t\). At no point have we let \(\phi_2\), \(\phi_3\), or any of the \(\theta_l\) do anything.
Module 2 also handed us two loose ends, and this module picks up both. First, the over-differencing trap named a boundary: differencing a trend-stationary series manufactured an MA(1) with \(\theta = -1\) — a “unit root in the MA polynomial” — and we promised that Module 3 would define the property being violated (invertibility, §3.3.4) and probe it experimentally (§3.5.3). Second, Module 2’s power studies showed that formal tests cannot reliably separate \(\phi = 0.95\) from \(\phi = 1.0\) in realistic samples. This module adds a complementary tool that works alongside testing rather than replacing it: the correlogram fingerprint — reading ACF/PACF pairs to identify AR and MA structure.
Today we turn on the rest. By the end of this module you will have seen the full AR(p) summation, the full MA(q) summation, and the mixed ARMA(p,q) process that combines them. The only piece of the master equation you will not have seen formally is the seasonal block, and that waits until Module 5.
The mixing-board picture from Module 1 is worth drawing one more time, because we are about to move several of the sliders:
| Term | Status after L2 | Status after L3 |
|---|---|---|
| \(\alpha\) | On | On |
| \(\delta t\) | On (in the trend ADF) | On |
| \(\phi_1 y_{t-1}\) | On | On |
| \(\phi_2 y_{t-2}, \ldots, \phi_p y_{t-p}\) | Off | On |
| \(\theta_1 \epsilon_{t-1}, \ldots, \theta_q \epsilon_{t-q}\) | Off | On |
| \(\epsilon_t\) | On | On |
Before the equations, the concept. AR processes and MA processes are both memory devices, but they remember different things.
A household-budget analogy. Let \(y_t\) be your bank balance at the end of day \(t\).
Keep that image in your head. We will return to it at the end of every major section.
The AR(p) process is the generalization of the AR(1) that uses \(p\) lags of \(y\) instead of one:
\[y_t = \alpha + \phi_1 y_{t-1} + \phi_2 y_{t-2} + \ldots + \phi_p y_{t-p} + \epsilon_t, \qquad \epsilon_t \sim \text{WN}(0, \sigma^2)\]
In summation form:
\[y_t = \alpha + \sum_{j=1}^{p} \phi_j y_{t-j} + \epsilon_t\]
And in the lag-operator notation from Module 1, using the course convention \(\Phi(L) = 1 - \phi_1 L - \phi_2 L^2 - \ldots - \phi_p L^p\):
\[\Phi(L) y_t = \alpha + \epsilon_t\]
The minus signs in \(\Phi(L)\) are not cosmetic. They come from the same move we made for the AR(1) in Module 1: subtract the AR terms from both sides to collect \(y_t\) on the left.
\[y_t - \phi_1 y_{t-1} - \phi_2 y_{t-2} - \ldots - \phi_p y_{t-p} = \alpha + \epsilon_t\]
\[(1 - \phi_1 L - \phi_2 L^2 - \ldots - \phi_p L^p) y_t = \alpha + \epsilon_t\]
The minuses come for free the moment you move the \(\phi_j y_{t-j}\) terms across the equals sign. The reference for all of this is notation_dictionary.md, which is the final word on sign conventions for the course. Other textbooks differ — some write \(\Phi(L) = 1 + \sum \phi_j L^j\) and then the \(\phi_j\) are the negatives of ours. When you read outside this course, always check which convention the author is using.
We use \(\epsilon_t\) throughout — the DGP innovation — and reserve \(e_t\) for residuals from an estimated model. That distinction, introduced in Module 1 and enforced in Module 2, matters here too.
Statement. A stationary AR(p) exists if and only if every root of the lag polynomial \(\Phi(L) = 0\) lies outside the unit circle in the complex plane.
For the AR(1), this reduces to something you already know. \(\Phi(L) = 1 - \phi_1 L\), so the single root is \(L^* = 1/\phi_1\). “Outside the unit circle” means \(|L^*| > 1\), which is the same as \(|\phi_1| < 1\) — the AR(1) stationarity condition from Module 1.
For \(p > 1\), the polynomial has \(p\) roots and we need all of them outside the unit circle. One warning: the condition \(|\phi_j| < 1\) applied to the individual coefficients is not the right generalization. You can have an AR(2) whose coefficients are both small in absolute value and yet whose lag polynomial has a root inside the unit circle, and vice versa. The condition is on the roots, not the coefficients.
A note on conventions, because this is a place where textbooks disagree and it confuses students. Some books — Hamilton, notably — state the condition as we have: “all roots of \(\Phi(L) = 0\) outside the unit circle.” Other books — Enders among them, along with the F2024 slides for this course — write the characteristic equation
\[r^p - \phi_1 r^{p-1} - \phi_2 r^{p-2} - \ldots - \phi_p = 0\]
and require all roots \(\lambda_1, \ldots, \lambda_p\) to lie inside the unit circle. The two conditions are identical — they are related by the substitution \(r = 1/L\), so a root of \(\Phi(L)\) outside the unit circle corresponds to a characteristic root inside. We will use the characteristic-equation form for the AR(2) worked examples below because the arithmetic is cleaner, and we will state answers in the “modulus less than one” language that both conventions agree on.
For an AR(2),
\[y_t = \phi_1 y_{t-1} + \phi_2 y_{t-2} + \epsilon_t, \qquad \Phi(L) = 1 - \phi_1 L - \phi_2 L^2\]
the characteristic equation is
\[r^2 - \phi_1 r - \phi_2 = 0\]
and the quadratic formula gives
\[\lambda_{1,2} = \frac{\phi_1 \pm \sqrt{\phi_1^2 + 4 \phi_2}}{2}.\]
The sign of the discriminant \(\phi_1^2 + 4\phi_2\) tells us whether the roots are real or complex. Stationarity requires \(|\lambda_1| < 1\) and \(|\lambda_2| < 1\) — both characteristic roots inside the unit circle.
We will work three examples by hand. These are the F2024 examples, and the arithmetic matters — this is one of the two places in the module where we do not just state a result. Work through it with a calculator or an R console as you read. (Examples 2 and 3 use \(z_t\) and \(v_t\) purely as fresh series labels so the three processes do not collide on the page — they play exactly the role \(y_t\) plays everywhere else.)
Example 1 — stationary, two real roots.
\[y_t = 0.5 y_{t-1} + 0.3 y_{t-2} + \epsilon_t\]
Here \(\phi_1 = 0.5\) and \(\phi_2 = 0.3\). The discriminant is
\[\phi_1^2 + 4\phi_2 = 0.25 + 4(0.3) = 0.25 + 1.2 = 1.45 > 0\]
so the roots are real. Applying the quadratic formula,
\[\lambda_{1,2} = \frac{0.5 \pm \sqrt{1.45}}{2} = \frac{0.5 \pm 1.2042}{2}\]
\[\lambda_1 = \frac{0.5 + 1.2042}{2} \approx 0.8521, \qquad \lambda_2 = \frac{0.5 - 1.2042}{2} \approx -0.3521.\]
Both have modulus less than 1, so the process is stationary. The ACF will decay as a mixture of \(0.8521^k\) and \((-0.3521)^k\) — smooth decay dominated by the larger root, with a small alternating component from the negative root.
Example 2 — non-stationary, one real root outside.
\[z_t = 0.8 z_{t-1} + 0.3 z_{t-2} + \epsilon_t\]
Now \(\phi_1 = 0.8\) and \(\phi_2 = 0.3\). The discriminant is
\[\phi_1^2 + 4\phi_2 = 0.64 + 4(0.3) = 0.64 + 1.2 = 1.84 > 0\]
so again the roots are real. The quadratic formula gives
\[\lambda_{1,2} = \frac{0.8 \pm \sqrt{1.84}}{2} = \frac{0.8 \pm 1.3565}{2}\]
\[\lambda_1 \approx \frac{0.8 + 1.3565}{2} \approx 1.0782, \qquad \lambda_2 \approx \frac{0.8 - 1.3565}{2} \approx -0.2782.\]
The root \(\lambda_1 \approx 1.078\) is outside the unit circle (in the “characteristic root inside unit circle” language, it is on the wrong side). The process is non-stationary. If you simulate it, you will see the series drift in a way that looks a lot like the random walks from Module 1, because it has a characteristic root only barely larger than one.
Notice how close this example is to the stationary one. The only change was \(\phi_1 = 0.5 \to 0.8\); the second coefficient stayed the same. This is exactly the warning from §3.2.2: you cannot read stationarity off the individual coefficients. \(\phi_1 = 0.8\) and \(\phi_2 = 0.3\) are both less than 1 in absolute value, and yet the process is non-stationary. The roots of the joint polynomial are what matter.
Example 3 — stationary, complex roots (the discriminant moment).
\[v_t = 0.6 v_{t-1} - 0.5 v_{t-2} + \epsilon_t\]
Here \(\phi_1 = 0.6\) and \(\phi_2 = -0.5\). The discriminant is
\[\phi_1^2 + 4\phi_2 = 0.36 + 4(-0.5) = 0.36 - 2 = -1.64 < 0\]
negative. This is the discriminant moment. A negative discriminant means the characteristic roots are a pair of complex conjugates, and a stationary process with complex characteristic roots has an ACF that oscillates — a damped sine wave instead of smooth geometric decay.
Plugging into the quadratic formula,
\[\lambda_{1,2} = \frac{0.6 \pm \sqrt{-1.64}}{2} = \frac{0.6 \pm i \sqrt{1.64}}{2} \approx 0.3 \pm 0.6403 i.\]
The roots are \(\lambda_1 = 0.3 + 0.6403 i\) and \(\lambda_2 = 0.3 - 0.6403 i\). For a complex number \(a + bi\), “inside the unit circle” means the modulus \(\sqrt{a^2 + b^2}\) is less than one. Here,
\[|\lambda| = \sqrt{0.3^2 + 0.6403^2} = \sqrt{0.09 + 0.4100} = \sqrt{0.5000} \approx 0.7071.\]
Both roots have modulus \(\approx 0.707\), safely inside the unit circle, so the process is stationary. But because the roots are complex, the dynamics are qualitatively different from Example 1:
This is the AR(2) version of “fingerprinting.” We will see it for real in the simulation gallery in §3.5.
The discriminant as a one-glance classifier. For AR(2), the discriminant is
\[\phi_1^2 + 4 \phi_2.\]
If it is positive, the roots are real and the ACF decays smoothly (monotonically or with a single sign pattern depending on the root signs). If it is negative, the roots are a complex-conjugate pair and the ACF is a damped sine wave. Stationarity is a separate question answered by the modulus of the roots, not the sign of the discriminant. Example 3 has a negative discriminant and is stationary — complex roots are perfectly fine, they just produce oscillatory dynamics.
We will not derive the ACF of a general AR(p) — the algebra involves the Yule-Walker equations and is covered in Enders Chapter 2, pp. 60–66. The pattern is what matters, so we state it and move on.
Pattern: the ACF of a stationary AR(p) tails off. It never cuts off sharply. The decay rate is set by the largest-modulus characteristic root, and the shape of the decay depends on whether the roots are real or complex.
The single sentence to remember: the ACF of an AR process tails off. It does not have a sharp cutoff. Holding on to that sentence is half of the identification rules we will build in §3.5.
The MA(q) process — moving average of order \(q\) — is the part of the master equation we have not touched yet. It defines \(y_t\) as a weighted sum of the current and the past \(q\) innovations:
\[y_t = \alpha + \epsilon_t + \theta_1 \epsilon_{t-1} + \theta_2 \epsilon_{t-2} + \ldots + \theta_q \epsilon_{t-q}\]
In summation form:
\[y_t = \alpha + \epsilon_t + \sum_{l=1}^{q} \theta_l \epsilon_{t-l}\]
And in lag-operator form, with \(\Theta(L) = 1 + \theta_1 L + \theta_2 L^2 + \ldots + \theta_q L^q\):
\[y_t = \alpha + \Theta(L) \epsilon_t\]
Sign convention, one more time. The course convention is \(\Phi(L) = 1 - \sum \phi_j L^j\) (minuses) and \(\Theta(L) = 1 + \sum \theta_l L^l\) (pluses). The minus signs in \(\Phi(L)\) came from moving the AR terms to the left side of the equation; the MA terms are already on the right side with plus signs, so \(\Theta(L)\) inherits plus signs naturally. Other textbooks differ — some absorb a sign into their \(\theta\)’s. Enders and Hamilton use the same plus-sign convention we do, but Box, Jenkins, and Reinsel write the MA polynomial with minus signs, so their \(\theta\) is the negative of ours. Students who read outside this course need to check which convention is in play. The full statement lives in notation_dictionary.md.
Put the two together and you get the compact form of the general ARMA(p,q) process, a preview of the gluing-together we will do in §3.5:
\[\Phi(L) y_t = \alpha + \Theta(L) \epsilon_t.\]
An MA(q) with finite \(q\) and bounded \(\theta\)’s is always weakly stationary. No conditions on the \(\theta\)’s are needed.
The mean is constant: \(\mathbb{E}[y_t] = \alpha\), because every innovation has mean zero. The variance is constant:
\[\text{Var}(y_t) = \sigma^2 (1 + \theta_1^2 + \theta_2^2 + \ldots + \theta_q^2),\]
which depends on the \(\theta\)’s but not on \(t\). And the autocovariance at lag \(k\) depends only on the number of overlapping \(\epsilon\)’s between \(y_t\) and \(y_{t-k}\), which depends on \(k\) and not on \(t\). All three conditions for weak stationarity are satisfied automatically.
This is a genuine contrast with AR processes. AR(p) stationarity is a condition — roots of \(\Phi(L)\) outside the unit circle — and if it fails, the process is non-stationary. MA(q) stationarity is automatic. Where MA(q) does have a condition is invertibility, and we will see in §3.3.4 that invertibility is a different property entirely from stationarity. Students often conflate the two; do not be one of those students.
This is the first of the two algebraic moments in the module. The calculation is short, and the result is so striking that doing it by hand once cements the intuition for the whole MA family.
Consider the MA(1):
\[y_t = \epsilon_t + \theta \epsilon_{t-1}, \qquad \epsilon_t \sim \text{WN}(0, \sigma^2)\]
(I have dropped \(\alpha\) for clarity — it does not affect the autocovariance calculations.) The innovations are uncorrelated: \(\mathbb{E}[\epsilon_s \epsilon_t] = 0\) for \(s \neq t\) and \(\mathbb{E}[\epsilon_t^2] = \sigma^2\). That single fact does all the work below.
Variance (\(\gamma_0\)):
\[\gamma_0 = \text{Var}(y_t) = \text{Var}(\epsilon_t + \theta \epsilon_{t-1}) = \text{Var}(\epsilon_t) + \theta^2 \text{Var}(\epsilon_{t-1}) = \sigma^2 + \theta^2 \sigma^2 = \sigma^2 (1 + \theta^2).\]
The cross-term vanishes because \(\mathbb{E}[\epsilon_t \epsilon_{t-1}] = 0\).
Autocovariance at lag 1 (\(\gamma_1\)):
\[\gamma_1 = \mathbb{E}[y_t y_{t-1}] = \mathbb{E}\big[(\epsilon_t + \theta \epsilon_{t-1})(\epsilon_{t-1} + \theta \epsilon_{t-2})\big].\]
Expand the product:
\[= \mathbb{E}[\epsilon_t \epsilon_{t-1}] + \theta \mathbb{E}[\epsilon_t \epsilon_{t-2}] + \theta \mathbb{E}[\epsilon_{t-1}^2] + \theta^2 \mathbb{E}[\epsilon_{t-1} \epsilon_{t-2}].\]
Three of these four terms are zero because different \(\epsilon\)’s are uncorrelated. Only \(\theta \mathbb{E}[\epsilon_{t-1}^2]\) survives, and that equals \(\theta \sigma^2\):
\[\gamma_1 = \theta \sigma^2.\]
Autocovariance at lag 2 (\(\gamma_2\)):
\[\gamma_2 = \mathbb{E}[y_t y_{t-2}] = \mathbb{E}\big[(\epsilon_t + \theta \epsilon_{t-1})(\epsilon_{t-2} + \theta \epsilon_{t-3})\big].\]
Expand again:
\[= \mathbb{E}[\epsilon_t \epsilon_{t-2}] + \theta \mathbb{E}[\epsilon_t \epsilon_{t-3}] + \theta \mathbb{E}[\epsilon_{t-1} \epsilon_{t-2}] + \theta^2 \mathbb{E}[\epsilon_{t-1} \epsilon_{t-3}].\]
Every single term is a product of two different innovations, and every single expectation is zero:
\[\gamma_2 = 0.\]
And the same argument gives \(\gamma_k = 0\) for every \(k \geq 2\). There is simply no overlap between the \(\epsilon\)’s that make up \(y_t\) and the \(\epsilon\)’s that make up \(y_{t-k}\) once the lag exceeds the MA order.
Dividing through by \(\gamma_0\), the autocorrelations are:
\[\rho_1 = \frac{\gamma_1}{\gamma_0} = \frac{\theta \sigma^2}{\sigma^2 (1 + \theta^2)} = \frac{\theta}{1 + \theta^2}, \qquad \rho_k = 0 \text{ for all } k \geq 2.\]
This is the cutoff. The autocorrelation function of an MA(1) is non-zero at lag 1 and exactly zero at every lag beyond. Generalizing to higher orders: the ACF of an MA(q) is non-zero at lags \(1, 2, \ldots, q\) and zero for all lags \(k > q\). MA processes have a sharp ACF cutoff at lag \(q\).
This is the direct opposite of the AR pattern. AR processes tail off forever; MA processes cut off at exactly lag \(q\). That asymmetry is the first of the two identification fingerprints for this module.
A small but useful gotcha. Look at \(\rho_1 = \theta / (1 + \theta^2)\) for a minute. Treat it as a function of \(\theta\) and ask what its maximum value is. By calculus (or by observation), the maximum of \(\theta / (1 + \theta^2)\) is \(0.5\) at \(\theta = 1\), and the minimum is \(-0.5\) at \(\theta = -1\). So for any MA(1), no matter what \(\theta\) is:
\[|\rho_1| \leq 0.5.\]
Practical consequence: if you compute the sample lag-1 autocorrelation of a series and find \(|\hat{\rho}_1| > 0.5\), the series cannot be a pure MA(1). That is a small, free diagnostic that you get just from the algebra above.
We said stationarity is automatic for MA(q) but that there is a separate condition called invertibility. It is time to pay off that claim.
The intuition. Given a realized MA(q) series \(\{y_t\}\), can we recover the innovations \(\{\epsilon_t\}\) from the past observations? If yes, the MA representation is unique — there is only one set of \(\theta\)’s that produces this series, and the parameters are identifiable. If no, then two different parameter configurations give the same likelihood, and maximum-likelihood estimation either refuses to converge or gets stuck on the boundary.
The condition (stated). An MA(q) is invertible if and only if every root of \(\Theta(L) = 0\) lies outside the unit circle.
Notice the mirror symmetry. AR stationarity: roots of \(\Phi(L)\) outside the unit circle. MA invertibility: roots of \(\Theta(L)\) outside the unit circle. Same language, same geometry, applied to two different polynomials, meaning two different things. Do not conflate them — they are related in shape but independent in content.
Why we care, without deriving anything:
In Module 2 we looked at what happens when you difference a series that is trend-stationary rather than difference-stationary. Starting from \(y_t = \alpha + \delta t + u_t\) with \(u_t\) stationary, the first difference is
\[\Delta y_t = \delta + u_t - u_{t-1}.\]
Take \(u_t = \epsilon_t\) to simplify. Then
\[\Delta y_t = \delta + \epsilon_t - \epsilon_{t-1}.\]
That is an MA(1) with \(\theta = -1\). Its MA polynomial is
\[\Theta(L) = 1 - L.\]
Where is the root of \(\Theta(L) = 0\)? Set \(1 - L = 0\), and the root is \(L = 1\) — exactly on the unit circle. Non-invertible.
This is the canonical non-invertible MA(1). It is not invertible by any margin; the root is sitting right at the boundary. And this is exactly the process you generate when you difference something that did not need to be differenced. In Module 2 we warned that over-differencing breaks ARMA estimation “because you end up with a root on the unit circle.” Now you can see exactly what that means: the implied MA(1) is at the very edge of invertibility, MLE struggles to identify \(\theta\), and the parameter estimate typically sticks close to \(-1\) with a huge standard error. When you see \(\hat{\theta} \approx -1\) after differencing, that is the over-differencing warning light.
Two modules ago you took the claim on faith. Now you have the vocabulary to talk about it.
This is the conceptual payoff of invertibility, and it is one of the most important pieces of intuition you will carry out of this module. The one-sentence version:
A finite stationary AR(p) and a finite invertible MA(q) are two ways of writing the same kinds of underlying dynamics. Each can be rewritten as an infinite-order version of the other.
We have actually seen one direction of this already, in Module 1. Recall that for the AR(1) with \(|\phi_1| < 1\), recursive substitution gave
\[y_t = \frac{\alpha}{1 - \phi_1} + \sum_{i=0}^{\infty} \phi_1^i \epsilon_{t-i}.\]
That is the AR(1) rewritten as an MA(\(\infty\)) — an infinite-order moving average with geometrically decaying weights. The same substitution trick, generalized, works for any stationary AR(p): every stationary AR can be expressed as an infinite-order moving average. The formal result behind this is the Wold decomposition theorem, which states that every covariance-stationary process has an MA(\(\infty\)) representation. We mentioned it in passing in Module 1; now you have one more reason to remember it.
Invertibility gives us the other direction. An invertible MA(q) can be rewritten as an AR(\(\infty\)):
\[\epsilon_t = \Theta(L)^{-1} y_t = \pi_0 y_t + \pi_1 y_{t-1} + \pi_2 y_{t-2} + \ldots\]
The \(\pi_k\) coefficients decay geometrically — and the condition that makes them decay geometrically, rather than blowing up, is precisely that the roots of \(\Theta(L)\) lie outside the unit circle. That is what invertibility means, operationally: it is the condition under which the inverse polynomial \(\Theta(L)^{-1}\) is a convergent power series in \(L\), so the implied AR(\(\infty\)) representation has finite coefficients that fade.
The parsimony argument. Here is why you should care beyond the algebra. A small AR(p) with two or three parameters can capture dynamics that would require an infinite number of weights in MA(\(\infty\)) form. A small invertible MA(q) with two or three parameters can capture dynamics that would require an infinite number of weights in AR(\(\infty\)) form. This is why time series models can describe rich, persistent behavior with surprisingly few parameters: the finite ARMA representation is a compact code for an infinite-order process.
It is also the deep reason ARMA(p,q) models are so powerful. An ARMA(1,1) has only two dynamic parameters, \(\phi\) and \(\theta\), and yet it can approximate dynamics that would need either a long AR or a long MA in the pure-class representations. When a series is well-described by a small ARMA, the small ARMA is the parsimonious description; a long AR or a long MA would be wasteful and would overfit.
One-line takeaway to keep on a sticky note: small \(p\) and small \(q\) go a long way. Don’t fit a big pure-AR or pure-MA model when a small ARMA will do the same job.
Take an MA(1) with \(\theta = 0.7\):
\[y_t = \epsilon_t + 0.7 \epsilon_{t-1}, \quad \Theta(L) = 1 + 0.7 L.\]
The root of \(\Theta(L)\) is \(L = -1/0.7 \approx -1.43\), which is outside the unit circle — invertible. Its AR(\(\infty\)) representation is formally \(\epsilon_t = \Theta(L)^{-1} y_t\), and for this specific \(\Theta(L)\) the power-series expansion is
\[\Theta(L)^{-1} = \frac{1}{1 + 0.7 L} = 1 - 0.7 L + 0.49 L^2 - 0.343 L^3 + 0.2401 L^4 - \ldots\]
(the standard geometric series with ratio \(-0.7\)). Applying this to \(y_t\) gives \((1 - 0.7L + 0.49L^2 - \ldots)y_t = \epsilon_t\). In the notation dictionary’s terms, the AR(\(\infty\)) weights are
\[\pi_k = (-0.7)^k: \qquad \pi_0 = 1, \quad \pi_1 = -0.7, \quad \pi_2 = +0.49, \quad \pi_3 = -0.343, \; \ldots\]
alternating in sign and decaying geometrically at rate \(|\theta| = 0.7\). Now rearrange to put \(y_t\) alone on the left — moving the lagged terms across the equals sign, which flips their signs — and you get the equation you would actually estimate:
\[y_t = 0.7 \, y_{t-1} - 0.49 \, y_{t-2} + 0.343 \, y_{t-3} - 0.2401 \, y_{t-4} + \ldots + \epsilon_t.\]
The coefficient on \(y_{t-k}\) in this \(y_t\)-equation is \(-\pi_k = (-1)^{k+1}(0.7)^k\) for \(k \geq 1\) — the same weights, signs flipped by the rearrangement. Keep the two versions straight: \(\pi_k\) is the weight in the \(\epsilon_t\)-equation (the dictionary’s definition), and \(-\pi_k\) is what appears once \(y_t\) is on the left. Either way, the “infinite AR” that a plain MA(1) is hiding has coefficients that alternate in sign and decay geometrically at rate \(|\theta| = 0.7\). Two parameters — really, one, if you set \(\alpha = 0\) — produce an infinite sequence of AR weights.
You can verify this in R. Simulate a long MA(1) with \(\theta = 0.7\), fit a high-order AR by OLS, and look at the coefficients:
set.seed(8675309)
y <- arima.sim(n = 5000, list(ma = 0.7))
ar_fit <- ar(y, order.max = 8, method = "ols")
ar_fit$ar, , 1
[,1]
[1,] 0.67962369
[2,] -0.47882046
[3,] 0.33649914
[4,] -0.21993663
[5,] 0.17120231
[6,] -0.09558180
[7,] 0.04054739Because ar() estimates the \(y_t\)-on-the-left equation, its fitted coefficients are the \(y\)-side weights \(-\pi_k\), not the dictionary’s \(\pi_k\). The first few should come out close to \(+0.7, -0.49, +0.343, -0.2401, \ldots\) — exactly the \(-\pi_k = (-1)^{k+1}(0.7)^k\) pattern. The match will not be perfect (we have finite data, the AR order is truncated at 8, and there is sampling noise), but the sign pattern and the geometric decay rate are unmistakable. That is the isomorphism made visible: an MA(1) is, in a very concrete sense, an AR(\(\infty\)) with a specific pattern of decaying coefficients.
Why this matters for the rest of the course:
Consider the problem the ACF leaves us with. The ACF of an AR(1) tails off. The ACF of an AR(2) tails off. The ACF of an AR(5) tails off. We can read “tails off” from the correlogram, and that tells us we are probably looking at an AR (or an ARMA). But we cannot read the order \(p\) from the ACF alone — every AR order has the same qualitative shape.
We need a second fingerprint, one that cuts off at \(p\) instead of tailing off. That fingerprint is the partial autocorrelation function, the PACF.
Plain-language definition. The partial autocorrelation at lag \(k\), written \(\phi_{kk}\), is the correlation between \(y_t\) and \(y_{t-k}\) after removing the linear effect of the intermediate lags \(y_{t-1}, y_{t-2}, \ldots, y_{t-k+1}\).
Think of it as the answer to the question: How much new information does \(y_{t-k}\) give me about \(y_t\) that I did not already get from the lags in between?
The partial-effect interview analogy. Imagine you are trying to predict whether someone will be late to work today. You start by asking, “Were they late yesterday?” That tells you something — lateness is persistent, lateness yesterday is informative about lateness today. Now you ask, “Were they late two days ago?” At first glance this also looks informative. But yesterday and two-days-ago are themselves correlated — if someone was late yesterday, they were probably also late two days ago. So the real question is: does two-days-ago lateness tell me anything beyond what yesterday already told me? The PACF at lag 2 is the answer to exactly that question.
This is the cutoff we wanted. The PACF of an AR(p) cuts off at lag \(p\), the same way the ACF of an MA(q) cuts off at lag \(q\). The two correlograms are mirror images of each other: AR cuts in the PACF and tails in the ACF; MA cuts in the ACF and tails in the PACF.
After the intuition, the operational definition. One concrete way to compute the sample PACF at lag \(k\) is to estimate the OLS regression of \(y_t\) on its first \(k\) lags — written here in estimated (hatted) form, with \(e_t\) the residual:
\[y_t = \hat{\beta}_0 + \hat{\beta}_1 y_{t-1} + \hat{\beta}_2 y_{t-2} + \ldots + \hat{\beta}_k y_{t-k} + e_t\]
and take the coefficient on the deepest lag:
\[\hat{\phi}_{kk} = \hat{\beta}_k.\]
To get the sample PACF out to lag \(K\), you run \(K\) such regressions, each one lag deeper than the last, and read off the deepest-lag coefficient from each one. That is a bit wasteful computationally — R’s pacf() uses the Yule-Walker equations or the Durbin-Levinson recursion under the hood to do it much faster — but the partial-regression definition is the one to hold in your head. The PACF at lag \(k\) is the OLS coefficient on \(y_{t-k}\) after you have controlled for all the shallower lags.
Live demo: building the PACF from regressions.
set.seed(8675309)
y <- arima.sim(n = 500, list(ar = c(0.6, -0.3))) # An AR(2)
y_t <- y[5:500]
y_lag1 <- y[4:499]
y_lag2 <- y[3:498]
y_lag3 <- y[2:497]
y_lag4 <- y[1:496]
phi_11 <- coef(lm(y_t ~ y_lag1))[2]
phi_22 <- coef(lm(y_t ~ y_lag1 + y_lag2))[3]
phi_33 <- coef(lm(y_t ~ y_lag1 + y_lag2 + y_lag3))[4]
phi_44 <- coef(lm(y_t ~ y_lag1 + y_lag2 + y_lag3 + y_lag4))[5]
c(phi_11, phi_22, phi_33, phi_44) y_lag1 y_lag2 y_lag3 y_lag4
0.45044406 -0.25877553 0.03763853 -0.05069753 pacf(y, lag.max = 4, plot = FALSE)
Partial autocorrelations of series 'y', by lag
1 2 3 4
0.446 -0.257 0.040 -0.052 The two should match, up to small differences in how pacf() handles observation alignment at the beginning of the sample. Because the DGP is an AR(2), you should see clearly non-zero values at lags 1 and 2 and much smaller values at lags 3 and 4 — fingerprinting in action.
For an MA(q), the PACF tails off — typically as a damped exponential or a damped sinusoid. The intuition is the exact dual of the AR case: an invertible MA(q) is an AR(\(\infty\)) in disguise (§3.3.6), and an AR(\(\infty\)) has non-zero partial autocorrelation at every lag, decaying geometrically. There is no finite order at which the PACF of an MA process suddenly goes to zero, because the underlying AR representation has non-zero coefficients at every lag.
Symmetry again: AR cuts off in the PACF, tails off in the ACF; MA cuts off in the ACF, tails off in the PACF. Neither process cuts off in both; only an ARMA with small \(p\) and \(q\) tails off in both. We will see all of this in pictures in §3.5.
There is a more elegant way to compute the PACF via the Yule-Walker equations, which express the PACF directly in terms of the ACF using a recursive algorithm (the Durbin-Levinson recursion). Software uses this because it is much faster than running \(K\) separate regressions. We will not need the Yule-Walker machinery in this course; for the curious, Enders Chapter 2, p. 65 has a clean derivation.
Everything in this module so far comes down to one table. Memorize it; we will use it for the rest of the course.
| Process | ACF | PACF |
|---|---|---|
| AR(p) | Tails off | Cuts off at lag \(p\) |
| MA(q) | Cuts off at lag \(q\) | Tails off |
| ARMA(p,q) | Tails off | Tails off |
Three rules. The AR and MA cases are each identified by a sharp cutoff in one of the two correlograms: AR cuts in the PACF at \(p\), MA cuts in the ACF at \(q\). The ARMA case is the hard one — both correlograms tail off, so you cannot read \(p\) and \(q\) directly. That is the motivation for information criteria in Module 4, where we will let AIC and BIC arbitrate when the correlogram fingerprint is ambiguous.
The best way to internalize the table is to see it. Below is a gallery of six processes, all simulated with the same length (\(T = 500\)) and the same random seed so that the visual comparison is clean. For each one we look at the series itself, the ACF, and the PACF, and we ask: does the fingerprint match what the table says?
| Process | Parameters | What to look for |
|---|---|---|
| AR(1) | \(\phi = 0.7\) | ACF geometric decay; PACF spike at 1, then nothing |
| AR(2) real | \(\phi_1 = 0.5, \phi_2 = 0.3\) | ACF smooth decay; PACF spikes at 1 and 2 |
| AR(2) complex | \(\phi_1 = 0.6, \phi_2 = -0.5\) | ACF oscillates / damped sine; PACF spikes at 1 and 2 |
| MA(1) | \(\theta = 0.7\) | ACF spike at 1; PACF tails off |
| MA(2) | \(\theta_1 = 0.5, \theta_2 = 0.4\) | ACF spikes at 1 and 2; PACF tails off |
| ARMA(1,1) | \(\phi = 0.6, \theta = 0.4\) | Both tail off — the L4 problem in one picture |
The code to generate and visualize all six is straightforward once arma_simulator() is built (see §3.6). A small helper function stitches together the series plot and the two correlograms into a single fingerprint panel. (A note on packages: patchwork, introduced in Module 2, does no modeling — it only arranges several ggplots into one figure.)
library(forecast); library(ggplot2); library(patchwork)
show_fingerprint <- function(y, title) {
p1 <- autoplot(y) + ggtitle(title) + theme_bw()
p2 <- ggAcf(y, lag.max = 24) + ggtitle("ACF") + theme_bw()
p3 <- ggPacf(y, lag.max = 24) + ggtitle("PACF") + theme_bw()
p1 / (p2 + p3)
}
set.seed(1985)
y_ar1 <- arma_simulator(n = 500, phi = 0.7)
y_ar2r <- arma_simulator(n = 500, phi = c(0.5, 0.3))
y_ar2c <- arma_simulator(n = 500, phi = c(0.6, -0.5))
y_ma1 <- arma_simulator(n = 500, theta = 0.7)
y_ma2 <- arma_simulator(n = 500, theta = c(0.5, 0.4))
y_arma <- arma_simulator(n = 500, phi = 0.6, theta = 0.4)
show_fingerprint(y_ar1, "AR(1), phi = 0.7")
show_fingerprint(y_ar2r, "AR(2) real roots, phi = (0.5, 0.3)")
show_fingerprint(y_ar2c, "AR(2) complex roots, phi = (0.6, -0.5)")
show_fingerprint(y_ma1, "MA(1), theta = 0.7")
show_fingerprint(y_ma2, "MA(2), theta = (0.5, 0.4)")
show_fingerprint(y_arma, "ARMA(1,1), phi = 0.6, theta = 0.4")
(show_fingerprint() is built live here; it also lives in helpers/diagnostics.R as the canonical course copy, so Module 4 onward can source() it instead of re-defining it.)
Reading the panels in order:
The first five panels validate the table cleanly. The sixth panel motivates the next module. That is the arc of the simulation gallery: see the rules work, then see the case where they do not decide the question for you.
The ARMA(1,1) panel is the whole motivation for Module 4. The identification rules give you “probably ARMA” but not “ARMA(1,1) versus ARMA(2,1) versus ARMA(1,2).” Module 4 introduces information criteria (AIC and BIC) and residual diagnostics (Ljung-Box) as the tools that let you choose among competing ARMA specifications when the correlogram is ambiguous. Keep this picture in your head when you read that module — you will want to remember what the problem looked like before we had the tools to solve it.
Everything in the gallery so far shows you what the theory promises. This final entry shows you what happens when you push the theory toward its boundary — and it is where the simulation step earns its keep in the four-step ladder (concept → math → simulate → real data). Simulations are not just for confirming that the math works; they are for stressing the math until you can see where it breaks.
The target is an MA(1) with \(\theta = -0.97\):
\[y_t = \epsilon_t - 0.97\,\epsilon_{t-1}.\]
On paper, the MA polynomial is \(\Theta(L) = 1 - 0.97 L\), which has root \(L^{*} = 1/0.97 \approx 1.031\). Technically this is outside the unit circle, so the process is invertible and everything in §3.3 applies. But \(L^{*}\) sits right next to \(L = 1\), which is the over-differencing trap root from §3.3 — and that proximity is going to matter.
The theoretical first-lag autocorrelation is
\[\rho_1 = \frac{\theta}{1 + \theta^{2}} = \frac{-0.97}{1 + 0.9409} \approx -0.4998,\]
essentially the \(|\rho_1| \leq 0.5\) boundary we derived earlier. In theory, this is still a clean MA(1): the ACF cuts off at lag 1, all other autocorrelations are zero.
In practice:
set.seed(1999)
y_bad <- arma_simulator(n = 500, theta = -0.97)
show_fingerprint(y_bad, "MA(1), theta = -0.97 — nearly non-invertible")
Two things go wrong, and both are worth seeing with your own eyes:
fit_bad <- arima(y_bad, order = c(0, 0, 1))
fit_bad
Call:
arima(x = y_bad, order = c(0, 0, 1))
Coefficients:
ma1 intercept
-1.000 -0.0012
s.e. 0.009 0.0003
sigma^2 estimated as 0.8816: log likelihood = -681.08, aic = 1368.16Depending on the seed, you will see one of two things: the optimizer pegs \(\hat{\theta}\) at or near \(-1\) (stuck on the boundary of invertibility), or it reports a standard error large enough to make the estimate meaningless. This is MLE running into the exact problem we warned about in §3.3: the likelihood function has multiple modes near the boundary, and the optimizer cannot resolve them.
You have now seen the over-differencing trap three times, in three different guises:
arima() fail to pin \(\hat{\theta}\) down. The theory and the algebra were not abstract warnings; they were predictions, and here are the predictions playing out in the console.This is the whole point of the simulation step. A clean gallery shows you what works. An assumption-break shows you why the assumptions are there. When you later encounter a model estimate where \(\hat{\theta}\) comes back at \(-0.98\) with a huge standard error, you will not be surprised — you will recognize it as this picture.
arma_simulator()This is the Module 3 version of the signature live-coding move from Module 1. In Module 1 we built ar1_simulator() from scratch instead of calling arima.sim(), on the grounds that typing the DGP into code is how you internalize what a DGP actually is. We are going to do the same thing here, but generalized. The function we build is an almost-literal transcription of the master equation into a for-loop.
arma_simulator <- function(
n = 500,
alpha = 0,
phi = c(),
theta = c(),
sigma = 1,
burn_in = 200) {
p <- length(phi)
q <- length(theta)
total <- burn_in + n
eps <- rnorm(total, 0, sigma)
y <- numeric(total)
for (t in seq_len(total)) {
ar_part <- 0
if (p > 0 && t > p) {
ar_part <- sum(phi * y[(t-1):(t-p)])
}
ma_part <- 0
if (q > 0 && t > q) {
ma_part <- sum(theta * eps[(t-1):(t-q)])
}
y[t] <- alpha + ar_part + ma_part + eps[t]
}
ts(y[(burn_in + 1):total])
}(Canonical copy: this function also lives in helpers/simulators.R. Per the course’s inline-first policy it is built inline here, in the module where it is born; Module 4 onward will source() it from helpers/ instead of re-defining it.)
Walk through it block by block.
n is the length of the final returned series, alpha is the intercept, phi is a vector of AR coefficients (possibly empty), theta is a vector of MA coefficients (possibly empty), sigma is the innovation standard deviation, and burn_in is the number of observations we simulate and then discard at the beginning. The burn-in is there to wash out the effect of the starting values — we initialize \(y[1]\) with nothing but \(\epsilon_1\), which is not a draw from the stationary distribution, and the first couple of hundred observations carry that contamination. Throwing them away gives us a clean draw from the stationary distribution (approximately, for stationary processes).p and q are the orders of the AR and MA parts. total is the total number of observations we actually simulate, including the burn-in. eps is the vector of innovations — one call to rnorm() for the whole simulation, so they are all drawn up front. y is the pre-allocated vector we will fill in.t > p and t > q prevent us from indexing off the beginning of the vectors during the first few time steps, where we do not yet have \(p\) or \(q\) lags of history. You could initialize more carefully (seed with draws from the stationary distribution, or burn in inside the loop), but the burn-in approach is conceptually cleaner and is what most textbook simulators use.The three lines inside the loop —
ar_part <- sum(phi * y[(t-1):(t-p)])
ma_part <- sum(theta * eps[(t-1):(t-q)])
y[t] <- alpha + ar_part + ma_part + eps[t]— are literally the master equation:
\[y_t = \alpha + \sum_{j=1}^{p} \phi_j y_{t-j} + \sum_{l=1}^{q} \theta_l \epsilon_{t-l} + \epsilon_t.\]
There is no hidden algebra, no cleverness. The AR sum is a dot product of phi with the last \(p\) values of \(y\). The MA sum is a dot product of theta with the last \(q\) innovations. The contemporaneous innovation is \(\epsilon_t\). Add them up, store in y[t], increment \(t\), repeat. If you can read this for-loop, you can read the master equation, and vice versa.
Once you have internalized the loop, you can (and should) switch to the packaged version for production work. arima.sim() is faster, better-tested, and handles edge cases you have not yet thought about:
y_pkg <- arima.sim(n = 500, list(ar = c(0.6, -0.3), ma = c(0.4)))But the home-built version is the pedagogically honest one. When you simulate something and it looks wrong, the first thing to check is whether your understanding of the DGP matches the three lines inside that loop.
This is the fourth and final step of the ladder: concept, math, simulation, and now real data. Every module in this course ends with a look at a FRED series that students would realistically work with, precisely because simulations are clean and real data is not. The gap between them is where most of your judgment as a time series practitioner lives.
In Module 2 we applied the full stationarity workflow to the US civilian unemployment rate from FRED and concluded that, on balance, we should treat the level series as \(I(1)\) and work with its first difference. Having spent this module learning to read fingerprints, it is worth asking: what does the ACF/PACF of \(\Delta\)(unemployment rate) actually look like?
To pull the series live, store your FRED key in the environment (never hardcode it in course files) and run:
library(fredr)
fredr_set_key(Sys.getenv("FRED_KEY"))
unrate <- fredr(series_id = "UNRATE",
observation_start = as.Date("1948-01-01"))
y <- ts(unrate$value, start = c(1948, 1), frequency = 12)
dy <- diff(y)So that these notes render identically for everyone — with or without an API key or an internet connection — the code below loads the same series from the course’s cached copy in data/UNRATE.csv (the same cache Module 2 used). The workflow from here on is identical either way.
library(forecast); library(ggplot2); library(patchwork)
unrate <- read.csv("../data/UNRATE.csv") # cached FRED pull; see data/README.md
# The cache has one missing month (2025-10, a federal data-release gap).
# Keep the complete run from 1948-01 through the last month before the gap
# so the monthly ts index stays aligned.
first_na <- which(is.na(unrate$UNRATE))
if (length(first_na) > 0) unrate <- unrate[seq_len(min(first_na) - 1), ]
y <- ts(unrate$UNRATE, start = c(1948, 1), frequency = 12)
dy <- diff(y)p1 <- autoplot(dy) + theme_bw() +
labs(title = "Differenced US Unemployment Rate", y = "")
p2 <- ggAcf(dy, lag.max = 24) + theme_bw() + ggtitle("ACF")
p3 <- ggPacf(dy, lag.max = 24) + theme_bw() + ggtitle("PACF")
p1 / (p2 + p3)
What you see is not a textbook fingerprint. The ACF has a significant spike at lag 1 (positive), some structure at seasonal lags 12 and 24, and then generally small correlations. The PACF also has a spike at lag 1, plus some seasonal echo. If you were forced to guess at a model, you would probably say “a low-order ARMA with a seasonal component” — which is exactly what it is, and which is exactly what Module 5 will equip you to write down properly.
Real data is honest about being messier than simulations. No sample correlogram ever reads as cleanly as the gallery in §3.5. The identification skill you are building is not “match the picture to the table exactly”; it is “form a defensible first guess at the orders, then let Module 4’s tools refine it.” The FRED unemployment series is a good reminder of that.
Closing the L2 → L3 arc. It is worth plotting the levels ACF next to the differenced ACF to see the whole story in two pictures:
p_levels <- ggAcf(y, lag.max = 36) + theme_bw() +
ggtitle("UNRATE (levels) — slow decay, unit-root signature")
p_diff <- ggAcf(dy, lag.max = 36) + theme_bw() +
ggtitle("Δ UNRATE — readable as a low-order ARMA")
p_levels / p_diff
The levels ACF decays glacially — that is the random-walk-like fingerprint you learned to recognize in Module 2, and it is exactly why we differenced. The differenced ACF is legible as a low-order ARMA. Two plots, and you have the full L2 → L3 arc: stationarity testing told you the level series was \(I(1)\); differencing turned an unreadable fingerprint into a readable one; the identification rules you learned in this module now give you a first guess at the orders. Module 4 will tell you how to pick among your candidates.
“An AR(2) is stationary if both \(|\phi_1| < 1\) and \(|\phi_2| < 1\).” No. The condition is on the roots of \(\Phi(L)\), not on the coefficients individually. Example 2 in §3.2.3 has \(|\phi_1| = 0.8 < 1\) and \(|\phi_2| = 0.3 < 1\), and is non-stationary.
“An MA process can be non-stationary.” No. A finite-order MA is always weakly stationary. What it can fail is invertibility, which is a different property.
“Stationarity and invertibility are the same thing.” No. Stationarity lives on the AR side and refers to roots of \(\Phi(L)\). Invertibility lives on the MA side and refers to roots of \(\Theta(L)\). They are mirror images in form, but they govern different objects.
“If the ACF cuts off at lag 2, the process is AR(2).” No. An ACF cutoff is the MA signature, not the AR signature. An AR(p) cuts off in the PACF at lag \(p\), not in the ACF.
“Complex characteristic roots mean the model is broken.” No. Complex roots are fine as long as their modulus is less than one. They produce the characteristic damped-sine ACF and are perfectly stationary — Example 3 in §3.2.3 is stationary.
“\(\rho_1\) for an MA(1) can be anything.” No. It is bounded: \(|\rho_1| \leq 0.5\). If you see a sample \(|\hat{\rho}_1| > 0.5\), the process is not a pure MA(1).
“AR and MA models are fundamentally different families.” Not quite. A stationary AR(p) has an MA(\(\infty\)) representation; an invertible MA(q) has an AR(\(\infty\)) representation. They are two finite encodings of overlapping families of infinite-order dynamics. We use small finite ARMA for parsimony, not because the classes are mutually exclusive.
“I should always use arima.sim() rather than write my own simulator.” For production work, yes. For learning, building the simulator is how you internalize what the DGP actually is. The three-line loop in arma_simulator() is the master equation.
A convention warning when you cross-reference. Enders writes AR coefficients as \(a_i\) (his \(a_1\) is our \(\phi_1\)) and states stationarity in terms of characteristic roots lying inside the unit circle, where this course uses roots of the lag polynomial \(\Phi(L) = 0\) lying outside the unit circle — equivalent framings related by \(r = 1/L\) (§3.2.2). On the MA side, Enders and Hamilton share our plus-sign convention for \(\Theta(L)\), but Box, Jenkins, and Reinsel write the MA polynomial with minus signs, so their \(\theta\)’s are the negatives of ours (§3.3.1). Hamilton Chapters 3–4 are the rigorous reference for the ACF/PACF results and invertibility; Hyndman & Athanasopoulos cover the same ACF/PACF identification rules from the forecasting-workflow angle, with forecast-package examples.
AR(2) roots. Given an AR(2) with \(\phi_1 = 0.4, \phi_2 = 0.2\), compute the discriminant and the characteristic roots by hand. Verify that both roots have modulus less than one, so that the process is stationary. Then simulate the process in R, plot the ACF, and check that the decay rate matches the largest-modulus root.
AR(2) with specified complex modulus. Find an AR(2) coefficient pair \((\phi_1, \phi_2)\) whose characteristic roots are complex conjugates with modulus exactly 0.9. (Hint: for complex roots \(\lambda = a \pm bi\), the characteristic equation is \(r^2 - 2ar + (a^2 + b^2) = 0\), so \(\phi_1 = 2a\) and \(\phi_2 = -(a^2 + b^2)\).) Simulate the process you found, plot its ACF, and describe the oscillatory pattern you see.
MA(2) autocorrelations analytically. For an MA(2) with \(\theta_1 = 0.6, \theta_2 = 0.3\), compute \(\rho_0, \rho_1, \rho_2, \rho_3\) analytically by the cross-product method used for the MA(1) in §3.3.3. Then simulate a long MA(2) (\(T = 5000\)) and compute the sample autocorrelations; verify that they match your analytical values to within sampling error.
The half bound for MA(1). Show by direct algebra or calculus that \(\rho_1 = \theta / (1 + \theta^2)\) cannot exceed 0.5 in absolute value for any real \(\theta\). At what values of \(\theta\) is the bound attained?
The ARMA(1,1) correlogram trap. Simulate an ARMA(1,1) with \(\phi = 0.5, \theta = 0.5\) (\(T = 500\)). Plot its ACF and PACF. Comment on what you see, and on why an automated procedure that reads only these two plots would struggle to identify \((p, q) = (1, 1)\) from them directly.
Over-differencing and non-invertibility in practice. Take an MA(1) with \(\theta\) very close to \(-1\), say \(\theta = -0.99\). Simulate it (\(T = 500\)). Try to estimate it with arima(y, order = c(0,0,1)). What happens to \(\hat{\theta}\)? What does its standard error look like? Relate what you see to the invertibility discussion in §3.3.4 and the L2 over-differencing callback in §3.3.5.
Extend the simulator. Modify arma_simulator() to accept an explicit innovation vector eps as an optional argument, so that the user can pass in the same \(\epsilon_t\) sequence to multiple parameter configurations and compare the resulting series directly. Use your modified simulator to generate an AR(1) with \(\phi = 0.5\) and an AR(1) with \(\phi = 0.9\) driven by the same innovations, and plot them together. What do you observe?
AR(p) is memory of \(y\). Stationarity requires every root of \(\Phi(L) = 1 - \phi_1 L - \ldots - \phi_p L^p\) to lie outside the unit circle; equivalently, every characteristic root of \(r^p - \phi_1 r^{p-1} - \ldots - \phi_p = 0\) must lie inside. The ACF tails off and the PACF cuts off at lag \(p\).
The AR(2) discriminant \(\phi_1^2 + 4 \phi_2\) distinguishes real roots (smooth ACF decay) from complex roots (damped-sinusoidal ACF). Stationarity is a separate question answered by the modulus of the roots.
MA(q) is memory of shocks. It is always weakly stationary, with no conditions on the \(\theta\)’s. It is invertible when every root of \(\Theta(L) = 1 + \theta_1 L + \ldots + \theta_q L^q\) lies outside the unit circle. The ACF cuts off at lag \(q\) and the PACF tails off.
The MA(1) ACF cutoff comes from cross-product expansion: \(\gamma_k\) is zero as soon as \(k > q\) because there is no overlap between the innovations in \(y_t\) and \(y_{t-k}\). The autocorrelation \(\rho_1 = \theta / (1 + \theta^2)\) is bounded by \(0.5\) in absolute value.
The over-differencing trap from L2 is the canonical non-invertible MA(1): \(\Delta y_t = \delta + \epsilon_t - \epsilon_{t-1}\) has \(\Theta(L) = 1 - L\), with a root at \(L = 1\) — exactly on the unit circle.
AR \(\leftrightarrow\) MA isomorphism. A stationary AR(p) is an MA(\(\infty\)) (Wold decomposition). An invertible MA(q) is an AR(\(\infty\)). A small finite ARMA(p,q) is a compact code for infinite-order dynamics, and that is why a two-parameter ARMA(1,1) can capture behavior that a long AR or a long MA would need many parameters to describe.
The PACF is the second fingerprint. It is the correlation between \(y_t\) and \(y_{t-k}\) after removing the linear effect of the intermediate lags; equivalently, the OLS coefficient on the deepest lag in a regression of \(y_t\) on \(y_{t-1}, \ldots, y_{t-k}\). Use ACF and PACF together.
The identification table. AR(p): ACF tails, PACF cuts at \(p\). MA(q): ACF cuts at \(q\), PACF tails. ARMA(p,q): both tail — and that is the Module 4 problem.
Sign conventions. \(\Phi(L) = 1 - \sum \phi_j L^j\) (minus), \(\Theta(L) = 1 + \sum \theta_l L^l\) (plus). \(\epsilon_t\) for DGP innovations, \(e_t\) for residuals of an estimated model. When in doubt, consult notation_dictionary.md.
This module ends on a deliberately unresolved note, and the next one picks it up:
arima() peg \(\hat{\theta}\) near \(-1\) previews Module 4’s treatment of maximum likelihood estimation at the concept level — enough to read arima() output and know when the optimizer is telling you something is wrong. And the practical symptom (\(\hat{\theta} \approx -1\) after differencing) returns in Module 5’s diagnostics as the over-differencing warning light in a fitted model.You now have the vocabulary for almost every univariate model in this course. Modules 4 and 5 are about fitting and checking what you just learned to write down.