Chapter 03

Medium

Linear Processes and ARMA Models

00 · Symbol Glossary

$\psi(z)$psi of z — MA polynomial / causal weights

$\psi(z)=\sum_{j=0}^{\infty}\psi_j z^j$ with $\psi_0=1$ . The process is $X_t-\mu=\psi(B)\varepsilon_t$ where $B$ is the backshift, $BX_t=X_{t-1}$ . Square-summable weights $\sum\psi_j^2<\infty$ give a well-defined WN-driven series.

$\phi(B)$phi of B — AR polynomial

$\phi(z)=1-\phi_1 z-\cdots-\phi_p z^p$ . Causal ARMA requires the roots of $\phi(z)=0$ to lie outside the unit circle.

$\theta(B)$theta of B — MA polynomial

$\theta(z)=1+\theta_1 z+\cdots+\theta_q z^q$ . Invertibility: roots of $\theta(z)=0$ outside the unit circle, so $\varepsilon_t$ is a function of current and past $X$ .

$B$B — backshift (lag) operator

$B^k X_t = X_{t-k}$ . Polynomials in $B$ are the language of ARMA: $\phi(B)X_t=\theta(B)\varepsilon_t$ .

01 · Wold’s Representation

Every weakly stationary, purely nondeterministic process is an infinite MA of its innovations. Purely nondeterministic means the one-step linear prediction error has positive variance — the process is not perfectly predictable from its infinite past using linear combinations of past values.

Theorem — Wold Decomposition (regular case)

If $\{X_t\}$ is weakly stationary, mean $\mu$ , and purely nondeterministic, then

X_t-\mu = \sum_{j=0}^{\infty}\psi_j\varepsilon_{t-j}, \qquad \psi_0=1, \quad \sum_{j=0}^{\infty}\psi_j^2<\infty

with $\varepsilon_t$ the one-step linear prediction error (white noise). The $\psi_j$ are the impulse responses: $\partial X_{t+j}/\partial\varepsilon_t = \psi_j$ in this linear world.

ARMA models are finite-parameter restrictions on $\{\psi_j\}$ : rational $\psi(z)=\theta(z)/\phi(z)$ .

02 · MA, AR, and ARMA

Definition — ARMA$(p,q)$

$\{X_t\}$ is ARMA $(p,q)$ if it is weakly stationary and

\phi(B)(X_t-\mu)=\theta(B)\varepsilon_t

with $\varepsilon_t\sim\mathrm{WN}(0,\sigma^2)$ , $\phi_p\neq 0$ , $\theta_q\neq 0$ , and $\phi$ and $\theta$ having no common roots.

MA $(q)$ : $\phi\equiv 1$ , so $X_t-\mu=\theta(B)\varepsilon_t$ . Always weakly stationary (finite MA of WN).
AR $(p)$ : $\theta\equiv 1$ , so $\phi(B)(X_t-\mu)=\varepsilon_t$ . Stationarity is not free — it is a root condition.

Definition — Causality (Stationarity of ARMA)

The ARMA is causal (a convergent function of current and past $\varepsilon$ ) iff every root of $\phi(z)=0$ satisfies $\lvert z\rvert>1$ . Then $\psi(z)=\theta(z)/\phi(z)$ has a power series in $\lvert z\rvert\leq 1$ , and $\gamma(h)$ is well-defined and absolutely summable.

Definition — Invertibility

The ARMA is invertible iff every root of $\theta(z)=0$ satisfies $\lvert z\rvert>1$ . Then $\varepsilon_t=\theta(B)^{-1}\phi(B)(X_t-\mu)$ depends on current and past $X$ only — required for residual diagnostics and for reading $\varepsilon_t$ as an innovation.

Causality gives a past-shock representation for $X$ ; invertibility makes that shock recoverable from observed $X$ . Both are root conditions on $\phi$ and $\theta$ respectively, and both matter before you trust forecasts or residual tests.

Example — AR(1) causality

$X_t=\phi X_{t-1}+\varepsilon_t$ has $\phi(z)=1-\phi z$ , root $z=1/\phi$ . Causality: $\lvert 1/\phi\rvert>1$ , i.e. $\lvert\phi\rvert<1$ . Then $\psi_j=\phi^j$ , $\gamma(0)=\sigma^2/(1-\phi^2)$ , $\rho(h)=\phi^{\lvert h\rvert}$ .

If $\lvert\phi\rvert>1$ one can still write a future-dependent expansion in $\varepsilon_{t+j}$ , which is useless for forecasting. If $\phi=\pm 1$ , the process is the unit-root / ARIMA case of Chapter 04, not a weakly stationary AR(1).

❌ Fitting AR(1) with |φ̂| = 1.04 and calling it stationary ARMA

Software will often still compute OLS $\hat\phi$ .

Why it breaks: $\lvert\phi\rvert\geq 1$ is outside the causal AR(1) model. The sample may be a short stretch of a random walk or an explosive path.

Consequence: inspect roots of $\hat\phi(z)$ , not only $t$ -stats. If a root sits on the unit circle, switch to the ARIMA toolkit (Chapter 04), do not “keep the ARMA and ignore the root.”

03 · Identification: ACF and PACF Patterns

Rule — Box–Jenkins signatures (causal, invertible)

Model	ACF	PACF
WN	$0$ for $h\neq 0$	$0$ for $h\neq 0$
MA $(q)$	zero after lag $q$	tails off
AR $(p)$	tails off	zero after lag $p$
ARMA $(p,q)$	tails off	tails off

“Tails off” means decay (geometric for AR(1); mixture of geometrics / damped sinusoids for higher order), not a hard cutoff.

These are population patterns. Sample plots are noisy; use them to propose a small set of $(p,q)$ , then compare with information criteria (Chapter 05).

04 · Yule–Walker for AR $(p)$

Definition — Yule–Walker equations

For causal AR $(p)$ , $\gamma(h)=\phi_1\gamma(h-1)+\cdots+\phi_p\gamma(h-p)$ for $h\geq 1$ , with $\gamma(-h)=\gamma(h)$ . In matrix form $\Gamma_p\phi=\gamma_p$ where $(\Gamma_p)_{ij}=\gamma(i-j)$ and $(\gamma_p)_i=\gamma(i)$ . Solving with $\hat\gamma$ gives the Yule–Walker estimator $\hat\phi$ .

Step-by-step — AR(1) Yule–Walker

The

h=1

equation is

\gamma(1)=\phi\gamma(0)

, so

\phi=\rho(1)

Replace by samples:

\hat\phi_{\mathrm{YW}}=\hat\rho(1)

Innovation variance:

\hat\sigma^2=\hat\gamma(0)(1-\hat\phi^2)

, from

\gamma(0)=\phi\gamma(1)+\sigma^2

and

\gamma(1)=\phi\gamma(0)

OLS on

x_t=\phi x_{t-1}+\varepsilon_t

is not identical to Yule–Walker (end-effect / divisor), but they agree as

T\to\infty

for causal AR(1).

Common factors

If $\phi(z)$ and $\theta(z)$ share a root, the ARMA is overparameterized — infinitely many $(p,q)$ pairs give the same $\psi(z)$ . Cancel common roots before interpreting lag order.

05 · Exercises

EXERCISE 3.1

$\phi(z)=1-1.3z+0.4z^2=(1-0.8z)(1-0.5z)$ .

Roots $z=1/0.8=1.25$ and $z=1/0.5=2$ , both outside the unit circle, so the AR(2) is causal (weakly stationary). Impulse responses decay as a mix of $0.8^j$ and $0.5^j$ .

Is $X_t=1.3 X_{t-1}-0.4 X_{t-2}+\varepsilon_t$ a causal AR(2)? Factor $\phi(z)$ and check roots.

EXERCISE 3.2

MA(1) invertibility is $\lvert\theta\rvert<1$ . The pair $(\theta,\sigma^2)$ and $(1/\theta, \theta^2\sigma^2)$ share the same $\gamma(h)$ when $\theta\neq 0$ .

$\gamma(0)=\sigma^2(1+\theta^2)$ , $\gamma(1)=\theta\sigma^2$ . Replacing $\theta$ by $1/\theta$ and $\sigma^2$ by $\theta^2\sigma^2$ yields the same $\gamma$ . The invertible choice is the unique one with $\lvert\theta\rvert<1$ (or $\lvert\theta\rvert\leq 1$ at the boundary). Always report the invertible MA.

Why does an MA(1) with $\theta=2$ and an MA(1) with $\theta=1/2$ (and a rescaled $\sigma^2$ ) have the same autocovariance? Which one is invertible?

EXERCISE 3.3

Match cutoff vs decay in both ACF and PACF.

ACF cutoff after lag 2, PACF decaying: MA(2) signature. Not AR(2), which would cut off in the PACF and decay in the ACF.

Population ACF is zero after lag 2; PACF decays. Which ARMA class is this the signature of?

06 · Chapter Summary

Piece	Condition / meaning
Wold	stationary regular process $= \psi(B)\varepsilon_t$
ARMA	rational $\psi=\theta/\phi$
Causal	AR roots $\lvert z\rvert>1$
Invertible	MA roots $\lvert z\rvert>1$
AR $(p)$ PACF	cuts off after $p$
MA $(q)$ ACF	cuts off after $q$
Yule–Walker	$\Gamma\phi=\gamma$ from $\gamma(h)$

Next: Chapter 04 — Unit Roots, Differencing, and ARIMA, when an AR root sits on the unit circle instead of outside it.