Time Series Foundations

Time series analysis studies observations indexed by time: $\{X_t: t \in \mathbb{Z}\}$, where $X_t$ is the value at time $t$. The foundations of time series cover three core concepts: stationarity (is the process stable over time?), dependence (how does $X_{t-k}$ affect $X_t$?), and parsimonious models that capture that dependence (AR, MA, ARMA, ARIMA). These let you move from raw data to forecasts and inference.

Why care? Because many applied problems are sequential: forecasting electricity load, modeling GDP, or controlling processes in engineering. A model that respects time-dependence is usually far more accurate than treating observations as iid. The prerequisites you already know — Covariance and Correlation (measuring linear relation) and Linear Regression (estimating linear coefficients via least squares) — are used heavily: sample autocovariances extend covariance to lags, and regression ideas underpin estimation of autoregressive parameters.

Stationarity: Intuitively, a stationary time series 'behaves the same' through time. We commonly use weak (covariance) stationarity: 1) constant mean $\mathbb{E}[X_t]=\mu$ for all $t$, 2) autocovariance $\gamma(s,t)=\operatorname{Cov}(X_s,X_t)$ depends only on lag $|t-s|$, i.e. $\gamma(h)=\operatorname{Cov}(X_{t},X_{t-h})$. Strict stationarity (distribution invariance under time shifts) is stronger but weaker in practice because many models are only second-order.

Concrete example: Random walk vs AR(1)

• •Random walk: $X_t = X_{t-1} + \varepsilon_t$ with $\varepsilon_t \sim (0,\sigma^2)$. The mean of $X_t$ depends on $t$ (it drifts with accumulated noise), so it's non-stationary.
• •AR(1): $X_t = \phi X_{t-1} + \varepsilon_t$, with $|\phi|<1$ is stationary. For $|\phi|<1$ and $\varepsilon_t$ white noise with variance $\sigma^2$, the stationary variance is

Numeric example: if $\phi=0.6$ and $\sigma^2=1$, then $\operatorname{Var}(X_t)=1/(1-0.36)=1/0.64=1.5625.$ This shows a finite, time-invariant variance.

Autocorrelation and autocovariance: the autocovariance at lag $k$ is

and the autocorrelation is

where $\gamma_0=\operatorname{Var}(X_t)$. These generalize the concepts from Covariance and Correlation. A quick sample calculation: suppose observed series $[2,4,6,8]$. The sample mean $\bar X=5$. The sample lag-1 autocovariance (using denominator $n$ for simplicity) is

Compute terms: $(4-5)(2-5)=(-1)(-3)=3$, $(6-5)(4-5)=(1)(-1)=-1$, $(8-5)(6-5)=(3)(1)=3$. Sum = 3-1+3=5. So $\hat\gamma_1 = 5/4 = 1.25$. The sample variance $\hat\gamma_0 = \frac{1}{4}[(2-5)^2+(4-5)^2+(6-5)^2+(8-5)^2] = (9+1+1+9)/4 = 20/4 = 5$. Thus sample autocorrelation at lag 1 is $\hat\rho_1 = 1.25/5 = 0.25$.

Partial Autocorrelation (PACF): the PACF at lag $k$, denoted $\phi_{kk}$ or $\alpha_k$, measures the correlation between $X_t$ and $X_{t-k}$ after removing linear effects of intermediate lags $1\ldots k-1$. A simple way to compute PACF is to fit the linear regression

and take the coefficient on $X_{t-k}$. This uses Linear Regression skills directly. Numeric example: for an AR(1) with coefficient $\phi=0.6$, PACF at lag 1 is $0.6$ and at higher lags it is (theoretically) 0.

Model types (intuition):

• •AR(p) (autoregressive of order p): $X_t = \phi_1 X_{t-1} + \dots + \phi_p X_{t-p} + \varepsilon_t$. Like regressing current value on p past values. If p=1, AR(1) as above.
• •MA(q) (moving average of order q): $X_t = \varepsilon_t + \theta_1 \varepsilon_{t-1} + \dots + \theta_q \varepsilon_{t-q}$. This is a linear filter of white noise and has dependence that dies after q lags in the PACF/ACF in characteristic ways.
• •ARMA(p,q): combination of both.
• •ARIMA(p,d,q): integrate non-stationary series d times (differences) then fit ARMA(p,q).

These building blocks let you represent many time series parsimoniously and prepare for forecasting. The rest of the lesson develops the main mechanical tools: ACF/PACF, estimation via Yule–Walker and least squares, and the Box–Jenkins cycle for practical modeling.

ACF and PACF are the primary diagnostic tools for identifying the dependence structure in a stationary time series. They are analogous to the correlation and partial correlation concepts from Covariance and Correlation and use Linear Regression for PACF computations.

Definitions and sample formulas

• •The theoretical autocovariance function (ACVF) is $\gamma_k = \mathrm{Cov}(X_t,X_{t-k})$. The theoretical autocorrelation function (ACF) is $\rho_k = \gamma_k/\gamma_0$.
• •For an observed series $x_1,\dots,x_n$, the sample autocovariance at lag $k$ is often taken as

Example: with $[2,4,6,8]$ we computed $\hat\gamma_1=1.25$ and $\hat\gamma_0=5$, so $\hat\rho_1=0.25$ (see Section 1). If you prefer the unbiased denominator $n-k$ replace $1/n$ with $1/(n-k)$; both are used in practice.

Properties of ACF for simple models (critical identification clues)

• •AR(1): $X_t=\phi X_{t-1}+\varepsilon_t$ with $|\phi|<1$ has theoretical autocorrelation $\rho_k = \phi^k$. Numeric example: $\phi=0.6$ gives $\rho_1=0.6$, $\rho_2=0.36$, $\rho_3=0.216$, etc.
• •MA(1): $X_t=\varepsilon_t+\theta\varepsilon_{t-1}$ has $\rho_1=\theta/(1+\theta^2)$ and $\rho_k=0$ for $k>1$. Numeric example: $\theta=0.5$ gives $\rho_1=0.5/(1+0.25)=0.5/1.25=0.4$, and $\rho_2=0$.

Thus the ACF of AR(1) decays exponentially, while the ACF of MA(1) cuts off after lag 1. This distinction is foundational for identification.

Partial Autocorrelation Function (PACF)

• •PACF at lag $k$, denoted $\alpha_k$ or $\phi_{kk}$, is the coefficient on $X_{t-k}$ in the best linear predictor of $X_t$ using $X_{t-1},\dots,X_{t-k}$:

then $\alpha_k = \beta_k$ from that regression. For stationary AR(p), the PACF cuts off after lag $p$ (i.e., $\alpha_k=0$ for $k>p$). For MA(q), the PACF decays gradually.

Computational methods

• •Sample ACF is computed directly from $\hat\gamma_k$ as $\hat\rho_k = \hat\gamma_k/\hat\gamma_0$.
• •Sample PACF can be estimated by fitting the regression above for each $k$ (this uses Linear Regression knowledge). Another route uses the Durbin–Levinson recursion or solving Yule–Walker linear systems; both rely on sample autocovariances.

Small numeric demonstration of PACF via regression: Data: $x=[1.2, 0.9, 1.1, 1.4, 1.3]$. Compute $\bar x=1.18$. To estimate PACF at lag 2 (i.e., coefficient on $X_{t-2}$ when regressing on lags 1 and 2), form the design matrix for t=3..5:

X = [[x_2,x_1],[x_3,x_2],[x_4,x_3]] = [[0.9,1.2],[1.1,0.9],[1.4,1.1]].

Response y = [x_3,x_4,x_5] = [1.1,1.4,1.3].

Run a small OLS: $\beta=(X^TX)^{-1}X^T y$. Numeric calculation (sketch): $X^TX = \begin{pmatrix}0.9^2+1.1^2+1.4^2 & 0.9\cdot1.2+1.1\cdot0.9+1.4\cdot1.1 \\ same & 1.2^2+0.9^2+1.1^2\end{pmatrix}$. Compute numbers: $0.81+1.21+1.96=3.98$, off-diagonal $1.08+0.99+1.54=3.61$, lower-right $1.44+0.81+1.21=3.46$. Solve to obtain coefficients; the lag-2 coefficient is the PACF estimate. This step explicitly uses Linear Regression.

Interpreting ACF/PACF plots

• •If ACF decays slowly and PACF cuts off after p lags: suggest AR(p).
• •If ACF cuts off after q lags and PACF decays: suggest MA(q).
• •Mixed decay patterns suggest ARMA models.

Statistical significance: For large n, sample autocorrelations under white noise are approximately $N(0,1/n)$. A rough 95% confidence band is $\pm 1.96/\sqrt{n}$. Example: if $n=100$, the band is $\pm 0.196$. Any sample ACF outside that band suggests significant autocorrelation.

Thus ACF and PACF, combined with numerical rules and plots, are your first discovery tools in the Box–Jenkins cycle (coming later). They depend directly on Covariance and Correlation ideas and on Linear Regression for PACF estimation.

We now define the principal parametric families and show core formulas and estimation strategies. These models are linear in either past values or past shocks and are the workhorses of time series forecasting.

Autoregressive (AR) models

• •AR(p) model:

where $\varepsilon_t$ is white noise with mean 0 and variance $\sigma^2$. This is analogous to regressing $X_t$ on its past p values (Linear Regression). To fit AR(p), one can use OLS on lagged regressors when the series is mean zero or include an intercept if needed. Another method uses the Yule–Walker equations derived from autocovariances.

Yule–Walker equations (for AR(p))

The theoretical autocovariances satisfy:

A practical way is to solve the linear system

Use sample autocovariances to get Yule–Walker estimators. Example: AR(1) Yule–Walker reduces to $\gamma_1 = \phi \gamma_0$, so $\phi = \gamma_1/\gamma_0$. Numerically, using the sample values from Section 1 where $\hat\gamma_1=1.25$ and $\hat\gamma_0=5$, the Yule–Walker estimate is $\hat\phi = 1.25/5 = 0.25$.

Moving Average (MA) models

• •MA(q) model:

The parameters $\theta_j$ multiply the unobserved past shocks $\varepsilon_{t-j}$, making direct linear regression impossible. Estimation typically proceeds via maximum likelihood or nonlinear least squares. However, the MA autocovariance has closed-form expressions in terms of $\theta$ and $\sigma^2$, which you can equate to sample autocovariances for method-of-moments estimation in small orders. Example: for MA(1) $\rho_1 = \theta/(1+\theta^2)$. If sample $\hat\rho_1=0.4$, solving $0.4 = \theta/(1+\theta^2)$ yields $0.4+0.4\theta^2=\theta \Rightarrow 0.4\theta^2 - \theta + 0.4=0$. Solve numerically: discriminant $=1-4\cdot0.4^2=1-0.64=0.36$, so $\theta=(1\pm 0.6)/(0.8)$. Positive root $(1-0.6)/0.8=0.4/0.8=0.5$ or $(1+0.6)/0.8=1.6/0.8=2$; invertibility constraint usually selects $|\theta|<1$, so $\theta=0.5$.

ARMA(p,q) models

• •ARMA(p,q): combines AR and MA parts:

Estimation is typically via maximum likelihood (often numerically executed) or conditional least squares.

ARIMA(p,d,q): dealing with nonstationarity

• •If the series is non-stationary, differencing can remove trends. A first difference is $Y_t = X_t - X_{t-1}$. The ARIMA(p,d,q) model means the d-th difference of $X_t$ follows an ARMA(p,q). For example, a random walk $X_t = X_{t-1} + \varepsilon_t$ is ARIMA(0,1,0).

Forecasting with AR models (closed-form)

• •For AR(1) with mean zero: $X_t = \phi X_{t-1} + \varepsilon_t$, the h-step ahead forecast from time t is

Estimation practicalities and model selection

• •Fit candidate models (identified via ACF/PACF), estimate parameters (Yule–Walker/OLS/ML), check diagnostics (residuals should resemble white noise), and compare models with AIC/BIC. Use overfitting caution: more parameters reduce residual variance but can harm out-of-sample forecasts.

Identifiability and invertibility

• •AR models require roots of the characteristic polynomial $1-\phi_1 z - \dots - \phi_p z^p$ lie outside the unit circle for stationarity. MA models require the invertibility condition (roots of $1+\theta_1 z + \dots$ outside unit circle). Checking these ensures unique parameterizations.

Concrete example illustrating estimation difference: Suppose sample ACF shows strong decay and PACF cuts after 2 lags: this suggests AR(2). Using sample autocovariances $\hat\gamma_1,\hat\gamma_2$ we solve Yule–Walker linear system to get $\hat\phi_1,\hat\phi_2$ and compute residuals $\hat\varepsilon_t$ then estimate $\hat\sigma^2$ as mean squared residual. This combines Covariance and Correlation (autocovariances) and Linear Regression (residual analysis).

In short, AR/MA/ARMA/ARIMA provide interpretable, mathematically tractable families; Yule–Walker and OLS give simple estimation for AR models, while MA parameters often need likelihood-based methods.

The Box–Jenkins (BJ) methodology is a structured four-step practical workflow for building ARIMA-class models. It turns the theoretical tools (ACF/PACF, AR/MA definitions, stationarity tests) into an applied recipe. Each step references ideas you've seen in previous sections and relies on Covariance and Correlation and Linear Regression for computations.

Box–Jenkins steps

1) Identification: Use plots of the time series, ACF, and PACF to decide whether differencing is needed and to propose model orders p and q. Look for patterns: slow ACF decay suggests nonstationarity or AR behavior; ACF cutoffs help detect MA terms. Example: If ACF decays slowly and first differences look stationary, set d=1 and then examine the ACF/PACF of differenced series.

2) Estimation: Fit candidate ARIMA(p,d,q) models via maximum likelihood or conditional least squares; estimate parameters $\phi,\theta,\sigma^2$. For AR parts you might use Yule–Walker to get initial estimates (a closed-form start) and then refine via ML. Example: Fit ARIMA(1,1,1) to data; the differencing makes the series stationary before estimation.

3) Diagnostic checking: Analyze residuals $\hat\varepsilon_t$ from the fitted model. Residual ACF should show no significant autocorrelation (randomness). Perform Ljung–Box test for joint autocorrelation up to lag m. Residuals should be approximately uncorrelated with constant variance and mean zero. If diagnostics fail, go back to step 1 and refine model orders.

4) Forecasting: With a validated model, compute forecasts and forecast intervals. ARIMA models allow iterative computation of point forecasts and mean squared error estimates for prediction intervals.

Concrete real-world examples and use cases

• •Economics: GDP and inflation series often modeled with ARIMA for medium-term forecasting. Example: GDP growth series are often stationary after first differencing (d=1), motivating ARIMA( p,1,q ) fits.
• •Finance: Daily returns are often roughly stationary and may require ARMA to capture serial dependence; however, volatility clustering is typically modeled with GARCH, which builds on AR foundations.
• •Energy: Electricity load often displays strong daily/weekly seasonality. Seasonal ARIMA (SARIMA) extends ARIMA with seasonal differencing and seasonal AR/MA terms: SARIMA(p,d,q)(P,D,Q)_s. Example: hourly load with daily seasonality s=24 might need D=1 or SAR terms.

Seasonality and SARIMA: When periodic patterns exist, include seasonal terms. For seasonal period s, a SARIMA model includes factors like $(1 - \Phi_1 B^s - \dots )$ for seasonal AR and seasonal differencing $(1-B^s)^D$. Numeric example: for monthly data with annual seasonality s=12, seasonal differencing with D=1 is $Y_t = (1-B^{12})X_t = X_t - X_{t-12}$.

Model evaluation and selection

• •Use information criteria: AIC = $-2\log L + 2k$, BIC = $-2\log L + k\log n$, where $k$ is number of parameters. Lower values are preferred. Example: Compare AR(1) (k small) vs ARMA(1,1) (k larger); pick model minimizing AIC with parsimony in mind.

Connections to machine learning and further topics

• •State-space models and the Kalman filter generalize ARIMA and provide efficient likelihood computation and handling of missing data; they underpin many time-series ML methods. Recurrent neural networks and sequence models are flexible alternatives but lack statistical interpretability.

Practical pitfalls and tips

• •Over-differencing can destroy structure and induce invertibility issues: only difference to achieve stationarity (use tests like the augmented Dickey–Fuller test).
• •Inspect residuals visually and with tests; small sample sizes make ACF/PACF noisy.
• •Start with simple models (parsimony) and gradually increase complexity.

A short end-to-end numeric illustration

Suppose you have monthly sales series and see a linear upward trend. First difference to remove trend (d=1). The differenced series shows ACF cutting off at lag 1 and PACF decaying: candidate ARIMA(0,1,1). Fit MA(1) on differenced data; estimate $\theta\approx0.5$ and residuals look white. Forecast by integrating predicted differences forward to recover level forecasts. This is the Box–Jenkins loop in action.

In summary, Box–Jenkins ties together stationarity tests, ACF/PACF-guided identification, estimation (Yule–Walker, OLS, ML), and diagnostic checking to produce reliable forecasts. The concrete algebra and numerical examples throughout this lesson show how concepts from Covariance and Correlation and Linear Regression are repurposed to handle temporal dependence and forecasting tasks.