Time Series Analysis: Foundations, ARIMA, GARCH Volatility, and Machine Learning Validation

Time series analysis is the discipline of extracting meaningful statistics and predictive signals from chronologically ordered data. Unlike cross-sectional data, time series observations violate the foundational statistical assumption of independent and identically distributed ($i.i.d.$) samples: observations close together in time are inherently correlated.

Whether predicting macroeconomic indicators, quantitative financial returns, or cloud infrastructure demand, mastering time series demands a systematic approach across three layers:

1. →Data Preparation & Foundations (Stationarity, Autocorrelation, and Decomposition)
2. →Classical Modeling (ARIMA, Diagnostics, and Information Criteria)
3. →Advanced Volatility & Machine Learning (GARCH Modeling and Chronological Validation)

Part 1: Data Preparation & Foundations

1. Why must a time series be stationary before modeling?

The Core Reason: Most classical statistical and econometric forecasting models (such as AR, MA, and ARIMA) assume that the underlying generative process is invariant over time.

A time series $\{Y_t\}$ is defined as strictly stationary if the joint distribution of $(Y_{t_1}, \dots, Y_{t_k})$ is identical to $(Y_{t_1 + \tau}, \dots, Y_{t_k + \tau})$ for all time shifts $\tau$. In practice, we evaluate weak (covariance) stationarity, which requires three conditions:

1. →Constant Mean: $\mathbb{E}[Y_t] = \mu \quad \forall t$
2. →Constant, Finite Variance: $\text{Var}(Y_t) = \sigma^2 < \infty \quad \forall t$
3. →Lag-Dependent Autocovariance: $\text{Cov}(Y_t, Y_{t+k}) = \gamma(k)$, depending solely on the lag $k$, not on the absolute time $t$.

What happens if data is non-stationary? If a time series contains a trend or drifting variance, historical sample averages are completely unrepresentative of future expectations. Ordinary least squares (OLS) regression between two independent non-stationary series frequently yields high $R^2$ values and statistically significant $t$-statistics purely due to shared temporal drift—a dangerous trap known as spurious regression.

How to achieve stationarity:

• →Differencing: $\Delta Y_t = Y_t - Y_{t-1}$ removes linear polynomial trends.
• →Logarithmic Transformation: $\ln(Y_t)$ stabilizes exponential variance growth.
• →Log Returns: $r_t = \ln(Y_t) - \ln(Y_{t-1})$ simultaneously stabilizes both trend and variance for financial asset prices.
• →Formal Testing: Check stationarity using the Augmented Dickey-Fuller (ADF) test ($H_0$: Unit root present, non-stationary) and the KPSS test ($H_0$: Trend stationary).

2. What is the difference between ACF and PACF plots, and how do you use them?

The Autocorrelation Function (ACF) and Partial Autocorrelation Function (PACF) are the foundational diagnostic plots used to identify the orders of autoregressive and moving average processes.

• →

  ACF (Autocorrelation Function): Measures the linear correlation between $Y_t$ and its lagged value $Y_{t-k}$, including both direct and indirect linear dependencies across all intervening time steps: $\rho_k = \frac{\text{Cov}(Y_t, Y_{t-k})}{\sqrt{\text{Var}(Y_t) \text{Var}(Y_{t-k})}}$ Example: The correlation $\rho_3$ between Day 1 and Day 4 includes the direct effect of Day 1 on Day 4, plus the indirect effect of Day 1 impacting Day 2, which impacts Day 3, which in turn impacts Day 4.

  • →Model Identification: The ACF cuts off sharply after lag $q$ for a pure Moving Average process $\text{MA}(q)$.
• →

  PACF (Partial Autocorrelation Function): Measures the correlation between $Y_t$ and $Y_{t-k}$ after removing the mutual linear influence of all intervening lags $Y_{t-1}, Y_{t-2}, \dots, Y_{t-k+1}$: $\alpha_k = \text{Corr}(Y_t - \hat{Y}_t, Y_{t-k} - \hat{Y}_{t-k})$ It isolates the pure, direct relationship between observations separated by exactly $k$ periods.

  • →Model Identification: The PACF cuts off sharply after lag $p$ for a pure Autoregressive process $\text{AR}(p)$.

Quick Diagnostic Summary Table:

3. How do you handle seasonality vs. trend?

Real-world series contain structural patterns that obscure underlying random processes:

• →Trend ($T_t$): Long-term upward or downward movement in the series over extended horizons.
• →Seasonality ($S_t$): Periodic fluctuations that repeat at fixed, known calendar intervals (e.g., retail spikes every December, electricity surges during summer afternoons).
• →Residual / Noise ($R_t$): The stationary, unpredictable stochastic component left after isolating trend and seasonality.

Decomposition Architectures:

1. →Additive Decomposition: $Y_t = T_t + S_t + R_t$ Use when: The amplitude of seasonal swings remains constant regardless of the overall level of the series.
2. →Multiplicative Decomposition: $Y_t = T_t \times S_t \times R_t \iff \ln(Y_t) = \ln(T_t) + \ln(S_t) + \ln(R_t)$ Use when: The magnitude of the seasonal variation scales proportionally with the trend (e.g., airline passenger counts growing 15% every summer as total base traffic doubles).

Modern decomposition methods like STL (Seasonal and Trend decomposition using Loess) offer robust handling of non-linear trends and evolving seasonal patterns over time.

Part 2: Classical Modeling (ARIMA & Friends)

1. What do the parameters in an $\text{ARIMA}(p, d, q)$ model mean?

The Autoregressive Integrated Moving Average model, denoted $\text{ARIMA}(p, d, q)$, is the workhorse of linear time series forecasting:

$(1 - \sum_{i=1}^p \phi_i L^i) (1 - L)^d Y_t = c + (1 + \sum_{j=1}^q \theta_j L^j) \epsilon_t$

Where $L$ is the lag operator ($L^k Y_t = Y_{t-k}$) and $\epsilon_t \sim \mathcal{WN}(0, \sigma^2)$ is white noise error.

• →$p$ (Autoregressive Order): The number of lagged observations of the dependent variable included in the regression equation. An $\text{AR}(p)$ model regresses current values on its own $p$ immediate past values: $Y_t = c + \phi_1 Y_{t-1} + \dots + \phi_p Y_{t-p} + \epsilon_t$
• →$d$ (Degree of Differencing): The number of times the raw series must be differenced to eliminate stochastic trends and achieve covariance stationarity. If $d = 1$, the model operates on first differences $\Delta Y_t = Y_t - Y_{t-1}$; if $d = 2$, it operates on second differences $\Delta^2 Y_t$.
• →$q$ (Moving Average Order): The size of the moving window applied to historical forecast shocks. An $\text{MA}(q)$ model expresses the current observation as a linear combination of the current shock and the past $q$ random error terms: $Y_t = \mu + \epsilon_t + \theta_1 \epsilon_{t-1} + \dots + \theta_q \epsilon_{t-q}$

2. How do AIC and BIC help in time series model selection?

When selecting parameters $(p, d, q)$, there is an inherent trade-off between maximizing fit and avoiding overparameterization. Estimating too many coefficients causes overfitting, capturing in-sample noise rather than generalizable signals.

Both the Akaike Information Criterion (AIC) and Bayesian Information Criterion (BIC) balance goodness of fit against complexity:

$\text{AIC} = 2k - 2\ln(\hat{L})$ $\text{BIC} = k \ln(n) - 2\ln(\hat{L})$

Where:

• →$\hat{L}$ is the maximized likelihood of the fitted model.
• →$k$ is the number of estimated parameters ($p + q + \text{constant}$).
• →$n$ is the total number of observations.

How to use AIC and BIC:

1. →Candidate models are ranked; the model with the lowest score is optimal.
2. →Penalty Comparison: For any sample size $n \ge 8$, $\ln(n) > 2$. Consequently, BIC penalizes model complexity substantially more severely than AIC.
3. →Rule of Thumb:
   • →Use AIC when the primary objective is out-of-sample predictive performance.
   • →Use BIC when the goal is consistent structural identification of the true underlying parsimonious model.

Part 3: Advanced Volatility & Machine Learning

1. When should you use a GARCH model instead of an ARIMA model?

Standard ARIMA models assume homoskedasticity: the error variance $\sigma^2$ is assumed to be constant across all time periods: $\text{Var}(\epsilon_t \mid \mathcal{F}_{t-1}) = \sigma^2$

In quantitative financial assets (equities, foreign exchange, commodities), this assumption fails dramatically due to volatility clustering:

"Large changes tend to be followed by large changes, of either sign, and small changes tend to be followed by small changes." — Benoit Mandelbrot

• →Use ARIMA when: Modeling and forecasting the conditional mean $\mathbb{E}[Y_t \mid \mathcal{F}_{t-1}]$ (predicting price trajectories or inventory levels).
• →Use GARCH when: The conditional mean is close to zero or unpredictable, but the conditional variance $\sigma_t^2 = \text{Var}(Y_t \mid \mathcal{F}_{t-1})$ changes dynamically over time.

The $\text{GARCH}(1, 1)$ Specification:

Let return residuals be $a_t = \sigma_t z_t$, where $z_t \sim \text{i.i.d. } \mathcal{N}(0, 1)$. The conditional variance evolves as: $\sigma_t^2 = \omega + \alpha a_{t-1}^2 + \beta \sigma_{t-1}^2$

Where:

• →$\omega > 0$ is the baseline variance.
• →$\alpha a_{t-1}^2$ (ARCH term): Measures the immediate reaction to market shocks.
• →$\beta \sigma_{t-1}^2$ (GARCH term): Measures volatility persistence.
• →Stability requirement: $\alpha + \beta < 1$. If $\alpha + \beta \approx 1$, volatility shocks persist for long horizons (Integrated GARCH).

GARCH models are essential for Value-at-Risk (VaR) calculations, option pricing volatility surfaces, and dynamic risk management.

2. Why can't you use standard K-Fold Cross-Validation for time series?

In classical cross-sectional machine learning, data points are assumed to be independent. Standard $K$-fold cross-validation randomly shuffles observations into $K$ subsets, training on $K-1$ folds and validating on the remaining fold.

Why this breaks time series:

1. →Temporal Leakage (Lookahead Bias): Random partitioning places future observations in the training set while evaluating on past observations. The model learns patterns informed by future information, generating artificially optimistic validation scores that collapse in production.
2. →Autocorrelation Disruption: Randomly dropping observations breaks the autocorrelation structure and creates artificial discontinuities in lag structures.

The Correct Method: Time Series Split (Rolling Forecast Origin)

To validate time series models without leakage, use Forward-Chaining (Walk-Forward Validation):

Fold 1: [ Train: t_1 ... t_k ] -> [ Test: t_{k+1} ]
Fold 2: [ Train: t_1 ... t_{k+1} ] -> [ Test: t_{k+2} ]
Fold 3: [ Train: t_1 ... t_{k+2} ] -> [ Test: t_{k+3} ]
Fold 4: [ Train: t_1 ... t_{k+3} ] -> [ Test: t_{k+4} ]

• →Expanding Window: The training set grows chronologically, incorporating each new validated point into historical context.
• →Rolling Window: If the data generating process exhibits regime shifts, maintain a fixed training window length $[t - W, t]$ while sliding forward.

Practitioner Summary

1. →Never fit ARIMA on non-stationary data: Apply differencing or log-returns; verify with ADF and KPSS tests.
2. →Inspect ACF/PACF systematically: Identify MA cutoffs in ACF and AR cutoffs in PACF.
3. →Balance fit with parsimony: Use AIC/BIC grids to prevent overparameterization.
4. →Separate mean from volatility: Use ARIMA for conditional mean; use GARCH for clustering conditional variance.
5. →Respect chronology in ML: Always employ temporal walk-forward splits to prevent lookahead bias.