09. Machine Learning

Data Science for Economists

Julian Hinz
Irene Iodice

2026-06-10

Road-map

  1. What is machine learning?
  2. Supervised learning: regression vs classification
  3. A first classifier: \(k\)-NN and the bias–variance trade-off
  4. Decision trees
  5. Evaluating model performance
  6. High-dimensional data and the limits of OLS
  7. Shrinkage: Ridge, LASSO, Elastic Net
  8. An economics application: growth and convergence
  9. Unsupervised learning: PCA and \(k\)-means
  10. The tidymodels pipeline
  11. A teaser: causal forests

Appendix: bias–variance math, OLS breakdown, LASSO geometry, post-double-selection LASSO, Double ML, more applications.

What is Machine Learning?

A Short Definition

“Machine Learning is the science of getting computers to learn without being explicitly programmed.”

\(\qquad\) – Arthur Samuel, 1959


Samuel built a checkers program in the 1950s that played better the more games it had seen. No new code, just more data.

A Working Definition

Mitchell (1997)

A computer program learns from experience \(E\) with respect to a class of tasks \(T\) and a performance measure \(P\), if its performance on \(T\), measured by \(P\), improves with \(E\).

Handwriting recognition

  • \(T\): classify handwritten words in images
  • \(P\): share of correctly classified words
  • \(E\): labelled dataset of handwritten words

Autonomous driving

  • \(T\): drive a highway from camera input
  • \(P\): average distance before human takes over
  • \(E\): sequence of (image, steering command) pairs from a human driver

Three Flavours of ML

  1. Supervised learning – learn \(f: X \to Y\) from labelled data
    • Regression (continuous \(Y\)): predict rent from features
    • Classification (discrete \(Y\)): spam vs ham, sepsis vs not
  2. Unsupervised learning – find structure in unlabelled data
    • Clustering, dimension reduction
  3. Reinforcement learning – an agent learns by trial-and-error from rewards
    • AlphaGo, robot locomotion

Today: mostly (1), a quick tour of (2), and (3) only by name.

Why economists care

  • Healthcare: predict sepsis risk from hundreds of sensor streams (Kleinberg et al., 2015).
  • Finance: predict loan default from mobile-phone metadata when credit history is missing (Bjorkegren & Grissen, 2017).
  • Urban policy: map poverty from satellite imagery with millions of pixel features (Naik et al., 2017).
  • Labour: identify minimum-wage workers from rich demographics (Cengiz et al., 2024).

Common pattern

Hundreds to millions of predictors, often \(p \gg n\). Classical tools wobble.

Supervised Learning

Regression vs Classification

Regression – continuous outcome

  • House prices from size, location, age
  • Wages from education, experience
  • GDP growth from macro indicators

Classification – discrete label

  • Spam vs ham email
  • Sepsis vs healthy patient
  • Iris setosa vs versicolor vs virginica

Same workflow

  1. Split data into training and test sets.
  2. Fit the model on training data.
  3. Score performance on test data.

The test split protects against overfitting – a model that memorises the training set but generalises poorly.

Types of Classification

Binary

Spam / not spam

Multi-class

Iris species

Multi-label

Movie genres (one film can be both thriller and romance)

Running Example: Iris

Iris setosa

Iris versicolor

Iris virginica
  • 150 flowers, 50 of each species
  • 4 features: sepal length/width, petal length/width
  • Goal: predict the species
library(ggplot2)
ggplot(iris, aes(Petal.Length, Petal.Width, color = Species)) +
  geom_point() + theme_minimal()

A First Classifier: \(k\)-NN

\(k\)-Nearest Neighbours

Idea: to classify a new point \(x_0\), look at the \(k\) closest training points and take a majority vote.

Euclidean distance

\[ d(p, q) = \sqrt{(p_1 - q_1)^2 + \cdots + (p_n - q_n)^2} \]

  • No parameters, no training – just store the data.
  • Choice of \(k\) controls how “smooth” the decision boundary is.
  • Sensitive to feature scale – standardise first.

The Bias–Variance Trade-Off, Visualised

\(k = 1\)