10. LLMs

Data Science for Economists

Julian Hinz
Irene Iodice

2026-07-01

Learning objectives

By the end of today you should be able to:

  1. Explain why economists moved from bag-of-words to embeddings to LLMs.
  2. Describe what word embeddings capture and how transformers improve on them.
  3. Use LLMs via APIs from R (the ellmer package) for classification, extraction, and annotation tasks.
  4. Design structured outputs (JSON schemas) for reproducible measurement.
  5. Apply prompt-engineering strategies to economics research problems.
  6. Describe at least two published economics papers that leverage modern NLP/LLM representations.

Motivation

Why move beyond Bag-of-Words?

  • Order and context: BoW ignores word order – “bank raises rates” vs “rates raise bank”.
  • Synonymy: “job”, “employment”, “work” treated as independent features.
  • Sparsity: high-dimensional DTM leads to curse of dimensionality.
  • Modern NLP compresses text into dense, low-dimensional vectors that encode semantic information.

Economic upside

Better text representations often improve predictive power (e.g. central-bank tone \(\to\) yields) and enable causal designs that exploit semantic shifts (e.g. narrative shocks).

Economic use cases

  • Forecasting & Nowcasting: Use news or earnings call transcripts with embeddings to predict macro variables (e.g. GDP, inflation).
  • Policy uncertainty: Dictionary vs embedding-based measures of uncertainty (Baker et al. 2016).
  • Industry competition: Similarity of 10-K filings to gauge product-market overlap (Hoberg & Phillips 2016).
  • Sentiment analysis: Gauge financial sentiment in firm filings or tweet corpora to predict returns (Ke et al. 2019).
  • Labor-market skill maps: Match job-posting text to O*NET tasks using embeddings (Hansen et al. 2021).

Why economists now mine text

  • Digital text – news, patents, 10-Ks, job ads – is exploding; numbers alone miss these “soft” signals. Gentzkow et al. 2019; Ash & Hansen 2023
  • NLP lets us turn unstructured words into structured variables we can graph, regress, and test.
  • Early tools (bag-of-words, \(n\)-grams) count words but ignore context.
  • Modern models capture meaning and nuance \(\to\) richer measures of innovation, policy tone, skills, etc.

From bag-of-words to LLMs: a leap in capability

  • Then: bag-of-words \(\approx\) word counts; good for frequency, weak on order and synonyms.
  • Now: transformer-based large language models (BERT, GPT) use attention to weigh each word in context (Vaswani et al. 2017).
  • Fine-tuned variants (FinBERT, ClimateBERT) already boost accuracy in financial and climate text. Yang et al. 2020; Webersinke et al. 2021
  • Instruction-tuned assistants (ChatGPT, Claude) make advanced NLP tasks almost turn-key for economists.

From counts to dense vectors

The problem with counting words

  • BoW ignores word order; documents are represented by token counts.
  • Simple, interpretable, widely used in economics.
  • Major limitations:
    • Synonymy: “weak” \(\neq\) “tepid”
    • Polysemy: “statistics lie” vs “cats lie”
    • No understanding of word order.

Fixes on top of BoW (topic models, n-grams, dependency parsing) help but remain brittle, high-dimensional, and context-insensitive.

Word embeddings: distributional semantics

“You shall know a word by the company it keeps.” – J. R. Firth

  • Word2Vec (Mikolov et al. 2013): CBOW & Skip-Gram architectures.
  • GloVe (Pennington et al. 2014): factorises global co-occurrence matrix.
  • Embedding dimension typically 100–300.
  • Cosine similarity captures analogies: \(\text{king} - \text{man} + \text{woman} \approx \text{queen}\).

Key idea

Move beyond which words occur to where words live in a low-dimensional space. Build a co-occurrence matrix and factorise it so that similar words sit close together.

Vector arithmetic and analogies

  • Embeddings support meaningful linear operations: \(\text{"king"} - \text{"man"} + \text{"woman"} \approx \text{"queen"}\).
  • Latent dimensions pick up interpretable traits (royalty, masculinity, age …).
  • Helps quantify bias: the “gender” direction separates stereotypically male vs female occupations, enabling post-processing debiasing (Bolukbasi et al. 2016).

Where words live: the embedding space

  • Every token is a point in a high-dimensional space; project down to 2D and semantic neighbourhoods appear.
  • Note the economics cluster — economy, growth, income, investment, money — sitting together, far from king / queen or city / town.
  • This geometry is what lets LLMs generalise: nearby words behave alike, so meaning is interpolated, not looked up.

Source: S. Wolfram, What Is ChatGPT Doing…? (writings.stephenwolfram.com, 2023)

Dimensionality reduction: topic models

  • LDA (Latent Dirichlet Allocation): each document is a mixture of topics; each topic a distribution over words. \[\theta_d \sim \text{Dir}(\alpha), \quad \beta_k \sim \text{Dir}(\eta)\]
  • Bayesian inference reduces overfitting in sparse, high-dimensional spaces.
  • Produces interpretable, human-readable topics.
  • Widely used in economics: e.g. Hansen et al. (2018) on Fed communications.

Other approaches

LSA (PCA on DTM), pLSA (probabilistic LSA), NMF (non-negative matrix factorisation) all share the same goal: reduce dimensionality from \(V\) (vocab size) to \(K\) (topic count). LDA adds Dirichlet priors and is the most widely adopted.

Transformers and Attention

How an LLM actually works: one pipeline

Strip away the hype and every LLM does one thing: given the text so far, predict the next token — then append it and repeat.

  1. Tokenize text into integer IDs
  2. Embed each token as a dense vector (+ position)
  3. Transformer blocks (attention + FFN), repeated \(N\) times, mix context across tokens
  4. Unembed + softmax \(\rightarrow\) a probability over the whole vocabulary
  5. Sample one token, append, and loop

. . .

“What ChatGPT is always fundamentally trying to do is produce a reasonable continuation of whatever text it’s got so far.” — Stephen Wolfram

Source: 0xkato, How LLMs Actually Work (0xkato.xyz)

Next-token prediction, concretely

Ask a model to continue *“The best thing about AI is its ability to ___“* and it returns a probability for every word in its vocabulary:

  • learn — 4.5%
  • predict — 3.5%
  • make — 3.2%
  • understand — 3.1%
  • do — 2.9%

Ranked probabilities fall off as a power law — a few plausible words, then a long tail.

Source: S. Wolfram, What Is ChatGPT Doing…? (writings.stephenwolfram.com, 2023)

Sampling and temperature

  • The model gives a distribution; to generate text it samples a word, appends it, and repeats — “a biased coin flip, at 100,000-way scale.”
  • Temperature controls the gamble:
    • 0 — always take the top word \(\rightarrow\) deterministic, but soon repetitive.
    • ~0.8 — sometimes pick lower-ranked words \(\rightarrow\) more fluent and varied.
  • Same prompt, temperature 0.8, five runs — five different continuations:

Source: S. Wolfram, What Is ChatGPT Doing…? (writings.stephenwolfram.com, 2023)

Tokenization: the model never sees letters

  • Text is first split into subword tokens, each mapped to an integer ID.
  • Common words are a single token; rare words fragment (e.g. “token” + “ization”).
  • The model only ever sees token IDs, never raw characters — which is why it fumbles letter-level tasks (famously: counting the r’s in “strawberry”).
  • Practical upshot: context windows and API costs are counted in tokens — roughly ¾ of a word in English.