06. Text as Data

Learning objectives

By the end of today you should be able to:

1. Tokenise raw text and build a Bag-of-Words document–term matrix in R.
2. Compute and interpret tf–idf weights.
3. Apply a domain-specific sentiment dictionary and evaluate its accuracy.
4. Explain two research designs that use dictionary counts or cosine similarity in economics.

Why text?

It has become increasingly affordable to store and process vast quantities of digital text, triggering an explosion of empirical research that leverages text as data.

Historical cost of computer memory and storage

Text as Data – Strengths

• “Always on”

Google Trends: Ukraine

Text as Data – Strengths (cont.)

• “Non-Reactive”

Google Trends: US abortion

Text as Data – Weaknesses

• Incomplete
• Inaccessible or sensitive
• Non-representative
• Confounding

Google Flu Trends – overprediction due to media-driven search behaviour

Check here the Nature article

Check here the article

Dimensionality challenge

Why is text data hard?

Imagine a document with \(w\) tokens, each drawn from a vocabulary of \(p\) distinct words.

Texts \(\rightarrow\) Feature matrix \(\rightarrow\) Analysis

Source: Kenneth Benoit, Course on Quantitative Text Analysis (TCD 2016)

Roadmap

1. Representing Text
   • Pre-processing: tokenisation, normalisation, stop words, n-grams, lemmatisation
   • Bag-of-Words, tf-idf, Document-Term Matrix
2. Statistical Methods
   • Dictionary-based classification
   • Supervised text classification
   • Unsupervised: similarity, clustering, topic modelling
3. Applications in Economics
   • Baker, Bloom & Davis (2016): Economic Policy Uncertainty
   • Hoberg & Phillips (2016): Product markets from 10-K filings

How to represent news?

library(wordcloud)

What would you have done differently?

Processing pipeline

1. Split corpus into documents (e.g. one article per day).
2. Tokenise documents (words, n-grams, or characters).
3. Normalise and reduce: lower-case, remove punctuation/stop words, stemming or lemmatisation.

Normalisation: lower-case and punctuation removal

library(tidytext)

raw <- tibble(text = "The ECONOMISTS were studying the Economies of Countries!")

# unnest_tokens() lowercases and strips punctuation automatically
raw |> unnest_tokens(word, text) |> pull(word)
#> [1] "the"        "economists" "were"       "studying"
#> [5] "the"        "economies"  "of"         "countries"

raw |> unnest_tokens(word, text, to_lower = FALSE)   # preserve case
raw |> unnest_tokens(word, text, strip_punct = FALSE) # keep punctuation

Tokenisation and stop-word removal

Tokenisation breaks text into smaller units – words, characters, or n-grams.

library(gutenbergr); library(tidytext)
gutenberg_download(1184)[1, ] |>
  unnest_tokens(input = text, output = word, token = "words")
#> # A tibble: 5 x 2
#>   gutenberg_id word
#>          <int> <chr>
#> 1         1184 the
#> 2         1184 count
#> 3         1184 of
#> 4         1184 monte
#> 5         1184 cristo

library(hcandersenr); library(tidytext)
tidy_fir_tree <- hca_fairytales() |>
  filter(book == "The fir tree") |>
  unnest_tokens(word, text) |>
  filter(!(word %in% stopwords(source = "snowball")))

Tokenisation and stop-word removal in practice

library(tidytext)

doc <- tibble(text = "It is a truth universally acknowledged, that a single man in possession of a good fortune, must be in want of a wife.")

tokens <- doc |> unnest_tokens(word, text)
nrow(tokens)
#> [1] 23

# Which tokens are stop words?
tokens |> semi_join(stop_words, by = "word") |> pull(word)
#>  [1] "it"   "is"   "a"    "that" "a"    "in"   "of"   "a"
#>  [9] "must" "be"   "in"   "want" "of"   "a"

# What remains after anti_join?
tokens |> anti_join(stop_words, by = "word") |> pull(word)
#> [1] "truth"        "universally"  "acknowledged" "single"
#> [5] "man"          "possession"   "good"         "fortune"
#> [9] "wife"

23 tokens \(\rightarrow\) 14 stop words removed \(\rightarrow\) 9 content words. The signal-to-noise ratio improves substantially.

Stemming vs. lemmatisation

Both reduce words to a base form:

• am, are \(\Rightarrow\) be
• car, cars, car’s, cars’ \(\Rightarrow\) car

library(textstem)
x <- c('doggies', ',', 'they', "aren't", 'Joyfully', 'running', '.')
stem_words(x)
#> [1] "doggi"  ","  "thei"  "aren't"  "Joyfulli"  "run"  "."

lemmatize_words(x)
#> [1] "doggy"  ","  "they"  "aren't"  "Joyfully"  "run"  "."

Lemmatisation in practice

library(tidytext)
library(textstem)

doc <- tibble(text = "The economists were studying the economies of countries that had been economically restructured.")

tokens <- doc |> unnest_tokens(word, text)
tokens |> pull(word)
#> [1] "the"          "economists"   "were"         "studying"
#> [5] "the"          "economies"    "of"           "countries"
#> [9] "that"         "had"          "been"         "economically"
#> [13] "restructured"

# Lemmatise: collapse inflected forms to their dictionary root
tokens |> mutate(lemma = lemmatize_words(word)) |> select(word, lemma)
#>           word         lemma
#>    economists    economist    # plural → singular
#>          were            be    # past → base verb
#>      studying         study    # gerund → base verb
#>      economies       economy    # plural → singular
#>      countries       country    # plural → singular
#>  economically     economic    # adverb → adjective
#>  restructured   restructure    # past participle → base verb

7 of 13 tokens change form – the vocabulary shrinks and variants of the same concept are now counted together.

Bag-of-Words representation

When text is represented as the bag (multiset) of its words:

• disregard grammar and word order
• keep multiplicity (multiset)

Computing tf-idf in R

library(tidytext)
library(dplyr)

# From a tidy text data frame with columns: document, word
word_counts <- tidy_books |>
  count(document, word, sort = TRUE)

# Compute tf-idf
tf_idf <- word_counts |>
  bind_tf_idf(word, document, n)

# Top distinctive words per document
tf_idf |> group_by(document) |> slice_max(tf_idf, n = 5)

The Document–Term Matrix (DTM)

After tokenisation, stop-word removal, stemming/lemmatisation, and (optionally) tf-idf weighting, your corpus becomes a document–term matrix:

\[\mathbf{C}_{n \times p} = \begin{pmatrix} c_{11} & c_{12} & \cdots & c_{1p} \\ c_{21} & c_{22} & \cdots & c_{2p} \\ \vdots & & \ddots & \vdots \\ c_{n1} & c_{n2} & \cdots & c_{np} \end{pmatrix}\]

• \(n\) documents (rows), \(p\) vocabulary terms (columns)
• Entries are raw counts, relative frequencies, or tf-idf weights
• Typically very sparse – most entries are zero

This matrix is the starting point for all downstream analysis: dictionaries, regression, clustering, topic models.

quanteda 4.0 – a modern R text toolkit

The quanteda package (Benoit et al.) provides an integrated, high-performance pipeline for text analysis in R:

library(quanteda)

# Build a corpus, tokenise, create DTM in three lines
corp  <- corpus(my_texts)
toks  <- tokens(corp, remove_punct = TRUE) |>
           tokens_remove(stopwords("en"))
dfmat <- dfm(toks) |> dfm_tfidf()

Key features:

• Fast C++ back-end; handles millions of tokens
• Built-in dictionaries, collocations, n-grams
• Seamless integration with quanteda.textplots, quanteda.textstats, and quanteda.textmodels
• Works naturally with tidy workflows via tidytext conversion

See: quanteda.io

Methods overview

Three families of methods – today covers 1 and 2; last class covers 3:

1. Dictionary-based classification – researcher-defined keyword lists; no labelled data required
   • Existing dictionaries, building your own, regex rules, validation
2. Supervised text classification – learn from labelled documents; BoW/tf-idf as features
   • Logistic regression, LASSO, random forests
3. Unsupervised text analysis – discover structure without labels
   • Document similarity and clustering
   • Topic modelling (LDA)

\(\rightarrow\) Methods 3 and 4 from Gentzkow et al. (generative models, word embeddings) are covered in the last class.

Dictionary-based methods

Classifying documents when categories are known:

1. Identify a set of words that correspond to each category
   • thesaurus: vote = {poll, suffrage, franchis*, ballot*, ^vot}
   • sentiment: positive or negative
   • emotions: sad, happy, angry, anxious
   • topics: economics, culture, etc.
2. Count how many times these words appear in each document
3. Normalise by document length
4. Validate:
   • Code a few documents manually and check alignment
   • Check sensitivity to exclusion of specific words
   • Decide sample size based on the power of your test

Regular expressions

Regex = algebraic notation for pattern-matching in strings. Essential for counting keyword occurrences and text cleaning. Key patterns: [0-9] (digits), ^ (start), $ (end), .* (any characters).

library(stringr)

x <- c("GDP grew by 2.3% in Q1 2020",
       "Unemployment rate: 4.1%",
       "Federal Reserve raised rates",
       "Vote on the bill: 234 in favour")

str_detect(x, "\\d+\\.?\\d*%")       # find percentage figures
#> [1]  TRUE  TRUE FALSE FALSE

str_extract(x, "\\d{4}")             # extract 4-digit years
#> [1] "2020" NA     NA     NA

str_replace(x, "Federal Reserve", "Fed")  # normalise an entity name
#> [1] "GDP grew by 2.3% in Q1 2020"
#> [2] "Unemployment rate: 4.1%"
#> [3] "Fed raised rates"
#> [4] "Vote on the bill: 234 in favour"

Reference: MIT regex cheatsheet (PDF)

Dictionaries are context-specific

Loughran and McDonald (2010) used the Harvard-IV-4 TagNeg (H4N) dictionary to classify sentiment for firms’ 10-K filings: three-quarters of the “negative” words were typically not negative in a financial context – e.g. cancer, tax, cost, capital, board, liability, foreign.

• Polysemes – words with multiple meanings cause misclassification
• H4N also lacked negative financial words: felony, litigation, restated, misstatement, unanticipated

\(\Rightarrow\) Always validate dictionaries against your specific domain.

Building your own dictionary

1. Identify “extreme texts” with known positions
2. Search for differentially occurring words using word frequencies
3. Use these words (or their lemmas) as category markers

Supervised text classification

Dictionary methods use researcher-defined keywords. Supervised methods learn from labelled documents:

• \(y_i\): human-coded label, e.g. relevant / not relevant, positive / negative
• \(X_i\): BoW or tf-idf feature vector
• Learn \(\hat{f}(X_i)\) to predict \(y_i\) on new documents

Similarity across documents

Define \(a = |BoW_{R1} \cap BoW_{R2}|\), \(b = |BoW_{R1}| - a\), \(c = |BoW_{R2}| - a\).

1. Cosine similarity: normalises by length – robust when documents differ in size \[s_{\cos} = \frac{a}{\sqrt{(a+b)(a+c)}}\]

2. Jaccard similarity: treats each unique word equally – penalises length differences \[s_{\text{jacc}} = \frac{a}{a+b+c}\]

How much in our example? What changes after stemming?

Topic modelling: discovering themes automatically

Goal: find latent topics in a corpus without pre-defined categories.

Key idea (LDA): assume each document is a mixture of topics, and each topic is a probability distribution over words. Infer the hidden structure from the observed word counts.

Applications overview

Two examples of text-based classification in economics:

1. Dictionary-based methods – Baker, Bloom & Davis (2016): Economic Policy Uncertainty Index
2. Clustering by text similarity – Hoberg & Phillips (2016): Product market definitions from 10-K filings

Application 1: Measuring Economic Policy Uncertainty

Can we measure policy uncertainty in the US? How does it look, and does it matter?

Baker, Bloom & Davis (2016)

The EPU Index

Validation of the EPU Index

Computer-coded vs. human-coded EPU

Testing economic hypotheses

Theory: \(\uparrow\) Uncertainty \(\rightarrow\) \(\uparrow\) real option to wait \(\rightarrow\) \(\downarrow\) investment (Bernanke 1983, Dixit & Pindyck 1994)

Regression: Firm-level estimates exploit differences in industry exposure to government:

\[Y_{it} = \text{FE}_i + \text{FE}_t + \beta\, (INT_i \times EPU_t) + \alpha\, (INT_i \times GS_t) + \varepsilon_{it}\]

where \(INT_i\) measures firm \(i\)’s dependence on government contracts, and \(Y_{it}\) is investment or hiring.

Results: EPU and firm-level outcomes

Application 2: Product markets from text (Hoberg & Phillips 2016)

How to define a product market that is endogenous to firms’ choices?

Validation: Text-based vs. traditional classification

Testing Sutton and Schumpeter with text

Text-based product similarity allows Hoberg & Phillips to test whether firms change their product-market position over time.

Sutton-style mechanism:
Advertising and R&D are endogenous sunk costs that differentiate products and soften competition.

Schumpeterian mechanism:
R&D creates new products and technologies, allowing firms to escape existing competitive pressure.

Prediction:

\[ \text{Advertising/R\&D today} \rightarrow \downarrow \text{future similarity} \rightarrow \downarrow \text{number of close rivals} \rightarrow \uparrow \text{profitability} \]

Results: Testing Sutton’s predictions

Application 3: Business news and business cycles

Figure: Topic taxonomy from Bybee et al. (2024).

From articles to topic attention

\(\Rightarrow\) Topic modelling measures what the news is about, not only whether it is positive or negative.

Key take-aways

• Text is high-dimensional; representation choices matter.
• Simple pre-processing + tf-idf already provides useful features.
• Dictionary methods are quick but require context-specific validation.
• Similarity metrics unlock clustering and network applications.
• Modern toolkits like quanteda make the full pipeline accessible in R.
• LLMs complement (but do not replace) interpretable bag-of-words approaches.
• Note: LLM-based classification is increasingly replacing hand-built dictionaries for many tasks — we’ll see this next.

Further reading

• Jurafsky & Martin, Speech and Language Processing (draft 3e).
• Gentzkow, Kelly & Taddy (2019), “Text as Data” (JEL).
• Benoit et al., quanteda documentation and tutorials.
• Ferrario and Stancheva (2022), “Eliciting People’s First-Order Concerns: Text Analysis of Open-Ended Survey Questions” (link).
• Ash & Hansen (2023), “Text Algorithms in Economics”, Annual Review of Economics.
• Ash & Hansen (2024), “Large Language Models in Economics”, CEPR DP19479.