03 — Large Structured Data

Working with large datasets using data.table, DuckDB, and Arrow.

A port with ships and containers.

This module covers techniques for handling large structured datasets: firm-level trade data, Pareto distributions, and modern tools like data.table, DuckDB, and Apache Arrow.

Lecture slides

View fullscreen | Source

Code

Check the course repository for the R scripts.

Key packages: data.table, arrow, duckdb, ggplot2

Further resources

data.table

• The official data.table vignette
• Grant McDermott’s data.table slides
• data.table cheat sheet

ggplot2

• Kieran Healy’s Data Visualization — A practical introduction
• R for Data Science: Data Visualization

DuckDB & Arrow

• DuckDB R API documentation
• Apache Arrow R cookbook