Tuesday, September 1: Introduction

No required reading before class.

Thursday, September 3: Background

• Pre-read by Arvind Narananan and Matt Salganik
• Breiman (2001) “Statistical Modeling: The Two Cultures”, Statistical Science. (Read article, skim comments and rejoinder)
• Hand (2006) “Classifier Technology and the Illusion of Progress”, Statistical Science. (Read article, skim comments and rejoinder)

Tuesday, September 8: Predicting geopolitical events (part 1)

• Watch pre-read video
• Read assignment 1
• Tetlock (2005) Expert Political Judgement: How Good Is It? How Can We Know?:
  • Preface, p xi-xvi.
  • Chapter 1, Quantifying the Unquantifiable p 1-24.
  • Chapter 2, The Ego-deflating Challenge of Radical Skepticism p 25-66.
  • Methodological Appendix Read only up to section “Coding Free-flowing Thoughts”, p 239-250.
  • Technical Appendix Read only up to section “Adjustments of Probability Scores to Address Conceptual Objections”, p 273-283.
  • Preface to the 2017 edition, p. xvii-xxii.

Reading question (these questions are designed to help you think about the reading, you don’t need to submit written responses):

1. Summarize in one paragraph the research design and main findings for the parts of Expert Political Judgement that you read.
2. What is the relationship between probability score, variability index, calibration index, and discrimination index? What do these terms capture about the relationship between a set of predictions and outcomes?

Thursday, September 10: Predicting geopolitical events (part 2)

• Watch pre-read video
• Mandel and Barnes (2014) “Accuracy of forecasts in strategic intelligence”, PNAS.
• Tetlock and Mellers (2014) “Judging political judgment”, PNAS.
• Risi, Sharma, Shah, Connelly, Watts (2019) “Predicting history”, Nature Human Behaviour.

Reading question (these questions are designed to help you think about the reading, you don’t need to submit written responses):

1. Tetlock, Mandel and Barnes, and Risi et al. all to try to assess the predictability of geopolitical events. Compare and contrast their approaches in terms of who is doing the predictions, what events are being predicted, and how the predictions are scored.
2. Tetlock and Risi et al. have a comparison of what could be called simple and complex algorithms. What was the result of these comparisons? What, if anything, does the relative performance tell us about the limits of prediction for these tasks?
3. Tetlock, Mandel and Barnes, and Risi et al. all move beyond simply taking the data as given and measuring accuracy and simply running a “horserace” between approaches (e.g., human vs simple algorithm vs complex algorithm). What is your favorite thing that was learned from each of these analyses?
4. Based on the results of these papers, how would you try to measure the difficulty of predicting geopolitical events? Do you notice any empirical patterns from these studies?

Tuesday, September 15: Computer vision and deep neural networks (part 1)

• Pre-read
• Kurenkov (2015) A ‘Brief’ History of Neural Nets and Deep Learning, 4 part blog series.
• Bengio (2009) Learning Deep Architectures for AI, Foundations and Trends in Machine Learning:. Chapters 1–3.

Thursday, September 17: Computer vision (part 2)

• Gershgorn (2017) The data that transformed AI research, Quartz
• Deng, Dong, Socher, Li, Li, Fei-Fei (2009) ImageNet: A Large-Scale Hierarchical Image Database, CVPR.
• Recht, Roelofs, Schmidt, Shankar (2019) Do ImageNet Classifiers Generalize to ImageNet?, ICML.
• Sutton (2019) The Bitter Lesson
• Brooks (2019) A Better Lesson

Tuesday, September 22: Armed conflict (part 1)

• Watch pre-read video
• Cederman and Weidmann (2017) “Predicting armed conflict: Time to adjust our expectations”, Science.
• Gartzke (1999) “War is in the error term”, International Organization.
• Coletta and Gatzke (2003) “Testing War In the Error Term”, International Organization.
• Ward, Greenhill, and Bakke (2010) “The perils of policy by p-value: Predicting civil conflicts”, Journal of Peace Research.

Thursday, September 24: Armed conflict (part 2)

• Watch pre-read video
• Read assignment 2
• Muchlinski et al. (2016) “Comparing Random Forest with Logistic Regression for Predicting Class-Imbalanced Civil War Onset Data”, Political Analysis.
• Neunhoeffer and Sternberg (2019) “How Cross-Validation Can Go Wrong and What to Do About It”, Political Analysis.
• Wang (2019) “Comparing Random Forest with Logistic Regression for Predicting Class-Imbalanced Civil War Onset Data: A Comment”, Political Analysis.
• Heuberger (2019) “Insufficiencies in Data Material: A Replication Analysis of Muchlinski, Siroky, He, and Kocher (2016)”, Political Analysis.
• Muchlinski et al. (2019) “Seeing the Forest through the Trees”, Political Analysis.

Tuesday, September 29: Online ads

• Pre-read
• McMahan et al. Ad Click Prediction: a View from the Trenches. KDD 2013. (Sections 4 and 9 are optional.)
• Frederik and Martijn. The new dot com bubble is here: it’s called online advertising. 2019.
• Lewis, Rao, Reiley. Measuring the Effects of Advertising: The Digital Frontier. NBER 2015 (book chapter; omit Section 7.6).

Thursday, October 1: Recommender systems; engineering limits

• Herlocker et al. Evaluating Collaborative Filtering Recommender Systems. TOIS 2004.
• Amatriain and Basilico. Netflix Recommendations: Beyond the five stars, Part 1. 2012.
• Sculley et al. Machine Learning: The High Interest Credit Card of Technical Debt. 2014.

Tuesday, October 6: Social fads (part 1)

• Watch pre-read video
• Hedstrom (2006) “Experimental Macro Sociology: Predicting the Next Best Seller”, Science.
• Salganik, Dodds, Watts (2006) “Inequality and unpredictability in an artificial cultural market”, Science.
• Martin, Hofman, Sharma, Anderson, Watts (2016) “Exploring Limits to Prediction in Complex Social Systems”, WWW.
• Hofman, Sharma, Watts (2017) “Prediction and explanation in social systems”, Science.

Thursday, October 8: Social fads (part 2)

• Watch pre-read video
• Cheng, Adamic, Dow, Kleinberg, Leskovec (2014) “Can cascades be predicted?”, WWW. (Section 4 is optional)
• Goel “What Can Search Predict?”
• Goel, Hofman, Lahaie, Pennock, Watts (2010) “Predicting consumer behavior with Web search”, PNAS.

Tuesday, October 13: No class for fall break

Thursday, October 15: Measuring predictability

• Pre-read
• Fawcett (2006) An introduction to ROC analysis, Pattern Recognition Letters.
• McCutcheon, The logarithmic scoring rule of decision theory (Section 5 is optional)
• Kvålseth, Cautionary note about R², The American Statistician.
• Ford (2015), Is R-squared Useless?.

Tuesday, October 20: Weather, the Lorenz attractor

• Watch pre-read video
• Watch The Science Behind the Butterfly Effect by Veritasium
• Lorenz (1993) The Essence of Chaos:
  • Appendix 1: “The Butterfly Effect”, p 179-182.
  • Chapter 1: “Glimpses of Chaos”, p 3-24.
  • Chapter 2: “A Journey into Chaos”, Read only the section “Chaos in Action”, p 25-39.
  • The Lorenz attractor by Mark McLure.

Thursday, October 22: Weather, empirical verification

• Watch pre-read video
• Lorenz (1993) The Essence of Chaos:
  • Chapter 3: “Our Chaotic Weather”, p 77-110.
• Bauer, Thorpe, and Brunet (2015) “The quiet revolution of numerical weather prediction”, Nature.
• Lorenz (1982) “Atmospheric predictability experiments with a large numerical model”, Tellus. (Skip section 4)

Tuesday, October 27: Healthcare (part 1)

• Pre-read
• Dawes et al. Clinical vs actuarial judgment. Science 1989.
• Caruana et al. Intelligible Models for HealthCare: Predicting Pneumonia Risk and Hospital 30-day Readmission. KDD 2015.
• Gulshan et al. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA 2016. (Read article as well as technical details in supplement.)

Thursday, October 29: Healthcare (part 2)

• Pre-read
• Muriello et al. Under the hood: Suicide prevention tools powered by AI. 2018.
• Marks. Artificial intelligence for suicide prevention. 2018.
• Nagendran et al. Artificial intelligence versus clinicians: systematic review of design, reporting standards, and claims of deep learning studies. BMJ 2020.
• Maher. Personal genomes: The case of the missing heritability. Nature News 2008.
• Grasby et al. Estimating Heritability from Twin Studies. (Chapter 9 of “Statistical Human Genetics: Methods and Protocols”; Section 1 only.)

Tuesday, November 3: Disease models

• Watch pre-read video
• Fisher. 2020. “R0, the Messy Metric That May Soon Shape Our Lives, Explained”, New York Times.
• Watts and Strogatz. 1998. “Collective dynamics of ‘small world’ networks”, Nature.
• Holme and Takaguichi. 2015. “Time evolution of predictability of epidemics on networks”, Physical Review E.
• Scarpino and Petri. 2019. “On the predictability of infectious disease outbreaks”, Nature Communications.
• Holmes, Rambaut, and Andersen. 2018. “Pandemics: spend on surveillance, not prediction”, Nature.

Thursday, November 5: Disease empirics

• Watch pre-read video
• Ginsberg et al. 2009. “Detecting influenza epidemics using search engine query data” Nature.
• Lazer et al. 2014. “The parable of Google Flu: Traps in big data analysis”, Science.
• Vibound and Vespignani. 2019. “The future of influenza forecasts”, PNAS.
• Reich et al. 2019. “A collaborative multiyear, multimodel assessment of seasonal influenza forecasting in the United States”, PNAS with Comment and Reply.
• Moran et al. 2016. “Epidemic forecasting is messier than weather forecasting: The role of human behavior and Internet data streams in epidemic forecasts”, Journal of Infectious Diseases.

Tuesday, November 10: Predictability of life trajectories

• Watch pre-read video
• Garip (2020) “What failure to predict life outcomes can teach us”, PNAS.
• Salganik et al. (2020) “Measuring the predictability of life outcomes with a scientific mass collaboration”, PNAS.

Thursday, November 12: Searching for dark matter

• Watch pre-read video (available from Piazza)
• Review interview guides (available from Piazza)
• Read interviews (materials available from instructors)
• Complete interview analysis template (available from Piazza) for each family

Tuesday, November 17: Privacy and ethics (part 1)

• Pre-read
• Christl et al. 2017. Corporate surveillance in everyday life. (OK to skim.)
• Duhigg. 2012. How companies learn your secrets. New York Times Magazine. (See instructions here for accessing the NYT.)
• Kosinski, Stillwell, Graepel. 2013. Private traits and attributes are predictable from digital records of human behavior. PNAS.

Thursday, November 19: Privacy and ethics (part 2)

• Pre-read
• Hardt. 2014. How big data is unfair. Medium.
• Obermeyer et al. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science.
• Benjamin. 2019. Assessing Risk, Automating Racism. Science.
• Angwin & Larson. 2016. Bias in Criminal Risk Scores Is Mathematically Inevitable, Researchers Say. ProPublica.
• Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data.
• Barabas et al. 2018. Interventions over Predictions: Reframing the Ethical Debate for Actuarial Risk Assessment. Conference on Fairness, Accountability, and Transparency.

Tuesday, November 24: To be determined (last day of class)