Lecture Roadmap

• Mapping the Topic
• Strategy
• Two Major Debates
  • Quant vs Qual
  • Experimental Turn
• Bonus slides

What is political science?

• What do we study?
• How do we study it?

How do we study it?

• Role of Theory
• Role of Empirics
  • How to leverage empirics to evaluate theories.

Past Questions

• 2015: In the study of comparative politics, what criteria should we use for selecting cases?
• 2016: ‘Single-country case studies that allow us to identify clear causal relationships contribute more to our knowledge than large-scale cross-country comparisons.’ Discuss.
• 2017: No question
• 2018: ‘Much more is lost than gained in political analysis by attempting to replace proper names, for example the names of countries, with variables.’ Discuss.
• 2019: No question
• 2020: Does the experimental turn in comparative politics mean that observational studies are no longer relevant?

Why NOT to choose this topic

• Unpopular: only 4/58 candidates answered it last year.
• Uncertain: 2017 and 2019 have no question for this topic.
• Expansive: 21 starred readings. Last year’s question was about 4 of the single-star papers.
• Contentious: most researchers are not methodologists, but have strong opinions about methodology.
• Difficult: the exam rewards theorising and synthesis. Hard to do with methods.

Reasons to choose it anyways

• You enjoy the debate on methods and methodology.
• Strong knowledge of methodology is foundational:
  • Provides a basis to critically engage with the rest of the literature.

My Recommendations

• Good back-up topic.
  • Complementary to all other topics
• If you haven’t revised a sub-section, avoid answering a question about that sub-section with the readings you do know.
• You can draw on your knowledge of other weeks.
  • Can show a deeper engagement.
  • Maybe use better-known debates/examples.

Some Complementary Questions:

• 2020: Is it possible to make generalisations about the causes of democratisation across different historical eras?
• 2019: Is it impossible to evaluate the causal effect of constitutions?
• 2018: Are electoral systems endogenous to party competition?
• 2017: Questions 5 (historical approaches) and 11 (institutionalism)

Locating the Debate

• This is debate about the method of:
  • Gathering data
  • Analysing data
  • Inferring theory from data
• It is tempting to make it about these things:
  • Theory vs Empirics
  • Induction vs Deduction
  • I recommend you keep these conceptually separated

Definitions

• What distinguishes quantitative and qualitative approaches?

• What tools are associated with each?

• Is this a false dichotomy?

• You must have an answer to these questions.

An Shot Across the Bow

Designing Social Inquiry may be the most important reading on this list, but it’s often misconstrued. Let’s play a game.

• Quantitative and qualitative methods differ fundamentally.
• Qualitative scholars can learn a lot from quantitative scholars.
• Quantitative methods are superior to qualitative methods.
• Qualitative scholars should construct scales and run regressions.

Bonus Round

KKV also claim that:

• Qualitative scholars should seek to increase their \(N\).
• All science seeks to make either descriptive or causal inferences.
• Single case studies are pointless.
• Quantitative research is less prone to fallacy.

What KKV Actually Claim

• Both quantitative and qualitative research can be judged by the same criteria because they have a common logic of inference.
  • These criteria or more explicitly stated/formulated in quantitative approaches.
• All scientific research seeks to make inferences.
  • Pure description is useful, but not scientific.
  • Descriptive inference is good, but the goal is causal inference.
• Theories must be falsifiable, and should maximise leverage.
  • Parsimony is not a must.
  • Not a great deal is said on theory generation.
• Goal is unbiased inference: correct on average.

Rethinking Social Inquiry (RSI)

The three highlighted chapters present distinct takes on the debate:

• Chapter 1: Brady, Collier and Seawright
• Chapter 5: Rogowski
• Chapter 9: McKeown

RSI 1: Omissions in the Debate

• Not a refutation of KKV. Authors agree that:
  • common framework and vocabulary are important.
  • both are founded on essentially similar epistemologies.
• Things KKV over-emphasizes:
  • Increasing N is not always helpful
  • Deductive approaches at the cost of inductive approaches.
• Things KKV under-emphasizes:
  • The authors distinguish between the quantitative approach and the statistical/experimental approach.
  • Conceptualization and measurement.
• Overlooked contributions of qualitative methodology
  • Knowledge of cases and contexts contributes to achieving valid inference.

RSI 5: (KKV on) Rogowski on KKV

• Use of Anomalous Cases:
  • Single anomalous observation is sufficient to disprove a theory.
  • Lijphart (1968) uses the Netherlands to discredit a theory linking cross-cutting cleavages with social peace.
  • Rogowski says this would be inadmissible for KKV as single observation.
  • KKV reply that Lijphart positions his observation in a wider literature; not a single observation.
• Historical Accounts
  • Rich description, no inference.
  • KKV say this is useful, but political science must strive further (inference).

RSI 5: cont.

• Selection on DV
  • Katzenstein (1985) looks only at small European states; Rogowski points out that this fails to vary the DV (economic success).
  • KKV point out that Katzenstein eschews causal claims (but retains ambiguously causal language).

RSI 9: Epistemological Challenges

• McKeown questions whether there is a common logic of inference at all.
• KKV essentially espouse a logical positivist epistemology based on Poppler and Hempel.
• Problematic to apply these ideas to research based on one or few cases:

Tripartite Consensus?

• This consensus, laid out in Laitin (2002), consist of three prongs:
  1. Large-N Statistical Analysis: not only for theory-testing, but for comparing the explanatory of competing explanations
  2. Theorization: comparative scholars now engage in a continuous and simultaneous process of theory testing and building.
     1. Relatedly, formalization plays an important role in endogenizing the core variables of an analysis.
  3. Case Study: to examine whether theories apply to real cases, and to build narative (e.g. by juxtaposing historical cases)

Lack of Consensus?

• Much of the previous debate is focused on empirics.
  • Not orthogonal to theory, but often holds theory constant.
• Munck (2007) notes a distinction between theories and theorising, and a lack of consensus on either.
• Quantitative/qualitative distinction exists here as well.

Quantitative Civil War

• Samii (2016) provides an excellent summary of the paradigm shift in causally-focused quantitative political science research.
• In particular, this paper addresses a false “trade-off” that is often used to either criticise experimental approaches or justify observational studies.

A Characterisation of Conventional Quantitative Research

• Collect data on DVs and IVs of interest.
• Identify relations between them using multiple regression.
• Include “standard” set of controls to “account for” omitted variable bias.

Internal Validity

• Not so clear how to interpret changes in coefficients resulting from changes in controls.
• Choice of control variables affects coefficients of interest.
• Using the example on pg 948, if we find Y ~ T + X shows an effect of T, but Y ~ T + X + W does not, there are multiple possible interpretations:
  • Omitted Variables: there was confounding W→T, W→Y, that has now been corrected.
  • Relevant subpopulation: there is an effect of T on Y, but it is very small for the subpopulation for which W.
  • Control misspecification: X or W are misspecified.
  • Bias amplification: there was residual confounding (E(e)|X != 0), but adding W made it even worse.

External (Ecological) Validity

• Trade-off: field experiment vs regional survey data
• Relevant comparison population:
  • Scope condition of claims are limited to those with covariate overlap on units with variation on outcome.
• Significant difference between nominal and effective sample, once positivity and overlap are accounted for.

Design-Based Causal Inference

• Is it possible to make causal claims from purely observational data?
• Focus is now on causal designs; using experimental methods or natural experiments to convincingly identify causal effects.
• Causal identification \(\ne\) micro-focus.

Experiments

• Lab, lab-in-the-field, and field experiments (Grossman and Paler 2015)
  • Trade-off: researcher control over mechanism vs realism
• Still small in number and entrepreneurial in nature; authors call for infrastructure.
  • Relatedly, Samii (2016) calls for theorising to be left to meta-analyses that account for many different causally identified mid-level results.

How do political scientists choose questions?

• Most of these readings have something to say on this.
• Interesting, consequential, and contributes to the literature.
• Should availability of causal design override this?

Some Current Stuff

• Grimmer, Roberts and Stewart (2021), Annual Review of Political Science. “Machine Learning for Social Science: An Agnostic Approach”
• Lundberg, Johnson and Stewart (forthcoming), American Sociological Review. “What is Your Estimand? Defining the Target Quantity Connects Statistical Evidence to Theory”

Discussing Questions

• In the study of comparative politics, what criteria should we use for selecting cases?
• ‘Single-country case studies that allow us to identify clear causal relationships contribute more to our knowledge than large-scale cross-country comparisons.’ Discuss.
• ‘Much more is lost than gained in political analysis by attempting to replace proper names, for example the names of countries, with variables.’ Discuss.
• Does the experimental turn in comparative politics mean that observational studies are no longer relevant?