Between Geddes (2003) and Brady and Collier (2004), we get a good feel for the qual-quant divide within political science. Of course, these are ideal types and most research agendas straddle this divide. Nonetheless, we can learn much about triangulation and constructing creative research designs from this debate. –Erik
Methods: Parsimony and Causality vs. Complexity and Context
Geddes, Barbara. 2003. Paradigms and Sand Castles.
Rethinking Social Inquiry, Brady and Collier (eds.)
King, Gary, Robert O. Keohane, and Sidney Verba. 2004 (1995). “The Importance of Research Design,” pp. 181-192.
McKeown, Timothy J. “Case Studies and the Limits of the Quantitative Worldview,” pp. 139-167.
Munck, Gerardo L. “Tools for Qualitative Research,” pp. 105-121.
Ragin, Charles C. “Turning the Tables: How Case-Oriented Research Challenges Variable-Oriented Research,” pp. 123-138.
The RSI contributors continue the volume’s ongoing critique of KKV on several counts. The contributions to this section of the book overlap and echo many of the same themes, such as the ability of qualitative research to provide more context for the selection of cases and independent variables, the unique tools by which qualitativists can define the universe of cases through within-case analysis and typologies, the sharpening of hypotheses and theories through refinement of original assumptions based on evaluation of evidence, and the general iterative nature of qualitative research that can lead to the identification of new variables or a more clearly articulated object of observation.
Munck (ch. 7) questions what he takes to be many of the operating assumptions of KKV, namely that qualitative research and methodology have a lot to learn from quantitativists. He argues that this subordinates or ignores the strengths specific to qualitative approaches, such as “contextually-grounded analysis, typologies, and process tracing.” (106) Munck further claims that an advantage of qualitative methods, and fundamental break with quantitative approaches, is the fact that “hypothesis testing is best seen as an iterative process that interacts with the development of theory, rather than as a process in which theory is more early treated as static.” (107) Munck believes that KKV treat qualitative research as being methodologically deficient, where it in fact has a set of “well-developed procedures—which in fact address every step in the research process. The problem is not that qualitative researchers lack tools to conduct their research, but rather that these tools have not been adequately systematized.” (120) These tools are outlined in Table 7.1 (pp. 108-9). Other issues addressed include the bounding of theory, deterministic vs. probabilistic models of causation, and qualitative comparative analysis (QCA).
Ragin (ch. 8) teases out the differences between variable-oriented inquiry and case-oriented inquiry. Ragin claims that variable-oriented inquiry is not a substitute for case-oriented research, because “the case-oriented approach is better understood as a different mode of inquiry with different operating assumptions.” (124) Ragin wants to examine practical concerns that are specific to case-oriented inquiry. Table 8.1 (p. 126) identifies the tasks and tools of case-oriented inquiry as defining the population of cases, focusing on positive cases, defining relevant negative cases, analyzing multiple and conjunctural causes, and addressing nonconforming cases. Ragin claims that the difference between variable- and case-oriented methods can make variable-oriented methods more rigorous, and indeed that it would become more dynamic and less static, reduce the error term, and allow for the examination of multiple sources of causation.
McKeown (ch. 9) contends that KKV’s claim to a common logic of inference that underpins all inquiry, reveals much about the quantitative worldview.
KKV (ch. 11) respond to criticism of their thesis and arguments. Though originally published in 1995, the KKV response addresses concerns that remain in the current volume.
Geddes (ch. 3) takes up the concern of case selection, and particularly the focus on selection of cases on the dependent variable. She claims that often the consequences of selection on the dependent variable are addressed by authors, especially in the qualification of the scope of their results. However, readers often ignore or minimize these boundaries/caveats and read bounded results as generalized theory that apply to wide universes of cases. Geddes looks particularly at three major bodies of work and the caveats that the authors have applied to their generalizability – labor repression, revolution, and _____ – to demonstrate the limits and possibilities of studies in which cases were selected on the dependent variable. The rationale, as strongly put forth by KKV, for avoiding case selection on the dependent variable “stem from the logic of inference.” (91) This can obscure the effects of other explanatory variables that produce the phenomena under observation and result in more attribution of effect to the independent variables under consideration than they deserve. Geddes also argues, though, that “selection on the dependent variable biases statistical results toward finding no relationship even when one does, in fact, exist.” (93)
- Problem of overfitting (95)
- “appropriate universe of observations on which to test a hypothesis depends on the domain implied by the hypothesis.” (96)
- large universes lend themselves to random samples, but “randomization does not guarantee the absence of correlation.” (97)
- “habit of writing down explicit coding rules” (100)
- “many of Skocpol’s cases violate her own criteria for limiting the domain of her argument.” (11)
- ideal test of Skocpol’s hypothesis (111)
- “If we accept the idea that the domain depends on the argument itself, then these findings suggest that if Skocpol had selected a broader range of cases to examine, rather than selecting on the dependent variable, she would have reached different conclusions.” (113)
- however, in Skocpol’s case, selection on dependent variable does not invalidate the original argument
- Bayesian analysis (114-117) – new data is required
- “When the hypothesized necessary cause is very common in the world, the increase in one’s level of belief in the argument is increased only very modestly when a few cases are examined.” (116)
- case studies as “nonquantitative time-series research designs” (117)
- problem of regression to the mean (123) – problem of building hypotheses/research based on extreme cases, which later turn out to be statistically insignificant
- two sources of regression to the mean (124) – every measurement has error; what is the second?
- “Relationships that seem to exist between causes and effects in a small sample selected on the dependent variable may disappear or be reverse when cases that span the full range of the dependent variable are examined.” (129)
- although useful for specific tasks, studies of cases selected on the dependent variable cannot alone test hypotheses (129)
- “In short, selecting cases without giving careful thought to the logical implications of the selection entails a serious risk of reaching false conclusions.” (129)