Legal scholars, lawyers, and judges frequently make positive claims about the state of legal doctrine. Yet despite the profligate citation norms of legal writing, these claims are often supported in a somewhat imprecise way—such that the exact evidence is unclear or difficult for others to probe or falsify. In response to similar issues, other disciplines have developed methodological standards for conducting “systematic reviews” that summarize the state of knowledge on a given subject. In this Essay we argue that methods for performing systematic reviews that are specifically tailored to legal analysis should be developed. We propose a simple four-step process that could be used whenever someone is trying to make objective claims about the state of legal doctrine, and we illustrate the value of this method by applying it to doctrinal claims that have been made in recent legal scholarship.
Legal scholars, advocates, and judges commonly make positive claims about the state of legal doctrine. For example, a legal scholar might claim that there is a trend in recent federal court decisions to allow a particular pretrial procedure, or a judge might claim that most courts endorse a given legal proposition. These claims, however, are frequently made without a systematic demonstration of supporting evidence. When this occurs, it not only makes it difficult for the reader to evaluate the validity of the claim, but also may impede future legal analysis and allow for either conscious or unconscious bias.
In response to analogous concerns about their literature, other disciplines have developed rigorous methods for conducting “systematic reviews.”1 A systematic review is a method of summarizing the results of prior literature on a research question.2 Typically employed in the medical and psychological sciences, but increasingly being used in the social sciences,3 systematic reviews have four key lessons for doctrinal legal work. First, the researcher should state clearly the question she is attempting to answer, as this affects the manner in which she conducts her review. Second, the researcher must justify and be transparent about how she defines and obtains the sample for her review. Third, the researcher ought to explain any weighting that is applied to the cases in the sample. Fourth, the researcher needs to justify and be transparent about the manner in which she analyzes the sample cases she reviews. Together these steps can prevent bias from case selection and improve the legitimacy of conclusions drawn from the review.
In this Essay we argue that legal scholars, lawyers, and judges should conduct a four-step systematic review when they are making positive claims about the state of legal doctrine. In Part I we survey the lack of systematic support for positive claims in the legal literature and explain the benefits of greater methodological rigor. In Part II we discuss systematic reviews in other fields and their applicability to legal analysis. In Part III we propose a four-step process for conducting systematic review of legal doctrine and then demonstrate its use by applying it to an example from one of our own articles. Finally, we conclude by discussing the areas that we believe would benefit most from the application of our proposed method.
- 1. See generally, for example, Cynthia D. Mulrow, Rationale for Systematic Reviews, 309 Brit Med J 597 (1994); Ruairidh Milne and Larry Chambers, Assessing the Scientific Quality of Review Articles, 47 J Epidemiology & Community Health 169 (1993); Gregg B. Jackson, Methods for Integrative Reviews, 50 Rev Educ Rsrch 438 (1980); Kenneth A. Feldman, Using the Work of Others: Some Observations on Reviewing and Integrating, 44 Sociology Educ 86 (1971).
- 2. For a more detailed definition of “systematic review,” see Amit X. Garg, Dan Hackam, and Marcello Tonelli, Systematic Review and Meta-Analysis: When One Study Is Just Not Enough, 3 Clinical J Am Society Nephrology 253, 253 (2008) (“A systematic review uses a process to identify comprehensively all studies for a specific focused question (drawn from research and other sources), appraise the methods of the studies, summarize the results, present key findings, identify reasons for different results across studies, and cite limitations of current knowledge.”).
- 3. See, for example, id at 253–54 (discussing systematic reviews in nephrology and other medical fields); Jeffrey C. Valentine, Therese D. Pigott, and Timothy Lau, Systematic Reviewing and Meta-Analysis, in James D. Wright, ed, 23 International Encyclopedia of the Social & Behavioral Sciences 906, 906–09 (Elsevier 2d ed 2015) (discussing the steps of systematic reviews in the context of social science questions).