Abstract

Demography’s population perspective, and the sampling methods that help produce it, are powerful but underutilized research tools. The first half of this article makes the case for more vigorous promotion of a population perspective throughout the sciences. It briefly reviews the basic elements of population sampling and then provides examples from both developed and developing countries of how population sampling can enrich random-assignment policy experiments, multisite studies, and qualitative research. At the same time, an ill-considered application of a population perspective to the problem of causal inference can hinder social and behavioral science. The second half of the article describes the “slippery slope” by which some demographic studies slide from providing a highly useful description about the population to using regressions to estimate causal models for that population. It then suggests that causal modeling is sometimes well served by a highly selective look at small subsets of a population with interesting variability in independent variables of interest. A robust understanding of causal effects, however, rests on convergence between selective and population-wide perspectives.

The text of this article is only available as a PDF.

References

Evaluating Head Start: A Recommended Framework for Studying the Impact of the Head Start Program
. (
1999
).
Washington, DC
:
U.S. Department of Health and Human Services
.
Agodini, R., & Dynarski, M. (
2004
).
Are Experiments the Only Option? A Look at Dropout Prevention Programs
.
Review of Economics and Statistics
,
86
,
180
94
. 10.1162/003465304323023741
Altonji, J., Elder, T., & Taber, C. (
2005
).
Selection on Observed and Unobserved Variables: Assessing the Effectiveness of Catholic Schools
.
Journal of Political Economy
,
113
,
151
84
. 10.1086/426036
Angrist, J., Lavy, V., & Schlosser, A. (
2005
).
New Evidence on the Causal Link Between the Quantity and Quality of Children
.
Cambridge, MA
:
National Bureau of Economic Research
.
Bendheim-Thoman Center for Research on Child Wellbeing. 2005. “Introduction to the Fragile Families Core Public Use Data: Baseline, One-Year, and Three-Year Files.” Available online at http://www.fragilefamilies.princeton.edu/Public%20Use%20Data/ff_public_3waves_100605 .pdf
Bernard, H.R. (
1995
).
Research Methods in Anthropology: Qualitative and Quantitative Approaches
. 2nd ed.
Walnut Creek, CA
:
Alta Mira Press
.
Bloom, D., & Michalopoulos, C. (
2001
).
How Welfare and Work Policies Affect Employment and Income: A Synthesis of Research
.
New York
:
MDRC
.
Boerma, J., Ghys, P., & Walker, N. (
2003
).
Estimates of HIV-1 Prevalence From National Population-Based Surveys as a New Gold Standard
.
Lancet
,
362
,
1929
31
. 10.1016/S0140-6736(03)14967-7
Bos, J.M., Huston, A.C., Granger, R., Duncan, G., Brock, T., & McLoyd, V. (
1999
).
New Hope for People With Low Incomes: Two-Year Results of a Program to Reduce Poverty and Reform Welfare
.
New York
:
MDRC
.
Box, G., & Draper, N. (
1987
).
Empirical Model-Building and Response Surfaces
.
New York
:
Wiley
.
Brooks-Gunn, J., Gross, R., Kraemer, H., Spiker, D., & Shapiro, S. (
1992
).
Enhancing the Cognitive Outcomes of Low Birth Weight, Premature Infants: For Whom Is Intervention Most Effective?
.
Pediatrics
,
89
,
1209
215
.
Card, D.E. (
1999
).
The Causal Effect of Education on Earnings
. In Ashenfelter, O., & Card, D. (Eds.),
Handbook of Labor Economics
(pp.
1801
63
).
Amsterdam
:
North-Holland
.
Card, D.E., & Krueger, A.B. (
1992
).
School Quality and Black-White Relative Earnings: A Direct Assessment
.
Quarterly Journal of Economics
,
107
,
151
200
. 10.2307/2118326
Cochran, W.G. (
1977
).
Sampling Techniques
.
New York
:
Wiley
.
Currie, J.T., & Thomas, D. (
1995
).
Does Head Start Make a Difference?
.
American Economic Review
,
85
,
341
64
.
Deming, W.E. (
1953
).
On the Distinction Between Enumerative and Analytic Surveys
.
Journal of the American Statistical Association
,
48
,
244
55
. 10.2307/2281285
Deming, W.E., & Stephan, F.F. (
1941
).
On the Interpretation of Censuses as Samples
.
Journal of the American Statistical Association
,
36
,
45
49
. 10.2307/2278811
DiPrete, T., & Gangl, A.M. (
2004
).
Assessing Bias in the Estimation of Causal Effects: Rosenbaum Bounds on Matching Estimators and Instrumental Variables Estimation With Imperfect Instruments
.
Sociological Methodology
,
34
,
271
310
. 10.1111/j.0081-1750.2004.00154.x
Duncan, G.J., Magnuson, K.A., & Ludwig, J. (
2004
).
The Endogeneity Problem in Developmental Studies
.
Research in Human Development
,
1
,
59
80
. 10.1207/s15427617rhd0101&2_5
Ellenberg, J.H., & Nelson, K.B. (
1980
).
Sample Selection and the Natural History of Disease: Studies of Febrile Seizures
.
Journal of the American Medical Association
,
243
,
1337
40
. 10.1001/jama.243.13.1337
Garces, E., Thomas, D., & Currie, J. (
2002
).
Longer-Term Effects of Head Start.
.
American Economic Review
,
92
,
999
1012
. 10.1257/00028280260344560
Gibson, C., & Duncan, G.J. (
2005
).
Qualitative/Quantitative Synergies in a Random-Assignment Program Evaluation
. In Weisner, T.S. (Ed.),
Discovering Successful Pathways in Children’s Development: New Methods in the Study of Childhood and Family Life
(pp.
283
303
).
Chicago
:
University of Chicago Press
.
Glaser, B., & Strauss, A. (
1967
).
The Discovery of Grounded Theory: Strategies for Qualitative Data Research
.
London
:
Weidenfeld and Nicholson
.
Gueron, J.M., & Pauly, E. (
1991
).
From Welfare to Work
.
New York
:
Russell Sage Foundation
.
Hamilton, B.E., Martin, J.A., & Ventura, S.J. (
2006
).
Births: Preliminary Data for 2005.
.
Hyattsville, MD
:
National Center for Health Statistics
.
Hansen, M.H. (
1987
).
Some History and Reminiscences on Survey Sampling
.
Statistical Science
,
2
,
180
90
. 10.1214/ss/1177013352
Heckman, J.J., LaLonde, R.J., & Smith, J. (
1999
).
The Economics and Econometrics of Active Labor Market Programs
. In Ashenfelter, O., & Card, D. (Eds.),
Handbook of Labor Economics
(pp.
1865
2073
).
Amsterdam
:
North Holland
.
Hedges, L. (
2008
).
Improving Generalizability of Social Experiments
.
Evanston, IL
:
Center for Improving Methods for Quantitative Policy Research, Northwestern University
.
Hill, H.D. (
2007
).
Infi delity and Sexual Jealousy
. In England, P., & Edin, K. (Eds.),
Unmarried Couples With Children
(pp.
104
32
).
New York
:
Russell Sage Foundation
.
Holland, P.W. (
1986
).
Statistics and Causal Inference
.
Journal of the American Statistical Association
,
81
,
945
60
. 10.2307/2289064
Imbens, G., & Angrist, J. (
1994
).
Identifi cation and Estimation of Local Average Treatment Effects
.
Econometrica
,
62
,
467
75
. 10.2307/2951620
Johnson, J.C. (
1990
).
Selecting Ethnographic Informants: Qualitative Research Methods Series No. 22
.
Newbury Park, CA
:
Sage Publications
.
Kish, L. (
1987
).
Statistical Designs for Research
.
New York
:
John Wiley
.
Lee, M. (
2005
).
Micro-Econometrics for Policy, Program and Treatment Effects
.
Oxford
:
Oxford University Press
.
Little, R.J.A. (
1992
).
Regression With Missing X’s: A Review
.
Journal of the American Statistical Association
,
87
,
1227
37
. 10.2307/2290664
Lofland, J., & Lofl, L.H. (
1995
).
Analyzing Social Settings: A Guide to Qualitative Observation and Analysis
. 3rd ed.
Belmont, CA
:
Wadsworth Publishing Company
.
Ludwig, J., & Miller, D.L. (
2007
).
Does Head Start Improve Children’s Life Chances? Evidence From a Regression Discontinuity Design
.
Quarterly Journal of Economics
,
122
,
159
208
. 10.1162/qjec.122.1.159
Ludwig, J., & Phillips, D. (
2007
).
The Benefits and Costs of Head Start
.
Social Policy Report
,
21
(
3
),
3
13
.
Magnuson, K., Meyers, M., Ruhm, C., & Waldfogel, J. (
2004
).
Inequality in Preschool Education and School Readiness
.
American Educational Research Journal
,
41
,
115
57
. 10.3102/00028312041001115
Manski, C. (
1990
).
Nonparametric Bounds on Treatment Effects
.
American Economic Review
,
80
,
319
23
.
Meyer, B. (
1995
).
Natural and Quasi-Experiments in Economics
.
Journal of Business and Economic Statistics
,
13
,
151
61
. 10.2307/1392369
Michael, R.T., & O’Muircheartaigh, C.A. (
2008
).
Design Priorities and Disciplinary Perspectives: The Case of the US National Children’s Study
.
Journal of the Royal Statistical Society: Series A (Statistics in Society)
,
171
,
465
80
. 10.1111/j.1467-985X.2007.00526.x
Million Women Study Collaborators
(
2003
).
Breast Cancer and Hormone-Replacement Therapy in the Million Women Study
.
Lancet
,
362
,
419
27
. 10.1016/S0140-6736(03)14065-2
Moffitt, R. (
2005
).
Remarks on the Analysis of Causal Relationship in Demographic Research
.
Demography
,
42
,
91
108
. 10.1353/dem.2005.0006
Morgan, S.L., & Winship, C. (
2007
).
Counterfactuals and Causal Inference: Methods and Principles for Social Research
.
Cambridge
:
Cambridge University Press
.
Myers, D., Olsen, R., Seftor, N., Young, J., & Tuttle, C. (
2004
).
The Impacts of Regular Upward Bound: Results From the Third Follow-Up Data Collection
.
Washington, DC
:
Mathematica Policy Research
.
Neyman, J., Iwaszkiewicz, K., & Kolodziejczyk, St. (
1935
).
Statistical Problems in Agricultural Experimentation (With Discussion)
.
Journal of the Royal Statistical Society, Series B
,
2
,
107
80
.
(
1994
).
Child Care and Child Development: The NICHD Study of Early Child Care
. In Friedman, S.L., & Haywood, H.C. (Eds.),
Developmental Follow-up: Concepts, Domains, and Methods
(pp.
377
96
).
New York
:
Academic Press
.
Nisbett, R., & Cohen, D. (
1996
).
Culture of Honor: The Psychology of Violence in the South
.
Boulder, CO
:
Westview Press
.
Orr, L., Feins, J.D., Jacob, R., Beecroft, E., Sanbonmatsu, L., Katz, L.F., Liebman, J.B., & Kling, J.R. (
2003
).
Moving to Opportunity: Interim Impacts Evaluation
.
Washington, DC
:
U.S. Department of Housing and Urban Development, Office of Policy Development and Research
.
Patton, M.Q. (
2002
).
Qualitative Research and Evaluation Methods
. 3rd ed.
Thousand Oaks, CA
:
Sage Publications
.
Pearl, J. (
2000
).
Causality: Models, Reasoning and Inference
.
Cambridge, UK
:
Cambridge University Press
.
Pelto, P.J., & Pelto, G.H. (
1978
).
Anthropological Inquiry: The Structure of Inquiry
. 2nd ed.
New York
:
Cambridge University Press
.
Reichman, N.E., Teitler, J.O., Garfi nkel, I., & McLanahan, S.S. (
2001
).
Fragile Families: Sample and Design
.
Children and Youth Services Review
,
23
,
303
26
. 10.1016/S0190-7409(01)00141-4
Rosenbaum, P. (
2002
).
Observational Studies
. 2nd ed.
New York
:
Springer
.
Rosenbaum, P.R., & Rubin, D.B. (
1983
).
The Central Role of the Propensity Score in Observational Studies for Causal Effects
.
Biometrika
,
70
,
41
55
. 10.1093/biomet/70.1.41
Rubin, D.B. (
1974
).
Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies
.
Journal of Educational Psychology
,
66
,
688
701
. 10.1037/h0037350
Shadish, W.R., Cook, T., & Campbell, D.T. (
2002
).
Experimental and Quasi-Experimental Designs for Generalized Causal Inference
.
New York
:
Houghton Mifflin
.
Shafer, E. (
2007
).
Data From the TLC3
. In England, P., & Edin, K. (Eds.),
Unmarried Couples With Children
(pp.
277
91
).
New York
:
Russell Sage
.
Spirtes, P., Glymour, C.N., & Scheines, R. (
2000
).
Causation, Prediction, and Search
. 2nd ed.
Cambridge, MA
:
MIT Press
.
U.S. Census Bureau. 2007. Current Population Survey 2007: Annual Social and Economic Supplement. Available online at http://pubdb3.census.gov/macro/032007/pov/toc.htm.
Head Start Impact Study: First Year Findings
. (
2005
).
Washington, DC
:
Author
.
Weisner, T., L. Bernheimer, C. Gibson, E. Howard, K. Magnuson, J. Romich, and E. Lieber. 1999. “From the Living Rooms and Daily Routines of the Economically Poor: An Ethnographic Study of the New Hope Effects on Families and Children.” Paper presented at the Society for Research in Child Development biannual meeting, Albuquerque, New Mexico, April 1999.
Wilde, E. and R. Hollister. 2002. “How Close is Close Enough? Testing Nonexperimental Estimates of Impact Against Experimental Estimates of Impact With Education Test Scores as Outcomes.” Discussion Paper 1242-02. Institute for Research on Poverty, University of Wisconsin-Madison