Studying Settings
The Foundation supports research to understand and improve settings for youth ages 8 to 25 in the United States. We also appreciate that this is a relatively new area for research, so we have been collecting and encouraging the creation of information that can be used by applicants, grantees, and others interested in learning more about youth settings. (Download fees charged by publishers are noted where applicable.)
Featured Resources
Setting Theory and Measurement:
- Measuring Social Settings by Edward Seidman with the Senior Program Team (6 pages, 276kb PDF)
- Social Setting Theory and Measurement by Senior Program Team members Edward Seidman, Vivian Tseng, and Thomas Weisner (4 pages, 128kb PDF)
- Doing Social Setting Research by the Senior Program Team (8 pages, 632kb PDF)
- Measuring Youth Program Quality: A Guide to Assessment Tools, 2nd Edition by Nicole Yohalem, Alicia Wilson-Ahlstrom, Sean Fischer, and Marybeth Shinn. 2009, January
The Forum for Youth Investment updated their report, first issued in March 2007, which compares the purpose, history, structure, methodology, content and technical properties of different program observation tools. (92 pages, 1.31mb PDF)
- Draft of An Eight-Step Paradigm for Studying the Reliability of Group-Level Measures by Stephen W. Raudenush, Andres Martinez, Howard Bloom, Pei Zhu, and Fen Lin (23 pages, 101kb PDF)
- Conceptualization, Measurement, and Improvement of Classroom Processes by Robert C. Pianta and Bridget K. Hamre, published on pages 109-119 of in Vol. 38, No. 2/March 2009 of Educational Researcher.
- Studying Reading Instruction with Teacher Logs: Lessons from the Study of Instructional Improvement by Brian Rowan and Richard Correnti, published on pages 120-131 in Vol. 38, No. 2/March 2009 of Educational Researcher.
- A Systems Framework for Understanding Social Settings by Senior Program Team members Vivian Tseng and Edward Seidman, published on pages 217-228 in Volume 39, Numbers 3-4/June, 2007 of the American Journal of Community Psychology. (27 pages, 239kb PDF; $ download fee)
Design and Analysis of Cluster-Randomized Trials:
- New Empirical Evidence for the Design of Group Randomized Trials in Education 2009, December
This MDRC working paper, written by Robin Jacob, Pei Zhu, and Howard S. Bloom, offers guidance for designing group-randomized studies to measure the impacts of educational interventions. Specifically, the authors provide new empirical information about the values of parameters that influence the precision of impact estimates, include a discussion of the error in estimates of key design parameters, and discuss the implications of those errors for design decisions. (57 pages, 396kb PDF)
- Modern Regression Discontinuity Analysis 2009, December
This MDRC working paper by Howard Bloom provides a detailed discussion of the theory and practice of modern regression discontinuity (RD) analysis for estimating the effects of interventions or treatments focused on individuals or clusters of individuals. (62 pages, 642kb PDF)
- Empirical Issues in the Design of Group-Randomized Studies to Measure the Effects of Interventions for Children 2008, July
This MDRC working paper on research methodogy, by Howard Bloom and colleagues, “provides practical guidance for researchers who are designing studies that randomize groups to measure the impacts of interventions on children.” (85 pages, 734kb PDF)
- The March 2007 issue of Education Evaluation and Policy Analysis, Volume 29, No. 1, features three articles that are recommended reading on this topic.
- Pages 5-29: "Strategies for Improving Precision in Group-Randomized Experiments" by Stephen W. Raudenbush, Andres Martinez, and Jessaca Spybrook (24 pages, $ download fee)
- Pages 30-59: "Using Covariates to Improve Precision for Studies That Randomize Schools to Evaluate Educational Interventions" by Howard S. Bloom, Lashawn Richburg-Hayes, and Alison Rebeck Black (29 pages, $ download fee)
- Pages 60-87: "Intraclass Correlation Values for Planning Group-Randomized Trials in Education" by Larry V. Hedges and E. C. Hedberg (27 pages, $ download fee)
- Using Covariates to Improve Precision 2005, November
This working paper, by Howard Bloom and colleagues, examines how controlling statistically for baseline covariates (especially pretests) improves the precision of studies that randomize schools to measure the impacts of educational interventions on student achievement. (121 pages, 573.99kb PDF)
- Strategies for Improving Precision in Group-Randomized Experiments 2005, November
This paper, by Stephen Raudenbush and colleagues, aims to clarify conditions under which the use of blocking and covariance adjustment can reduce the number of groups required to achieve adequate power in group-randomized studies. (42 pages, 326kb PDF)
- Randomizing Groups to Evaluate Place-Based Trials 2005, June
Howard Bloom presents a discussion of the rationale for, operating procedures for, and statistical implications of evaluation studies that randomize intact groups to measure the impact of interventions. The discussion focuses on a wide range of issues with respect to the design and analysis of such studies. "Randomizing Groups to Evaluate Place-Based Trials" is a chapter from the book, Learning More From Social Experiments: Evolving Analytic Approaches, edited by Howard Bloom. (90 pages, 746.55kb PDF)
- Some Food for Thought About Effect Size 2004, March
This document by Howard Bloom presents a series of vignettes that illustrate how the widely-used standardized measure of "effect size" can have markedly different implications in different settings. These vignettes are intended to stimulate thinking about how small the minimum detectable effect must be for experiments in different settings and thus what their sample requirements are. (6 pages, 24kb PDF)
- Statistical Analysis and Optimal Design for Cluster Randomized Trials 1997; American Psychological Association
This document by Stephen Raudenbush gives a more detailed account of design concerns for these studies with attention to the potential value of covariates in increasing precision and power and therefore reducing the number of clusters required. (15 pages, 961.79kb PDF)
More Resources on Setting Theory and Measurement - Chan, D. 1998. Functional relations among constructs in the same content domain at different levels of analysis: A typology of composition models. Journal of Applied Psychology 83 (2): 234-246.
- Henry, D. B., J. Cartland, H. Ruchross, and K. Monahan. 2004. A return potential measure of setting norms for aggression. American Journal of Community Psychology 33: 131-149.
- Henry, D. B., N. G. Guerra, L. R. Huesmann, P. H. Tolan, R. VanAcker, and L. D. Eron. 2000. Normative influences on aggression in urban elementary school classrooms. American Journal of Community Psychology 28: 59-81.
- Raudenbush, S. W., B. Rowan, and S. J. Kang. 1991. A multilevel, multivariate model for studying school climate with estimation via the EM algorithm and application to U.S. high-school data. Journal of Educational Statistics 16 (4): 295-330.
- Raudenbush, S. W., and R. J. Sampson. 1999. Ecometrics: Toward a science of assessing ecological settings, with application to the systematic social observations of neighborhoods. Sociological Methodology 29: 1-41.
- Rowan, B., S. W. Raudenbush, and S. J. Kang. 1991. Organizational design in high schools: A multilevel analysis. American Journal of Education 99 (2): 238-266.
- Shinn, M. 1990. "Mixing and matching: Levels of conceptualization, measurement, and statistical analysis in community research." In Researching Community Psychology, P. Tolan, C. Keys, F. Chertok, and L. Jason, eds. Washington, DC: American Psychological Association, 111-126.
- Shinn, M., and B. D. Rapkin. 2000. Cross-level research without cross-ups in community psychology. In Handbook of Community Psychology, J. Rappaport and E. Seidman, eds. New York, NY: Kluwer Academic/Plenum Publishers, 669-695.