Identifying the Conditions That Contribute to Performance Mobility
This is the second in a series of CenterLine posts based on the assessment and accountability issues addressed this summer by our 2022 summer interns and their Center mentors. Michael Fienberg, from the University of Southern California Rossier School of Education, worked with Chris Domaleski to study performance mobility in contemporary school accountability systems.
A thoughtful examination of equity often considers the concept of mobility. A focus on creating more equitable school accountability systems should consider the question, what is the likelihood that schools will transition from lower- to higher-performing classifications? Hereafter, I describe this as performance mobility. In the same way, for example, to study economic mobility, researchers may examine the extent to which someone’s income changes over time to better understand the factors that increase the odds of escaping poverty.
To create more equitable school accountability systems, it’s important to understand what conditions are likely to contribute to performance mobility. This is the primary question I set out to address in my summer internship at the Center for Assessment with guidance from my mentor Chris Domaleski.
In particular, I focused on performance mobility as it pertains to school accountability required by the Every Student Succeeds Act (ESSA). ESSA requires that states designate at least the lowest-performing 5% of their Title I schools as Comprehensive Support and Improvement (CSI) schools. These schools get extra support with the goal of improving enough to exit CSI. But how frequently do schools improve enough to exit low-performing status?
Mobility may be impacted by the statistical features of state-selected performance indicators, indicator weighting, and numerous other factors. For my project, I focused on the first two issues, using simulations to ascertain which indicator statistical factors relate to performance mobility. The end goal of this work was to contribute a set of empirically-backed principles to help states design accountability systems that provide schools opportunities to convert legitimate improvement into performance mobility.
Examining Performance Mobility
To better understand performance mobility, I constructed a simulation that replicated the key features of school accountability systems, using five generic indicators selected to mimic the accountability requirements in ESSA. Using synthetic parameters, established to reflect characteristics commonly observed in state systems, 1,000 simulated schools were randomly assigned a starting value on each of the five indicators, which were aggregated into a composite score based on each indicator’s assigned weighting. Schools were then ranked and designated as CSI (bottom 5%), at-risk (5-15%), or passing (top 85%). These designations aligned with how the states I studied to build the simulation tended to grade their schools. In Wisconsin for example, approximately 85% of schools were typically given a “meets standards” grade or better; about 10% were given a “nearly meets standards” grade; and about 5% were given the lowest, failing grade.
Next, I simulated yearly changes for indicators, generating updated composite scores, ranks, and designations. At the end of each simulation, the 1,000 schools had six years of generated data. I ran 30 different versions of the simulation to examine a range of factors that might be important, such as starting performance, relationships between and among indicators, characteristics of yearly change, and indicator weighting. Ultimately, I focused on two questions:
- What percentage of schools initially classified as CSI were still classified as CSI after six years?
- What percentage of schools were classified as CSI for 0, 1, 2, 3, 4, 5, and 6 of the six simulation years?
Principles to Promote Performance Mobility
After running the simulations, I looked for characteristics of the school accountability system and individual indicators that had the most impact on performance mobility. Based on these findings, I developed some initial recommendations or principles that may help promote performance mobility. Here, I’ll describe four key principles; more details will be available when the full study is published.
- Select indicators that are well-suited to detect progress
Accountability indicators have to be sensitive enough to detect progress in order for schools to show improvement, which means indicators should have sufficient variability to detect even modest changes. Express performance as a broad range of values instead of a few categories.
- Select diverse indicators
Systems that better detect performance mobility include distinct but important indicators of school quality that are not too duplicative. It is especially important that these indicators are not highly correlated with status, such as proficiency rates.
- Avoid an overemphasis on system weights
In the simulations I created, I found that the weights assigned to indicators had little impact on mobility. However, in combination with other effects, weighting did amplify performance mobility when it was present. For example, if an indicator is positively contributing to performance mobility, such as academic growth, then heavily weighting that category would further improve mobility relative to an equal-weight system.
- Address interactions among design features
Adjusting one indicator will inevitably affect other parts of the system, which will necessitate further adjustments, which will again affect the system. To better detect mobility, design decisions have to be considered as a whole, not just with specific indicators.
Final Thoughts
The goal of my research at the Center was to identify approaches to improve the degree to which state school accountability systems can detect and promote performance mobility. The simulations I produced helped identify some promising practices supported by evidence.
Ideally, application of these practices will help ensure schools are recognized for their progress and help states identify the right schools for support based on the most accurate and defensible data.
The post Mobility Matters When Designing More Equitable School Accountability Systems appeared first on Center for Assessment.
Baca Juga :