The federal GEAR UP program in Rhode Island led to large advantages for students who participated in the program in terms of persistence through the middle and high school years, high school graduation and college enrollment.
The Gaining Early Awareness and Readiness for Undergraduate Program, more commonly known as GEAR UP, is a product of federal legislation designed to increase high school completion and college attendance of low-income youth. It is a federally funded discretionary grant program that is planned, organized and operated at the state and local level. This means that while the objectives of GEAR UP are the same across the country, the program structure, program services and connections to local school districts vary widely. This variation results in considerable differences in program models and program outcomes across state and local program sites.
The College Crusade of Rhode Island provides GEAR UP program services to about 4,000 low-income, middle school and high school students each year. The College Crusade offers programs of academic enrichment as well as personal and social development, complemented by substantial college scholarship support to participants.
Since millions of dollars are spent annually around the nation supporting GEAR UP programs, its seems fair to ask if a state or local GEAR UP program can improve high school completion and college enrollment of low-income students, and indeed considerable efforts are underway to provide some answers to this question. State and local GEAR UP programs are all under pressure to provide some solid—evidence-based—answers.
This paper presents the findings of a first of its kind study of the impacts of a state Gear-Up program on student participant outcomes. The study uses statistically rigorous methodology (Propensity Score Matching) to measure program impacts on its core outcomes of student persistence in middle and high school, high school graduation, and college enrollment.
Evaluation mandate
In the current round of funding, the U.S. Department of Education (DOE) placed a strong emphasis on engaging a third-party evaluator that had the capability of measuring the net impact of program participation. “Net impact evaluations” are much more rigorous than the normal requirements of tracking program enrollment, persistence, completion and outcomes. Net impact studies are designed to measure the impact of a program on participant outcomes by creating a counterfactual to measure outcomes as if the program intervention had not occurred. As such, these studies are very data-intensive and use sophisticated statistical methods to estimate the contribution of program participation to student outcomes—and therefore rarely undertaken.
Random-assignment evaluation is often considered the best way to measure the impact of program participation on student outcomes; since (at least conceptually) it eliminates bias associated with the process of student selection into the program.[1] But random assignment doesn’t fit into this GEAR UP model very well, since the College Crusade operates a ‘universal access’ program and does not turn down applications from any eligible low income students who want to enroll—thus eliminating the random-assignment alternative.
When random assignment is not an option, a range of quasi-experimental evaluation methods are available that can be used to control for factors other than program participation (such as race, gender, basic skills, and other co-variates) that are known to influence the programs basic goals. The three most important outcomes of GEAR UP are persistence through the middle and high school years, high school graduation and college enrollment.
Serving as the third-party evaluators of the College Crusade of Rhode Island’s GEAR UP program, we opted to use propensity score matching (PSM) to measure the impact of program participation. PSM is a statistical method that creates a comparison group of non-participating students with traits that are matched to traits of students who participate in the program in order to minimize the effects of selection bias. We used the PSM method to match key demographic, SES, academic performance, behavioral traits and school climate measures to identify a comparison group of classmates of GEAR UP participants with traits nearly identical to the participant group, allowing us to more accurately isolate the impact of program participation on measures of student outcomes.
Measuring GEAR UP outcomes in Rhode Island
Even the most rigorous evaluation methods are limited by the availability of high-quality student level data. The data needed to conduct a PSM study is quite extensive: It must measure a wide number of variables on an individual student basis, including student characteristics and key measures of outcomes and track year-by-year progress of students over an extended period of time; in this instance, following students from the sixth grade through high school graduation and into college.
Student record data are, by their very nature, personal and confidential. Consequently, these data are subject to strict security and confidentiality requirements. The Rhode Island Department of Education (RIDE) and other agencies in the state have been working with The Providence Plan, a local nonprofit organization, to develop a centralized data hub (RI Data HUB) containing a wide array of student-level data. The RI DataHUB is a secure and confidential source of unit-record information that can provide researchers with de-identified student-level record data while still maintaining strict student confidentiality and a strong secure environment for these data.
The College Crusade of Rhode Island has developed a data-sharing agreement with RIDE and The Providence Plan, which has provided us access to de-identified student-level data for Rhode Island GEAR UP participants as well as their classmates for the quasi-experimental evaluation. Our objective in using these data was to measure the net contribution or impact of GEAR UP participation on middle and high school progress, high school completion and college enrollment outcomes of participants.
Our analysis of the first cohort of Rhode Island GEAR UP participants used student-record data of participants and their classmates who had entered the sixth grade in school year 2007-08. Using these data, we selected a matched comparison group from sixth grade classmates of GEAR UP participants who were not enrolled in the program. We tracked these students over seven years using RIDE student records from the RI DataHUB measuring their progress through middle school and high school, then using National Student Clearinghouse (NSC) data to measure their enrollment in college after high school.
Impact of RI GEAR UP program participation
We found that the GEAR UP program in Rhode Island led to large advantages for students who participated in the program. On all three of our basic outcomes measures, GEAR UP participants outperformed their matched comparison group classmates.
Our evaluation found that while participants and matched comparison group students were equivalent at the sixth grade starting line, in each successive year, Rhode Island GEAR UP participants raced ahead of their classmates. From school year 2007-08 onward, the gap between the on-time grade promotion rate of Rhode Island GEAR UP participants and the comparison group grew steadily in favor of GEAR UP participants. By 2014, GEAR UP participants had an 8.8 percentage point advantage in the fraction of students who entered the 12th grade on time; a relative increase of more than 12% in the chance of progressing on time to the last year of high school.[2]
The steady on-time grade-promotion advantage of Rhode Island GEAR UP participants accumulated over the years into a sizable advantage in high school graduation. Seven years after sixth grade enrollment in the state’s GEAR UP program, three-quarters of participants graduated from high school on time (spring 2014), compared with just two-thirds of their classmates in the matched comparison group. Participation in the Rhode Island GEAR UP program increased the chance of on-time graduation from high school by about 12%.
The most powerful net impact of the RI GEAR UP program is on students’ college-going rates.
In the fall of 2014, nearly 70% of Rhode Island GEAR UP high school graduates had enrolled in college compared with the 56% share of college-enrolled high school graduates in the matched comparison group. GEAR UP participants who graduated from high school were 25% more likely to enroll in college than their counterparts in the matched comparison group.
Cumulative impact
Our evaluation also measured the cumulative net impact of RI GEAR UP participation by constructing a second measure of the cohort college-going rate. In this instance, we measured the fraction of all sixth grade (in 2007-08) participants and matched comparison group members who had enrolled in college by fall 2014. (The measure of college enrollment discussed above, is restricted to just those who graduated from high school).
Impact of the Rhode Island Gear Up Program on Selected Outcomes for the 2007-08 Entering Cohort of Sixth Grade Participants
The advantage of Rhode Island GEAR UP participants over the comparison group in on-time grade promotion, on-time high school graduation and college enrollment after high school graduation resulted in a very large cumulative impact on college-going rates among the entire cohort. Students who enrolled in the GEAR UP program in the sixth grade (in 2007-08) had a college enrollment rate of 56%. Among their classmates in the matched comparison group, the proportion enrolled in college was just 41%.
The cumulative impact of participation in the Rhode Island GEAR UP program was to sharply raise the probability of enrolling in college among low-income urban youth. Our findings reveal a remarkable 37% increase in the probability of college enrollment among GEAR UP participants.
It is important to note that we also measured the outcomes of Rhode Island GEAR UP participants against all their sixth grade classmates—the matched comparison group as well as those not selected in the matched comparison group. These unadjusted comparisons of outcomes showed gains that were much greater than the gains against the matched comparison group using the PSM evaluation method that are presented above. This illustrates the benefit of the PSM method which is designed to establish equivalence of the comparison group with the participant group at the starting line (sixth grade for this evaluation) so that differences at the finish line (in outcomes) are attributable to the treatment; in this instance participation in the Rhode Island GEAR UP program.
The fundamental goal of the GEAR UP program is to increase the likelihood that low-income students enroll in college through interventions beginning in middle school. The Rhode Island GEAR UP Program has created very strong gains for this cohort of middle school students. These youngsters progressed through middle and high school with a reduced chance of faltering on the way. They were substantially more likely to earn their high school diploma on time and, upon graduating high school had a large absolute and relative college enrollment advantage. The cumulative impact of GEAR UP on these youngsters who entered the sixth grade back in 2007 was to increase their chance of enrolling in college by more than one-third (37%). These are very encouraging findings for the College Crusade of Rhode Island, but more importantly, they suggest a path forward for New England educators in improving college opportunity for the region’s economically disadvantaged students.
Neeta P. Fogg is a research professor in the Center for Labor Markets and Policy at Drexel University. Paul E. Harrington is director of the center.
[1] Real world applications of random assignment suggest that actually assigning students to treatment and non-treatment groups is pretty difficult as potential enrollees and their parents often try quite hard to work around the random assignment process or seek similar program services at other locations.
[2] All findings presented in the paper are statistically significant at 95% or 99% levels.
[ssba]