Mayor de Blasio's announcement on specialized high schools (photo: Benjamin Kanter/Mayor's Office)

The recently-released Metis validation study of the Specialized High School Admissions Test has portrayed the SHSAT in a more favorable light than the facts warrant.

Although Metis found that SHSAT scores were correlated with high school grades, it did not investigate whether other factors might be stronger predictors. In my validation study, the most important finding was that seventh grade GPA predicts more than twice as much of the variance in high school grades (44%) when compared to the SHSAT, which predicted 20% in my study and 21% in the Metis study. This finding is critical to the current debate over the admission system at New York City’s Specialized High Schools.

Mayor Bill de Blasio, along with many advocates and civil rights groups, have proposed doing away with the test, with a three-year phase-out period where the test is used along with grades for admissions purposes. Their opponents have argued that allowing grades as a factor would dilute the quality of the students admitted and thus the quality of these schools, an argument that is rebutted by my analysis.

This study uses data supplied by the New York City Department of Education, including the 27,818 SHSAT scores from 2013, and grades for New York City public school students in that cohort.

The chart below compares mean SHSAT scores for students at Stuyvesant High School with their Freshman Grade Point average (FGPA) in different FGPA categories. Students with FGPAs below 75 actually had higher SHSAT scores than students with FGPAs from 75 to 85, while students with FGPAs of 85 to 90 had barely higher scores. Only the students with grades above 95 had much higher scores. This suggests that the exam does not predict very well for students with scores below the very top. 

The next chart, which compares the same categories of FGPA by seventh grade GPA (GPA7) of Stuyvesant students, instead of their SHSAT scores, shows a much more linear relationship. Higher FGPAs were associated with higher GPA7 in a regular progression.

As is well known, girls are admitted to the specialized high schools at much lower rates than boys. Below is the chart with the rates by gender for those students who took the test last year, when girls represented 51% of applicants and only 44% of those admitted.

A good study of predictive validity should address the gender bias of the exam. According to American Educational Research Association standards, the test user is “responsible for evaluating the possibility of differential prediction for relevant subgroups for which there is prior evidence or theory suggesting differential prediction.” Prior evidence with respect to the validity of the SAT suggests that possibility. Yet, the Metis study failed to do this, despite the fact that girls are under-represented in the Specialized High Schools.

In my study, girls had lower SHSAT scores than boys with the same GPA. For example, at Stuyvesant High School, girls had GPAs equal to GPAs of boys whose SHSAT scores were 6 to 7 points higher. The following chart shows that girls (red bars) achieved each FGPA level with lower SHSAT scores than boys (blue bars). Among students in 95+ FGPA group, girls (549) had mean SHSAT scores 16 points lower than boys (565). This chart helps explain why fewer girls are admitted to the Specialized High Schools than boys.

The gender bias of the SHSAT parallels findings with respect to the predictive validity of the SAT. However, there is a crucial difference in how the scores are used. The SAT is only one of multiple criteria considered in college admissions, including high school grades, which enables admissions officers to take that statistical bias into account. In New York City, where the SHSAT is the sole admissions criterion for eight schools, girls with lower SHSAT scores than boys are denied admission despite the fact that they are expected to do as well as boys admitted with higher scores. It is clear that as the sole admissions criterion to the Specialized High Schools, the SHSAT is unfair to girls.

A variety of analyses were done to rule out the possibility that the under-prediction in girls’ grades resulted from differences in course selection. Because the under-representation of females in STEM majors and occupations has been a source of societal concern, it was of special importance to examine the relationship between SHSAT scores and girls’ grades in STEM areas, and to ascertain whether girls in New York City’s Specialized High Schools enroll as frequently and do as well as boys in challenging STEM courses.

Gender comparisons were done across different academic domains for all students who took the SHSAT in 2013 and subsequently attended New York City high schools. Similar comparisons were done for those who actually attended Specialized High Schools. Finally, analyses were done at Stuyvesant High School, which, because it is the most selective of the schools, might possibly have the most difficult courses. Overall, girls earned significantly higher grades than boys in each of these categories. At the Specialized High Schools, girls had higher mean STEM grades (89.6) than boys (86.7) and enrolled in STEM courses in proportion to their overall numbers in these elite schools.

Grades in specific courses offer the best test of the hypothesis that girls are less capable of succeeding in STEM subjects, and that the under-prediction of their grades by standardized tests such as the SHSAT is due to enrollment in easier courses. At Stuyvesant High School, girls, on average, earned better grades than boys in each of the ten specific STEM courses analyzed, including Geometry, Integrated Algebra, Biology, and Physics. In four of the ten courses, girls were represented in greater numbers than in the overall Stuyvesant ninth grade cohort. Furthermore, their grades were significantly under-predicted by the SHSAT.

In 2005, Lawrence Summers, then president of Harvard, suggested that the reason there were fewer women in STEM fields was their under-representation at the highest level of the distribution of ability in STEM subjects. Because one of the goals of the Specialized High School admissions process is to identify truly exceptional students, it is important to determine if the higher mean grades earned by girls in STEM classes were achieved without those extreme high-achievers. It would be possible for girls to have higher mean grades, but fewer exceptional grades. However, this is not the case. Girls represented only 42% of the cohort and only 41% of the top 3% of SHSAT scores, but in STEM courses they earned 50% of the grades of 95 or better, and only 30% of grades 80 or lower.

Why should females earn better grades than predicted by their performance on the SHSAT and SAT? Research has repeatedly shown that women do poorly relative to men on multiple choice questions. In addition to the SAT, this phenomenon has been found on a wide range of exams, including AP exams, placement exams, and the California State Bar Exam. One researcher attributed this to women on average being more risk-averse, and therefore not guessing as often as males, although guessing on multiple choice tests is often a useful strategy.

Because women often outperform men on constructed answer questions, it seems possible that multiple choice and constructed answer questions tap different competencies on which there are real gender differences. Constructed answer questions, particularly essays, may require multiple abilities, and may reflect a more complex understanding of material than do questions that require only the selection of a correct choice. If multiple choice and open constructed answer questions measure important, but different abilities, in the interest of fairness, exams should include both types of questions. Given that the purpose of the SHSAT is to admit a class with the greatest chance of success at school, it is apparent that the same metric cannot be used for boys and girls, and that the exam should be reformulated to include questions in different formats.

Our analysis also found that New York state exams were somewhat less gender-biased than the SHSAT, and somewhat more predictive of success in high school, perhaps because they contain more open-constructed answer questions, though again, 7th grade GPA is the most valid predictive factor for success at New York City high schools in general, including the Specialized High Schools.

A hypothetical admissions index was constructed weighting SHSAT and GPA7 by coefficients from the regression of FGPA. The index was then used to simulate admissions to the Specialized High Schools, resulting in the proportion of girls “admitted” rising from 45% to 62%, a difference of 716 girls. At Stuyvesant, using the index, the representation of girls increased from 44% to 65%, a difference of 177 girls. If GPA7 were the sole admissions criterion, the proportion of girls would have been 68%, a difference of 202 girls.

Use of this index would also have resulted in substantially different racial/ethnic proportions. At Stuyvesant, for example, an additional 10 African-American and 25 Hispanic students would have been admitted. Asian students, though reduced in numbers from 78% would still comprise 66% of those admitted. Similar shifts would occur in the entire cohort of students admitted to Specialized High Schools, with increases of 40 African-American, 209 Hispanic, and 205 white students. With 49% of the hypothetically admitted class, Asian students would still constitute the largest segment and would be admitted in numbers far exceeding their proportion of applicants (32%).

Because the hypothetical criterion predicts far more variance in FGPA than the actual criterion, with a smaller standard error of estimate, its use should not dilute the quality of the entering class. In fact, it might even result in a stronger cohort, while simultaneously increasing diversity and gender equity.

With changes to the SHSAT and consideration of additional criteria, it may be possible to select a group of students who will be more representative of the community the school system serves, and the pool of students who apply, without sacrificing the excellence for which New York City’s Specialized High Schools are so justifiably famous.

***
Jonathan Taylor, Ph.D. is an educational research analyst.