Ofqual grades algorithm: A recipe for unfairness

17th August 2020
upReach

Award-winning social mobility charity, upReach, comments on their recently released research revealing the unfairness at the heart of the Ofqual A-Level grading algorithm

Read the research report here

John Craven, Chief Executive of upReach, comments:

“Ofqual’s flawed methodology resulted in rampant grade inflation in subjects more popular at independent schools such as Classics. By their own definition, Ofqual and the government have thus failed in their attempt at maintaining a “Gold Standard” by capping grades. Popular subjects more commonly studied at Sixth Form colleges saw no grade inflation. This, together with the use of a flawed algorithm based on historical data, has unfairly destroyed the dreams of thousands of ordinary students.”

“The government need to urgently do the right thing, giving all students their teacher-awarded grades, rather than favouring private school students.”

In summary, upReach research released over the last three days shows that:

  1. The algorithm’s reliance on historical schools data, meant that outstanding students at schools where few if any people had ever got top grades were unfairly capped, and have missed out on top university places as a direct result. This will have been one factor behind the widening of the Attainment Gap between students eligible for free school meals and the rest for A* and A grades, reversing years of hard won progress by schools.
  2. This reliance on historical grades and on prior attainment data, meant the algorithm particularly disadvantaged students at schools that have been improving in recent years, had a particularly strong cohort in a subject this year, and/or that had weaker prior attainment data.
  3. The algorithm had a bias that favoured those in small subject cohorts. This favoured smaller schools, and those doing niche subjects. Overall, this favoured private schools, where A* and A grades rose 4.7%, compared to only 0.3% at Sixth Form colleges. The model created by upReach estimated that this “small subject” effect alone meant that Sixth Form colleges students were 20% more likely to have their teacher-assessed grade downgraded than private schools.
  4. Based on the “small subject cohort” issue alone, A-Levels studied at Sixth Form colleges were 20% more likely to be downgraded than those at independent schools. Other factors that require data not yet published by the DfE or Ofqual, are likely to make this an under-estimate.

What drove outperformance in Independent schools compared to Sixth Form colleges?

Based on the “small subject cohort” factor alone, our model estimates that A-Levels grades at Sixth Form colleges were 20% more likely to be downgraded than those at Independent schools. Other factors that require data not yet published by the DfE or Ofqual, are likely to make this an under-estimate. Our analysis suggest it is likely to be a combination of two factors:

  1. The smaller independent schools having “small subject cohorts” for even averagely popular subjects such as Economics, meaning that they were more likely to have the teacher awarded grades used in the calculation, and less likely to be downgraded.
  2. Rampant grade inflation in subjects more commonly studied by small cohorts at Independent schools and higher attaining state schools, such as Latin and Classics saw significant grade inflation, that was not found in other more popular subjects.

There was more easily accessible on the second factor, so this was our initial focus. In Classical subjects there were 7.7% more A* and 10.4% more A* or A grades than the prior year. This is most likely because in these subjects, students at almost all schools, were given grades that factored in teacher-assessed grades that Ofqual admitted were “over-optimistic.” In Latin, 97.4% of schools factored in teacher-assessed grades, in Classical Greek it was 98.9%, and in History of Art it was 81.2%. Analysis of our historical grades database showed that over 70% of schools offering Latin and History of Art A-Levels in 2019 were independent schools.

In contrast, subjects more commonly taught at Sixth Form colleges, such as Psychology, Sociology and Business Studies, saw little or no grade inflation (0.5% more students got A* or A grades in Psychology, and just 0.2% in Sociology). Our findings suggest this is almost certainly because most schools (and almost all Sixth Form colleges) had their teacher-assessed grades completely ignored by the algorithm. In Psychology only 28.8% of schools factored in teacher awarded grades rather than just rankings, with just 35.2% in Sociology.

Around 30 times more students studied Sociology A-Level at Sixth Form and FE colleges than at private schools. Students studying these subjects at larger schools and colleges didn’t benefit from the grade inflation evident in subjects more commonly taught at private schools or the most academic state schools.

On this issue, Ofqual stated in their Technical Report that “this is indicative of the modelling approach operating as intended.”

CovidCohort Grades Evaluation

To address the fact that 39% of A-Level grades were “downgraded”, upReach created the CovidCohort Grades Evaluation in order to help students, schools, universities and employers better understand the likelihood that their A-Level grades have been unfairly capped by the system in place for 2020. Students can visit the website to enter their details, in order to receive a free four page evaluation.

The evaluation includes an A-Level Grades Evaluation based on their school’s historic results data; a flag for Likelihood of Lowered Grade; a “Statistical Outperfomer” Flag for risk of Under-predicted grade; and a Contextualisation Adjustment Score through REAL (see below). The methodology was informed by the precise details of the standardisation process being used by Ofqual.

Notes to editors

upReach are an award-winning social mobility charity that support over 1,600 students to secure top graduate jobs, in partnership with leading universities and employers such as Bank of America, Civil Service Fast Stream, Deloitte, Goldman Sachs, McKinsey and Slaughter and May. A team of 34 employees provide personalised support helping to broaden horizons, raise aspirations and enable disadvantaged students to develop the skills, networks and experiences to succeed on merit. upReach won Charity of the Year in 2019 at the Charity Times Awards, and won the 2020 Embracing Digital Award in May’s Charity Governance Awards.

REAL (Relative Educational Attainment Level) is a contextualisation platform built by upReach in 2017 to help employers identify hidden potential. A research and evidence-backed algorithm takes into account 14 different indicators of disadvantage to calculate a net indicator of socio-economic disadvantage called a Contextualised Adjustment Score. This is used to calculate contextualised A-Level grades, which are used by employers as a better indicator of academic potential than A-Level grades alone. Further details, including student case studies, can be found here. Learn more at https://realrating.co.uk/