The qualifications fiasco

 To everyone’s relief, the Government eventually caved in over the awarding of grades to A/AS level and GCSE students who had not been able to take their exams. The algorithm that awarded aspiring young people grades they did not expect and did not deserve was discarded in favour of grades set by their schools and colleges. But only if the grades awarded were too low. Those to whom the algorithm awarded over-high grades got to keep them. As a result, the rampant “grade inflation” that the Education Secretary was so desperate to prevent is now even worse than it would be if only the centre-assessed grades were used. What a mess. 

But it is not the current mess that bothers me, bad though it is. Nor even the mess there will probably be next year, when universities are faced with cramming next year’s cohort into intakes already partially filled by deferrals from this year. No, it is what this algorithm tells us about the way officials at the Department of Education, at OfQual and at examination boards think. The  algorithm they devised has been consigned to the compost heap, but they are still there. Their inhuman reasoning will continue to influence policy for years to come.  

I say “inhuman”, because for me that is what is fundamentally wrong with the “algorithm”, or rather with the logical process of which the mathematical model is part. Both the model itself and its attendent processes were biased and unfair. But they are merely illustrative of the underlying – and fatal – problem. The algorithm did not treat people as individuals. It reduced them to points on a curve. It was, in short, inhuman. 

A fundamentally flawed model 

The mathematical model itself is deceptively simple. Here it is:

ð‘ƒð‘˜ð‘— = (1 − ð‘Ÿð‘—)ð‘𑘠+ ð‘Ÿð‘—(ð‘ð‘˜ð‘— + ð‘žð‘˜ð‘— − ð‘ð‘˜ð‘—) 

where Pkj is the proportion of students at the school or college allocated a particular grade in a particular subject, ckj is the historical proportion of the school or college’s students that got that grade in that subject, rj is the proportion of students at the school or college for which there exists “prior attainment” data (GCSEs for A/AS levels, KS2 SATs for GCSEs), and qkj – pkj adjusts the grades awarded  to those students by the difference between their cohort’s “prior attainment” and that of the 2019 cohort. The tech spec helpfully annotates the equation to this effect:

This equation will result in unfair outcomes in two dimensions. 

1. Historical performance cap

If a high proportion of students in a school or college lack “prior attainment” data in the subject under consideration, the performance of past students of that school or college will determine their grades. The predicted performance of their own cohort will make little or no difference. In the tech spec, this effect is described thus: 

From this is can be easily seen that, in a situation where a centre has no prior attainment matched students, the centre-level prediction is defined entirely by the historical centre outcome since ð‘Ÿð‘— = 0 leading to the second term collapsing to zero resulting in 

{ð‘ƒð‘— } = {ð‘ð‘— }

As one mother ruefully commented on Twitter, it seemed grossly unfair that the results of her clever, hard-working younger child would be determined by the awful results achieved by her lazy, “head-in-the-clouds” older one. Of course, this algorithm doesn’t operate at an individual level like that. But in aggregate, that is its effect. 

Furthermore, even when there is “prior attainment” data, the algorithm’s primary reliance on historical data prevents students in the 2020 cohort from achieving grades in that subject higher than those achieved by previous cohorts in that school or college. So if the highest grade anyone ever got in biology at your school was B, that’s all you could get – even if your school predicted an A* and you had a place waiting for you at Cambridge. There were stories after stories of kids at historically poorly-performing schools being awarded lower than expected grades because their predicted grade was higher than anyone had ever achieved at the school, and losing prized university places because of it. But these downgrades were not accidental. They were not a “bug” in the algorithm. They were designed into it. 

2. “Prior attainment” distortions

The algothim also created unfairness for students at schools that lacked historical data in their subjects (for example if they had not offered those subjects before, or had switched exam board or syllabus). In this case, the cjk term in the equation collapsed to zero, reducing the equation to

ð‘ƒð‘˜ð‘— =  ð‘Ÿð‘—(ð‘žð‘˜ð‘— − ð‘ð‘˜ð‘—) 

The students’ grades were entirely determined by the predicted performance of the 2020 cohort based on their GCSE results compared to those of the 2019 cohort. Why is this unfair? Because qkj – pkj was the predicted value-added performance for the whole country, not for the individual school or college. If the school or college generally delivered higher grades than the national average across all subjects – as might be expected in a selective school, whether in the state or independent sector – students in subjects where the school or college lacked historical data received lower grades than their teachers predicted. The downgrades could be considerable.  

This actually happened at, of all places, Eton College. Here’s what the Headmaster had to say about it, in a letter to parents: 

However, regrettably a number of our candidates saw their CAGs downgraded – sometimes by more than one grade and in a way in which on many occasions we feel is manifestly unfair. One particularly extreme example related to a subject in which this was the first year Eton had followed that particular syllabus and so there was no direct historic data. Rather than accept our CAGs and/or consider alternative historic data in the previous syllabus we had been following (from the same examination board), the board chose instead to take the global spread of results for 2019 and apply that to our cohort. This failed to take any account of the fact that Eton is an academically selective school with a much narrower ability range than the global spread. The results awarded to many boys in this subject bore no relation at all to their CAGs or to their ability. Several lost university places as a result. 

But the algorithm didn’t just award unreasonably low grades to students at high-performing schools which lacked historical data. It also geneated spuriously low grades for high-performing schools which did have historical data. There were numerous reports of students being awarded “U” grades at schools where there had never previously been a “U” grade. How did this happen?

To understand this, it’s necessary to dig a little deeper into how that “prior attainment” adjustment worked. George Constantinides explains how, for A-level students, a slightly lower GCSE profile than in previous years could generate anomalously bad results:

Imagine that Centre A has a historical transition matrix looking like this – all of its 200 students have walked away with A*s in this subject in recent years, whether they were in the first or second GCSE decile (and half were in each). Well done Centre A!