Let’s admit that students may have learned less

This post was first published on Wonkhe, https://wonkhe.com/blogs/lets-admit-that-students-have-learned-less/

CDE Fellow, David Baume says we should acknowledge the Covid-19-shaped asterisk beside students' qualifications this year.

Gavin Williamson said something very helpful on February 25. Stay with me.

He said that 2021 A level students will be assessed on what they had been taught, not on what they had missed.

Maybe he will take this to its logical conclusion. Perhaps A level results will be accompanied by a transcript showing not just marks and grades in subjects, but additionally what outcomes students had achieved. Assuming that there is a close relationship between learning outcomes and assessment tasks, this could be done largely mechanically rather than individually.

This transcript could be accompanied by an account of what they had been taught. The users of A level results could then compare these two accounts to the syllabus, and thus be able to make better use of the results. They could calibrate their expectations of what grades mean this year. They could be clear in which subset of the syllabus each student had shown proficiency, and in which not. And be clear in which subsets it would be unreasonable to expect students to demonstrate attainment, because that subset had not been taught.

Students graduating from university in 2020, 2021 and (insert your own closing date here) will, in their own minds and in the minds of employers and others, have a large Covid-19-shaped asterisk by their degree. The Minister’s plan, with my modest extension to it, will mean that we know what this asterisk means for A-levels.

The obvious question is – what will that asterisk mean for university graduates?

A better question may be – what will graduates, employers and others take that asterisk to mean?
The best kind of question, a Wonkhe-type question, may be – what kind of assessment policy would help here?

Deconstructing education and assessment

The University contract of education with the student may be, essentially, this:

• Universities admit students who will have a reasonable prospect of graduating;

• Universities teach students what the students need to know or be able to do, and how well, to enable them to graduate, and more broadly support them to learn;

• Students, meanwhile, do the necessary listening and studying and learning; and

• Most of those students get some sort of a degree.

Similarly crudely, there may be two dimensions to assessment judgments:

• Quantity – how much have the students learned / how much can they do?

• And quality – how well do they know it / how well can they do it?

Talk of standards in assessment generally and unhelpfully crunches these two dimensions of assessment together.

A giant finger of fudge

There are proposals to convert that Covid-shaped asterisk into a giant finger of fudge. The idea is that we adjust / inflate / normalize students’ grades. In other words, that we pretend that they have learned and achieved more / better than they have actually learned and achieved. In fact, we pretend that they have learned pretty much what they would have learned in a non-Covid year.

Of course, we fudge assessment anyway. Under the seemingly benign label of “norm-referenced” assessment, we fudge marks up or down, each year, so that, on balance, overall, more or less, the distribution of marks looks pretty much the same as it did last year.

But what happens when we fudge marks upwards in Covid times? We don’t fool anybody.

Universities, students and employers all know that graduates from the Covid years may well have learned less, and / or learned to perform less well, than graduates from previous years, because teaching and learning were disrupted.

Employers can cope – they will adjust their view of the students, to reflect what they turn out actually to have learned, what they actually know, what they can actually do, and how well. That will confirm the already widely-held (and pretty accurate) view that university grades are measured on rulers made mostly from rubber, although that’s probably not a message that universities, as among other roles guardians of standards, want to reinforce.

Universities, who in this respect are being flexible with the truth, and the students, who know that their current state of knowledge and capabilities are probably being flattered by their award, are both likely to feel a bit grubby. This finger of fudge is no treat at all.
Of course, some students may do better under the new regime. Even if the questions are the same, a Covid-era twenty-four-hour unseen open-world un-invigilated exam, answered via a keyboard, is a very different task from the pre-Covid two-hour unseen invigilated handwritten version.

Tell the truth

Alternatively, we could mark against the same inviolable-ish standards we use every year, before we norm reference. What would happen?

Most students would (presumably) get somewhat lower grades in the Covid years (although see the caution above about assessment methods). Because they had learned less. Because their teaching and learning had been disrupted.

Employers, and others who use degree qualifications as a basis for selection, might have to change their selection criteria – reduce their expectations of graduates; maybe also provide their new recruits with more initial training and development to fill the Covid-19-induced gaps in what their incoming graduates had learned. Some students will have managed to learn very well, despite the disruptions to their teaching.

“But that’s not fair!”

There would be concerns about fairness – but Covid isn’t fair, and the disruption to education caused by restrictions arising from it isn’t fair. The fact that some students are better than others at independent learning isn’t fair. The fact that some universities have done a better job at taking education online than others isn’t fair. The fact that different assessment methods may have been used isn’t fair. Very little here is fair.

We can’t honestly “no-detriment” for Covid. Nor can we do the opposite, no-unfair-advantage. Covid happened, is happening and will continue to happen, for a while yet. And it has detrimental effects, and maybe a few positive effects, probably different effects on different students, subjects, courses, universities.

Can we meaningfully safety-net? To some extent.

We can surely allow more preparation time for assessment, perhaps early sight of the paper – very few learning outcomes specify preparation time, although assessment regulations may. Regulations can be reviewed, and changed, as long as we acknowledge that regulations affect the assessment task.

We can surely allow more resubmissions – very few learning outcomes say that the outcome has to be achieved at the first or second attempt, although again current regulations may.

But I don’t think we can sensibly safety-net by, for example, adjusting everyone’s post-mid-March-2020 marks to line up with the marks they were getting before that date.

Marks mean outcomes and standards achieved, things learned, capabilities developed and demonstrated. To adjust in this way would be implicitly to pretend that students had learned things they hadn’t learned, learned to do better than they had actually done, achieved outcomes that they hadn’t achieved.

Discomfort

Telling the truth here may feel uncomfortable. That may be because we confuse grades with worth, with intelligence, with academic ability, with other desirable qualities. A student is not a worse human being because they got a lower rather than an upper second. A student who might otherwise, in a non-Covid year, have got an upper second is not academically poorer or weaker just because, in a Covid year, they got a lower second.

They may have different (less) current knowledge, different (fewer) current capabilities, different (lower) current levels of proven ability. Because their teaching and learning were disrupted. And so they have different development needs for whatever they decide to do next. They are the same person. With the same potential. They may just be a few months behind.

Policy implications?

We should be truthful. Let’s acknowledge and declare that the Covid-19-shaped asterisk beside their qualification means that, because of the pandemic, their teaching was disrupted, and so they (may have) learned less. No shame.

If we exaggerate what has been learned; if we fudge what Covid-era qualifications actually mean – then we damage the credibility of universities. I don’t think we want to do that.

The 2021 A-level approach may show a way, with my modest extensions to it. A transcript, showing what students have actually learned, and what they have been taught, and under what conditions they have demonstrated what they have learned. No fudge.