Best Practice

The misuse of international studies in UK education

Government policy
The misuse and oversimplification of international surveys and comparisons can lead to misleading statements being made about English education. Dr Newman Burdett looks at how we should be interpreting international data.

“The Programme for International Student Assessment (PISA) figures show that the standard of education offered to young people in this country declined relative to our international competitors. Literacy, down; numeracy, down; science, down: fail, fail, fail.” Education secretary Michael Gove, (February 7, 2011)

“Increasing the social mix in schools is the way to close performance gaps. Ministers should return to comprehensive ideals if they are serious about concern for the poor – and Britain’s global ranking. The results from the latest of the influential surveys in the Organisation for Economic Co-operation and Development (OECD) and PISA makes this abundantly clear and is chilling reading.” The Guardian (April 19, 2012)

 

Media coverage of the UK’s performance in international educational surveys often makes depressing reading. But if we scratch beneath the surface, it becomes clear that many of these gloomy reports have, arguably, oversimplified the interpretation of the data.

This is an all-too-common problem neatly summed up recently by an NFER colleague, who said: “Probably the greatest risk in the use of large-scale international datasets is the ease with which it is possible to draw overly simplistic – or erroneous – conclusions.”

The tragedy, here, is that such oversimplification detracts from the huge amount of useful and relevant information the international survey data contain, and what they can usefully tell us about our education system – especially where we are succeeding and what we should be doing more of. 

These surveys help us to develop research-informed policies when looking at reforming and evolving our education system, and enable us to move to a position where everyone can agree on the best way forward; one that delivers genuine benefits to learners. For a start there are a number of different surveys, including:

  • The OECD’s PISA study, which provides evidence about reading, mathematics and science for 15-year-olds.

  • TIMSS (the Trends in International Mathematics and Science Study) which provides evidence about mathematics and science for year 5 and year 9 pupils.

  • PIRLS (Progress in International Reading Literacy Study), which provides evidence of the reading ability of year 5 pupils.

In each of these studies we perform differently. In some performance appears to be improving, in others it is largely stable, and in some we seem to be going down.

If we look in more detail at some common misconceptions, starting with whether we are really plummeting down the tables, it quickly becomes apparent that it is not a sensible question – in fact the reports make it clear it is not possible to give any country an absolute rank. 

At best we can say our achievement in a given survey is similar to this group of countries which performs well, but not as good as these countries and better than these countries. And, it all depends on what aspect of performance you value. For example:

  • Do you want to be top of an international league table but not worry that young children are studying 13-hours-a-day and are disengaged? 

  • Do you want parity of esteem between vocational and academic routes?

  • Do you want to be producing future world leaders? 

Achievement

Let’s turn again to the idea that achievement is falling. An investigation of the data over time shows clearly that the change in rankings quoted above by Mr Gove is as much a result of other countries improving their achievement scores or a differing mix of countries participating rather than a drop in educational standards here. 

Looking at how other countries are achieving this improvement is very interesting. Many of the countries that have improved have done so by targeting their low achievers – not just for educational reasons, but for very sound economic and social reasons too. 

Wherever the UK stands overall on the international surveys, it does have a higher range of scores between its highest and lowest performers than we should be happy with. 

So a key message to take from the performance of other countries is that if we want to improve, a good starting point would be to target those who are not achieving at the basic level (PISA In Focus 2: Improving Performance: Leading from the bottom) – those who are at risk of disengaging from school and becoming NEET (not in education, employment or training).

The issue is far more complex than simply requiring a better social mix in schools, as the extract from the Guardian would have us believe.

What we need to do is to understand why a significant minority of our young people do not achieve at school, and put in place strategies to keep them engaged. It may be possible to learn from other countries that have done this well.

The Guardian quote is right in that a wider social mix in schools is linked to better social mobility, but this is as much about quality of teaching, aspiration and equity of access as about social mixing. That is to say, equitable systems allow social mixing; altering the school system to enforce social mixing will not necessarily ensure equity. There are some very complex economic and social factors in play here as well as the educational ones, and this highlights one of the big challenges in how we use the international surveys to inform the debate.

It cannot just be a simple case of “policy tourism”. Many of the top performers have small and relatively homogenous populations – the UK is large and has issues of diversity (geographical, social, economic etc) that many of the top ranking jurisdictions do not need to worry about. 

Engagement

So far we have only been talking about achievement. But education is also about engagement, motivation and the development of wider skills beyond the academic curriculum. The international surveys have very rich data and there are some surprising findings in there.

Take STEM education as an example. We have been making great efforts in the UK to make science and mathematics education more relevant to learners – and the latest data from TIMSS suggests we are succeeding. 

If we look at how students judge mathematics as relevant to everyday life and its importance in finding a job, students in England rate it as more important than students in the high-performing jurisdictions that England is most often compared with. 

In Finland, the golden child of international education, less than 28 per cent of students think that mathematics is really useful in daily life compared to 

61 per cent of their English counterparts (for relevance to employment it is similar: 22 vs 56 per cent). 

Even in the Pacific Rim countries, where we assume mathematics and science have huge cachet, only Singapore comes close, and scores lower than England on these two important student perspectives. 

We are obviously doing something right in how we teach mathematics and science in England – let’s not lose sight of that when we try to boost achievement. 

So how do we keep our education system as one we can continue to be rightfully proud of? How can we ensure we improve achievement – especially at the lower end of the achievement spectrum?

One of the best ways of achieving greater social equity is to address the problem at the start. Evidence shows that good quality early childhood education is a significant factor in later achievement (PISA In Focus 1: Does participation in pre-primary education translate into better learning outcomes at school?) .

The evidence also shows that making this freely available can go a long way to addressing the issues of socio-economic background on educational achievement. 

Accountability

Other factors that need to be considered are more subtle and complex. Accountability is a word that is very popular (or unpopular depending on your standpoint) in the current debate. 

Most would agree that while accountability is very important, and whole-heartedly support it as a means of improving the educational experience of our young people, there remain serious concerns about how it is currently being used in schools.

Unfortunately, the international surveys do not tell us how to introduce good accountability, but from the teaching profession’s point of view these surveys do give a very clear and very interesting message for policy-makers to contemplate.

As the OECD has said: “It is a combination of several autonomy and accountability policies, not just a single, isolated policy, that is related to better student outcomes.”

In other words, accountability needs to be coupled with autonomy to be useful, and you have to look at the whole educational picture.

I think it is this idea – that we need to look at the complex web of policies and how they interact – that is the real take-home message from international surveys.

  • Dr Newman Burdett is head of the Centre of International Comparisons at National Foundation for Educational Research. NFER has a long history of delivering international surveys and benchmarking studies, including TIMSS, PISA and PIRLS.
Further information