You would be better off flipping a coin to judge the quality of a lesson than listening to the views of an Ofsted inspector, it was claimed this week.
A far-reaching report into the future of England’s schools inspectorate has issued a damning verdict on both the quality of school inspectors and the reliability of lesson observation judgements.
The study, by the think-tank Policy Exchange, says that Ofsted should stop observing lessons during school inspections and should “abolish or radically reduce” the number of additional inspectors – those contracted to private outsourcing companies.
It claims that many inspectors lack the necessary skills to make a fair judgement of schools – especially the ability to analyse data.
The study, Watching the Watchmen: The future of school inspections in England, proposes that all future Ofsted inspectors have relevant and recent teaching experience in the type of school they are assessing.
Inspectors should also have to pass an exam that tests their ability to analyse data and make reliable judgements.
The report quotes research by Professor Rob Coe of Durham University showing that the results of lesson observations are unreliable.
It highlights that when comparing a lesson observation judgement of teacher quality against its “actual quality” as defined by the value-added progress made by pupils in the class, the results show you’d be better off flipping a coin.
It states: “Overall the results are worse than flipping a coin – there is a 49 per cent chance that the quality of the lesson will be empirically the same as judged, and a 51 per cent chance that it will not be the same as that assessed by an observation.”
Other research quoted by the report’s authors shows that in order to produce a reliable judgement, a teacher would need to undergo one 45-minute observation, and then three further observations by different observers. However, this practice was “highly unlikely, veering towards unprecedented” during an Ofsted inspection, it said.
The report adds: “One headteacher we spoke to timed all the lesson observations during her school’s inspection and found that the longest lasted for 14 minutes; many of the responses to our call for evidence cited the brevity of lesson observations as one of the reasons they could not be used as a fair representation of a teacher’s competence.”
Ofsted has recently moved to stop its inspectors from grading lesson observations, but the report says that this is not enough.
It states: “The consequences that flow from the practice of observations – whether schools preparing checklists of ‘outstanding lessons’, conducting mock inspections, or teachers preparing ‘Ofsted lessons’ – are all both nugatory and avoidable.
“Why should a school conduct carefully planned and exhaustive processes around monitoring and assessing its own staff’s performance, if Ofsted can make a judgement that trumps with a series of 20-minute drop-ins?”
Instead, the report says that headteachers should be at the forefront of making judgements about their school, with the inspectors’ role being one of scrutiny rather than making the judgements themselves “without sufficient time and evidence”.
The report also attacks the quality of many inspectors, listing concerns in two main areas – the ability to “understand, interpret and draw conclusions from statistical data” and a fixed mindset in some inspectors of how a school should be run with an “unwillingness to engage in different structures”.
It states: “Much of the weakness stems from poor quality control among the inspectors themselves, which include large variance in recent school experience, levels of training, and the extent to which personal preferences and behaviours as to models of school improvement affect what ought to be impartial judgements
“One of the most significant concerns raised by headteachers and schools ... was that inspectors simply did not understand their data, in particular progress measures. This is deeply concerning, given the reliance placed by inspectors on data.”
It calls for inspectors to have to pass a “data interpretation test” in order to inspect schools, with this accreditation being time-limited and regularly reviewed every five years.
The report raises further concerns that many inspectors cannot be held to account either. This is because around 3,000 additional inspectors, 1,500 of whom carry out school inspections, are employed by three regional contractors who are paid £30 million a year for their services. Ofsted itself directly employs just 300 to 400 HMIs, of whom around 140 work in schools.
The report calls this a “twice removed form of accountability” because many additional inspectors are freelance workers under contract themselves to the service provider.
It states that few details are available as to the level of training of many additional inspectors or about their previous inspections. It adds: “Moreover, as private companies and because of a commercial contract between them and the (FOI-eligible) Ofsted, none of this information is subject to FOI (Freedom of Information requests).”
Elsewhere, the report puts forward ideas for a new two-stage inspection process which it says would enable Ofsted to focus more of its resources on struggling schools and free-up better schools from the “bureaucracy of full inspection”.
The first stage would be a “short inspection” for all schools every two years yielding two judgements – an overall grade and a new “school capability” grade that would assess its ability to continue to perform.
The second stage would then be a “tailored inspection” for all schools that are not rated good or outstanding in both these judgements.
It says that the resources saved by this model would allow the tailored inspections to be twice as long, meaning inspectors could “really understand the school and its data, and explore issues that the school may feel a brief inspection does not cover”.
Jonathan Simons, one of the report’s authors, said: “At the moment a team of external observers watching a handful of lessons can make a judgement on the quality of teaching which trumps the view of the school itself. The evidence suggests that when it comes to relying on the judgement of a trained Ofsted inspector on how effective a lesson is, you would be better off flipping a coin.
“More needs to be done to drive up the quality of inspectors. Heads and teachers must feel confident that the person running their eye over their school is a specialist, preferably with recent teaching experience.
“Inspectors don’t need to be rocket scientists, but they must also have the ability to interpret the increasing amounts of data on the performance of schools, and understand the different ways in which schools are now operating. Schools should not be forced to second guess what the inspector coming through the door will be like.”
Responding to the report, Ofsted’s national director for schools, Michael Cladingbowl, said: “Ofsted has played a major part in raising standards in England’s schools over the past 21 years. We are now looking at how inspection should develop in the coming years to reflect the fact that eight out of 10 schools are now good or outstanding. We welcome Policy Exchange’s recommendations – many of which chime with our own – and will be studying them more closely in the coming days.
“Headteachers tell me that schools would benefit from more regular contact with Her Majesty’s Inspectors and we know parents would value more up-to-date information. Our proposals will draw on our experience of carrying out more than 7,000 school inspections each year. We will consult on any changes to our inspection arrangements in due course.
“We are also determined to build on our recent success in getting many more excellent serving school leaders onto our inspection teams.
“In my view, parents will always expect inspectors to spend time in classrooms when they visit a school because teaching is the heart of what schools do. While we do not judge individual teachers, visiting lessons is a key way of gathering evidence about the quality of teaching in the school overall. Inspectors also take account of the school’s own views of teaching, undertake joint evidence-gathering with senior leaders, look at children’s work and teachers’ marking, discuss test and examination results, and talk to parents, pupils and staff.”
Brian Lightman, general secretary of the Association of School and College Leaders, said the report raised a number of “interesting points”. He explained: “In particular we agree that all inspectors should have recent and relevant experience as a practitioner. This needs to be more than just having been a teacher, inspectors should have experience of school leadership.”
However, he disagreed with the findings on lesson observations: “Lesson observation will always have a place within inspection. I cannot see how an inspector could make an overall judgement about a school without seeing what happens in the classroom.”
Dr Mary Bousted, general secretary of the Association of Teachers and Lecturers, said the findings reflected their concerns: “We have for some time highlighted the problem of inspectors judging lessons and stages of education in which they have little or no experience. Ofsted suffers a major problem with inconsistency in the quality and reliability of its judgements which it struggles to address. There is now too much concern both inside and outside Ofsted with using private sector contractors for inspection that this issue can no longer be ignored.”
Chris Keates, general secretary of the NASUWT, said, however, that the report’s proposed alternative would “create as many problems as they aim to solve”.
She explained: “In particular, the idea that inspectors can form secure judgements about the quality of education provided by a school through the narrow prism of performance data would actually exacerbate the pressures that league tables and floor targets create. These already lead to a narrowing of the curriculum which is at the heart of the serious flaws within the current inspection system.”
The report is available at www.policyexchange.org.uk