Those of us who complain that edtech gets too little attention in the national press should perhaps have been beware of what we wished for. Two recent studies to have hit the headlines both say that the general impact of technology on learning is negative.
Advocates of technology enhanced learning have tended to brush this research aside. Various online commentators have said that the reports were “flawed”, “confused” and “tosh”, and those who reported them were guilty of “lazy and irrelevant journalism”.
The two reports are:
Communication: Technology, Distraction & Student Performance
- written by Louis Philippe Beland and Richard Murphy
- published by the Centre for Economic Performance at the London School of Economics
- in May 2015
- which I will referred to as “the LSE report”.
Students, Computers and Learning—Making the Connection
- published by the OECD under the direction of Andreas Schleicher
- in September 2015
- which I will refer to as “the OECD report”.
In this post I will:
- review what the reports say and why they said it,
- assess the arguments put forward for dismissing them,
- and suggest some conclusions to take away.
At over 14,000 words, I cannot pretend that this is anything other than a long article—but the length is unavoidable given the nature of the topic. What is more, I believe that the reaction of the edtech community to these reports is sufficiently important to justify the effort of writing it. I thank you in advance for making the effort to read it and hope that by the time you finish, you will also think that it was worthwhile.
If you want to link to any section header, right click on it and select “Copy link address”.
The LSE report was based on a survey of schools that were asked whether they had implemented a ban on mobile devices in the last 13 years. The results of this survey were cross-referenced against data on student performance in terminal exams.
The researchers contacted all the Local Authorities in Birmingham, Leicester, London and Manchester and then approached all the schools in those local authorities that gave their permission. 91 schools (21% of those approached) responded. Schools were asked whether they had introduced a ban on mobile phones over the last 13 years and, if so, how rigorously it had been enforced.
This information on mobile phone bans was cross referenced against the achievements in terminal exams of the 130,000 students who had attended those 91 schools over the same period, as recorded in the National Pupil Database between 2001 and 2011.
Part of the strength of the study comes from the fact that it includes both a lateral and a longitudinal methodologies: the change in performance of individual schools can be monitored as they introduced bans, as well as against the performance of similar types of school (with or without robust mobile bans).
The report found that, overall, the introduction of mobile phone bans correlated with an improved performance of 5.67% of a standard deviation (or an effect size of 0.06). It should be noted that this is a small effect size: the received wisdom is that in the social sciences, an effect size of:
- 1 is “small”,
- 3 is “medium”
- 5 is “large”.
However, this generic scale might be a little too demanding when applied to education, as Dylan Wiliam notes in his ResearchEd 2014 keynote (at 21:40) that:
An effect size of 0.3, in a secondary school, would be doubling the speed of student learning. I think doubling the speed of student learning is worth having.
Students were divided into 5 quintiles by their prior attainment in national exams. The report found that the lowest 20% of students improved by 14.23% of a standard deviation (an effect size of 0.14), while the top 20% of students were neither positively nor negatively affected by any ban.
Hence, the overall finding of the report was that allowing the use of mobile phones had no effect on the strongest students but did have a significant negative effect on the weakest students.
The report noted that these findings should not be taken to imply that there might not be positive uses for mobile phones. It mentions in particular a study in which parents were informed of students’ homework assignments by text message, a practice that was associated with higher test scores (though it should be noted that this practice did not require any mobile phones to be brought into school).
While the LSE report is a skinny 43 pages, the OECD comes in at a heftier 200. Instead of being a single focused study, it attempts to review the effect of technology on learning, using all the evidence from the 2012 PISA tests.
In 2012, the PISA tests were administered in 65 countries (not including the UK in that year). The core tests were supplemented by an optional ICT Familiarity Questionnaire, which was distributed to students in a total of 42 countries. The study compares the performance statistics achieved on the core PISA tests against the answers given in the familiarity tests. In addition, trends in ICT familiarity were available in the 28 countries that had also administered the ICT questionnaire in 2009.
Given that the OECD report is more of a baggy monster than the LSE report, there are a number of different conclusions.
The report found that top performance in the PISA tests was achieved by countries that reported less than average computer use. In the case of maths, the best performances on paper-based mathematics were achieved by students who made the very lowest use of computers in school, so the relationship between technology and achievement was always negative, though the report notes that this inverse relationship was reversed in Belgium, Denmark and Norway, where there was a positive correlation between the use of technology and performance in Maths.
In the case of reading, the best performance was reported by students who made just slightly less than average use of computers. The three graphs below show performance mapped against use of computers
- in school;
- out of school for schoolwork;
- out of school for leisure.
The generally anti-tech conclusion was re-enforced by correlating the amount of improvement between 2003 and 2012 made by different countries in their average PISA scores against computer use in school, as reported in the survey. The mapping produces a broad scatter of results—but with a clear trend-line suggesting an inverse relationship.
This relationship is investigated in more detail with reference to reading ability, mapping the frequency of different sorts of computer use against performance in the tests for “digital reading” (a skill in which you would have thought that familiarity with computers was important).
The first graph looks at the frequency of the use of computers for schoolwork. This suggests that the lack of any access to the internet for the purposes of browsing for information was correlated with a sharp reduction in performance. The lack of access to computers as a productivity tool (“doing homework on the computer”) was also damaging. On the other hand, not using computers for social networking did not appear to have any negative effects on performance.
At the same time, the frequent use of the computer (i.e. every day) was always correlated with poorer performance. So we can conclude that the best performances are correlated with moderate use of computers for information browsing and word-processing, while avoiding social networking altogether.
The second graph looks shows the same performance in digital reading compared to the frequency of the use of computers for leisure activities. This shows only a slight drop in performance when computers were used in this way “every day”. The counter-intuitive conclusion from a comparison between the two graphs might be that students do better if they use computers for leisure than for learning. But one should bear in mind that these graphs do not show the total amount of time used for leisure and schoolwork. They hint, rather, at the proportion of schoolwork and leisure time respectively that is spent on computers. My guess is that very frequent computer use for schoolwork might suggest an opportunity cost, replacing more valuable forms of instruction by:
- an over-reliance on collecting information or chatting online rather than useful practice,
- and/or a corresponding lack of engagement from a human teacher.
Conversely, the use of computers during leisure time does not imply any reduction in more useful forms of instruction or study.
So long as the use of the computer for leisure does not unduly extend the total amount of time spent on leisure (see however, the Damaging effects of overuse, below), then it should not interfere with schoolwork.
Looking in more detail at the different uses made of computers during leisure time, sending emails and browsing the internet (both potentially more serious uses) tend to correlate with higher performance that downloading films, music and software, or playing collaborative games.
Indeed, playing collaborative games is the only online leisure activity that is shown to have a wholly detrimental effect on learning—a finding that will doubtless surprise those (including the government’s own e-skills champion, Ian Livingstone and the people at Learning Without Frontiers), who have argued that the use of commercial games in school is a really good way to improve educational outcomes.
The OECD report suggests that the digital divide is not a serious issue for education, for two reasons:
- because the gap in access to technology at a basic level between poorer and richer students is closing rapidly;
- because in spite of this greater equality in access to basic digital tools, the gap in educational achievement remains just as great as ever (as one would expect, given the lack of correlation between technology use and useful educational outcomes).
This finding confirms what many of us have been arguing for a long time, that government projects like Becta’s Home Access project, that focused on handing out laptops to poorer students, were a waste of time and money.
The report contains a useful summary of how computers are used in school.
I have simplified OECD’s 9 categories into 5 and represented them on the chart below.
|49.1%||Browse the internet for schoolwork||Research|
|22.6%||Download, upload or browse material on the school website|
|15.2%||Post work on the school website|
|25.0%||Use email at school|
|21.1%||Chat online at school|
|27.3%||Use school computers for group work and communication with other students|
|25.9%||Do individual homework on a school computer||Productivity|
|21.4%||Practice and drilling, such as for foreign language learning or mathematics||Courseware|
|13.2%||Play simulations at school|
What most advocates of education technology would call “internet research” represents by far the most frequent use of computers in schools—about twice as frequent as any other use. The second most frequent use of computers was for social networking (email, chatting online, working in groups).
I have repeatedly made the case on this blog that both these uses of computers tend to be ineffective in formal settings. And while the OECD report suggests that not having access to the internet as an information resource is damaging, it confirms
- that an over-reliance on the internet as an information resource is not helpful;
- that the use of the internet for social networking has not been shown to have any educational value at all.
The two most frequent uses of computers in schools are probably the least educationally valuable.
Categories 3 and 4 (use of computers as a productivity tool and for administrative purposes) I regard as uncontroversial. The value of productivity tools will depend on the tool, but I don’t think that anyone would get exercised about using word processors for writing up essays or the internet for checking on the dates of half term. Both are useful but no-one is claiming that either are likely to be transformational.
The final category of computer use, for drill, practice and simulation, represents the sort of education-specific software for which I have argued in this blog. And it represents the least common use of computers in schools. Even within the category, “drill and practice” (which I suspect represents the use of simplistic multiple choice) is much more common that “simulations”, which is the only sort of activity that requires any significant development of education-specific software.
In short, the use of computers in school for “looking things up on the internet” is about four times as frequent as the use of computers for subject-specific simulations (and even this statistic does not tell us anything about the quality of the simulations being used or the quality of assignment, sequencing, tasking and data analysis that contribute to the effective use of such software in formal education).
These results support my contention that the reason why computers are having so little impact on educational outcomes is that they are being used for the wrong things.
I picked out one graph that seemed to me to summarise what will strike many as a piece of common sense—that using technology to assist presentation is only of any use when the presentation is graphical. When it comes to the actual doing something, technology should be used to support student activity in preference to teacher demonstration.
In all cases, high performance correlated most strongly with not using any technology at all (this reflects the inverse relationship reported above for computer use in maths). But where technology was used, it correlated with the weakest performance where the teacher used the technology to demonstrate the entering of data on a spreadsheet or how to calculate with numbers. In both these cases, performance was higher when the students were able to use the technology themselves. In the other two cases, drawing the graph of a function and constructing geometrical figures (both of which are visual), the use of technology as a demonstration medium was more effective that letting the students engage with the technology directly.
You might argue that the student needs to have a method demonstrated before they can be expected to implement it successfully themselves. The opposition of “teacher demonstration” to “student practice” may be a false one in many circumstances. This might suggest another conclusion that might be drawn about the government’s imposition of interactive whiteboards in the mid 2000s: it might have worked much better than it did if the whiteboards could have been used to demonstrate in plenary contexts interactive content which students were also able to use themselves in individual study time. The trouble is that this sort of interactive content was never created, and no-one put in place the open interoperability standards that would allow such content to be controlled from the proprietary clickers that many of the whiteboard suppliers provided to support interaction in the plenary context.
The same conclusion could be applied to the use of video online, as with services like the Khan Academy. The demonstration alone is of little use, unless it is backed up with interactive practice of a sort that can engage the student.
Finally, the report takes a deep dive into assessing students’ aptitude with digital text. What it finds to be particularly important is the ability to navigate hypertext: a skill that in turn depends on:
assessing the credibility of sources and predicting the likely content of a series of unseen screens, based on hints such as the explicit name assigned to a link, the surrounding text, and the URL that appears by hovering over the link with a mouse. They also include organisational and spatial skills, such as the ability to construct a mental representation of the structure of a website in order to move confidently across the different pages of which it is composed.
This issue plays to what proponents of technology-enhanced independent learning tend to call “digital literacy”: those basic skills that students need to operate in the modern world and to support their learning across the curriculum.
The OECD report tracked student hypertext movements through a website as they searched for the information required to solve a particular task. They then quantified:
- the length of navigation route followed, or quantity of navigation (suggestive of perseverance);
- the relevance & efficiency of the navigation route followed, or quality of navigation.
High quality navigation is characterised by students who take no mis-steps in following the optimal navigation sequence to solve a task, or who quickly correct any mis-steps which are taken.
On the whole, ability in digital reading was found to correlate with following capabilities:
|Ability in print-reading||80%|
|Quality of navigation||10%|
|Quantity of navigation||5%|
This analysis suggests one (fairly obvious) conclusion and suggests one further question.
The conclusion is that aptitude in the digital realm depends heavily on the mastery of traditional academic skills: the ability to read digital text is 80% dependent on the ability to read traditional print. No surprises there to anyone except those who have argued that the 21st century turns our curriculum on its head.
The question is “what helps students make good quality navigation decisions?” The answer seems to be more of the same:
good navigation relies on the same cognitive skills and motivational aspects that are prerequisites for success in the paper-based assessment of reading as well… Statistical analyses show that students’ reading skills in print documents strongly predict their navigation behaviour in a digital environment.
This is not quite the whole story, however, as some countries perform better in digital navigation than would be predicted by their print-reading scores, and this is attributed to their problem-solving.
That does not mean that the OECD report is arguing the case for transferrable skills because:
For the most part, the problem-solving skills that students demonstrate when navigating complex online texts are specific, and are likely best learned in the context of reading from the Internet.
From which one can conclude that the best way to be a good consumer of digital text is to have good traditional literacy skills, to have a sense of perseverance, and to have the chance to familiarise yourself with the requirements of reading digital text.
You might find this a rather underwhelming conclusion from such a detailed analysis—but it does at least stand against some of the more revolutionary claims that have been made by proponents of digital literacy. It confirms that the twenty-first century does not change everything and that operating online does not require a completely different set of skills to those which have been taught traditionally in schools. Good digital literacy requires good traditional literacy with the occasional opportunity to practice these traditional skills in an online context.
The OECD report also looked briefly at the sort of overuse and misuse of computers that might be termed pathological.
Basing their conclusions partly on their own findings and partly on other research, the report notes that the use of the internet for more than 6 hours per day tends to be associated with social alienation, poor punctuality, truancy, physical weakness and insufficient sleep. It notes, however, that in some circumstances, the use of the internet in these ways might be symptom as much as cause of the underlying problems.
The substantive findings of the report are therefore:
- that the effect of technology on learning outcomes was “mixed at best”—worse than mixed when the use of computers in schools was over-emphasized;
- that there was strong continuity between digital skills and traditional literacy and thinking skills;
- that problems in equity of outcomes are reducible to the unequal attainment of these traditional literacies and not to the unequal distribution of technology or to any need to redefine our educational objectives;
- that parents and schools should be mindful of the potential to abuse the online environment.
All of these findings strike me as eminently sensible and many repeat arguments that I have been making on this blog for the last 3½ years.
My criticism comes with the final part of the report’s Executive Summary, which presents the authors’ own opinions on how we should respond to these findings. This passage is at best bland and at worst misleading.
One interpretation of these findings is that it takes educators time and effort to learn how to use technology in education while staying firmly focused on student learning. Meanwhile, online tools can help teachers and school leaders exchange ideas and inspire each other, transforming what used to be an individual’s problem into a collaborative process. In the end, technology can amplify great teaching, but great technology cannot replace poor teaching.
The passage repeats what have become common orthodoxies in the teaching world, that:
- technology is a given (not something that needs to be developed and improved);
- that improvements in use have to be led by teachers, not technologists or specialist instructional designers;
- that what matters is good teaching, which is something that is distinct from technology.
In all these judgements (none of which are based on the evidence produced in the report), I think Andreas Schleicher’s conclusion is wrong. But these are just personal opinions—a gloss that is added to the report’s substantive research findings. It is the substantive findings that should interest us and in my next two sections I will examine the reasons why many in the edtech community have been dismissive of the substantive findings of both reports, but in particular of the LSE report.
I will look in much more detail at criticisms of the first, LSE report, of which there have been many more. This might be because the report has been published for longer; it might be that its findings were starker and therefore more provocative; it might be that people felt that the OECD was a more formidable target to attack.
The criticisms that I am going to examine are in:
- Mike Cameron’s blog article On research and banning things
- José Picardo’s blog article Banning mobile phones is cargo cult science;
- Gerald Haigh’s article in Sec-Ed Mobile phones: To ban or not to ban
- Discussion on the ICT Research Network email reflector, which was originally set up by Becta but is now run by the Association for Learning Technology (when quoting from these emails, which are not fully in the public domain, I will give titles and dates of emails but not names of authors).
Before I look in detail at the sources above, I must return to the thorny issue of correlation and causation. Almost all of the critics show a basic misunderstanding of the nature of scientific evidence, and in particular the relationship between correlation and causation.
On the ICTRN, a member of a Faculty of Education criticised the report’s “potential confusion between correlation and causation” , while this point was supported by other subscribers who referenced a cartoon on causality , a list of supposedly spurious correlations  and a recollection that
I remember my statistics lecturer telling us that in one particular year there was a correlation between the increase in the birth rate and the increase in the number of storks visiting New Zealand.
José Picardo comments that the report:
simply states a correlation. Correlation is not causation. It reminds me of that sketch in Father Ted when Dougal and Father Ted are sitting at a hospital waiting room and Dougal says to Ted “do you ever notice that it’s usually sick people that end up in hospitals?”
Gerald Haigh complains that in the media coverage, it is difficult to find:
anyone who points out that correlation does not imply causation, or that the research does not attempt to compare “banning” schools with the many that regard student phones as valuable additional technology.
What this shows is a widespread misunderstanding of the relationship between correlation and causation—indeed of the nature of evidence itself—among teachers and commentators on education. In the face of such a general stampede of ignorance, it seems to have little effect to point out the misconception (as I have tried to do many times before on this blog) because no one shows any interested in debating the truth of their position. Everyone avoids any serious challenge and merely continues to repeat their existing prejudices.
So it is somewhat wearily that I return to a subject which I have covered many times before—most recently at How technology will revolutionize research, section 2.4.
The fact is that the relationship between correlation and causation is very close indeed:
- causation is nothing other than a particular sort of very secure correlation between two types of event, in which one type of event always occurs before the other.
- correlation always implies causation of some sort (though the direction of causation may not be clear from a single correlation);
- correlation is ultimately the only evidence we ever have of a causal relationship existing.
If you do not understand these points or think they are wrong, then you might like to look at my previous post, but you really need to read David Hume’s Enquiry Concerning Human Understanding. Spark Notes offers a good summary at (read through to the end of section 7), or the original text is available in PDF at or in a free Kindle edition.
The key point that Hume makes is that correlation (or “constant conjunction”) is the only evidence we have for causation (or “necessary connection”) but that our pattern-forming brains imagine that we understand the secret reason for the necessary connection. This imagined understanding lies at the heart of our false reasoning. If you are an educationalist, you are probably familiar with the thought of Jean Piaget, who argues that our understanding is represented by a model of the world, which develops as we either assimilate or accommodate the evidence that we derive from experience. In this process, the model offers a vision of how everything fits together, while the evidence is fragmentary.
The fundamental error of everyone who argues that correlation does not imply causation is to think that the model is the evidence—that we have no reason to say that A causes B until we understand how A causes B, how everything fits together, what is really going on. But all that stuff is the model, the product of our imagination, it is how we represent the world internally. The model is not the evidence.
The evidence comes from observations of isolated phenomena. In situations in which we do not have reason to believe that a single observation will necessarily be repeated, evidence comes from correlations. As noted in my previous post on this subject, the website of spurious correlations are not correlations at all—they are datasets that are:
- specially selected because they fit (if the significance of this escapes you, listen to Dylan Wiliam on the beneficial effect of green jelly beans on acne, from 22:50);
- each contain very few data points (10 or 11);
- represent trend lines, which dramatically reduces the statistical power of the data sample still further (they are reducible to half a dozen basic patters: rising, falling, peaking, bottoming-out…).
José Picardo tells me on Twitter that assuming correlation implies causation is
like saying Finns do better cos they eat reindeer.
No it isn’t:
- “Finns do better in education” is a single data point;
- “Finns eat more reindeer” is another single data point.
The coincidence of two data points does not establish a correlation.
But what is a profound misunderstanding, one that shows a failure to grasp the essential character of scientific evidence, is almost universally accepted as gospel truth among teachers and educationalists. Exactly the same elementary mistake is made by the subscriber to the ICT Research Network who understood so little from her statistics course that she thinks that:
- an increase in the New Zealand birth-rate in one particular year
- and an increase in migrating storks visiting New Zealand in the same year
establishes a correlation.
In fact, the example of the storks is a distant and misunderstood echo of a 1952 article by J Neyman, which I summarised on an email to the ICTRN reflector as follows.
I am very grateful to Harvey Mellar for pointing me to the source of your stork correlation, which is a humorous 1952 article by J Neyman in Lectures and Conferences on Mathematical Statistics and Probability. This article is described in a 1993 article by Richard Kronmal, Spurious Correlation and the Fallacy of the Ratio Standard Revisited, in Vol 156, No. 3 of the Journal of the Royal Statistical Society, Series A (Statistics in Society), which can be found on JSTOR.
Neyman’s 1952 article showed a correlation between different countries based on the number of babies per 10,000 women and the number of storks per 10,000 women. The paper recounts how the finding supposedly led to a proposal by the Zero Population Group, campaigning for better birth control, to eradicate storks, and the statistical argument that followed between the ZPG and the Trust for the Preservation of Wildlife Species.
This study was designed to illustrate a phenomenon identified by 1897 as “spurious correlation”. Kronmal’s 1993 article suggests that, even though the phenomenon of spurious correlation has been known for over a century, researchers continue to ignore the problem – for example in much research based on the Body Mass Index.
The problem with the stork correlation is that both are expressed as ratios, using the same denominator – “per 10,000 women” – and, which may make the situation worse, in a situation where the ratio between storks and the number of women is meaningless. Neyman showed that there was no correlation between the number of storks and the number of births, but there was a correlation when these were expressed “per 10,000 women”. The lesson for researchers is to investigate raw metrics and not ratios, avoiding correlations between ratios especially where they both share the same denominator.
I do not pretend to understand the argument, which is a technical one, or exactly what is happening – but its importance for our discussion seems clear enough. The storks are the exception that proves the rule. The particular reason why this sort of “spurious correlation” is of such interest is precisely because of the general assumption that correlations DO imply causation is fundamental to the practice of quantitative research. If, as most of this list seems to assume, correlation does not imply causation, there would be no reason to make such a fuss about just one more irrelevant correlation.
To summarise the lesson for educationalists of the Neyman article: spurious correlations are correlations between datasets populated by ratios that share the same denominator. Such datasets cannot be interpreted as supporting the causal inferences—even though these are implied by non-spurious correlations which are not based on the comparison between ratios.
The case of causal direction always has to be considered, of course. José Picardo’ supports his assertion that “correlation is not causation” by recounting his Father Ted joke.
It reminds me of that sketch in Father Ted when Dougal and Father Ted are sitting at a hospital waiting room and Dougal says to Ted “do you ever notice that it’s usually sick people that end up in hospitals?”
Presumably José means that people are not sick because they are in hospital (even this may often be the case)—but it cannot have escaped his attention that the reason that most people are in hospital is because they are sick. An example chosen to demonstrate that correlation does not imply causation only succeeds in demonstrating the exact opposite.
I want to finish this reprise of the discussion about correlation by reiterating a rather more subtle point. Many of the more sophisticated objections to the LSE report consist in pointing out things that the report did not look at, things that the research left out. It was not sufficiently holistic. I shall refer to these arguments as the holistic fallacy.
Empirical evidence (of which correlation is a formal type) is of its nature fragmentary. No one ever comes across evidence that supports the scientific theory of heat. The evidence comes in small packages, like touching a hot ring and getting burnt, or observing the transfer of heat from one object to another. Correlations could be compared to the results of an electrical test in which the electrician compares the volts in two electrical wires: it is a binary comparison between two things—and it is only a succession of such comparisons that will form a complete theory (e.g. of where the electrical fault has occurred).
The holistic fallacy is a variant of the point that Hume makes, that the non-scientist is confused into thinking that the model—what puts us into the emotionally comforting state of mind in which we persuade ourselves that we understand what is really going on—is the evidence. Our mental model is holistic: the evidence is fragmentary. The holistic fallacy is one way in which we muddle the two. It is like dismissing the significance of the fingerprint on the window sill because it does not take into account the torn clothing in the thorn bush. Both are important pieces of evidence that need to be accommodated into the blow by blow narrative of the crime by which the sleuth stuns the assembled house-guests with the brilliance of his or her reasoning.
A further variation of the holistic fallacy (which I envisage as an infinite outwards expansion of all the different influences that have to be taken into account) is the belief that you cannot demonstrate causality without burrowing infinitely inwards and tracing every minute step of the causal pathway. This variant is illustrated by another correspondent on the ICTRN reflector:
The real problem of the correlation v causation issue in research into digital technologies in education is that to prove causation one would have to dig inside pupils’ brains.
Wrong. We can never follow the minute tendrils of causality, which are forever invisible to us. Even if we got inside students’ brains, it would never be enough for us to reach out and touch the infinitely small, the infinitely many and inevitably secret causal couplings. All we can do, like the electrician with a voltmeter connected to two crocodile clips, is show the binary dependency between two phenomena occurring at different points along what we then infer to be a causal pathway.
I should make clear that such arguments about the whole picture are important when we consider how to accommodate the research into our existing intellectual models. The research may be true but not significant because it tested the wrong things. But pointing out things that the research did not consider does not invalidate the causal relationship between the two phenomena that the research did test. If people want to argue that the conclusions of the research are not relevant, then they need to start by assuming that the conclusions of the research are true, and argue the case that in spite of being true, they may not matter. That is a different sort of argument to one that says that the research is not valid because it did not take into account all the many facets of our model.
Mike Cameron’s post contains a list of notes and questions which, when I first read them, struck me as somewhat insubstantial musings: Mike thinking aloud. I passed over the opportunity to comment. But in a recent Twitter exchange, Mike told me that he thought the LSE report was “tosh” and cited these questions as justification for what had by then developed into a very robust position. They are also cited by Gerald Haigh as constituting a good reason to question the validity of the research. So I shall start by responding inline to Mikes notes and questions.
1. No data on who owned mobile phones. Uses phone ownership among over 13s as a proxy for 13-16yr olds.
Strictly speaking, data on mobile phone ownership is irrelevant to the conclusions of the study. The study shows a correlation between mobile phone bans and student performance. It does not seek to examine every link in the chain in what Hume called the “secret connexion” between the two phenomena.
It is nevertheless true that if children do not own phones, then the mechanism that the report suggests (distraction in class) would be implausible. The correlation between mobile phone bans and stronger performance would still be valid—but we would have very great trouble in explaining the relationship.
Given our informal observations of the modern world and the fact that schools have been busy banning mobile phones since the early 2000s, it hardly seems plausible to suggest that children simply don’t have them and are not bringing them into school. If that were the case, why should anyone ban them? And if it were true that only a small minority of children were bringing phones into the classroom, the only consequence of that fact on the conclusions of the report would be that those phones that were being brought into class would have to be having a much more significant deleterious influence that one would otherwise assume, in order to account for what is a statistically significant effect across the whole population.
Having said all of that, the issue is explicitly considered by the report, which acknowledges on page 9 that
any impact a school mobile phone ban could have would be tempered if teenagers did not use phones in the first instance.
The report reproduces research into phone ownership undertaken by Ofcom into “individuals 13 years and older”.
Mike’s criticism suggests that it is not justified to assume ownership rates among 13-16 year olds from data based on a survey of people 13 and older (including adults). But this point is also explicitly considered by the report, which cites :
Survey research by the Office of Communications (Ofcom) finds that teenagers in the UK have similar mobile ownership rates as adults since mid-2000s (Ofcom 2011).
A further survey of teenagers in 2005 found that 82% of 12-16 years old owned a mobile phone, being slightly higher than the overall rate of 80% (Ofcom, 2006)
- the extent phone ownership is irrelevant to the finding of a correlation between mobile phone bans and examination results at GCSE—even if it is relevant to the way we choose to apply that finding to our model;
- any reduction of phone ownership among school children would only have the effect of exaggerating the deleterious influence of those phones that were present;
- Mike is incorrect in suggesting that the study does not cite adequate evidence to demonstrate that a large proportion of teenagers own mobile phones;
- which will surely come as a surprise to no-one.
The explanation of this variable is given in the note to Table 2 on page 25:
Head teachers were asked to rate the extent to which the policy is adhered to by students on a seven-point scale with 1 representing “Not at all” and 7 representing “Completely.” A school was considered to have a high compliance ban if the response was greater than four.
In an ideal world, of course, we would have an objective measure of compliance, based perhaps on independent evaluators going into all 91 schools at yearly intervals. Not only would such objective data be impractical to collect but it would not add much to the force of the study. I suggest that the most likely bias introduced by relying on the subjective judgement of Head Teachers is that it will tend to overstate compliance with the mobile bans. If the finding of the report is broadly accurate, the true effect size will then be larger than the one found, given that many of the bans that were reported as having high compliance in fact had low compliance. If the way that schools reported compliance is not accurate, Mike has to explain why there is a correlation between performance and schools who report high-compliance bans, when there is no such statistically significant correlation with the sample as a whole.
This is irrelevant and is another example of what I have called the holistic fallacy. The whole point of taking a large sample is to randomize the contextual influences that you are not testing. We are not interested in context, we want to get rid of it as an influence on the outcome of the report and this is achieved by randomizing it.
We are interested, of course, to check that the sampled schools are representative of the wider population in respect of wider context (in other words, checking that this randomization has been done effectively) and there are numerous passages in the report which are dedicated to checking this point.
The report is aware of this issue and gives a clear definition of “ban” on page 7:
There are a multitude of ways in which schools have restricted phone use, from asking for them to be set on silent to not allowing them on school premises. We define a school as introducing a school ban if that school did not allow them on the premises or required them to be handed in at the start of the day.
It does not test the effect of visibility bans but that is irrelevant to its purpose. Any suggestion that this compromises the reliability of the report would be another example of the holistic fallacy.
This is true—but hardly significant without considering how the sample might be biased as a consequence.
It is plausible that children in the city might have more mobiles—but as discussed above, this would not change the influence of mobile phones on learning, except to the extent that if children do not have mobile phones, a ban would be expected to have a smaller effect.
It might be that the performance of city schools is different from country schools. I address this point in the context of the London Challenge, below.
In other respects, I do not understand how the selection of city schools is supposed to have biased the conclusions regarding the effect of mobiles on learning. It does not strike me as plausible to suggest that the influence of mobile phones on learning will change between the town and the country.
6. Survey sought from 450 schools with 91 respondents – is this all the secondary schools in the four cities?
No. The report makes clear that the research team first sought permission from Local Authorities to approach schools. On page 6 it is stated that:
Before approaching schools, we obtained permission from the relevant Local Authorities. Every secondary school from Local Authorities where permission was granted was then contacted.
And in a note to this passage:
We did not obtain permission from five Local Authorities in London (Hackney, Lewisham, Newham, Redbridge and Tower Hamlets), which combined have 77 secondary schools. The City of London Authority does not contain any public schools and therefore was not approached. The remaining 27 London Local Authorities gave permission, with 337 secondary schools being approached.
It certainly seems that there is an element of self-selection in those schools participating in the survey. This would be a serious criticism if the report were examining the proportion of schools that had introduced bans. But it isn’t. It is examining the effect of those bans on performance, as judged against other schools both nationally and in the same cities, and for this purpose, the proportion of the sample that had introduced bans is not significant.
Although 90 out of the 91 respondents had introduced bans, there was nevertheless a more equal spread of those who had introduced high compliance bans (56) and those who had introduced “low compliance” bans (34).
This is the significant criterion from the point of view of learning and the distinction between the two groups validates the report’s findings by showing that performance improvements were only correlated in schools with “high compliance” bans and not those with “low compliance” bans.
8. Sample schools contain significantly more minority, SEN and FSM -eligible pupils – answers the context question above.
The recent decision to abandon criterion referencing, in the form of levels, illustrates the point that individual criteria or definitions do not necessarily improve the consistent understanding of the intervals on a scale. It is actually quite hard to create short rubrics that define performance criteria reliably. The two ends of the scale (“not at all” and “completely”) can certainly be defined precisely—but the intermediate points are much more challenging. Imagining a smooth progression is probably the most consistent way of defining the intermediate intervals. See also Daisy Christodoulou’s recent arguments on the benefits of comparative marking.
See next point.
11. Paper suggests no upswing in results in these schools in previous years results – given London Challenge this might suggest that these schools are not representative of the whole.
The following graph shows that grade inflation was significant during this period across the whole country but occurred slightly more rapidly in London than elsewhere.
Figure two in the LSE report charts the performance of students at schools with high-compliance bans, charted from 7 years before the ban was introduced to 9 years after.
It would certainly represent a serious challenge to the report if the only reason that the performance of students at the schools with high-compliance bans improved was because the performance of all students was improving, particularly in London where many of the sample schools were located.
But this is not what the data in Figure 2 shows. The Y axis of this graph shows the effect size of the introduction of the ban relative to other schools in the sample. I was not able to establish this from the report itself and so checked with Richard Murphy, one of the authors of the report, who stated to me in an email that:
Figure 2 is showing the plot of the estimated effect size of the ban before and after it was introduced. It is not showing the test scores. This is estimated on the sample of schools that responded to the survey, rather than the whole population over which test scores were standardised. If there was general grade inflation amongst these schools, that is accounted for through the inclusion of year fixed effects in the estimating equation. This allows for the mean achievement in schools to be increasing (or decreasing) over time. [my emphasis]
In other words, the graph shows the improvement in performance of students at schools with high-compliance bans relative to the performance of students at all the schools in the sample, which includes those schools with low-compliance bans or in a single case, no ban at all. To the extent that grades were improving at the sampled schools anyway, either because grades were improving everywhere or because many of the schools were in London, this level of increasing performance was taken as the baseline.
General grade inflation or London-specific effects, such as the London Challenge, therefore cannot be used to account for the improvement in performance shown in the LSE study.
12. Provides no evidence of pre-ban in-school use of phones (though one could take the need to introduce the ban as evidence it was required).
This information is irrelevant, again demonstrating the holistic fallacy. The evidence consists in the correlation between the ban and the improving performance—and not in how we interpreted that evidence. As Mike himself acknowledges, the suggestion that all these schools introduced bans when (contrary to the Ofcom evidence of mobile phone use by teenagers) no-one in those schools was using mobile phones—is pretty far-fetched.
13. “It needs not be the case for an individual to use a phone to be distracted by them, their use by others in the classroom may cause disruptions. ” – evidence for this?
Common sense, even though this point is not material to the conclusion of the report and is therefore another example of the holistic fallacy. The sentence occurs in the same paragraph that deals with the level of mobile ownership among school children (see point 1).
14. “There may be a concern that only high-achieving schools introduce mobile phone bans, which could lead to overestimating the effects of a mobile phone ban.” – I would have assumed the opposite.
See my response to point 8. The evidence of the report strongly suggests that the assumption (made very widely by those trying to condemn the report) that mobile phone bans were introduced in under-performing schools is simply false.
15. How the paper handles the issue of other policy changes is “interesting” – ‘The variable OtherPolicy takes a value of 1 for the years after a change at a school occurs. We combine information coming from our survey of headteachers and information from school’s website. We do not observe multiple change of policies/leader in addition to the phone policy change, hence a binary variable can be used.‘ – So there are either other policy changes or there aren’t, no consideration of what they are.
This again demonstrates the holistic fallacy – the idea that the research is not valid if it does not measure everything that is going on. It shows a failure to appreciate that the evidence is focussed purely on the relationship between the ban and the improving performance.
If there were other significant policy initiatives introduced at the same time, then there is a risk that any change in performance could be attributed to the other policy initiatives and not to the introduction of the ban. The significance of the results from schools that report concurrent major policy initiatives is therefore given reduced weighting. What the other policy initiatives were is irrelevant and the attempt to provide evidence for which of these other policy initiatives might have been more or less successful is way beyond the scope of this survey.
16. The statement – “We add to this by illustrating that a highly multipurpose technology, such as mobile phones, can have a negative impact on productivity through distraction. “ is unevidenced – the results in the paper show that there is an effect. They do not show why the effect occurs.
It is true that this research does not produce evidence for why the effect occurs—but the quoted passage does not claim that it does. They say it “illustrates” a theory that is already widespread in the existing literature.
This passage is quoted from the report’s conclusion, which comments on the significance of the survey findings when placed against the background research. The sentence immediately preceding the one quoted states:
The existing literature on the impact of technology in the classroom implies that the unstructured presence of technology has ambiguous impacts on student achievement.
The context of the quotation therefore clearly establishes that the report is commenting on the significance of this reports conclusions in relation to the existing research. Although no mention is made in this concluding passage to the extent to which that research demonstrates the importance of distraction, this relationship has been covered explicitly on pages 4-5:
Recent experimental papers present evidence that mobile phone use while executing another task decreases learning and task completion (e.g. Ophir et al. (2009); Smith et al. (2011); Levine et al. (2013); and Lee et al. (2014)). The distracting nature of mobile phones has been previously examined in other context such as incidence of road accidents.
Bhargava and Pathania (2013) exploit a pricing discontinuity in call plans and show that there is a large jump in phone use after 9 p.m. This jump, however, is not followed by an increase in car accidents. Using vehicular fatality data from across the United States and standard difference-indifferences techniques, Abouk & Adams (2013) find that texting bans have only a temporary impact on car accident fatalities, suggesting that drivers react to the announcement of a legislation only to return to old habits shortly afterward.
The paper does not claim to show that the negative impact occurs as a result of distraction. But it interprets the correlation between mobile phone bans and improved performance by reference to the effects of distraction, which has been widely evidenced elsewhere. Such an interpretation is at least plausible and, like all scientific theories, will stand until it is proven to be false.
At the end of the piece, Mike summarizes his explanation of the report’s findings.
My experience is this. Schools introduce phone bans in response to an existing issue within the school. Such a ban is rarely introduced as the sole change and consequently any changes need to be seen in that light.
More meat has been put on this theory by others. Another regular contributor to the ICT Research Network said:
Which of course leads on to your question of why the achievement levels in these schools were poor originally [I had said no such thing—I only said that results had improved]. I suggest to you that the schools that decided to ban mobiles are schools where behaviour was poor leading to low results.
The theory is given another outing by José, who argues:
By separating the mobile phone ban from the wider toughening of these schools’ behaviour policies, we may be fooling ourselves into believing that by replicating single aspects of these policies – e.g. enforcing a mobile phone ban – test results will improve. It just doesn’t work like that.
There are a number of problems with this theory, which seems to have been generally adopted.
- It is at best pure supposition—there is no evidence whatsoever of the initial indiscipline or of the raft of other policies that are supposed to have been introduced.
- The author of the ICTRN email assumes that because the performance of a school improves, it must have been poor to start with.
- Most significantly, the idea that the mobile bans were introduced in underperforming schools is flatly contradicted by the evidence of the LSE report. Although the report states that
the sample schools have a significantly more disadvantaged population than other schools in the cities and nationally, enrolling more minority, SEN and FSM -eligible pupils
…it also states that…
Comparing standardized age 16 test scores, we see…that the schools in our sample over the whole period achieve significantly higher scores than other schools within these cities (0.07σ)…[even though] the sampled schools have an even lower intake quality (-0.09σ) [than other schools in the same cities].
Even before the introduction of mobile phone bans, these schools were performing better, on average, than other schools in the same cities on value-add measures. These were over-performing schools not under-performing schools. The supposition that they were making all sorts of other unusual interventions in order to deal with serious behaviour problems becomes even more implausible.
4. The report specifically reduces the weighting given to schools that report introducing significant initiatives or who experienced changes of leadership at the same time as introducing the mobile phone bans (see point 15 above).
I have examined Mike’s questions in painstaking detail because, although they are presented as an exercise in thinking aloud, they have subsequently been put forward as a rationale for why the LSE report is “tosh”.
The truth is that none of these supposed criticisms of the report carry any force whatsoever. In my book, which follows the principles of the scientific method, subjecting something to a really hostile grilling and finding that it comes away totally unscathed is the strongest evidence you can ever find for its validity.
Others in the edtech community have picked up the general hue and cry against the LSE paper, often with much less thought than Mike.
An email to the ICTRN reflector draws attention to a sentence in a Guardian report of the paper:
“Murphy, an assistant professor of economics at the University of Texas in Austin, said the study did not take into account any positive effects for pupils using mobile phones for research”. No bias there then!
It is incidental to this comment that the author clearly believes that the ability to use mobile phones for in-class research offers a significant benefit, when the findings of the OECD report cast serious doubt on this position.
The main purpose of the comment is to suggest that the report is biased because one of the authors works in Texas rather than London, is an Economist rather than an Educationist, and/or because the report does not specifically consider something that would be completely irrelevant to the study being conducted—another example of the holistic fallacy. It then wraps all this false reasoning in a pernicious ad hominem attack on the integrity of the report’s authors. Yet this unworthy attack has been widely repeated on Twitter by prominent members of the the edtech community.
Gerald Haigh cites Mike’s questions as “16 searching questions about the detail of the LSE paper” in justification for the position that much of the media reporting has been “lamentably ill-informed”. In fact, the newspaper headlines are on the whole accurate and it is the critics from the edtech community who are, on the whole, “lamentably ill-informed”.
José Picardo invites readers to:
Delve into hoo-ha further and it appears that what these schools have done is to toughen their behaviour policies generally, with the ban on mobile phones being a part of it
This is the same theory that appears to be based on Mike’s musings but which (as I have demonstrated above) is supported by no evidence at all and is flatly contradicted by the report.
Having shown his failure to understand what a correlation is, José quotes a Tweet by Carl Hendrick, a teacher at Wellington College, who refers to “the undoubted usefulness in class” of mobile phones. For evidence of this position, José cites one of his other blogs and a couple of inspection reports on individual schools.
The Tweet cites no evidence at all but appears to be based on the theory that, if you say it with confidence, it must be true. Of the two inspection reports, the first reports on José’s own school, Surbiton High. This makes a fairly tangential reference to digital technology in the classroom, saying that
Excellent use of available resources, including digital media, enables interest to be stimulated and pupils’ learning to be enhanced,
Are highly proficient in the use of information and communication technology as a research tool,
The school uses a range of digital media to provide real-time updates about school life.
The second report into the Stephen Perse Foundation, has numerous references to the fact that:
The move towards digital learning…is central to the Foundation’s vision,
…concluding in its main findings that
The excellent teaching, using a wide variety of teaching methods combined with extensive use of digital technology, fosters an enquiring and creative learning environment in which pupils thrive.
These are both positive reports. However, it has to be said that we have learnt to be very sceptical of inspection reports as a source of evidence. Many reports by Ofsted have been shown to support classroom pedagogies such as group-work and learning styles, which are not supported by any serious research evidence. All the inspection report shows is the opinions of the inspector.
It is also significant that both these reports are written not by Ofsted but by the Independent Schools Inspectorate—because these both refer to top independent schools. As is Wellington College, where Carl Hendrick works. At my recent talk at ResearchEd14, a well-argued contribution was made from the audience by a floor by a teacher who worked at Radley, arguing for the usefulness of mobile devices at his school.
Stephen Perse, Surbiton High, Wellington, Radley. The fact that much of the anecdotal evidence for the beneficial use of mobiles seems to be emerging from top private schools is consistent with the LSE report, not contrary to it. As the report found that learners in the top 20% of prior attainment did not experience any negative effects from mobile phones, it is surely plausible to imagine that those in the top 5%, attending schools in the top 1%, may even experience positive effects. It does not follow that these same effects can be transferred to mainstream state education, into contexts in which the teacher-ratios and the attitudes and prior attainment of the pupils are often very different from those found in the top independent schools.
I shall return later to the relationship between technology and the quality of teaching—but the key point is one that I have frequently made on this blog before. We know what good teaching looks like and we can do it well enough for the few. The problem with our education system is scale. But if you are interested on the effect of mobile phones on the average state school, then you have to look at the evidence of what happens when you introduce mobile phones to the average state school—not what happens when you introduce mobiles phones in Surbiton High and Wellington College.
The only broader quantitative evidence cited by José is to another of José ’s blogs which states that
research suggests that there is a strong correlation between the effective use of technology and improved outcomes.
The link in this passage is to the EEF’s review of the Impact of Digital Technology on Learning. If you search this publication for the word “correlation”, the only substantive result states:
Taken together, the correlational and experimental evidence does not offer a convincing case for the general impact of digital technology on learning outcomes.
José concludes his post by suggesting that the LSE report is “cargo cult science”. Judging by the quality of the research evidence offered, it is José not the LSE report that is offering up cargo cult science.
The OECD report has not attracted the sort of methodological criticism that has been directed at the LSE report but it has been sidelined by a range of arguments about interpretation that could be applied equally to both reports.
That, at least, was what I was always told in the school changing rooms. The truth is rather different: as I have argued on this blog (ad nauseam, I’m afraid—but here and here for starters): it’s the technology that really matters.
In the context of these reports, the argument is that you cannot assess the effect of BYOD unless you also look at the quality of the teaching in the classrooms in which BYOD is deployed. The key point that José emphasises in his article is that
the use of mobile devices to support teaching and learning and robust behaviour policies are not mutually exclusive concepts
Digital technology works when backed up by good teaching and robust behaviour policies. Carl Hendrick follows up his first tweet by arguing that
Another way of interpreting this research is that we need better ways of incorporating it not an outright ban
Mike Cameron concludes his blog by commenting that people often misinterpret the effect of research by listening to people who say
“We do A, B, C, D, E, F and a little bit of G.” After which the visitors go back to their own school and vigorously implement lots of D and expect their school to become great as a result.
Among the A, B, C, E, F and G that people fail to transfer, it is clear that what really matters is really good teaching. As José also makes the argument:
this research completely ignores how mobile devices are being used effectively
…referring also in his second blog post to
the Sutton Trust’s most recent report on what makes great teaching…accompanied by ways in which mobile technology might, on occasion, help along the way.
The problem with this argument is that it tries to make out that what really matters is the use of technology from evidence that suggests what really matters is good teaching. If you take four types of school, those to do and do not make extensive use of technology and those who do and do not have good standards of teaching, the current research shows the incidence of improved outcomes coincides with the prevalence of good teaching, not technology use.
You cannot argue that technology creates good outcomes by using anecdotal evidence of schools drawn from the top right hand quadrant (good teaching in technology rich environments)—you have to show that schools in the top right hand quadrant achieve better outcomes than schools in the bottom right hand quadrant; or that schools in the top left hand quadrant achieve better outcomes than schools in the bottom left hand quadrant. If this evidence exists anywhere, I should be grateful if someone could draw it to my attention.
It might be possible to argue that the introduction of technology encourages better teaching by showing that improving schools tend to move clockwise from the bottom left, through the top-left to the top-right quadrants (a process representing tech-led improvement); and not anti-clockwise, from the bottom-left through the bottom-right to the top left quadrants (training-led improvement). But again, I have not seen any evidence to suggest that this is what tends to happen.
On the contrary, the general complaint one sees from education technologists is that their technology doesn’t work in practice because good training, a prerequisite for technology, has not been put in place first. So technologists themselves seem to assume that improvement is training-led. See, for example, Mark Anderson’s article, My response to everyone – Don’t bin IT or ban IT, get IT right!
It’s not the fault of the many teachers who choose to not use technology. As mentioned many times previously, it’s a problem of confidence. A problem of competence. A problem of training.
If it is the confidence, competence and training that matters (all issues which are extremely hard to change at scale), then what is the evidence that the technology itself is making any difference at all? Why not do the training and forget about the tech?
José Picardo’s fond hope that mobile technology “might, on occasion, help along the way” is an admirably tentative position but it does absolutely nothing to counter robust, quantitative evidence which shows that, as generally applied in mainstream education, mobile technology doesn’t help along the way—not for most teachers in most schools. From a policy point of view, it is what helps most teachers in most schools that counts.
Having taken some poorly aimed shots at the methodology of the LSE report, the detractors of both the LSE and the OECD reports generally come back to the assertion that it is irrelevant because what matters is curriculum not pedagogy. As Gerald Haigh puts the argument:
Rumbling distantly at the heart of this debate is something much more profound – a search for the real purpose of education, and a voice which says, in effect: “What if banning phones does improve GCSE results? Is that the only measure that matters?”
He then quotes Graham Brown-Martin’s sarcastic comment on the Guardian article:
Well that proves it then – banning 21st century technology improves 19th century school exams. Who knew?
José Picardo slips the same argument into a number of asides to his main argument:
Leaving to one side my concern about the belief that whatever improves results in high-stakes testing must be good for children…
Damning…the use of mobile devices as “a distraction” betrays a narrow view of what constitutes a good education…
A more thorough treatment of the argument is provided by Oliver Quinlan in his post Ignore the headlines – computers in classrooms are a good idea, and here’s why. I respond to Oliver’s argument below with a series of inline comments.
There is a tendency in discussions of education to refer to ‘learning outcomes’ and ‘results’ as if we have a universally accepted view of what these are. A fundamental challenge of education systems is the agreement with broad and uncontestable aims such as ‘preparing young people for the future’.
This is an important point, which I have also raised in my post at How technology will revolutionize research—although I would suggest “learning objectives” as a more accurate term for what we are really talking about than “learning outcomes”. Oliver continues:
It seems likely many teachers are implementing technology with the intention of achieving quite different ‘learning outcomes’ than those the OECD measure.
And Oliver, siding with those who argue for new definitions of curriculum objectives, brings the argument to its culmination:
Regardless of any potential benefits to skills in Reading, Maths and Science, learning about technology is in itself a reason for its use.
Only it isn’t—Oliver doesn’t quite manage to ground the ball for a try. The fact that I might want to learn about nuclear weapons isn’t a reason for using them. Turning the same relationship on its head, the fact that I drive to work in a car doesn’t mean I know anything at all about the internal combustion engine. So I don’t necessarily have to use something in order to learn about it, nor do I necessarily learn about it by using it.
The only reason to use digital technology in the classroom is because it benefits my learning, whether that learning is about reading, maths and science, or whether that learning is about technology itself.
We might assume, it is true, that learning about technology will involve some use of technology. But if we are to make progress in describing our learning objectives so that we can pursue them more consistently and more systematically, then we need to analyse very high-level and vague terms like “preparing young people for the future” into their more precise, constituent parts. As I have argued in my previous post, How technology will revolutionize research, this is fundamental to any effective application of education technology.
If we are going to let mobile phones into the classroom in order that we may better teach students about technology, then we need to describe precisely what our technology-related learning objectives are. To say that we should use technology in order that students should learn about technology and then, when challenged to describe exactly what they should learn, reply whatever they happen to pick up from the experience of using technology—then our argument is a circular one: we should have technology in the classroom merely for the sake of having technology in the classroom.
Oliver recognises that the Computing curriculum has just undergone a major overhaul.
In England, the relevant subject has recently been refocused to include understanding of computer science, programming and digital literacy.
The nature of this new Computing curriculum that has been introduced by the government is, broadly speaking, more academic than the old ICT. Computer Science is heavily based on Mathematics, not chatting to your friends on Facebook or looking up a random selection of factoids on Google. It is not at all clear to me that these learning objectives will be well served by having mobile phones in the technology classroom, any more than having them in the English classroom.
A second point which flows from the review of the Computing curriculum is that fundamental curriculum decisions are not for teachers to make according to their own lights. Broadly speaking, the job of the teacher is to teach effectively against educational objectives set by governments and exam boards and represented by high-stakes assessment.
If those objectives are often poorly described, then this represents an opportunity to apply technology effectively to describing our educational objectives more clearly (see my How technology will revolutionize research, section 5). If our assessments are not good at representing softer skills like creativity, teamwork and problem-solving, this is another opportunity to apply technology to improving our assessments. But neither of these arguments justifies teachers in ignoring the educational objectives that users and other interested educational stakeholders require the education service to address, in order to pursue their own personal curricula. It is a very poor reason to dismiss PISA’s assessments of literacy and numeracy to argue that many teachers in the UK may not be that bothered to teach literacy and numeracy.
Even when you look at the digital literacy, the educational objective most closely aligned with BYOD, the evidence of the OECD report that digital literacy is heavily correlated with traditional literacy and requires only moderate computer use.
The fact that today’s young people need to learn new skills does not mean that the old skills are no longer relevant; and even if those skills can best be taught by the use of digital devices in particular circumstances, this does not amount to an argument for allowing digital devices in classrooms more generally, where their use cannot be shown to promote learning against the objectives that are appropriate to that particular lesson.
The argument that digital devices should be allowed in every classroom by default because our children need to learn new digital skills is deeply flawed. Its proponents have failed to provide a coherent description of the new “21st century skills” that they suggest are so important; they fail to recognise that the definition of such curriculum objectives is not down to the classroom teacher, who is constrained by the educational objectives set by parents, exam boards and governments; they fail to acknowledge the vital importance of traditional skills, which are underlined by the findings of the OECD report; and they fail to acknowledge that the presence of technology in the classroom ultimately always comes down to pedagogy, not curriculum.
Some have dismissed the LSE and OECD reports on the basis of the weight of evidence that is said to contradict their findings. Professor Margaret Cox of the Institute of Education commented on the report of the OECD report on the BBC website:
The OECD report is flawed. Today, the 4th conference of 100 international experts in research, policy and practice in the use of Computers in Schools: Edusummit2015 in Bangkok (Sept 14-15th) sponsored by UNESCO shows that 40+ years of research from over 65 countries has shown that appropriate uses of computers and other IT technologies can enhance students’ learning.
But what exactly is it that shows the OECD report to be flawed? The fact that 100 academics have gone to Bankok for a jamboree? Or is it the 40+ years of research? Research is not measured in inches or years: it is measured in the strength of its arguments—and no reference is made in this comment to any argument or solid evidence at all.
Donald Clarke takes this same argument to another level in a tweet:
Irrelevant and lazy journalism, we have our own research and reports
This portrays a relativist position in which people produce research papers, like weapons, to defend their pre-existing beliefs, and not as methods of discovering truths which are valid for everyone. Dodge the enemy bullets and fire back as quick as you can with your own. It explains much about the failure of the edtech community to engage with arguments that do not suit it.
In fact, almost all of the research that can be found in the academic journals is based on small-scale studies that rely heavily on attitudinal surveys—the sort of research that is summed up in this sort of infographic:
Some teachers may be very keen on technology and many clearly think that it is improving their teaching. In isolated cases, they may be right. But equally, they may be wrong. Educational research is blighted by the Hawthorne and Pygmalion effects, where respondents tend to tell researchers what the respondents think the researchers want to hear. The views of teachers on what works in the classroom is extremely unreliable, as I have covered in Why teachers don’t know best. This means that research based on attitudinal surveys should be treated with scepticism when it is presented as robust evidence. The Tooley Report (search for “That the research has been done” in the link above), made a strong case that the quality of most academic research is extremely poor and that little reliance can be placed on a great deal of it. If the last 40 years of research has produced strong arguments, then give us the links and let us see whether it amounts to any more than a row of beans.
The arguments that have been made for ignoring research that does not suit us show a lack of intellectual integrity—which is a shame because it is made by people who would have much to contribute to an honest and constructive debate.
Mark Anderson (@ICTEvangelist) wrote a blog urging us: Don’t bin IT or ban IT—get IT right! I couldn’t agree more. José Picardo wrote a follow-up article in the TES, Headache Tablets, reprinted on his blog, in which he complained that the debate over technology had become polarized, with everyone being characterized either as “crazy-eyed zealot or a complete and utter Luddite”. Again, I couldn’t agree more.
Both Mark and José urge us to find the middle way. But finding the middle way is difficult: it is not a case of just splitting the Mars bar in half. How do we get IT right? What is the middle way? In these respects, I think that Mark and José and most of the rest of the edtech community have got it wrong.
Their first mistake is to denounce what in the case of both LSE and OECD reports is rigorous and well-founded research. They will never get their response right so long as they close their eyes to the evidence that they find inconvenient.
Not only is their denunciation of this research unjustified, it is also imprudent. Because neither report advocates blanket bans or denies the potential for education technology to have a profound effect on education. In the OECD report, Andreas Schleicher writes:
Technology is the only way to dramatically expand access to knowledge. To deliver on the promises technology holds, countries need to invest more effectively and ensure that teachers are at the forefront of designing and implementing this change”.
While I might disagree with Schleicher about the role of teachers in leading this change, this is hardly the voice of someone trying to consign technology to the garbage bin. Similarly, the LSE report does not urge us to bin education technology but rather to make better use of it:
The common theme in these education papers is that the mere introduction of technology [i.e. hardware] has a negligible impact on student test scores, but when incorporated into the curriculum and being put to a well-defined use, technology has the potential to improve student outcomes.
This point is reiterated in the conclusion:
These findings do not discount the possibility that mobile phones could be a useful learning tool if their use is properly structured. Our findings suggest that the presence of mobile phones in schools should not be ignored.
Part of the confusion of those who are hostile to the report is to think that because the LSE report evaluated the effect of mobile phones on learning by studying mobile phone bans, they are therefore advocating mobile phone bans. At no point does the report recommend banning mobile phones: it merely says that the presence in the classroom of mobile phones (and their potential to have negative impacts) should not be ignored.
Tom Bennett, UK education’s new Behaviour Tsar, has been asked by the DfE to draw up a set of recommendations on mobile devices. His initial comments suggest that he is not minded to recommend a blanket ban, but might urge schools to consider restricting mobile phone use for the under 16s. Such a general recommendation would recognise, in a broad-brush way, the LSE finding that mobile phones are unlikely to do any harm to well motivated and skilled learners.
So Tom’s approach strikes me as a common-sense one: let individual schools decide on the basis of the evidence and their own context. And I do not see that it differs so much even from the position of José Picardo, who says that in his own school:
though children are given a tablet, the use of mobile phones is generally not allowed unless a teacher specifically allows it,
continuing to ask for more research which assesses the exam performance of schools with bans
in comparison with other schools that do allow their [mobile phone] use on occasion.
So even those who are the most vociferous in opposing bans in fact support them in some circumstances. There is clearly plenty of middle ground for sensible people to occupy here—and this is exactly the sort of “heads up” approach to the issue that the LSE report is also advocating.
The polarisation of this debate has not come either from the LSE or the OECD reports, but from those in the edtech community who argue (in the words of the recent ETAG report), that
The use of digital technology in education is not optional
…and that all Schools, Colleges and Universities should adopt BYO policies.
This is the opposite of an informed, evidence-based approach. The dogmatism and the consequent polarisation of the debate has been coming from certain sections of the edtech community and not from those who are conducting quantitative research.
Many schools may decide against bans on grounds other than ones of pure pedagogy. A ban might be seen as too hard to enforce; undermining the relationship of trust that the school wants to build with its students; or important for home-school communications. But these decisions should be made on the basis of a careful weighing of the evidence and not because of a poorly justified view that electronic devices are an essential component of every 21st century classroom.
It is essential that we stop conflating arguments about the effectiveness of technology with arguments about what we should teach. They are separate issues.
It is not for teachers to decide what they want to teach in their classrooms—they are employed to teach towards the educational objectives that are set by exam boards and governments and encoded within their high-stakes exams. These formal educational objectives may certainly be supplemented by teaching other, softer skills and attitudes, which parents are likely to value highly when making their choice of schools. As I have argued elsewhere, we need to make more progress in describing these soft skills more clearly, in ways that can ensure more consistent teaching across the education service. By all means, let us campaign for better summative assessment, that does more to test creativity and genuine problem solving. But that doesn’t mean that teachers have the right to turn their backs on the requirement to teach the subjects that are tested in formal examinations, preferring to devote class time to their own private curricula. In this regard, I am sympathetic to the arguments made recently by Dame Sally Coates for a more centralized curriculum—a project which I believe will not be achievable unless we can use data analytics systems to support our description of educational objectives more clearly (See my recent Setting the Curriculum).
If people want to justify the more extensive use of digital technology in terms of the new skills that students will acquire in this way, then they have a great deal of work to do in order to construct a convincing argument. There has been a strong reaction—amongst employers, Computer Science faculties and government—against the old “dull and boring” ICT curriculum. The arguments made by people like Sir Ken Robinson for new sorts of 21st Century Skills are deeply flawed and implausible (see my article rebutting Sir Ken, a guest article on my website by Scott Goodman, or Tom Bennett’s more recent piece). And the new evidence of the OECD report suggests that the most important prerequisites for operating effectively online are traditional literacy skills (if that wasn’t blindingly obvious to everyone in the first place).
The arguments for 21st Century Skills have bombed: no-one outside a small clique of acolytes takes them seriously. Stop pretending that digital technology has completely transformed the nature of academic subjects across the curriculum: they haven’t. Finding the middle way does not mean that we have to compromise with nonsense.
Instead of denying the blindingly obvious or continuing to advocate discredited curricula, proponents of edtech should be welcoming these reports and responding to their challenge to produce more structured uses of edtech, better integrated into the practice of teachers, which can be shown by quantitative research to be effective as a pedagogical tool when used to teach traditional, non-technical educational objectives—the sort of objective that the rest of the world still values. The sort of objective that the OECD report shows that they are right to value, even in the digital world.
I agree with José Picardo, writing in Headache Tablets, that it is vital that teachers are able to take control of the technology if they are to integrate it successfully into their teaching (which remains vital, of course).
I think I disagree with José Picardo, Andreas Schleicher and Mark Anderson that this means that it is for teachers to lead change, if by leading change it is meant that teachers should effect change. Teachers do not have the necessary understanding of technology, nor the necessary resources to effect the sort of change that is required. The reason why mobile devices are failing to deliver benefits in the classroom is not just because of the negatives (such as distraction) but also because of the absence of positives. We have failed to develop compelling educational-specific software that will make the difference. Such software must be subject to teacher control—to the ability to assign and track work. But this requirement cannot be met by training or by creating new school policy documents: it has to be met by the development of classroom management and software along the lines that I describe in How technology will revolutionize research, section 4.5.—and this has to be done by industry, not by teachers.
There is a single purpose in using an iPad in any situation at all. That purpose is to run software. The problem that teachers face is that they lack the software that is appropriate to their requirements. While I agree with José that schools are “doing technology wrong”, I disagree that this means that teachers have the means to rescue themselves.
I understand if at this stage, having struggled through nearly 15,000 words, teachers reading this article might feel a little cheated. Not only am I suggesting that edtech (as currently implemented) does not work, but also that there is nothing that they can do about it.
Well—I think there is something that teachers can do about it. David Steel will be remembered for telling the Liberal Party to go back to their consituencies and prepare for government. He now wishes he didn’t. What teachers should not do is to go back to their classrooms and hand out the iPads—not at least in the expectation that it will have any real impact. What it is very important that teachers do is to start to contribute to making a compelling argument to Government about what Government should be doing to help bring about the sort of edtech that will genuinely help. I amnot saying that Government will effect the change any more than teachers will effect the change: but it is Government that needs to take the lead and to create the environment in which we teachers can be given access to the right sort of edtech. Structured edtech. Edtech focused on delivering recognised educational objectives. Edtech that can demonstrate its value in quantitative trials. Edtech that works.
When teachers have access to that sort of edtech, then they will be able to lead change in their classrooms and their schools—because then they will have in their hands the tools of the trade that any profession requires if it is to do a good job, faced with the demands and expectations of the modern world.
This sort of reasoned argument for pragmatic action is exactly what the Government asked the ETAG group to provide last year—and the ETAG group flunked what ought to have been a golden opportunity. The door is still open, the requirement is still there, the edtech community still has an opportunity to respond to the challenge—but it will only be taken seriously if it stops talking nonsense for most of the time. It cannot turn its back on robust research. It has to engage with those with whom it disagrees. Only in this way will it build a credible argument that will be heard by Ministers.
If this is what José Picardo, Andreas Schleicher and Mark Anderson mean by teachers leading change, then I withdraw my objection. The tech matters and teachers cannot create the tech that we need—but they can define the requirement, make the argument, and ultimately it will be the teachers that select and manage the tech that is produced. If we are ever to see successful education technology, it will be about empowering teachers, not replacing them. And if teachers are to play a role in leading that development, leading in the sense that I have used the word, their first step must be to engage constructively with the findings of what are two robust and helpful reports.