The widespread criticisms of the recent LSE and OECD reports are poorly founded and do nothing but undermine the argument for serious edtech
Those of us who complain that edtech gets too little attention in the national press should perhaps have been beware of what we wished for. Two recent studies to have hit the headlines both say that the general impact of technology on learning is negative.
Advocates of technology enhanced learning have tended to brush this research aside. Various online commentators have said that the reports were “flawed”, “confused” and “tosh”, and those who reported them were guilty of “lazy and irrelevant journalism”.
The two reports are:
Communication: Technology, Distraction & Student Performance
- written by Louis Philippe Beland and Richard Murphy
- published by the Centre for Economic Performance at the London School of Economics
- in May 2015
- which I will referred to as “the LSE report”.
Students, Computers and Learning—Making the Connection
- published by the OECD under the direction of Andreas Schleicher
- in September 2015
- which I will refer to as “the OECD report”.
In this post I will:
- review what the reports say and why they said it,
- assess the arguments put forward for dismissing them,
- and suggest some conclusions to take away.
At over 14,000 words, I cannot pretend that this is anything other than a long article—but the length is unavoidable given the nature of the topic. What is more, I believe that the reaction of the edtech community to these reports is sufficiently important to justify the effort of writing it. I thank you in advance for making the effort to read it and hope that by the time you finish, you will also think that it was worthwhile.
Contents
If you want to link to any section header, right click on it and select “Copy link address”.
What the reports say
The LSE report
The LSE report was based on a survey of schools that were asked whether they had implemented a ban on mobile devices in the last 13 years. The results of this survey were cross-referenced against data on student performance in terminal exams.
The researchers contacted all the Local Authorities in Birmingham, Leicester, London and Manchester and then approached all the schools in those local authorities that gave their permission. 91 schools (21% of those approached) responded. Schools were asked whether they had introduced a ban on mobile phones over the last 13 years and, if so, how rigorously it had been enforced.
This information on mobile phone bans was cross referenced against the achievements in terminal exams of the 130,000 students who had attended those 91 schools over the same period, as recorded in the National Pupil Database between 2001 and 2011.
Part of the strength of the study comes from the fact that it includes both a lateral and a longitudinal methodologies: the change in performance of individual schools can be monitored as they introduced bans, as well as against the performance of similar types of school (with or without robust mobile bans).
The report found that, overall, the introduction of mobile phone bans correlated with an improved performance of 5.67% of a standard deviation (or an effect size of 0.06). It should be noted that this is a small effect size: the received wisdom is that in the social sciences, an effect size of:
- 1 is “small”,
- 3 is “medium”
- 5 is “large”.
However, this generic scale might be a little too demanding when applied to education, as Dylan Wiliam notes in his ResearchEd 2014 keynote (at 21:40) that:
An effect size of 0.3, in a secondary school, would be doubling the speed of student learning. I think doubling the speed of student learning is worth having.
Students were divided into 5 quintiles by their prior attainment in national exams. The report found that the lowest 20% of students improved by 14.23% of a standard deviation (an effect size of 0.14), while the top 20% of students were neither positively nor negatively affected by any ban.
Hence, the overall finding of the report was that allowing the use of mobile phones had no effect on the strongest students but did have a significant negative effect on the weakest students.
The report noted that these findings should not be taken to imply that there might not be positive uses for mobile phones. It mentions in particular a study in which parents were informed of students’ homework assignments by text message, a practice that was associated with higher test scores (though it should be noted that this practice did not require any mobile phones to be brought into school).
The OECD report
Overview
While the LSE report is a skinny 43 pages, the OECD comes in at a heftier 200. Instead of being a single focused study, it attempts to review the effect of technology on learning, using all the evidence from the 2012 PISA tests.
In 2012, the PISA tests were administered in 65 countries (not including the UK in that year). The core tests were supplemented by an optional ICT Familiarity Questionnaire, which was distributed to students in a total of 42 countries. The study compares the performance statistics achieved on the core PISA tests against the answers given in the familiarity tests. In addition, trends in ICT familiarity were available in the 28 countries that had also administered the ICT questionnaire in 2009.
Given that the OECD report is more of a baggy monster than the LSE report, there are a number of different conclusions.
Relating technology and achievement
The report found that top performance in the PISA tests was achieved by countries that reported less than average computer use. In the case of maths, the best performances on paper-based mathematics were achieved by students who made the very lowest use of computers in school, so the relationship between technology and achievement was always negative, though the report notes that this inverse relationship was reversed in Belgium, Denmark and Norway, where there was a positive correlation between the use of technology and performance in Maths.
In the case of reading, the best performance was reported by students who made just slightly less than average use of computers. The three graphs below show performance mapped against use of computers
- in school;
- out of school for schoolwork;
- out of school for leisure.
The generally anti-tech conclusion was re-enforced by correlating the amount of improvement between 2003 and 2012 made by different countries in their average PISA scores against computer use in school, as reported in the survey. The mapping produces a broad scatter of results—but with a clear trend-line suggesting an inverse relationship.
This relationship is investigated in more detail with reference to reading ability, mapping the frequency of different sorts of computer use against performance in the tests for “digital reading” (a skill in which you would have thought that familiarity with computers was important).
The first graph looks at the frequency of the use of computers for schoolwork. This suggests that the lack of any access to the internet for the purposes of browsing for information was correlated with a sharp reduction in performance. The lack of access to computers as a productivity tool (“doing homework on the computer”) was also damaging. On the other hand, not using computers for social networking did not appear to have any negative effects on performance.
At the same time, the frequent use of the computer (i.e. every day) was always correlated with poorer performance. So we can conclude that the best performances are correlated with moderate use of computers for information browsing and word-processing, while avoiding social networking altogether.
The second graph looks shows the same performance in digital reading compared to the frequency of the use of computers for leisure activities. This shows only a slight drop in performance when computers were used in this way “every day”. The counter-intuitive conclusion from a comparison between the two graphs might be that students do better if they use computers for leisure than for learning. But one should bear in mind that these graphs do not show the total amount of time used for leisure and schoolwork. They hint, rather, at the proportion of schoolwork and leisure time respectively that is spent on computers. My guess is that very frequent computer use for schoolwork might suggest an opportunity cost, replacing more valuable forms of instruction by:
- an over-reliance on collecting information or chatting online rather than useful practice,
- and/or a corresponding lack of engagement from a human teacher.
Conversely, the use of computers during leisure time does not imply any reduction in more useful forms of instruction or study.
So long as the use of the computer for leisure does not unduly extend the total amount of time spent on leisure (see however, the Damaging effects of overuse, below), then it should not interfere with schoolwork.
Looking in more detail at the different uses made of computers during leisure time, sending emails and browsing the internet (both potentially more serious uses) tend to correlate with higher performance that downloading films, music and software, or playing collaborative games.
Indeed, playing collaborative games is the only online leisure activity that is shown to have a wholly detrimental effect on learning—a finding that will doubtless surprise those (including the government’s own e-skills champion, Ian Livingstone and the people at Learning Without Frontiers), who have argued that the use of commercial games in school is a really good way to improve educational outcomes.
The digital divide
The OECD report suggests that the digital divide is not a serious issue for education, for two reasons:
- because the gap in access to technology at a basic level between poorer and richer students is closing rapidly;
- because in spite of this greater equality in access to basic digital tools, the gap in educational achievement remains just as great as ever (as one would expect, given the lack of correlation between technology use and useful educational outcomes).
This finding confirms what many of us have been arguing for a long time, that government projects like Becta’s Home Access project, that focused on handing out laptops to poorer students, were a waste of time and money.
How computers are used for learning
The report contains a useful summary of how computers are used in school.
I have simplified OECD’s 9 categories into 5 and represented them on the chart below.
| 49.1% | Browse the internet for schoolwork | Research | |
| 22.6% | Download, upload or browse material on the school website | ||
| Administration | |||
| 15.2% | Post work on the school website | ||
| 25.0% | Use email at school | ||
| Social | |||
| 21.1% | Chat online at school | ||
| 27.3% | Use school computers for group work and communication with other students | ||
| 25.9% | Do individual homework on a school computer | Productivity | |
| 21.4% | Practice and drilling, such as for foreign language learning or mathematics | Courseware | |
| 13.2% | Play simulations at school |
What most advocates of education technology would call “internet research” represents by far the most frequent use of computers in schools—about twice as frequent as any other use. The second most frequent use of computers was for social networking (email, chatting online, working in groups).
I have repeatedly made the case on this blog that both these uses of computers tend to be ineffective in formal settings. And while the OECD report suggests that not having access to the internet as an information resource is damaging, it confirms
- that an over-reliance on the internet as an information resource is not helpful;
- that the use of the internet for social networking has not been shown to have any educational value at all.
The two most frequent uses of computers in schools are probably the least educationally valuable.
Categories 3 and 4 (use of computers as a productivity tool and for administrative purposes) I regard as uncontroversial. The value of productivity tools will depend on the tool, but I don’t think that anyone would get exercised about using word processors for writing up essays or the internet for checking on the dates of half term. Both are useful but no-one is claiming that either are likely to be transformational.
The final category of computer use, for drill, practice and simulation, represents the sort of education-specific software for which I have argued in this blog. And it represents the least common use of computers in schools. Even within the category, “drill and practice” (which I suspect represents the use of simplistic multiple choice) is much more common that “simulations”, which is the only sort of activity that requires any significant development of education-specific software.
In short, the use of computers in school for “looking things up on the internet” is about four times as frequent as the use of computers for subject-specific simulations (and even this statistic does not tell us anything about the quality of the simulations being used or the quality of assignment, sequencing, tasking and data analysis that contribute to the effective use of such software in formal education).
These results support my contention that the reason why computers are having so little impact on educational outcomes is that they are being used for the wrong things.
Uses and abuses of presentations
I picked out one graph that seemed to me to summarise what will strike many as a piece of common sense—that using technology to assist presentation is only of any use when the presentation is graphical. When it comes to the actual doing something, technology should be used to support student activity in preference to teacher demonstration.
In all cases, high performance correlated most strongly with not using any technology at all (this reflects the inverse relationship reported above for computer use in maths). But where technology was used, it correlated with the weakest performance where the teacher used the technology to demonstrate the entering of data on a spreadsheet or how to calculate with numbers. In both these cases, performance was higher when the students were able to use the technology themselves. In the other two cases, drawing the graph of a function and constructing geometrical figures (both of which are visual), the use of technology as a demonstration medium was more effective that letting the students engage with the technology directly.
You might argue that the student needs to have a method demonstrated before they can be expected to implement it successfully themselves. The opposition of “teacher demonstration” to “student practice” may be a false one in many circumstances. This might suggest another conclusion that might be drawn about the government’s imposition of interactive whiteboards in the mid 2000s: it might have worked much better than it did if the whiteboards could have been used to demonstrate in plenary contexts interactive content which students were also able to use themselves in individual study time. The trouble is that this sort of interactive content was never created, and no-one put in place the open interoperability standards that would allow such content to be controlled from the proprietary clickers that many of the whiteboard suppliers provided to support interaction in the plenary context.
The same conclusion could be applied to the use of video online, as with services like the Khan Academy. The demonstration alone is of little use, unless it is backed up with interactive practice of a sort that can engage the student.
Aptitude at navigating digital texts
Finally, the report takes a deep dive into assessing students’ aptitude with digital text. What it finds to be particularly important is the ability to navigate hypertext: a skill that in turn depends on:
assessing the credibility of sources and predicting the likely content of a series of unseen screens, based on hints such as the explicit name assigned to a link, the surrounding text, and the URL that appears by hovering over the link with a mouse. They also include organisational and spatial skills, such as the ability to construct a mental representation of the structure of a website in order to move confidently across the different pages of which it is composed.
This issue plays to what proponents of technology-enhanced independent learning tend to call “digital literacy”: those basic skills that students need to operate in the modern world and to support their learning across the curriculum.
The OECD report tracked student hypertext movements through a website as they searched for the information required to solve a particular task. They then quantified:
- the length of navigation route followed, or quantity of navigation (suggestive of perseverance);
- the relevance & efficiency of the navigation route followed, or quality of navigation.
High quality navigation is characterised by students who take no mis-steps in following the optimal navigation sequence to solve a task, or who quickly correct any mis-steps which are taken.
On the whole, ability in digital reading was found to correlate with following capabilities:
| Ability in print-reading | 80% |
| Quality of navigation | 10% |
| Quantity of navigation | 5% |
| Unexplained | 5% |
This analysis suggests one (fairly obvious) conclusion and suggests one further question.
The conclusion is that aptitude in the digital realm depends heavily on the mastery of traditional academic skills: the ability to read digital text is 80% dependent on the ability to read traditional print. No surprises there to anyone except those who have argued that the 21st century turns our curriculum on its head.
The question is “what helps students make good quality navigation decisions?” The answer seems to be more of the same:
good navigation relies on the same cognitive skills and motivational aspects that are prerequisites for success in the paper-based assessment of reading as well… Statistical analyses show that students’ reading skills in print documents strongly predict their navigation behaviour in a digital environment.
This is not quite the whole story, however, as some countries perform better in digital navigation than would be predicted by their print-reading scores, and this is attributed to their problem-solving.
That does not mean that the OECD report is arguing the case for transferrable skills because:
For the most part, the problem-solving skills that students demonstrate when navigating complex online texts are specific, and are likely best learned in the context of reading from the Internet.
From which one can conclude that the best way to be a good consumer of digital text is to have good traditional literacy skills, to have a sense of perseverance, and to have the chance to familiarise yourself with the requirements of reading digital text.
You might find this a rather underwhelming conclusion from such a detailed analysis—but it does at least stand against some of the more revolutionary claims that have been made by proponents of digital literacy. It confirms that the twenty-first century does not change everything and that operating online does not require a completely different set of skills to those which have been taught traditionally in schools. Good digital literacy requires good traditional literacy with the occasional opportunity to practice these traditional skills in an online context.
Damaging effects of overuse
The OECD report also looked briefly at the sort of overuse and misuse of computers that might be termed pathological.
Basing their conclusions partly on their own findings and partly on other research, the report notes that the use of the internet for more than 6 hours per day tends to be associated with social alienation, poor punctuality, truancy, physical weakness and insufficient sleep. It notes, however, that in some circumstances, the use of the internet in these ways might be symptom as much as cause of the underlying problems.
Summary
The substantive findings of the report are therefore:
- that the effect of technology on learning outcomes was “mixed at best”—worse than mixed when the use of computers in schools was over-emphasized;
- that there was strong continuity between digital skills and traditional literacy and thinking skills;
- that problems in equity of outcomes are reducible to the unequal attainment of these traditional literacies and not to the unequal distribution of technology or to any need to redefine our educational objectives;
- that parents and schools should be mindful of the potential to abuse the online environment.
All of these findings strike me as eminently sensible and many repeat arguments that I have been making on this blog for the last 3½ years.
My criticism comes with the final part of the report’s Executive Summary, which presents the authors’ own opinions on how we should respond to these findings. This passage is at best bland and at worst misleading.
One interpretation of these findings is that it takes educators time and effort to learn how to use technology in education while staying firmly focused on student learning. Meanwhile, online tools can help teachers and school leaders exchange ideas and inspire each other, transforming what used to be an individual’s problem into a collaborative process. In the end, technology can amplify great teaching, but great technology cannot replace poor teaching.
The passage repeats what have become common orthodoxies in the teaching world, that:
- technology is a given (not something that needs to be developed and improved);
- that improvements in use have to be led by teachers, not technologists or specialist instructional designers;
- that what matters is good teaching, which is something that is distinct from technology.
In all these judgements (none of which are based on the evidence produced in the report), I think Andreas Schleicher’s conclusion is wrong. But these are just personal opinions—a gloss that is added to the report’s substantive research findings. It is the substantive findings that should interest us and in my next two sections I will examine the reasons why many in the edtech community have been dismissive of the substantive findings of both reports, but in particular of the LSE report.
Criticisms of the LSE report
Sources
I will look in much more detail at criticisms of the first, LSE report, of which there have been many more. This might be because the report has been published for longer; it might be that its findings were starker and therefore more provocative; it might be that people felt that the OECD was a more formidable target to attack.
The criticisms that I am going to examine are in:
- Mike Cameron’s blog article On research and banning things
- José Picardo’s blog article Banning mobile phones is cargo cult science;
- Gerald Haigh’s article in Sec-Ed Mobile phones: To ban or not to ban
- Discussion on the ICT Research Network email reflector, which was originally set up by Becta but is now run by the Association for Learning Technology (when quoting from these emails, which are not fully in the public domain, I will give titles and dates of emails but not names of authors).
Correlation and causation
Before I look in detail at the sources above, I must return to the thorny issue of correlation and causation. Almost all of the critics show a basic misunderstanding of the nature of scientific evidence, and in particular the relationship between correlation and causation.
On the ICTRN, a member of a Faculty of Education criticised the report’s “potential confusion between correlation and causation” [1], while this point was supported by other subscribers who referenced a cartoon on causality [2], a list of supposedly spurious correlations [3] and a recollection that[4]
I remember my statistics lecturer telling us that in one particular year there was a correlation between the increase in the birth rate and the increase in the number of storks visiting New Zealand.
José Picardo comments that the report:
simply states a correlation. Correlation is not causation. It reminds me of that sketch in Father Ted when Dougal and Father Ted are sitting at a hospital waiting room and Dougal says to Ted “do you ever notice that it’s usually sick people that end up in hospitals?”
Gerald Haigh complains that in the media coverage, it is difficult to find:
anyone who points out that correlation does not imply causation, or that the research does not attempt to compare “banning” schools with the many that regard student phones as valuable additional technology.
What this shows is a widespread misunderstanding of the relationship between correlation and causation—indeed of the nature of evidence itself—among teachers and commentators on education. In the face of such a general stampede of ignorance, it seems to have little effect to point out the misconception (as I have tried to do many times before on this blog) because no one shows any interested in debating the truth of their position. Everyone avoids any serious challenge and merely continues to repeat their existing prejudices.
So it is somewhat wearily that I return to a subject which I have covered many times before—most recently at How technology will revolutionize research, section 2.4.
The fact is that the relationship between correlation and causation is very close indeed:
- causation is nothing other than a particular sort of very secure correlation between two types of event, in which one type of event always occurs before the other.
- correlation always implies causation of some sort (though the direction of causation may not be clear from a single correlation);
- correlation is ultimately the only evidence we ever have of a causal relationship existing.
If you do not understand these points or think they are wrong, then you might like to look at my previous post, but you really need to read David Hume’s Enquiry Concerning Human Understanding. Spark Notes offers a good summary at (read through to the end of section 7), or the original text is available in PDF at or in a free Kindle edition.
The key point that Hume makes is that correlation (or “constant conjunction”) is the only evidence we have for causation (or “necessary connection”) but that our pattern-forming brains imagine that we understand the secret reason for the necessary connection. This imagined understanding lies at the heart of our false reasoning. If you are an educationalist, you are probably familiar with the thought of Jean Piaget, who argues that our understanding is represented by a model of the world, which develops as we either assimilate or accommodate the evidence that we derive from experience. In this process, the model offers a vision of how everything fits together, while the evidence is fragmentary.
The fundamental error of everyone who argues that correlation does not imply causation is to think that the model is the evidence—that we have no reason to say that A causes B until we understand how A causes B, how everything fits together, what is really going on. But all that stuff is the model, the product of our imagination, it is how we represent the world internally. The model is not the evidence.
The evidence comes from observations of isolated phenomena. In situations in which we do not have reason to believe that a single observation will necessarily be repeated, evidence comes from correlations. As noted in my previous post on this subject, the website of spurious correlations are not correlations at all—they are datasets that are:
- specially selected because they fit (if the significance of this escapes you, listen to Dylan Wiliam on the beneficial effect of green jelly beans on acne, from 22:50);
- each contain very few data points (10 or 11);
- represent trend lines, which dramatically reduces the statistical power of the data sample still further (they are reducible to half a dozen basic patters: rising, falling, peaking, bottoming-out…).
José Picardo tells me on Twitter that assuming correlation implies causation is
like saying Finns do better cos they eat reindeer.
No it isn’t:
- “Finns do better in education” is a single data point;
- “Finns eat more reindeer” is another single data point.
The coincidence of two data points does not establish a correlation.
But what is a profound misunderstanding, one that shows a failure to grasp the essential character of scientific evidence, is almost universally accepted as gospel truth among teachers and educationalists. Exactly the same elementary mistake is made by the subscriber to the ICT Research Network who understood so little from her statistics course that she thinks that:
- an increase in the New Zealand birth-rate in one particular year
- and an increase in migrating storks visiting New Zealand in the same year
establishes a correlation.
In fact, the example of the storks is a distant and misunderstood echo of a 1952 article by J Neyman, which I summarised on an email to the ICTRN reflector as follows.
I am very grateful to Harvey Mellar for pointing me to the source of your stork correlation, which is a humorous 1952 article by J Neyman in Lectures and Conferences on Mathematical Statistics and Probability. This article is described in a 1993 article by Richard Kronmal, Spurious Correlation and the Fallacy of the Ratio Standard Revisited, in Vol 156, No. 3 of the Journal of the Royal Statistical Society, Series A (Statistics in Society), which can be found on JSTOR.
Neyman’s 1952 article showed a correlation between different countries based on the number of babies per 10,000 women and the number of storks per 10,000 women. The paper recounts how the finding supposedly led to a proposal by the Zero Population Group, campaigning for better birth control, to eradicate storks, and the statistical argument that followed between the ZPG and the Trust for the Preservation of Wildlife Species.
This study was designed to illustrate a phenomenon identified by 1897 as “spurious correlation”. Kronmal’s 1993 article suggests that, even though the phenomenon of spurious correlation has been known for over a century, researchers continue to ignore the problem – for example in much research based on the Body Mass Index.
The problem with the stork correlation is that both are expressed as ratios, using the same denominator – “per 10,000 women” – and, which may make the situation worse, in a situation where the ratio between storks and the number of women is meaningless. Neyman showed that there was no correlation between the number of storks and the number of births, but there was a correlation when these were expressed “per 10,000 women”. The lesson for researchers is to investigate raw metrics and not ratios, avoiding correlations between ratios especially where they both share the same denominator.
I do not pretend to understand the argument, which is a technical one, or exactly what is happening – but its importance for our discussion seems clear enough. The storks are the exception that proves the rule. The particular reason why this sort of “spurious correlation” is of such interest is precisely because of the general assumption that correlations DO imply causation is fundamental to the practice of quantitative research. If, as most of this list seems to assume, correlation does not imply causation, there would be no reason to make such a fuss about just one more irrelevant correlation.
To summarise the lesson for educationalists of the Neyman article: spurious correlations are correlations between datasets populated by ratios that share the same denominator. Such datasets cannot be interpreted as supporting the causal inferences—even though these are implied by non-spurious correlations which are not based on the comparison between ratios.
The case of causal direction always has to be considered, of course. José Picardo’ supports his assertion that “correlation is not causation” by recounting his Father Ted joke.
It reminds me of that sketch in Father Ted when Dougal and Father Ted are sitting at a hospital waiting room and Dougal says to Ted “do you ever notice that it’s usually sick people that end up in hospitals?”
Presumably José means that people are not sick because they are in hospital (even this may often be the case)—but it cannot have escaped his attention that the reason that most people are in hospital is because they are sick. An example chosen to demonstrate that correlation does not imply causation only succeeds in demonstrating the exact opposite.
The holistic fallacy
I want to finish this reprise of the discussion about correlation by reiterating a rather more subtle point. Many of the more sophisticated objections to the LSE report consist in pointing out things that the report did not look at, things that the research left out. It was not sufficiently holistic. I shall refer to these arguments as the holistic fallacy.
Empirical evidence (of which correlation is a formal type) is of its nature fragmentary. No one ever comes across evidence that supports the scientific theory of heat. The evidence comes in small packages, like touching a hot ring and getting burnt, or observing the transfer of heat from one object to another. Correlations could be compared to the results of an electrical test in which the electrician compares the volts in two electrical wires: it is a binary comparison between two things—and it is only a succession of such comparisons that will form a complete theory (e.g. of where the electrical fault has occurred).
The holistic fallacy is a variant of the point that Hume makes, that the non-scientist is confused into thinking that the model—what puts us into the emotionally comforting state of mind in which we persuade ourselves that we understand what is really going on—is the evidence. Our mental model is holistic: the evidence is fragmentary. The holistic fallacy is one way in which we muddle the two. It is like dismissing the significance of the fingerprint on the window sill because it does not take into account the torn clothing in the thorn bush. Both are important pieces of evidence that need to be accommodated into the blow by blow narrative of the crime by which the sleuth stuns the assembled house-guests with the brilliance of his or her reasoning.
A further variation of the holistic fallacy (which I envisage as an infinite outwards expansion of all the different influences that have to be taken into account) is the belief that you cannot demonstrate causality without burrowing infinitely inwards and tracing every minute step of the causal pathway. This variant is illustrated by another correspondent on the ICTRN reflector[5]:
The real problem of the correlation v causation issue in research into digital technologies in education is that to prove causation one would have to dig inside pupils’ brains.
Wrong. We can never follow the minute tendrils of causality, which are forever invisible to us. Even if we got inside students’ brains, it would never be enough for us to reach out and touch the infinitely small, the infinitely many and inevitably secret causal couplings. All we can do, like the electrician with a voltmeter connected to two crocodile clips, is show the binary dependency between two phenomena occurring at different points along what we then infer to be a causal pathway.
I should make clear that such arguments about the whole picture are important when we consider how to accommodate the research into our existing intellectual models. The research may be true but not significant because it tested the wrong things. But pointing out things that the research did not consider does not invalidate the causal relationship between the two phenomena that the research did test. If people want to argue that the conclusions of the research are not relevant, then they need to start by assuming that the conclusions of the research are true, and argue the case that in spite of being true, they may not matter. That is a different sort of argument to one that says that the research is not valid because it did not take into account all the many facets of our model.
Mike Cameron’s notes and questions
Mike Cameron’s post contains a list of notes and questions which, when I first read them, struck me as somewhat insubstantial musings: Mike thinking aloud. I passed over the opportunity to comment. But in a recent Twitter exchange, Mike told me that he thought the LSE report was “tosh” and cited these questions as justification for what had by then developed into a very robust position. They are also cited by Gerald Haigh as constituting a good reason to question the validity of the research. So I shall start by responding inline to Mikes notes and questions.
1. No data on who owned mobile phones. Uses phone ownership among over 13s as a proxy for 13-16yr olds.
Strictly speaking, data on mobile phone ownership is irrelevant to the conclusions of the study. The study shows a correlation between mobile phone bans and student performance. It does not seek to examine every link in the chain in what Hume called the “secret connexion” between the two phenomena.
It is nevertheless true that if children do not own phones, then the mechanism that the report suggests (distraction in class) would be implausible. The correlation between mobile phone bans and stronger performance would still be valid—but we would have very great trouble in explaining the relationship.
Given our informal observations of the modern world and the fact that schools have been busy banning mobile phones since the early 2000s, it hardly seems plausible to suggest that children simply don’t have them and are not bringing them into school. If that were the case, why should anyone ban them? And if it were true that only a small minority of children were bringing phones into the classroom, the only consequence of that fact on the conclusions of the report would be that those phones that were being brought into class would have to be having a much more significant deleterious influence that one would otherwise assume, in order to account for what is a statistically significant effect across the whole population.
Having said all of that, the issue is explicitly considered by the report, which acknowledges on page 9 that
any impact a school mobile phone ban could have would be tempered if teenagers did not use phones in the first instance.
The report reproduces research into phone ownership undertaken by Ofcom into “individuals 13 years and older”.
Mike’s criticism suggests that it is not justified to assume ownership rates among 13-16 year olds from data based on a survey of people 13 and older (including adults). But this point is also explicitly considered by the report, which cites :
Survey research by the Office of Communications (Ofcom) finds that teenagers in the UK have similar mobile ownership rates as adults since mid-2000s (Ofcom 2011).
A further survey of teenagers in 2005 found that 82% of 12-16 years old owned a mobile phone, being slightly higher than the overall rate of 80% (Ofcom, 2006)
In summary:
- the extent phone ownership is irrelevant to the finding of a correlation between mobile phone bans and examination results at GCSE—even if it is relevant to the way we choose to apply that finding to our model;
- any reduction of phone ownership among school children would only have the effect of exaggerating the deleterious influence of those phones that were present;
- Mike is incorrect in suggesting that the study does not cite adequate evidence to demonstrate that a large proportion of teenagers own mobile phones;
- which will surely come as a surprise to no-one.
2. Definition of ‘widely complied with’?
The explanation of this variable is given in the note to Table 2 on page 25:
Head teachers were asked to rate the extent to which the policy is adhered to by students on a seven-point scale with 1 representing “Not at all” and 7 representing “Completely.” A school was considered to have a high compliance ban if the response was greater than four.
In an ideal world, of course, we would have an objective measure of compliance, based perhaps on independent evaluators going into all 91 schools at yearly intervals. Not only would such objective data be impractical to collect but it would not add much to the force of the study. I suggest that the most likely bias introduced by relying on the subjective judgement of Head Teachers is that it will tend to overstate compliance with the mobile bans. If the finding of the report is broadly accurate, the true effect size will then be larger than the one found, given that many of the bans that were reported as having high compliance in fact had low compliance. If the way that schools reported compliance is not accurate, Mike has to explain why there is a correlation between performance and schools who report high-compliance bans, when there is no such statistically significant correlation with the sample as a whole.
3. Context of the schools that imposed the bans
This is irrelevant and is another example of what I have called the holistic fallacy. The whole point of taking a large sample is to randomize the contextual influences that you are not testing. We are not interested in context, we want to get rid of it as an influence on the outcome of the report and this is achieved by randomizing it.
We are interested, of course, to check that the sampled schools are representative of the wider population in respect of wider context (in other words, checking that this randomization has been done effectively) and there are numerous passages in the report which are dedicated to checking this point.
4. Wholesale ban vs visibility ban – Is this tested in the report?
The report is aware of this issue and gives a clear definition of “ban” on page 7:
There are a multitude of ways in which schools have restricted phone use, from asking for them to be set on silent to not allowing them on school premises. We define a school as introducing a school ban if that school did not allow them on the premises or required them to be handed in at the start of the day.
It does not test the effect of visibility bans but that is irrelevant to its purpose. Any suggestion that this compromises the reliability of the report would be another example of the holistic fallacy.
5. Only schools in cities were targeted
This is true—but hardly significant without considering how the sample might be biased as a consequence.
It is plausible that children in the city might have more mobiles—but as discussed above, this would not change the influence of mobile phones on learning, except to the extent that if children do not have mobile phones, a ban would be expected to have a smaller effect.
It might be that the performance of city schools is different from country schools. I address this point in the context of the London Challenge, below.
In other respects, I do not understand how the selection of city schools is supposed to have biased the conclusions regarding the effect of mobiles on learning. It does not strike me as plausible to suggest that the influence of mobile phones on learning will change between the town and the country.
6. Survey sought from 450 schools with 91 respondents – is this all the secondary schools in the four cities?
No. The report makes clear that the research team first sought permission from Local Authorities to approach schools. On page 6 it is stated that:
Before approaching schools, we obtained permission from the relevant Local Authorities. Every secondary school from Local Authorities where permission was granted was then contacted.
And in a note to this passage:
We did not obtain permission from five Local Authorities in London (Hackney, Lewisham, Newham, Redbridge and Tower Hamlets), which combined have 77 secondary schools. The City of London Authority does not contain any public schools and therefore was not approached. The remaining 27 London Local Authorities gave permission, with 337 secondary schools being approached.
It certainly seems that there is an element of self-selection in those schools participating in the survey. This would be a serious criticism if the report were examining the proportion of schools that had introduced bans. But it isn’t. It is examining the effect of those bans on performance, as judged against other schools both nationally and in the same cities, and for this purpose, the proportion of the sample that had introduced bans is not significant.
Although 90 out of the 91 respondents had introduced bans, there was nevertheless a more equal spread of those who had introduced high compliance bans (56) and those who had introduced “low compliance” bans (34).
This is the significant criterion from the point of view of learning and the distinction between the two groups validates the report’s findings by showing that performance improvements were only correlated in schools with “high compliance” bans and not those with “low compliance” bans.
8. Sample schools contain significantly more minority, SEN and FSM -eligible pupils – answers the context question above.
No comment.
9. Compliance on a 7 point scale, 1 not at all, 7 Completely. Intermediate points not defined.
The recent decision to abandon criterion referencing, in the form of levels, illustrates the point that individual criteria or definitions do not necessarily improve the consistent understanding of the intervals on a scale. It is actually quite hard to create short rubrics that define performance criteria reliably. The two ends of the scale (“not at all” and “completely”) can certainly be defined precisely—but the intermediate points are much more challenging. Imagining a smooth progression is probably the most consistent way of defining the intermediate intervals. See also Daisy Christodoulou’s recent arguments on the benefits of comparative marking.
10. Most bans introduced in 2005-10 – coincides with London Challenge time?
See next point.
11. Paper suggests no upswing in results in these schools in previous years results – given London Challenge this might suggest that these schools are not representative of the whole.
The following graph[7] shows that grade inflation was significant during this period across the whole country but occurred slightly more rapidly in London than elsewhere.
Figure two in the LSE report charts the performance of students at schools with high-compliance bans, charted from 7 years before the ban was introduced to 9 years after.
It would certainly represent a serious challenge to the report if the only reason that the performance of students at the schools with high-compliance bans improved was because the performance of all students was improving, particularly in London where many of the sample schools were located.
But this is not what the data in Figure 2 shows. The Y axis of this graph shows the effect size of the introduction of the ban relative to other schools in the sample. I was not able to establish this from the report itself and so checked with Richard Murphy, one of the authors of the report, who stated to me in an email that:
Figure 2 is showing the plot of the estimated effect size of the ban before and after it was introduced. It is not showing the test scores. This is estimated on the sample of schools that responded to the survey, rather than the whole population over which test scores were standardised. If there was general grade inflation amongst these schools, that is accounted for through the inclusion of year fixed effects in the estimating equation. This allows for the mean achievement in schools to be increasing (or decreasing) over time. [my emphasis]
In other words, the graph shows the improvement in performance of students at schools with high-compliance bans relative to the performance of students at all the schools in the sample, which includes those schools with low-compliance bans or in a single case, no ban at all. To the extent that grades were improving at the sampled schools anyway, either because grades were improving everywhere or because many of the schools were in London, this level of increasing performance was taken as the baseline.
General grade inflation or London-specific effects, such as the London Challenge, therefore cannot be used to account for the improvement in performance shown in the LSE study.
12. Provides no evidence of pre-ban in-school use of phones (though one could take the need to introduce the ban as evidence it was required).
This information is irrelevant, again demonstrating the holistic fallacy. The evidence consists in the correlation between the ban and the improving performance—and not in how we interpreted that evidence. As Mike himself acknowledges, the suggestion that all these schools introduced bans when (contrary to the Ofcom evidence of mobile phone use by teenagers) no-one in those schools was using mobile phones—is pretty far-fetched.
13. “It needs not be the case for an individual to use a phone to be distracted by them, their use by others in the classroom may cause disruptions. ” – evidence for this?
Common sense, even though this point is not material to the conclusion of the report and is therefore another example of the holistic fallacy. The sentence occurs in the same paragraph that deals with the level of mobile ownership among school children (see point 1).
14. “There may be a concern that only high-achieving schools introduce mobile phone bans, which could lead to overestimating the effects of a mobile phone ban.” – I would have assumed the opposite.
See my response to point 8. The evidence of the report strongly suggests that the assumption (made very widely by those trying to condemn the report) that mobile phone bans were introduced in under-performing schools is simply false.
15. How the paper handles the issue of other policy changes is “interesting” – ‘The variable OtherPolicy takes a value of 1 for the years after a change at a school occurs. We combine information coming from our survey of headteachers and information from school’s website. We do not observe multiple change of policies/leader in addition to the phone policy change, hence a binary variable can be used.‘ – So there are either other policy changes or there aren’t, no consideration of what they are.
This again demonstrates the holistic fallacy – the idea that the research is not valid if it does not measure everything that is going on. It shows a failure to appreciate that the evidence is focussed purely on the relationship between the ban and the improving performance.
If there were other significant policy initiatives introduced at the same time, then there is a risk that any change in performance could be attributed to the other policy initiatives and not to the introduction of the ban. The significance of the results from schools that report concurrent major policy initiatives is therefore given reduced weighting. What the other policy initiatives were is irrelevant and the attempt to provide evidence for which of these other policy initiatives might have been more or less successful is way beyond the scope of this survey.
16. The statement – “We add to this by illustrating that a highly multipurpose technology, such as mobile phones, can have a negative impact on productivity through distraction. “ is unevidenced – the results in the paper show that there is an effect. They do not show why the effect occurs.
It is true that this research does not produce evidence for why the effect occurs—but the quoted passage does not claim that it does. They say it “illustrates” a theory that is already widespread in the existing literature.
This passage is quoted from the report’s conclusion, which comments on the significance of the survey findings when placed against the background research. The sentence immediately preceding the one quoted states:
The existing literature on the impact of technology in the classroom implies that the unstructured presence of technology has ambiguous impacts on student achievement.
The context of the quotation therefore clearly establishes that the report is commenting on the significance of this reports conclusions in relation to the existing research. Although no mention is made in this concluding passage to the extent to which that research demonstrates the importance of distraction, this relationship has been covered explicitly on pages 4-5:
Recent experimental papers present evidence that mobile phone use while executing another task decreases learning and task completion (e.g. Ophir et al. (2009); Smith et al. (2011); Levine et al. (2013); and Lee et al. (2014)). The distracting nature of mobile phones has been previously examined in other context such as incidence of road accidents.
Bhargava and Pathania (2013) exploit a pricing discontinuity in call plans and show that there is a large jump in phone use after 9 p.m. This jump, however, is not followed by an increase in car accidents. Using vehicular fatality data from across the United States and standard difference-indifferences techniques, Abouk & Adams (2013) find that texting bans have only a temporary impact on car accident fatalities, suggesting that drivers react to the announcement of a legislation only to return to old habits shortly afterward.
The paper does not claim to show that the negative impact occurs as a result of distraction. But it interprets the correlation between mobile phone bans and improved performance by reference to the effects of distraction, which has been widely evidenced elsewhere. Such an interpretation is at least plausible and, like all scientific theories, will stand until it is proven to be false.
Mike’s summary
At the end of the piece, Mike summarizes his explanation of the report’s findings.
My experience is this. Schools introduce phone bans in response to an existing issue within the school. Such a ban is rarely introduced as the sole change and consequently any changes need to be seen in that light.
More meat has been put on this theory by others. Another regular contributor to the ICT Research Network said:
Which of course leads on to your question of why the achievement levels in these schools were poor originally [I had said no such thing—I only said that results had improved]. I suggest to you that the schools that decided to ban mobiles are schools where behaviour was poor leading to low results.[6]
The theory is given another outing by José, who argues:
By separating the mobile phone ban from the wider toughening of these schools’ behaviour policies, we may be fooling ourselves into believing that by replicating single aspects of these policies – e.g. enforcing a mobile phone ban – test results will improve. It just doesn’t work like that.
There are a number of problems with this theory, which seems to have been generally adopted.
- It is at best pure supposition—there is no evidence whatsoever of the initial indiscipline or of the raft of other policies that are supposed to have been introduced.
- The author of the ICTRN email assumes that because the performance of a school improves, it must have been poor to start with.
- Most significantly, the idea that the mobile bans were introduced in underperforming schools is flatly contradicted by the evidence of the LSE report. Although the report states that
the sample schools have a significantly more disadvantaged population than other schools in the cities and nationally, enrolling more minority, SEN and FSM -eligible pupils
…it also states that…
Comparing standardized age 16 test scores, we see…that the schools in our sample over the whole period achieve significantly higher scores than other schools within these cities (0.07σ)…[even though] the sampled schools have an even lower intake quality (-0.09σ) [than other schools in the same cities].
Even before the introduction of mobile phone bans, these schools were performing better, on average, than other schools in the same cities on value-add measures. These were over-performing schools not under-performing schools. The supposition that they were making all sorts of other unusual interventions in order to deal with serious behaviour problems becomes even more implausible.
4. The report specifically reduces the weighting given to schools that report introducing significant initiatives or who experienced changes of leadership at the same time as introducing the mobile phone bans (see point 15 above).
Summary
I have examined Mike’s questions in painstaking detail because, although they are presented as an exercise in thinking aloud, they have subsequently been put forward as a rationale for why the LSE report is “tosh”.
The truth is that none of these supposed criticisms of the report carry any force whatsoever. In my book, which follows the principles of the scientific method, subjecting something to a really hostile grilling and finding that it comes away totally unscathed is the strongest evidence you can ever find for its validity.
Others in the edtech community have picked up the general hue and cry against the LSE paper, often with much less thought than Mike.
An email to the ICTRN reflector[8] draws attention to a sentence in a Guardian report of the paper:
“Murphy, an assistant professor of economics at the University of Texas in Austin, said the study did not take into account any positive effects for pupils using mobile phones for research”. No bias there then!
It is incidental to this comment that the author clearly believes that the ability to use mobile phones for in-class research offers a significant benefit, when the findings of the OECD report cast serious doubt on this position.
The main purpose of the comment is to suggest that the report is biased because one of the authors works in Texas rather than London, is an Economist rather than an Educationist, and/or because the report does not specifically consider something that would be completely irrelevant to the study being conducted—another example of the holistic fallacy. It then wraps all this false reasoning in a pernicious ad hominem attack on the integrity of the report’s authors. Yet this unworthy attack has been widely repeated on Twitter by prominent members of the the edtech community.
Gerald Haigh cites Mike’s questions as “16 searching questions about the detail of the LSE paper” in justification for the position that much of the media reporting has been “lamentably ill-informed”. In fact, the newspaper headlines are on the whole accurate and it is the critics from the edtech community who are, on the whole, “lamentably ill-informed”.
José Picardo invites readers to:
Delve into hoo-ha further and it appears that what these schools have done is to toughen their behaviour policies generally, with the ban on mobile phones being a part of it
This is the same theory that appears to be based on Mike’s musings but which (as I have demonstrated above) is supported by no evidence at all and is flatly contradicted by the report.
Having shown his failure to understand what a correlation is, José quotes a Tweet by Carl Hendrick, a teacher at Wellington College, who refers to “the undoubted usefulness in class” of mobile phones. For evidence of this position, José cites one of his other blogs and a couple of inspection reports on individual schools.
The Tweet cites no evidence at all but appears to be based on the theory that, if you say it with confidence, it must be true. Of the two inspection reports, the first reports on José’s own school, Surbiton High. This makes a fairly tangential reference to digital technology in the classroom, saying that
Excellent use of available resources, including digital media, enables interest to be stimulated and pupils’ learning to be enhanced,
…that students…
Are highly proficient in the use of information and communication technology as a research tool,
…and that…
The school uses a range of digital media to provide real-time updates about school life.
The second report into the Stephen Perse Foundation, has numerous references to the fact that:
The move towards digital learning…is central to the Foundation’s vision,
…concluding in its main findings that
The excellent teaching, using a wide variety of teaching methods combined with extensive use of digital technology, fosters an enquiring and creative learning environment in which pupils thrive.
These are both positive reports. However, it has to be said that we have learnt to be very sceptical of inspection reports as a source of evidence. Many reports by Ofsted have been shown to support classroom pedagogies such as group-work and learning styles, which are not supported by any serious research evidence. All the inspection report shows is the opinions of the inspector.
It is also significant that both these reports are written not by Ofsted but by the Independent Schools Inspectorate—because these both refer to top independent schools. As is Wellington College, where Carl Hendrick works. At my recent talk at ResearchEd14, a well-argued contribution was made from the audience by a floor by a teacher who worked at Radley, arguing for the usefulness of mobile devices at his school.
Stephen Perse, Surbiton High, Wellington, Radley. The fact that much of the anecdotal evidence for the beneficial use of mobiles seems to be emerging from top private schools is consistent with the LSE report, not contrary to it. As the report found that learners in the top 20% of prior attainment did not experience any negative effects from mobile phones, it is surely plausible to imagine that those in the top 5%, attending schools in the top 1%, may even experience positive effects. It does not follow that these same effects can be transferred to mainstream state education, into contexts in which the teacher-ratios and the attitudes and prior attainment of the pupils are often very different from those found in the top independent schools.
I shall return later to the relationship between technology and the quality of teaching—but the key point is one that I have frequently made on this blog before. We know what good teaching looks like and we can do it well enough for the few. The problem with our education system is scale. But if you are interested on the effect of mobile phones on the average state school, then you have to look at the evidence of what happens when you introduce mobile phones to the average state school—not what happens when you introduce mobiles phones in Surbiton High and Wellington College.
The only broader quantitative evidence cited by José is to another of José ’s blogs which states that
research suggests that there is a strong correlation between the effective use of technology and improved outcomes.
The link in this passage is to the EEF’s review of the Impact of Digital Technology on Learning. If you search this publication for the word “correlation”, the only substantive result states:
Taken together, the correlational and experimental evidence does not offer a convincing case for the general impact of digital technology on learning outcomes.
Whoops.
José concludes his post by suggesting that the LSE report is “cargo cult science”. Judging by the quality of the research evidence offered, it is José not the LSE report that is offering up cargo cult science.
General arguments on interpretation
The OECD report has not attracted the sort of methodological criticism that has been directed at the LSE report but it has been sidelined by a range of arguments about interpretation that could be applied equally to both reports.
Its not what you have, its what you do with it that counts
That, at least, was what I was always told in the school changing rooms. The truth is rather different: as I have argued on this blog (ad nauseam, I’m afraid—but here and here for starters): it’s the technology that really matters.
In the context of these reports, the argument is that you cannot assess the effect of BYOD unless you also look at the quality of the teaching in the classrooms in which BYOD is deployed. The key point that José emphasises in his article is that
the use of mobile devices to support teaching and learning and robust behaviour policies are not mutually exclusive concepts
Digital technology works when backed up by good teaching and robust behaviour policies. Carl Hendrick follows up his first tweet by arguing that
Another way of interpreting this research is that we need better ways of incorporating it not an outright ban
Mike Cameron concludes his blog by commenting that people often misinterpret the effect of research by listening to people who say
“We do A, B, C, D, E, F and a little bit of G.” After which the visitors go back to their own school and vigorously implement lots of D and expect their school to become great as a result.
Among the A, B, C, E, F and G that people fail to transfer, it is clear that what really matters is really good teaching. As José also makes the argument:
this research completely ignores how mobile devices are being used effectively
…referring also in his second blog post to
the Sutton Trust’s most recent report on what makes great teaching…accompanied by ways in which mobile technology might, on occasion, help along the way.
The problem with this argument is that it tries to make out that what really matters is the use of technology from evidence that suggests what really matters is good teaching. If you take four types of school, those to do and do not make extensive use of technology and those who do and do not have good standards of teaching, the current research shows the incidence of improved outcomes coincides with the prevalence of good teaching, not technology use.
You cannot argue that technology creates good outcomes by using anecdotal evidence of schools drawn from the top right hand quadrant (good teaching in technology rich environments)—you have to show that schools in the top right hand quadrant achieve better outcomes than schools in the bottom right hand quadrant; or that schools in the top left hand quadrant achieve better outcomes than schools in the bottom left hand quadrant. If this evidence exists anywhere, I should be grateful if someone could draw it to my attention.
It might be possible to argue that the introduction of technology encourages better teaching by showing that improving schools tend to move clockwise from the bottom left, through the top-left to the top-right quadrants (a process representing tech-led improvement); and not anti-clockwise, from the bottom-left through the bottom-right to the top left quadrants (training-led improvement). But again, I have not seen any evidence to suggest that this is what tends to happen.
On the contrary, the general complaint one sees from education technologists is that their technology doesn’t work in practice because good training, a prerequisite for technology, has not been put in place first. So technologists themselves seem to assume that improvement is training-led. See, for example, Mark Anderson’s article, My response to everyone – Don’t bin IT or ban IT, get IT right!
It’s not the fault of the many teachers who choose to not use technology. As mentioned many times previously, it’s a problem of confidence. A problem of competence. A problem of training.
If it is the confidence, competence and training that matters (all issues which are extremely hard to change at scale), then what is the evidence that the technology itself is making any difference at all? Why not do the training and forget about the tech?
José Picardo’s fond hope that mobile technology “might, on occasion, help along the way” is an admirably tentative position but it does absolutely nothing to counter robust, quantitative evidence which shows that, as generally applied in mainstream education, mobile technology doesn’t help along the way—not for most teachers in most schools. From a policy point of view, it is what helps most teachers in most schools that counts.
That pedagogy is irrelevant
Having taken some poorly aimed shots at the methodology of the LSE report, the detractors of both the LSE and the OECD reports generally come back to the assertion that it is irrelevant because what matters is curriculum not pedagogy. As Gerald Haigh puts the argument:
Rumbling distantly at the heart of this debate is something much more profound – a search for the real purpose of education, and a voice which says, in effect: “What if banning phones does improve GCSE results? Is that the only measure that matters?”
He then quotes Graham Brown-Martin’s sarcastic comment on the Guardian article:
Well that proves it then – banning 21st century technology improves 19th century school exams. Who knew?
José Picardo slips the same argument into a number of asides to his main argument:
Leaving to one side my concern about the belief that whatever improves results in high-stakes testing must be good for children…
Damning…the use of mobile devices as “a distraction” betrays a narrow view of what constitutes a good education…
A more thorough treatment of the argument is provided by Oliver Quinlan in his post Ignore the headlines – computers in classrooms are a good idea, and here’s why. I respond to Oliver’s argument below with a series of inline comments.
There is a tendency in discussions of education to refer to ‘learning outcomes’ and ‘results’ as if we have a universally accepted view of what these are. A fundamental challenge of education systems is the agreement with broad and uncontestable aims such as ‘preparing young people for the future’.
This is an important point, which I have also raised in my post at How technology will revolutionize research—although I would suggest “learning objectives” as a more accurate term for what we are really talking about than “learning outcomes”. Oliver continues:
It seems likely many teachers are implementing technology with the intention of achieving quite different ‘learning outcomes’ than those the OECD measure.
And Oliver, siding with those who argue for new definitions of curriculum objectives, brings the argument to its culmination:
Regardless of any potential benefits to skills in Reading, Maths and Science, learning about technology is in itself a reason for its use.
Only it isn’t—Oliver doesn’t quite manage to ground the ball for a try. The fact that I might want to learn about nuclear weapons isn’t a reason for using them. Turning the same relationship on its head, the fact that I drive to work in a car doesn’t mean I know anything at all about the internal combustion engine. So I don’t necessarily have to use something in order to learn about it, nor do I necessarily learn about it by using it.
The only reason to use digital technology in the classroom is because it benefits my learning, whether that learning is about reading, maths and science, or whether that learning is about technology itself.
We might assume, it is true, that learning about technology will involve some use of technology. But if we are to make progress in describing our learning objectives so that we can pursue them more consistently and more systematically, then we need to analyse very high-level and vague terms like “preparing young people for the future” into their more precise, constituent parts. As I have argued in my previous post, How technology will revolutionize research, this is fundamental to any effective application of education technology.
If we are going to let mobile phones into the classroom in order that we may better teach students about technology, then we need to describe precisely what our technology-related learning objectives are. To say that we should use technology in order that students should learn about technology and then, when challenged to describe exactly what they should learn, reply whatever they happen to pick up from the experience of using technology—then our argument is a circular one: we should have technology in the classroom merely for the sake of having technology in the classroom.
Oliver recognises that the Computing curriculum has just undergone a major overhaul.
In England, the relevant subject has recently been refocused to include understanding of computer science, programming and digital literacy.
The nature of this new Computing curriculum that has been introduced by the government is, broadly speaking, more academic than the old ICT. Computer Science is heavily based on Mathematics, not chatting to your friends on Facebook or looking up a random selection of factoids on Google. It is not at all clear to me that these learning objectives will be well served by having mobile phones in the technology classroom, any more than having them in the English classroom.
A second point which flows from the review of the Computing curriculum is that fundamental curriculum decisions are not for teachers to make according to their own lights. Broadly speaking, the job of the teacher is to teach effectively against educational objectives set by governments and exam boards and represented by high-stakes assessment.
If those objectives are often poorly described, then this represents an opportunity to apply technology effectively to describing our educational objectives more clearly (see my How technology will revolutionize research, section 5). If our assessments are not good at representing softer skills like creativity, teamwork and problem-solving, this is another opportunity to apply technology to improving our assessments. But neither of these arguments justifies teachers in ignoring the educational objectives that users and other interested educational stakeholders require the education service to address, in order to pursue their own personal curricula. It is a very poor reason to dismiss PISA’s assessments of literacy and numeracy to argue that many teachers in the UK may not be that bothered to teach literacy and numeracy.
Even when you look at the digital literacy, the educational objective most closely aligned with BYOD, the evidence of the OECD report that digital literacy is heavily correlated with traditional literacy and requires only moderate computer use.
The fact that today’s young people need to learn new skills does not mean that the old skills are no longer relevant; and even if those skills can best be taught by the use of digital devices in particular circumstances, this does not amount to an argument for allowing digital devices in classrooms more generally, where their use cannot be shown to promote learning against the objectives that are appropriate to that particular lesson.
The argument that digital devices should be allowed in every classroom by default because our children need to learn new digital skills is deeply flawed. Its proponents have failed to provide a coherent description of the new “21st century skills” that they suggest are so important; they fail to recognise that the definition of such curriculum objectives is not down to the classroom teacher, who is constrained by the educational objectives set by parents, exam boards and governments; they fail to acknowledge the vital importance of traditional skills, which are underlined by the findings of the OECD report; and they fail to acknowledge that the presence of technology in the classroom ultimately always comes down to pedagogy, not curriculum.
The weight of research
Some have dismissed the LSE and OECD reports on the basis of the weight of evidence that is said to contradict their findings. Professor Margaret Cox of the Institute of Education commented on the report of the OECD report on the BBC website:
The OECD report is flawed. Today, the 4th conference of 100 international experts in research, policy and practice in the use of Computers in Schools: Edusummit2015 in Bangkok (Sept 14-15th) sponsored by UNESCO shows that 40+ years of research from over 65 countries has shown that appropriate uses of computers and other IT technologies can enhance students’ learning.
But what exactly is it that shows the OECD report to be flawed? The fact that 100 academics have gone to Bankok for a jamboree? Or is it the 40+ years of research? Research is not measured in inches or years: it is measured in the strength of its arguments—and no reference is made in this comment to any argument or solid evidence at all.
Donald Clarke takes this same argument to another level in a tweet:
Irrelevant and lazy journalism, we have our own research and reports
This portrays a relativist position in which people produce research papers, like weapons, to defend their pre-existing beliefs, and not as methods of discovering truths which are valid for everyone. Dodge the enemy bullets and fire back as quick as you can with your own. It explains much about the failure of the edtech community to engage with arguments that do not suit it.
In fact, almost all of the research that can be found in the academic journals is based on small-scale studies that rely heavily on attitudinal surveys—the sort of research that is summed up in this sort of infographic:
Some teachers may be very keen on technology and many clearly think that it is improving their teaching. In isolated cases, they may be right. But equally, they may be wrong. Educational research is blighted by the Hawthorne and Pygmalion effects, where respondents tend to tell researchers what the respondents think the researchers want to hear. The views of teachers on what works in the classroom is extremely unreliable, as I have covered in Why teachers don’t know best. This means that research based on attitudinal surveys should be treated with scepticism when it is presented as robust evidence. The Tooley Report (search for “That the research has been done” in the link above), made a strong case that the quality of most academic research is extremely poor and that little reliance can be placed on a great deal of it. If the last 40 years of research has produced strong arguments, then give us the links and let us see whether it amounts to any more than a row of beans.
The arguments that have been made for ignoring research that does not suit us show a lack of intellectual integrity—which is a shame because it is made by people who would have much to contribute to an honest and constructive debate.
What conclusions we should draw
We need to find the middle way
Mark Anderson (@ICTEvangelist) wrote a blog urging us: Don’t bin IT or ban IT—get IT right! I couldn’t agree more. José Picardo wrote a follow-up article in the TES, Headache Tablets, reprinted on his blog, in which he complained that the debate over technology had become polarized, with everyone being characterized either as “crazy-eyed zealot or a complete and utter Luddite”. Again, I couldn’t agree more.
Both Mark and José urge us to find the middle way. But finding the middle way is difficult: it is not a case of just splitting the Mars bar in half. How do we get IT right? What is the middle way? In these respects, I think that Mark and José and most of the rest of the edtech community have got it wrong.
Their first mistake is to denounce what in the case of both LSE and OECD reports is rigorous and well-founded research. They will never get their response right so long as they close their eyes to the evidence that they find inconvenient.
Not only is their denunciation of this research unjustified, it is also imprudent. Because neither report advocates blanket bans or denies the potential for education technology to have a profound effect on education. In the OECD report, Andreas Schleicher writes:
Technology is the only way to dramatically expand access to knowledge. To deliver on the promises technology holds, countries need to invest more effectively and ensure that teachers are at the forefront of designing and implementing this change”.
While I might disagree with Schleicher about the role of teachers in leading this change, this is hardly the voice of someone trying to consign technology to the garbage bin. Similarly, the LSE report does not urge us to bin education technology but rather to make better use of it:
The common theme in these education papers is that the mere introduction of technology [i.e. hardware] has a negligible impact on student test scores, but when incorporated into the curriculum and being put to a well-defined use, technology has the potential to improve student outcomes.
This point is reiterated in the conclusion:
These findings do not discount the possibility that mobile phones could be a useful learning tool if their use is properly structured. Our findings suggest that the presence of mobile phones in schools should not be ignored.
Part of the confusion of those who are hostile to the report is to think that because the LSE report evaluated the effect of mobile phones on learning by studying mobile phone bans, they are therefore advocating mobile phone bans. At no point does the report recommend banning mobile phones: it merely says that the presence in the classroom of mobile phones (and their potential to have negative impacts) should not be ignored.
To ban or not to ban
Tom Bennett, UK education’s new Behaviour Tsar, has been asked by the DfE to draw up a set of recommendations on mobile devices. His initial comments suggest that he is not minded to recommend a blanket ban, but might urge schools to consider restricting mobile phone use for the under 16s. Such a general recommendation would recognise, in a broad-brush way, the LSE finding that mobile phones are unlikely to do any harm to well motivated and skilled learners.
So Tom’s approach strikes me as a common-sense one: let individual schools decide on the basis of the evidence and their own context. And I do not see that it differs so much even from the position of José Picardo, who says that in his own school:
though children are given a tablet, the use of mobile phones is generally not allowed unless a teacher specifically allows it,
continuing to ask for more research which assesses the exam performance of schools with bans
in comparison with other schools that do allow their [mobile phone] use on occasion.
So even those who are the most vociferous in opposing bans in fact support them in some circumstances. There is clearly plenty of middle ground for sensible people to occupy here—and this is exactly the sort of “heads up” approach to the issue that the LSE report is also advocating.
The polarisation of this debate has not come either from the LSE or the OECD reports, but from those in the edtech community who argue (in the words of the recent ETAG report), that
The use of digital technology in education is not optional
…and that all Schools, Colleges and Universities should adopt BYO policies.
This is the opposite of an informed, evidence-based approach. The dogmatism and the consequent polarisation of the debate has been coming from certain sections of the edtech community and not from those who are conducting quantitative research.
Many schools may decide against bans on grounds other than ones of pure pedagogy. A ban might be seen as too hard to enforce; undermining the relationship of trust that the school wants to build with its students; or important for home-school communications. But these decisions should be made on the basis of a careful weighing of the evidence and not because of a poorly justified view that electronic devices are an essential component of every 21st century classroom.
Separating edtech from computing
It is essential that we stop conflating arguments about the effectiveness of technology with arguments about what we should teach. They are separate issues.
It is not for teachers to decide what they want to teach in their classrooms—they are employed to teach towards the educational objectives that are set by exam boards and governments and encoded within their high-stakes exams. These formal educational objectives may certainly be supplemented by teaching other, softer skills and attitudes, which parents are likely to value highly when making their choice of schools. As I have argued elsewhere, we need to make more progress in describing these soft skills more clearly, in ways that can ensure more consistent teaching across the education service. By all means, let us campaign for better summative assessment, that does more to test creativity and genuine problem solving. But that doesn’t mean that teachers have the right to turn their backs on the requirement to teach the subjects that are tested in formal examinations, preferring to devote class time to their own private curricula. In this regard, I am sympathetic to the arguments made recently by Dame Sally Coates for a more centralized curriculum—a project which I believe will not be achievable unless we can use data analytics systems to support our description of educational objectives more clearly (See my recent Setting the Curriculum).
If people want to justify the more extensive use of digital technology in terms of the new skills that students will acquire in this way, then they have a great deal of work to do in order to construct a convincing argument. There has been a strong reaction—amongst employers, Computer Science faculties and government—against the old “dull and boring” ICT curriculum. The arguments made by people like Sir Ken Robinson for new sorts of 21st Century Skills are deeply flawed and implausible (see my article rebutting Sir Ken, a guest article on my website by Scott Goodman, or Tom Bennett’s more recent piece). And the new evidence of the OECD report suggests that the most important prerequisites for operating effectively online are traditional literacy skills (if that wasn’t blindingly obvious to everyone in the first place).
The arguments for 21st Century Skills have bombed: no-one outside a small clique of acolytes takes them seriously. Stop pretending that digital technology has completely transformed the nature of academic subjects across the curriculum: they haven’t. Finding the middle way does not mean that we have to compromise with nonsense.
Constructing a coherent argument for edtech
Instead of denying the blindingly obvious or continuing to advocate discredited curricula, proponents of edtech should be welcoming these reports and responding to their challenge to produce more structured uses of edtech, better integrated into the practice of teachers, which can be shown by quantitative research to be effective as a pedagogical tool when used to teach traditional, non-technical educational objectives—the sort of objective that the rest of the world still values. The sort of objective that the OECD report shows that they are right to value, even in the digital world.
I agree with José Picardo, writing in Headache Tablets, that it is vital that teachers are able to take control of the technology if they are to integrate it successfully into their teaching (which remains vital, of course).
I think I disagree with José Picardo, Andreas Schleicher and Mark Anderson that this means that it is for teachers to lead change, if by leading change it is meant that teachers should effect change. Teachers do not have the necessary understanding of technology, nor the necessary resources to effect the sort of change that is required. The reason why mobile devices are failing to deliver benefits in the classroom is not just because of the negatives (such as distraction) but also because of the absence of positives. We have failed to develop compelling educational-specific software that will make the difference. Such software must be subject to teacher control—to the ability to assign and track work. But this requirement cannot be met by training or by creating new school policy documents: it has to be met by the development of classroom management and software along the lines that I describe in How technology will revolutionize research, section 4.5.—and this has to be done by industry, not by teachers.
There is a single purpose in using an iPad in any situation at all. That purpose is to run software. The problem that teachers face is that they lack the software that is appropriate to their requirements. While I agree with José that schools are “doing technology wrong”, I disagree that this means that teachers have the means to rescue themselves.
I understand if at this stage, having struggled through nearly 15,000 words, teachers reading this article might feel a little cheated. Not only am I suggesting that edtech (as currently implemented) does not work, but also that there is nothing that they can do about it.
Well—I think there is something that teachers can do about it. David Steel will be remembered for telling the Liberal Party to go back to their consituencies and prepare for government. He now wishes he didn’t. What teachers should not do is to go back to their classrooms and hand out the iPads—not at least in the expectation that it will have any real impact. What it is very important that teachers do is to start to contribute to making a compelling argument to Government about what Government should be doing to help bring about the sort of edtech that will genuinely help. I amnot saying that Government will effect the change any more than teachers will effect the change: but it is Government that needs to take the lead and to create the environment in which we teachers can be given access to the right sort of edtech. Structured edtech. Edtech focused on delivering recognised educational objectives. Edtech that can demonstrate its value in quantitative trials. Edtech that works.
When teachers have access to that sort of edtech, then they will be able to lead change in their classrooms and their schools—because then they will have in their hands the tools of the trade that any profession requires if it is to do a good job, faced with the demands and expectations of the modern world.
This sort of reasoned argument for pragmatic action is exactly what the Government asked the ETAG group to provide last year—and the ETAG group flunked what ought to have been a golden opportunity. The door is still open, the requirement is still there, the edtech community still has an opportunity to respond to the challenge—but it will only be taken seriously if it stops talking nonsense for most of the time. It cannot turn its back on robust research. It has to engage with those with whom it disagrees. Only in this way will it build a credible argument that will be heard by Ministers.
If this is what José Picardo, Andreas Schleicher and Mark Anderson mean by teachers leading change, then I withdraw my objection. The tech matters and teachers cannot create the tech that we need—but they can define the requirement, make the argument, and ultimately it will be the teachers that select and manage the tech that is produced. If we are ever to see successful education technology, it will be about empowering teachers, not replacing them. And if teachers are to play a role in leading that development, leading in the sense that I have used the word, their first step must be to engage constructively with the findings of what are two robust and helpful reports.
Ed Tech Now 
As ever, both exhaustive and exhausting, Crispin.
One of the few aspects left uncovered, possibly because of its age, was the Becta finding on 2009 that in their survey of ICT usage, some 34% of secondary pupils said that they used their mobile phones in the classroom to further their research, and help their learning.
So what, do I hear? The survey was carried out in schools where the use of mobile phones in lessons was banned. The awe-inspiring assumption by those in formal education that, whatever the pedagogical merits, they can hold off the 21st Cnetury indefinitely is a little sad, like Canute and the tide?
And before you say it, yes, I know these days it’s Cnut, and that he was probably trying to demonstrate the futility of passing laws to hold back the tide to his nobles. Which makes it an even more relevant point in my book.
Hi Tony, Thanks for a link to this Becta report (without the data) – http://bit.ly/1hlpdkX. I understand from you on Twitter that the 34% figure was given in a presentation by a senior Becta official.
The first thing I would say is that in this period, 2008-9, Becta was paddling away like mad to try and justify its own existence (partly after the catastrophic fall in its approval rating following its Learning Services procurement) and much very disappointing research was being sexed up to the point of downright falsification – not by the academics conducting the research but by the Becta people who were presenting the results.
As an example, the ICT Testbeds report in 2008 produced very disappointing results. In spite of an average of £1 million being spent on technology at each of 30 schools, the project found no significant effects at all on standardised test scores at KS3 or KS4, but a small effect at KS2 by which the overall performance in SATs increased by twice the national rate of grade inflation. This meant that the under-performing schools deliberately included in the trials thereby caught up with national averages. Most statisticians would have explained this effect as regression to the mean, but Bill Rammell, the Minister of State for Higher Education at the time, was given a script at the launch of the Next Generation Learning Campaign which said that the research had showed that children learnt “twice as fast with ICT”. Ha! So one has to be very sceptical of the gloss that Becta tended to put on its research.
In this case, the report speaks of “positive impact” in the Executive Summary but makes clear that there was *no* positive impact on standardised test scores (KS2 SATs & GCSE) – not that they took a very systematic approach to measuring impact:
“Some high-level users achieved beyond their predicted grades; others did not. Similarly, some low-level users exceeded their predicted grades; others did not.”
This outcome was backed up by widespread teacher judgments to stop using mobile devices in the lead-up to the exams because they felt they would not help children get better results:
“Many teachers argue that device use in class does not contribute to ensuring learners’ success in SATs and GCSE…Teachers think that the more open-ended, learner-centred approaches and increased learner autonomy that device-use affords require more time than is available”.
The report pushes back against this finding by arguing that the exams did not give sufficient reward for digitally produced artefacts. But this is to confuse ends and means, it is an exercise in confusing pedagogy and the curriculum or, to put it more crudely, in moving the goal-posts. I discuss this frequent tendency at some length in the article.
The report mentions that what I would call “unstructured” use of IT was strongly coupled with a particular style of independent or autonomous learning, and that this caused many difficulties. One of the most popular ways that children chose to use the autonomy that digital devices conferred on them was not to choose not to use the digital devices, often on account of a lack of confidence in how to use them:
“The ‘digital native’ narrative does not apply to a large number of students in the project”.
The report says that students got bored of doing internet searches and making presentations. “There are signs that they thought this activity was becoming repetitive”. This again confirms the argument I make in the article about the poor pedagogical value of so-called “internet research”.
Yet it is this particular point that the Becta executive chose to trumpet. In a survey of a project in which students *were specifically given digital devices and asked to search the internet*, 34% of them said that they did so. In this context, the figure seems to me to be to be catastrophically low!
The report notes the amount of time that had to be devoted to teaching (and perhaps encouraging) the children to use the devices, and that “A considerable investment of teachers’ time was needed at first. Those prepared to make this investment reaped rewards later”. It is not clear what the rewards were – but whatever they were, they did not include better exam results.
On the positive side there are mentions of some of the *potentially* beneficial effects that the report thinks the digital devices might be having: the ability to show work to parents, to repeat and review exercises, to personalise the device, to blur the boundary between formal and informal learning etc. All this stuff may well be significant in designing future uses of technology. But it does not amount to evidence that the experiment was successful this time round.
The report also mentions “interviews with teachers and students and from student surveys [that] indicate a belief that the use of devices had a positive impact”. I think these answers are what revealed the potentially beneficial effects noted above. It sounds like they got a few answers to a question “What did you like about this project”. But as evidence of effectiveness, it really has no value at all.
The whole exercise strikes me as an absolute disaster – and I am not surprised (putting minor events like elections aside) that Becta was closed within a year of the report being published. One can only admire the Becta executive who managed to convince his audience that it had been a terrific success.
More broadly speaking, I think this is representative of all of that great weight of research which looks impressive when stacked on the shelves but which normally falls apart when you actually read it.
Do you disagree with my analysis?
With respect to Canute, it is true that no-one can hold back the 21st century – it is already here. But you are using “21st century” to refer to something that isn’t the 21st century. You are using the term to refer to your view of what the future will look like. By calling this “the 21st century”, you are trying to give the impression that your particular view of the future is inevitable. For my part, I doubt this very much.
Crispin
You mistake my points…
1. I wasn’t arguing whether or no mobiles were effective for learning. My point was that the students used them anyway, in schools that were banning them. Which makes much of this debate pointless, they will use them anyway, whatever the evidence for or against.
2. My use of 21st Century was somewhat tongue in cheek and a reference to the interminable use of 21st C Learning, even though we are 15 years into the century. I must remember to flag up such jokes with little smileys…. And it wasn’t the century I meant it was effectively impossible for teachers to stop, I meant the students’ use of their mobile devices.
OK – point now taken.
I am sorry to appear humorless – but the joke sometimes get entangled with the argument, and I don’t want to concede the point accidentally, by appearing to appreciate the joke.
I have now found the body of the report at http://dera.ioe.ac.uk/1472/1/becta_2009_mobilelearning_report.pdf.
I have also extracted some data on usage in class and for homework, both as a result of teacher direction and autonomously. My Excel is at https://edtechnowdotnet.files.wordpress.com/2015/10/2009-usage-data1.xlsx.
This shows that mobiles were used in Primary classrooms as a result of teacher direction 77% of the time, but autonomously only 18% of the time. In secondary classrooms, as a result of teacher direction 22% of time and autonomously 10% of the time.
In other words, usage was being driven by teacher direction and not the other way round.
This impression is confirmed by the text.
“The pattern of data here suggests that where teachers make use of
devices in class or for homework, students are more likely to employ them
autonomously for school work”.
It should be borne in mind that all these schools were part of a project in which use of mobiles was being explicitly sanctioned and free mobiles had been handed out. So not only were students being encouraged to use the mobiles by many teachers (especially in Primary) but there was a background assumption that this was a good thing to be doing. Such assumptions can be expected to show up in student responses to questionnaires through the Hawthorne effect.
The bans occurred at classroom level and not school level.
“Teachers’ attitudes to autonomous use of PDAs in class varied
from encouraging, through limited tolerance to outright prohibition. Where there was no expectation of use in class, students – even high users – began to leave their devices at home”.
The lesson here is the opposite to the one you suggest (bans are pointless).
“Students’ attempts at autonomous use of devices in secondary classes were rarer. When asked how their teachers would respond if they did want to use the device in lessons, secondary learners’ answers varied. Most expected to have to ask permission. Some teachers, they suggested, would think it was a good idea. More teachers, they thought, would have a negative response and ask them to put the devices away, or remove them.”
The report notes how *difficult* it was to get children to use the mobiles for learning, largely because of their lack of confidence in their use.
In short, the productive use of mobiles requires a great deal of effort and leadership on the part of teachers, and the corollary of that position is that bans work. Which, again, is what the LSE report suggests.
I think much of this argument is heading in the wrong direction. It focuses on the relationship between the student, their tech and the outcomes. Students don’t so much use the tech to overcome problems on their own. They use tech to construct a group response to a problem. It is only by comparing HE with schools that this becomes teased out as HE are focuses on the student experience rather than on the school league table. I think you need a fresh constructivist approach to all of this.
You won’t be surprised that I am not a fan of constructivism, the problems with which I have touched on elsewhere in my blog (e.g. https://edtechnow.net/2013/11/10/wheel/ and https://edtechnow.net/2012/12/05/tel/).
I also disagree in respect of the specifics of your comment. The argument does not depend on the means by which learning occurs (individual or group). But it is based on the assumption that the end of education is learning that occurs in respect of the individual. Even if we accept that the group might learn (as indeed, I accept that it might when we look at the ways that group cohesion and teamwork improves), this is not the objective in an educational context because the group is transient and does not survive the end of the course.
Similarly, the response to a challenge (whether of a group or an individual) and the experience of the individual are means to an end and not the end itself. If the quality of the experience is all that education is about, then lets just head down to Alton Towers or all slip into a warm bath.
Oh Crispin, Crispin, whatever next? 🙂 “this is not the objective in an educational context because the group is transient and does not survive the end of the course.”
So helping students develop teamworking and collaborative skills, and the ability to work more constructively in short-term project groups, is NOT part of the educational remit or context? Nonsense!
This is one of the things the CBI and many others have complained about… students NEED to learn to work in such a fashion, and some in education are not doing enough to make this part of their educational context! It must be AN educational objective, if not THE educational objective. Anjd since when was this a singular concept? 🙂
You thought you caught me out there, Tony, but I’m slipperier than that. Yes of course teamwork is a very important learning objective but it is an individual skill/capability, something I take away with me into the workplace and express in the context of all the new teams that I join.
What I was talking about as being transient was the capability that the team develops in itself, the extent to which that particular team becomes better than the sum of its parts. I guess we’ve all been in teams like that, when everyone just knows instinctively how everyone else works. That is what I mean by the learning that is vested in the team itself and not in the individuals within it – and that is what is lost at the end of the course.
Of course, the experience of working in a great team will help bring on individual capabilities – so these things work synergistically. But that doesn’t change the fact that the objective of formal education, even in respect of teamwork, is the capability of the individual and not of the team.
Dear Mr. Weston.
Thank you for your detailed analysis of the LSE and the OECD reports! I sympathize with your views on the necessity of creating coherent arguments for EdTech. EdTech entrepreneurs are often very much the C21st skills acolytes you describe, and balanced arguments from the community are few and far between.
Based on that, I suspect the answer must lie somewhere between academia and the industry. To date, however, there is not enough resources being put into research on the effect of EdTech (including anything from subject learning outcomes to hard computer skills, like coding, spread sheets, and the ability to differentiate between online sources, etc.). EdTech (meaning using computers/devices) is a very young industry, and in my opinion still needs to prove it’s value to the education community and the world. It seems obvious that the potential is huge, especially considering the announced lack of qualified teachers a decade from now.
Will you be at BETT 2016? I would be very interested in hearing your opinion on current EdTech trends and your thought on which direction you think the EdTech community should be taking.
Also, this coming friday (Dec 4th), Norwegian EdTech meets English educators in London, Hackney, in a free event. I’ll be pitching in the EdTech-part.
https://www.eventbrite.co.uk/e/oslo-meets-hackney-2015-tickets-19313817147
I think (in all modesty 🙂 ) that we have a few really promising EdTech tools emerging from Norway now, and you might be interested in hearing about them.
Hello Arnfinn,
I very much agree with you – we need both academia and business but they both need to be pointed in the right direction: business to producing technology that is useful and popular with teachers and helps achieve clearly described educational objectives (not which conforms to poorly written government specifications) and academia to providing the expertise that will help business create useful products, not creating meaningless theories on “agency” etc. Although I also agree that the industry is still very immature and there might well be some good stand-alone tools around, I think governments need to do much more in helping point both in the right direction.
In this respect, I am planning a series of blogs on how we understand learning objectives, assessment and the curriculum. I think that one of the most important barriers to developing useful edtech has been our problems in describing our educational objectives. Who can say what is good and what is not, when no-one can be sure what we are trying to achieve? When I have completed this series (perhaps over Christmas), I will be looking to invite appropriate guest bloggers to submit use cases to the blog. Perhaps you might be interested?
On Friday, I am busy in the morning but could make it to East London by the middle of the afternoon. When is your EdTech presentation? I would certainly be interested in seeing what tools are coming out of Norway.
I will be at BETT at some point. DM me with your plans nearer the time and we could arrange to meet. Crispin.
I think you’re right in your assessment that business and academia need to be pointed in the right direction. I’m not sure that the government are the ones who should be pointing, though, at least not based on their current history in that matter (I can only speak for my perception of Norway’s policies here, I am not familiar with UK policies other than from your blog).
A related issue I hope you’ll include in your series is the current room for interpretation and local adjustments to the curriculum (across all subjects, and disregarding any relation with EdTech). Is it possible to write education policies and curriculum and describe educational objectives in such a way that the same values, content and skillsets will be taught in all schools? (How much variation is tolerable in a democracy?) If that is to be possible, I think teacher education must change.
The idea of the teacher as an autonomous professional with the knowledge, skills and experience to make the right choices is widespread but (in my opinion) not very well grounded. As long as dated and disproven pedagogical ideologies are taught in teacher education, teaching will not likely improve much. And, I’m sorry to add, teacher unions have not very useful in this respect.
In addition to this, policy makers must attempt to shake off their own ideology based ideas about education, and focus on what they want the children to learn: Which values, which skills, and what knowledge. Leave pedagogy and didactics to the teaching profession (oh, the irony). In short, policy making must be approached in a very different way than how it has been done till now.
I look forward to reading your planned blog series, and I’m happy to contribute any way I can.
Re: this Friday, the EdTech presentations start at 3.50 pm, and I’m the second presenter, so I’ll probably be on at 4 pm, or a few minutes later.
Re: BETT: Great! I’ll be in touch with you in January.
Arnfinn
Still agreeing with pretty much all of this, including the fact that policy makers have a very bad record too. I am not saying that they can be trusted to get it right, but that only they can establish the framework that is needed. And I have some sympathy with politicians, who are not normally specialists and who must feel themselves to be dependent on the advice of so-called experts, who time and time again give them lousy advice. I am not sure how this vicious cycle can be broken. It needs a few strong politicians who can think this stuff through from first principles and who have the intellectual integrity to follow their (hopefully well informed) convictions.
I agree with what I think you are saying that autonomy is overrated. It should be applied to pedagogy (means) but not curriculum (ends) and even when applied to pedagogy is not incompatible with accountability (ie for the success of whatever means the teacher chooses).
Who sets the ends I believe is (exactly as you suggest) an interesting and complex question. Some will be centrally set and some more locally set – but the one group that I believe must NOT be responsible for setting educational objectives are the teachers. Their job is to help learners achieve the objectives and as suppliers of a service, it is not for them to establish the success criteria.
I shall argue that the main responsibility of education technologists (and a key interest of democratic governments) is that learning objectives should be clearly described. Without this, teachers cannot be held to account by those they are meant to serve. That, I believe, is the situation we are in at the moment. I increasingly think that teachers’ obfuscation of educational objectives is where all the rot originates.
But I am ploughing through some more background reading, so it is going to take a little while to publish. I hope it will be done by the time we meet at BETT.
But not before East London on Friday. I shall try to make it for 3.30. Crispin.
As I said on twitter: You’re preaching to the choir. I look forward to reading your series, and to meeting.
Arnfinn