Why most of what currently excites the ed-tech world is hot air: MOOCs, Learning Analytics and Open Education Resources, amongst other fads.
I already know what my new year’s resolution will be. As well as losing a stone in weight (the same resolution every year), it will be to stop writing almost exclusively on why education technology has so far failed to transform education, and to focus more on arguing how education technology will transform education, when it is properly implemented. As the song has it:
You’ve got to accentuate the positiveEliminate the negative
Latch on to the affirmative
Don’t mess with Mr In-between[1].
The predominantly negative copy of 2012 has been no more satisfying to write than I imagine it has been to read. But it has been necessary. It is impossible to make progress with a cogent argument for how education technology will transform education while most of the community accepts as self-evident half-baked notions of “independent learners” and “21st century skills”, believes that creativity is possible without knowledge, or that testing is a dirty word. Before making a start on constructing the new you need to demolish the old.
That will be my resolution on 1st January—but for the last few days of 2012, I will follow the prayer of St Augustine (“Lord make me chaste but not yet”) and take one last swing with the old ball and chain.
The bubble is a fashionable idea (or investment) which become ever-more popular mainly because of its existing popularity. This self-inflating dynamic allows bubbles to expand at an exponential rate, floating away from any solid ground of reality, their substance becoming ever thinner until they eventually pop.
Even though bubbles are deceptive, their existence may nevertheless be significant. The first was the South Sea Bubble of 1720—an investment scam based on a company to which the British government had granted a monopoly to trade with South America, even though war with Spain meant that trade was impossible. Although the company shares were sold on a false prospectus, the scheme was popular because the proposal that money was to be made in overseas trade was plausible. The subsequent history of British global commercial expansion in the eighteenth and nineteenth centuries was to prove that this perception was justified in the long-term. A similar point could be made about the dot-com boom of the late 1990s: although a lot of money was lost in the short-term frenzy, the perception that the internet was going to transform business was fundamentally correct.
In describing in some detail three prominent ed-tech bubbles, I shall not only cover:
- a bit of background
- and why the bubble will pop;
- but also what are the pre-requisites required to achieve a more substantive development
- and what might be their long-term significance.
Massive Open Online Courses (MOOCs)
Background
Of all the bubbles listed in this post, only MOOCs really deserve the label in terms of attracting real investment. After 160,000 students enrolled for Sebastian Thrun’s course on Artificial Intelligence in the autumn of 2011, three new MOOC-delivery platforms were established in 2012: Udacity was set up by Thrun with more than $15m venture capital; Coursera was set up by Stanford with $16m of venture capital; and edX was set up by MIT and Harvard with $60m funding provided by the founders. In December 2012, a consortium of UK universities led by the Open University declared its intention to set up FutureLearn to rival the US initiatives.
The phenomenon has aroused excitement, not only ed-tech circles but also in the mainstream press. The New York Times declared that 2012 was “The year of the MOOC” (2 November 2012), while the Economist concludes that “Online courses are transforming higher education, creating new opportunities for the best and huge problems for the rest” (22 December 2012).
Why the bubble will pop
The following is adapted from a comment I left on the Economist article.
MOOCs do not work, either commercially or pedagogically.
much of what’s being lauded as ‘revolutionary’ simply involves videotaping lectures and putting them online
As the Economist article acknowledges, MOOCs do not yet have a business model. The fact that they are free is nothing to get excited about – it is easy to give things away until the money runs out.
When it comes to pedagogy, the article is hopelessly optimistic. It skates over the drop-out rate (which is in the order of 90-93%), blaming this on the lack of a proper qualification (which students will discover at the end of the course, if they did not realise it before they start). It should blame the drop-out rate on the poor pedagogy, which is what they discover during the course.
The Economist article states that that
MOOCs are more than good university lectures available online. The real innovation comes from integrating academics talking with interactive coursework, such as automated tests, quizzes and even games.But, like the Khan academy before it, “lectures available online” is predominantly what they are. As Audrey Waters writes in a characteristically well-researched post, “much of what’s being lauded as ‘revolutionary’ and as ‘disrupting’ traditional teaching practices here simply involves videotaping lectures and putting them online”. Audrey has attended 15 MOOCs in 2012, so she should know.
The lesson of the UK Open University and University for Industry is that while distance learning addresses the problems of the isolated learner (predominantly the adult in the workplace), it does not come cheaper than face-to-face delivery[2] owing to the need for expensive one-to-one tutoring support.
At a speech at Online Educa Berlin in November, Gary Matkin, Dean of Continuing Education at the University of California, commented that many were signing contracts with with the MOOC companies that were not compatible with their status as leading universities, in that they were losing control of the quality of the courses that they were agreeing to certify.
Robert Cummings, Assistant Professor of English at the University of Mississippi said that many universities were getting into these contracts due to the advocacy of individual faculty members, who were motivated by a combination of vanity and a desire to publicise their own books. He summarised, “Everyone wants to jump on-board and no-one is quite sure what they are jumping onto”.
Before jumping on-board, the MOOC-enthusiastic universities should consider the words of William Cory, Assistant Master of Eton, who wrote in 1861:
You go to school at the age of twelve or thirteen and for the next four or five years you are engaged not so much in acquiring knowledge as in making mental efforts under criticism.[3].If this is true of schoolboys in 1861, how much more true should it be of undergraduates in 2012. And yet those following MOOCs have no opportunity to make mental effort under the criticism of anyone except their peers—i.e. no-one who can relied on to have domain knowledge superior to the learner.
an academic education is not equivalent to a trip to the public library, digital or otherwise
Diana Laurillard, Professor of Learning with Digital Technologies at the London Institute of Education, highlights the same misconception in a book published earlier this year:
There is a danger that technology could undermine formal education…Arguments against formal education are now current again but, uninformed by any understanding of the theory of teaching and learning, they plunge us back into traditional approaches. Technology opportunists who challenge formal education argue that, with wide access to information and ideas on the web, the learner can pick and choose their education – thereby demonstrating their faith the transmission model of teaching. An academic education is not equivalent to a trip to the public library, digital or otherwise. The educationalist has to attack this kind of nonsense, but not by rejecting technology[4].Missing prerequisites
The key challenge is scalability. Peer mentoring (discussion groups and grading each other’s essays) might be a good ingredient in a broader mix, but without the long-stop of proper tuition, it doesn’t offer a sufficiently authoritative and informed conversation. When none of the peers is an expert, there is too much risk of misconceptions and bad habits becoming established within the cohort.
The staple that is required to handle large numbers of students successfully is machine-instruction, blended with expert tuition. Machine-instruction is what the Economist refers to in passing as “automated tests, quizzes and even games”. The critical aspect is not the “testing” itself but the analysis of the test results and the ability to offer appropriate feedback, guidance and adaptive instruction. In an environment which blends digital and human instruction, the learning analytics layer should produce recommendations for human intervention. You could have a seminar (maybe delivered online), with its topic, learning objectives, participants and tutor all automatically selected by a data-driven learning management system.
until now education technology has been regarded by the mainstream press as a backwater, inhabited by starry-eyed visionaries
These are technologies which have not yet been developed, largely because education (at least in Europe) is funded by governments giving money to teachers and academic educationalists, who are very good at spending it on their own research projects but which have been disconnected, if not actively hostile, to the industry responsible for the supply of education-specific software and other technology.
Long-term significance
The first (and not to be underestimated) achievement of MOOCs is to get the New York Times and the Economist interested in education technology. Until now, however crazy government policy was in this area, it has been regarded by the mainstream press as a backwater, inhabited by starry-eyed visionaries and not by serious investors. The venture capitalists who have backed Udacity and Coursera may well get their fingers burnt—but at least they or their successors will bring some feet-on-the-ground business analysis to the task of finding out what went wrong.
The failure of MOOCs will concentrate minds on what are the prerequisites for success. In the long term, we can hope that the movement will lead to proper R&D that is commercially-funded and responds to market requirements.
Learning analytics
Background
Learning analytics (which I take to be synonymous with Education Data Mining) has been attracting significant interest over the last couple of years. The Society of Learning Analytics Research (SoLAR) has been holding annual conferences since 2010; in 2011, the New Media Consortium’s (NMC) annual Horizon Report for emerging technology in Higher Education predicted that learning analytics would be a key technology for education in 4–5 years and in 2012, it reduced the timescale to 2–3 years; in October 2012, the US Department for Education published a significant briefing paper on Enhancing Teaching and Learning through Educational Data Mining and Learning Analytics; the US National Science Foundation and the European Commission are both investing in academic research into learning analytics[5]; and in the commercial sphere, Pearson ad McGraw Hill are both investing in learning analytics and adaptive learning systems.
analytics is predicated on “big data” but in education, big data will not exist until we sort out the current failure of interoperability
Learning analytics aims to apply to learning the same techniques that are now used by online companies to target their online marketing. By spotting patterns in the data produced by students’ online learning activity, learning analytics systems should be able to help:
- predict student progress;
- inform adaptive learning strategies (sequencing digital learning activities or recommending human interventions);
- profile a student’s current capabilities;
- automatically group students, depending on their learning needs;
- identify the most effective learning strategies in different situations;
- aggregate and present complex data in ways which helps administrators, teachers and students manage instructional processes.
Why the bubble will pop
Being a “research bubble” and not an “investment bubble”, it is unlikely that the failure of the current round of predictions will be so obvious as I suggest it will be for MOOCs. It is more likely that the current round of projects will finish, everyone will pat each other on the back, bank their research grants, add another project to their CVs, and move on. If you doubt that the concept of the “bubble” is applicable to the academic world, consider carefully the appeal made by Professor Erik Duval to his academic audience at the recent ITK conference:
So hey, one reason to try and take notice is you may be able to apply for some funding and do some work on learning analytics, even though, you know, you may not be completely sure what it is[6].The reason why that learning analytics will not deliver on its promise is that analytics is predicated on “big data”. In education, big data does not yet exist and will not exist until we sort out the current failure of interoperability. This is an argument which I have made at some length in an earlier post, Learning analytics for better learning content.
The crippling effect of poor interoperability on learning analytics was illustrated by the first of the SoLAR online webinars on learning analytics, given by Chris Ballard, who presented the system developed by Tribal for use in Higher Education. This produces a single output “at risk” metric for HE students, based on five input metrics, shown in the screenshot below.
In the second screenshot, a fictional student, Juliana McWilliam, is categorised as being “at risk” in all of her five current modules.
Such an “at risk” register may be of some use in a high-volume education system in which students have little personal contact with lecturers—but I find it hard to imagine that in any halfway-decent university, the fact that Juliana was struggling would not already be well known to her supervisors. Even if this data were useful, the metric produced is simplistic and one-dimensional. There is no attempt to profile Juliana’s different abilities, to suggest what sort of interventions (other than a one-size-fits-all kick up the backside) would be appropriate, or to gauge the success of different instructional strategies. If this is learning analytics, then it is clear that we at the stage of taking the first, faltering steps.
Missing prerequisites
The problem in the case above does not lie in the modelling being done by the developers at Tribal but in the lack of data which they can access. The lack of data comes down to the lack of interoperability standards which would allow semantically meaningful data to be harvested from educationally significant activities.
The video of Professor Duval’s presentation of learning analytics illustrates exactly this point. At 16:13, he shows the metrics being tracked, which covers the extent to which a student is using a variety of blogging and discussion tools. This is bit like marking an essay by its length. Learning analytics only works as a layer in a broader architecture in which learning activities are making educationally-significant judgements about student performance, and passing those judgements on in the form of semantically meaningful data.
Erik Duval himself recognises this problem. Having listed all the quantitative metrics that the current systems are tracking, he comments (21:31):
…and that’s sort-of interesting…but of course, what we really like to track is how well they learnt, and that’s much more difficult.From the perspective of learning analytics, creating good input metrics is more than difficult—it is out of scope.
Long-term significance
Learning analytics is critical to the business of education: to that extent, the learning analytics community has got it right. It is not that the advocates of learning analytics are wrong—it is that, like the advocates of MOOCs, that they are premature.
If the organisations currently funding research on learning analytics are sufficiently self-critical, then they may even come to ask themselves why the projects they are currently setting up eventually fail. If they do ask themselves that question, then we may start to see money starting to be invested more wisely in five years’ time.
However, there is at least an equal chance that the funding organisations will draw what would appear to be the safer conclusion from a future failure of research: that analytics is not applicable to education. Such a conclusion could set back the progress of education technology another decade.
Open Education Resources
Background
According to Wikipedia, the term “Open Education Resources” (OER) was first formally recognised by UNESCO’s 2002 Forum, which was held in response to the Massachusetts Institute of Technology Open Courseware project. This was initially funded by the William and Flora Hewlett Foundation, the Andrew W. Mellon Foundation, and MIT itself. Costing about $4 million a year, the MIT Open Courseware project is scheduled to run out of funds in 2014.
The OER movement has spawned many other initiatives. The Khan Academy was established in 2006, funded principally by the Bill & Melinda Gates Foundation and Google. In 2008, the Cape Town Open Education Declaration urged governments around the world to make education resources available free of charge. Many responded, not only aspiring to make education available to developing countries, but also perceiving that OER represented a cost-effective means of providing learning resources to their own education systems. In 2003, the UK government approved the abortive BBC Jam project, worth £150 million; [deleted text] the European Commission has been prominent in funding OER projects, announcing at a recent Ministerial conference in Norway its intention to launch a major new strategy, “focused on the use of ICT and open educational resources (OER) to enhance education and skills development”, by mid-2013.
in nature, weak animals die but in a government-funded OER ecosystem, useless resources that nobody wants survive and multiply
The “open” in OER is defined by Wikipedia “freely accessible, openly formatted and openly licensed”. But there is no such thing as a free lunch. OER has absorbed significant amounts of funding in what is still a very immature market for digital content.
Why the bubble will pop
OER represents a bubble for the same reason as the MOOCs that grew out of the same movement: the quality of the resources themselves and the pedagogies they represent are poor. A document released by the JISC in support of a UK HE OER project, explains that resources “can take the form of text, images, audio and video, and may even be interactive”. The fact that interactivity (so essential to the process of learning) is claimed as a rare bonus reveals the dreary truth, that the vast majority of resources are expositive. The assumption made by such programmes falls into the naïve fallacy highlighted above by Diana Laurillard that education is about the dissemination of information. It is the same fallacy that is promulgated by the original Cape Town declaration, when it decares that
Educators worldwide are developing a vast pool of educational resources on the Internet…creating a world where each and every person on earth can access and contribute to the sum of all human knowledge.The development of online information is the achievement of the World Wide Web: the job of educators is not to add yet more information but to manage the interactive process of learning how to use that information.
The failure of this type of initiative to achieve, in the words of Diana Laurillard, “any understanding of the theory of teaching and learning”, is concealed by the assumption of a certain moral superiority amongst the OER community. OER is supposed to save the honest practitioner from the piratical depredations of the commercial salesman, while, according to David Wiley, founder of MIT Open Courseware, quoted in the JISC paper:
Openness is really the only means of doing education [because] if I’m not sharing what I know, if I’m not giving you feedback, if I’m not engaging in this give and take with you there is no education.This piece of sophistry confuses a decision about how you want to fund a service with the need for good feedback within a learning process—a critical element of any instruction. This conflation of business models with pedagogy is a surprisingly popular piece of muddled thinking: many people talk about “learner generated content” as a subset of OER, when what they are really referring to is student product: artifacts which are created by a student as an output of an instructional process but which are neither intended nor useful as an input for new processes.
The lack of commercial investment in many OER resources is aggravated by the fact that, as no-one has to buy anything, there is no automatic quality filter. In nature, weak animals die; in an efficient market, companies producing poor quality products go out of business; in a government-funded OER ecosystem, useless resources that nobody wants survive and multiply. This problem is recognised implicitly by the briefing paper produced for the European Ministerial meeting in Oslo in December 2012 (written in typically opaque Eurospeak):
The very large heterogeneity of Open Educational Resources available and types of contributors to it, is leading to an increased difficulty in adopting quality standards and quality assurance tools. For the users identifying quality resources and/or sources may be a very difficult constraint to overpass. It is important establishing quality parameters for OER, as well as assessment and certification processes which can lead to the validation of skills acquired through OER.This extract not only supports my assertion that OER is being undermined by poor quality. It also demonstrates why the OER bubble is such a tough one to pop—that consummation that in this case is so devoutly to be wished. Instead of being funded by venture capitalists, who will at least learn quickly from their mistakes, OER is increasingly funded by governmental bodies, whose very existence depends on the continued funding of these programmes. Instead of recognising the error of funding supply-side initiatives, these bodies propose to introduce even more government-funded programmes, introducing “quality parameters” and “assessment and certification processes” that will validate skills, not for their own sake, but for the fact that they were “acquired through OER”.
The state funding of OER is nothing short of a racket. A majority of those who are sufficiently closely involved to know what is going on are themselves implicated—this includes most of the Higher Education community who, under the pretence of being consumers of useless OER are in fact being funded to produce useless OER (or conduct associated research). That is the reason that no-one (not even the academics who advise governments under the pretence of being ed-tech experts) is prepared to come clean and admit that these programmes have achieved next to nothing and should be abandoned.
the state funding of OER is nothing short of a racket
OER cannot be isolated from a much wider and more troublesome “bubble”: the tendency of all public sector projects that are not subject to rigorous political control to grow, regardless of whether or not they are productive. The OER bubble will only ever “pop” for the reason that this wider bubble will pop: the need to implement radical austerity programmes in order to avert the real and present danger of sovereign bankruptcy.
Missing prerequisites
What is pernicious about OER is its funding by government and misguided charitable foundations. There is nothing wrong with OER itself, so long as it’s production is not funded externally.
Unfunded voluntary production may be expected to occur naturally: OER may be produced by commercial organisations in order to achieve publicity, by enthusiasts for personal kudos; and by practitioners in order to save time through collaboration with peers.
In none of these cases will the production process attract substantial investment and for this reason, no-one should look to OER as a means of addressing significant deficiencies in current forms of education technology. The critical—and still largely missing—characteristic for effective learning content is interactivity (see my earlier post, What do we mean by “content”?). This will be provided by commercial software, developed in response to a healthy ed-tech market. One particular type of educational software that needs to be developed will be high-level authoring tools that embed complex interactivity and good pedagogical design. Once the heavy lifting has been done by commercially funded R&D, users of such software will be able to share the content that has been authored by such tools, confident that this content will at least be sound technically and pedagogically. Community networks will have no trouble recognising academic quality through the management of the reputation of the academic (and frequently volunteer) authors.
I have made the argument for high-level authoring tools in an earlier post, Learning analytics for better learning content. That post also highlights an associated and fundamental prerequisite for any content that is to be used productively in formal education: technical interoperability.
Long-term significance
The significance of the government funding of OER programmes is entirely negative, discrediting education technology by flooding the market with poor quality materials and undermining the commercial incentives required for the emergence of the ed-tech market.
Conclusion
There are other intellectual bubbles that I could list such as e-Portfolios, serious games, and adaptive learning. Most of these are similar to learning analytics: they contain good ideas which are still premature, principally because we lack genuine interoperability and a market that allows the development of robust, education-specific technologies.
e-Portfolios
e-Portfolios will not work without:
- creative tools that can interoperate with learning management systems to automate the assignment of creative activities and the handling of creative product through a complex cycle of review, redrafting, assessment, reflection and sharing;
- competency definitions against which student-created artifacts can be assessed.
Serious games
Serious games will not work without:
- run-time protocols that allow games to be assigned and launched automatically, allocating students to particular groups and roles;
- run-time protocols that allow outcomes to be reported automatically to learning analytics systems.
Adaptive learning systems
Adaptive learning systems will not work without:
- protocols that clearly define and allow the automatic launch of all kinds of learning activity;
- learning analytics systems which are capable of tracking student learning through an instructional process, creating the predictive profiling information required to control selection of appropriate learning activities.
Next steps
In all of these cases, what really matters is:
- the emergence of an ed-tech market that will address in the correct order all the different pieces of this complex jigsaw;
- ending interference by government funding programmes in the development of technologies that governments do not understand and that in any case require risky, entrepreneurial investments;
- instead, establishing processes that challenge uncompetitive markets and encourage the industry to develop appropriate interoperability specifications.
The withdrawal of government interference is already happening in the UK, which puts the UK in a particularly strong position to move on to the next step.
better interoperability is the next affirmative step that we need to take
Interoperability is not only necessary to the emergence of a true ed-tech market—different types of interoperability, as listed above, are also as essential prerequisites for many types of development, such as learning analytics, e-portfolios, serious games and adaptive learning.
It is therefore clear from this out-with-the-old tour of ed-tech bubbles that better interoperability is the next affirmative step that we need to take. It will therefore be to the theme of interoperability that I shall return (brimming with positivity) in the new year.
Related posts
 The problem with “Technology Enhanced Learning”, criticises this acronym and the academic-led initiatives that are commonly subsumed by the term. A shorter version of the analysis of MOOCs is included. |
 What do we mean by “content”? analyses the use of this poorly-defined term, which often excuses low-grade, non-interactive, information-bearing resources. The post makes the argument that there are many types of content and the sort we really need bears not information but activity. |
 Learning Analytics for better learning content explores the preconditions for enabling learning analytics to be used to improve the quality of learning content (one of which is commercial provision of content)—concluding that interoperability is the most important. |
 Home page, with a full listing of posts on this blog. |
Notes
[1] A song by Johnny Mercer and the Pied Pipers, later used in the BBC’s Pennies from Heaven
[3] Quoted in Resources for Learning, by L C Taylor, Penguin Education, 1971, p. 22. L C Taylor takes the quotation from Anthony Chevenix-Trench, the Public Schools, in Peter Blander (ed.), Looking Forward to the Seventies. Smythe, 1968, p. 76.
[4] Teaching as a Design Science, Diana Laurillard, Routledge, 2012, p. 4.
[5] See Erik Duval speaking at ITK 2012.
[7—deleted text] “while the Higher Education Funding Council for England has funded an OER programme worth £50 million; “. I have been told by David Kernohan, who ran this programme, that my valuation of this programme was inaccurate, that “they didn’t even spend 15 million” and this was on distribution infrastructure and not content. See https://edtechnow.net/2012/12/29/moocs-and-other-ed-tech-bubbles/#comment-924.
Ed Tech Now 
Reblogged this on Classroom Aid and commented:
I believe passionately that education technology has the potential massively to improve the efficiency of our education system—but that this improvement is being held back by misguided, prescriptive bureaucracy. I believe that government needs to support the emergence of an open and innovative market—and that one of the most important things that government can do in this respect is to support the emergence of capable data standards for better interoperability.
Reblogged this on Things I grab, motley collection and commented:
an opinion I don’t totally subscribe to but is worth, still, reading if only for the sake of having a new perspective of things.
I like your article/comments… You articulate the thoughts I’ve been having over the past several months! I’m a math tutor in the suburbs of Chicago — ages 9-19, so I’m working with students of various abilities who are taking different classes (that have different textbooks, teachers, approaches, etc..)… The MOOC’s alone cannot be a one size fits all… (By the way, I had no idea the drop-out rate was so high!)…. Learning must come from different resources — teachers, effective explanations/examples, practice tests, video, adaptive learning (i.e. the ability to go backward or forward depending on the student), and more… Personally, I believe it’s all out there — BUT, it’s not in one place (yet)… I know that my 30+ students get all of that from me! But, when will the complete, effective education reach 30+ million? (I suppose a lot of money will be spent/invested/wasted trying to get there…)
Hi Lance, I like your point about different resources/learning styles. I think talk about different learning styles rather misses the point. It is not that we are so very different, one from another, but that we all benefit from learning/applying new concepts in a range of different ways and contexts: abstract and concrete; collaborative and individual; didactic and experiential; auditory, visual and kinesthetic etc.
I think the characteristic of current teaching practice in which “my 30+ students get all of that from me!” goes to the root of the current problem. It places huge burdens on the teacher, results in a very inconsistent service (some teachers are great, others have to be endured), isolates teachers one from another, places productivity ceilings on the best teachers, and places barriers in the way of efficient institutional management. This (what I call the craftsman model of teaching) is the subject of my next post.
I agree that money needs to be spent to implement the technologies that will lead change. The problem (in my eyes) with the current situation is that we have a bureaucratic way of managing education in which money is poorly spent – and we need more efficient markets that direct the investment to the right places.
Hi Crispin. Thanks for your interesting and insightful post. I agree that it is essential that we take a balanced view of educational technology, and your post is a timely New Year reflection which questions why we are doing what we are doing. It is vital that educational technology is not just done for its own sake, and that we don’t ride a wave of excitement and hype without taking a step back to consider whether it will deliver real tangible benefits for the education sector and, crucially, for students.
Focusing on learning analytics, I’d like to respond to your comments about the learning analytics prototype I presented at the recent SoLAR online webinar. I agree with you that learning analytics is at an early stage – there is no disputing that. I also agree that having access to sufficient and relevant data that allows us to draw appropriate conclusions is essential to the success of a learning analytics project. You refer to the metric as being “simplistic and one-dimensional” – to which I think you refer to the overall success prediction against each module. Although helpful, I agree that a single overall prediction is too limiting, and for this reason we break it down into separate predictions against each of the 5 success themes. The aim is to give staff some insight into which areas the student may be struggling, and inform any discussions between the staff and student. This was discussed in the presentation and example screenshots were shown in later slides.
You also refer to the need to consider interventions in any analytics. I absolutely agree, and for this reason we will be looking at this area in a future stage of the project. In my presentation I highlighted such “actionable insights” as being a key objective for learning analytics solutions and this was discussed at various points during my presentation.
Learning Analytics is still in its infancy and its success relies not just on sufficient data being available, but also how that data can be presented to ensure that staff and students make appropriate conclusions. Indeed, most learning analytics practitioners are at the stage of evaluating how available data can be best exploited, and are exploring how it can be mapped to student learning capabilities or outcomes. I believe that analytics can be helpful if it is used to give staff information that encourages staff-student dialogue, and highlights potential issues which can help to inform and focus any resulting discussions.
Hi Chris, Many thanks for the comment, which I completely agree with. The purpose of my post was not to knock the principle of learning analytics (or any of the other “bubbles”) but suggest that it would be premature to present them as market-ready solutions until we have solved the problem of interoperablility. Which does not in any way knock the useful R&D you are doing at Tribal. I am sure that the academic community also has an important contribution to make, with the proviso that we need commercially-exploitable work if we are to apply innovative techniques in a sustainable way.
Two further thoughts.
1. When the interoperability log-jam is broken, I predict that you will get a lot of innovation in the instructional software “layer”. Different “learning activity delivery services” will produce different kinds of learning outcome data. This will mean that the learning analytics layer will find itself standing on an unsteady platform (i.e. a constantly changing set of input metrics). So I think that LA software will need to decouple the LA algorithms from the particular input metrics that they are using, aiming for more generic ways of spotting predictive relationships between input metrics [w, x, y] and output metric [z].
2. I think that, in interoperability terms, a vital prerequisite for capable LA systems is what I call “competency definitions” – the ability to identify different types of ability, knowledge domain, behaviour etc, posit relationships between them, and start to posit relationships between ephemeral performance outcomes (8/10 in a Maths test on Wednesday morning), with more durable competency profiling information. The authoring of those competency definitions may often be done by different agencies (particularly exam boards), in a form that anyone (instructional software developers, learning analytics software developers etc) can reference.
There are two “competency definitions” draft specifications in preparation, one in ISO/IEC/JTC1/SC36 and one being led by JISC/CETIS in Europe’s CEN Learning Technologies Workshop (see http://wiki.teria.no/display/inloc/Home). The latter has a meeting in Brussels on 17 January (anyone can turn up) – though I hope that we might be able to get some UK discussions going around this topic over the next couple of months, hosted by BSI/IST/43 (which I chair) which – if we can get a quorum of UK stakeholders, might be a more useful forum to participate in. Either way, I am keen to get more commercial implementers – and particularly those involved in analytics, involved in these processes. Ping me at crispin.weston@saltis.org if you might be interested.
Best, Crispin.
Crispin, this post is great. Too many American discussions tend to the extremes: excessively politicized or quixotic hype. It is incredibly refreshing to hear a rational and balanced discussion of both the challenges and the opportunities edtech presents.
Am I correct that you’re based in the UK? If so, consider this an open invitation to visit us and infect us with your sensibility.
Heather, Many thanks for your kind words, here and on Twitter. Yes, I am based in the UK, though I have weekly calls with a US standards community that straddles LETSI (an organisation which came out of the US DoD SCORM community) and the IEEE Learning Technology Standards Committee) – and occasionally I get over to your side of the pond for a face-to-face.
I think the problem of “politicized or quixotic hype” is not just a US one! I guess an emerging technology allows everyone to project their own aspirations onto the future. Its not that dreaming should be discouraged but that we need an environment which weeds out the realistic dreams from the unrealistic ones. Which I think is the market, not the bureaucratic tender or government-funded research project.
Do drop me a line at crispin.weston@saltis.org – it would be great to talk further.
Pingback: What would be the implications of MOOCs on Higher Education? | Learner Weblog
Thanks Crispin, I really enjoyed this well-argued post and will definitely be following your blog going forward. I also see the MOOCs as somewhat overhyped and would emphasise the importance of having proper instructors. I recently blogged about this at http://wp.me/p1PeiH-2X, where I saw them as symptomatic of a modern tendency to attempt to automate aspects of human interaction which simply cannot be automated effectively. I do think, however, that MOOCs may increase pressure on HE institutions to demonstrate the clear value they are adding in terms of teaching and support. This is not always obvious at present.
Thanks again, Daniel
Thanks for the link to your blog Daniel, which I thought was on the button. I also agree that the current MOOC phenomenon is not altogether bad – it at least gets people’s attention, highlights the requirement, emphasises (if ever it was needed) that education is a serious business, and that there *are* ways – which I am sure will emerge over the next decade – in which we will see serious, transformative change.
It’s ironic that the first comment, which you took some time to reply to, is actually boilerplate spam. (Google the phrases in it, and you’ll find the exact same comment’s been made at lots of other blogs. The commenter isn’t posting it because they’ve taken some time to engage with your thinking, but because they can get a SEO-boosting link back to their website– the one under their name– if they paste something in that sounds somewhat plausibly related to the post.)
The relationship between this sort of large-scale pseudo-“engagement” and the sort of automated or uninformed feedback being passed off as a substitute for expert evaluation in many large-scale online courses is one that’s worth thinking about.
Great Point John
Note how seductive machine feedback can be even to a reputable critic of machine feedback. – An apt demonstration of the growing power of machines to meet some of the criteria (or straw dog) that Dianne Laurillard posits marks REAL education. But rather than compare MOOCs, analytics or OERs against unattainable, unsaleable or unreal expectations of the “perfect” education which we know permeates existing campus based education (not!!) why not compare them to learning without these resources.
I am certainly not defending current education practice (though school, rather than HE, is my particular interest). But I am not sure that “anything is better than nothing”. My concern is that the selling of half-cock solutions based on poorly thought-out strategies (or, more often, “excessively politicized or quixotic hype”, as Heather Gilchrist puts it), actually inhibits more productive approaches.
As my post argues, I welcome the entrance on stage of the venture capitalists because they are good at recognising and managing failure. Bureaucracies are very bad at this.
Thanks for fingering the spam John. I have deleted that thread – not to cover the traces of my own gullibility – but to avoid getting other readers to wade through a pointless discussion between me and the wall.
For those who are interested in the exchange in the light of this discussion, the original spam comment was “I was wondering if you ever thought of changing the layout of your blog? Its very well written; I love what youve got to say. But maybe you could a little more in the way of content so people could connect with it better. Youve got an awful lot of text for only having one or 2 pictures. Maybe you could space it out better?” My first reply was a lengthy set of reflections on the problem and my second was “Hello again chemietoilette. I have added some “take-outs” which I hope might stimulate the interest of the flagging reader. Let me know what you think”.
I was taken in because the comment complained that the post was too long and wordy, which something I was worried about already. It may therefore be interesting that in response to my “pseudo-interlocutor”, I considered what I could do about this problem and added the new tabloid-style “take-outs”. If you accept that these improve the accessibility of the article in an age of short attention spans, then you might also agree that I learnt something, even though I was interacting with a machine.
You might reply that it was lucky that the spam message was relevant in any way and I agree with your criticism of “pseudo-engagement” (though I have no objection myself to polite robots that say “Hello Crispin” when I interact with them). But if the machine reaction *were* based on a genuine analysis of my work, maybe involving some sort of AI, then do you not think that this would be useful? I am not denying the importance of genuine interaction with an expert tutor, which I think will always need to provide the sheet anchor of any significant educational process – but suggesting that the “productivity” of those human teachers can be significantly increased by the assistance of appropriately sophisticated machines. I guess this is what Terry is also referring to by “the growing power of machines to meet some of the criteria that Dianna Laurillard posits marks REAL education”.
Reblogged this on From experience to meaning… and commented:
Well, this post looks more critical than it is, as it acknowledges that there are possible benefits but that there are a lot of things to sort out.
Thanks Pedro. Spot on. I would go further and say that the possible benefits are massive (even if the individual online courses won’t be).
Interesting reading, thanks for sharing and I see a lot of sense in what you are saying. But just to pick you up on one factual error. HEFCE didn’t spend 50m on an OER programme, they didn’t even spend 15 million. I know because I was running the bloody thing, which is also why I am confident that not one penny went towards paying academics to produce OER – our interest was in establishing scalable and sustainable approaches to releasing OER beyond the comparatively small amounts of funding we were offering. And as OER is still being released in large quantities, months after our funding ceased, I’d say we were fairly successful in that!
I might come back to leave a more considered comment when I have my work head back on. Best wishes for 2013!
p.s: You’re dead on about a lot of the MOOC stuff, if you ask me,
David, Thanks for the information – I stand corrected and clearly should have checked my information more carefully. I have corrected the main text, moving the original text to a footnote with caveat.
I agree that appropriate spending on OER infrastructure is justifiable, so long as it responds to production that is not artificially boosted by external funding) and does not skew the market in a way which inhibits commercial investment in innovation. It also raises interesting questions such as will your OER distribution system allow the circulation of content that depends on the purchase of proprietary reader/viewer software? My own view is that – for a “level playing field” it is important that it should – while also allowing searchers to filter results of course. Failing to allow this type of content traps users in a low-tech environment.
I am involved in school-level (K-12) ed-tech rather HE, but my impression that HE suffers somewhat from the researcher-hobbyist and that a dose of commercialisation in terms of delivery platforms would do no harm at all in creating more capable educational software appropriate to HE needs.
I look forward to your considered comment (am happy to give a guest-post slot to more extensive piece). I think that finding the right balance (or even better, synergy) between commercial content and OER is an important objective.
Pingback: MOOCs and other ed-tech bubbles | Metaglossia
Pingback: Diigo links 01/04/2013 : DrAlb
Pingback: Lectuur op zaterdag: marketing en het brein, MOOC’s, intuïtie « X, Y of Einstein?
Reblogged this on moocdesign.
Pingback: MOOCs and other ed-tech bubbles « The 21st Century Education
Pingback: Can We Keep the EdTech Bubble From Bursting? - Online Universities.com
Cross posting a comment I left on an interesting blog on MOOCs at http://ryan2point0.wordpress.com/2012/11/26/the-future-of-moocs/#comment-4092
Preamble (what about the disadvantaged?). This is a reason for *wanting* MOOCs to work, not a reason why they *do*. My argument is that MOOCs are premature – they *will* work when we have capable, interactive software in place. At the moment, we don’t. MOOC platforms might invest their $15 in developing these new digital pedagogies, but the money will run out, especially as innovation is inevitably a hit-and-miss business and much of it will be wasted. What they need is a supply chain.
In response to your points.
1. “Why would I pay tens of thousands of dollars [to go to university]?”Because what MOOCs offer is in no way equivalent to a university education.
2. “The vast majority of students will be overseas”. When the online pedagogy is sorted, maybe, but for serious education I suspect that they will still need local tuition.
3. “The pecking order will be reshuffled”. Maybe – but the brand advantage gives the old universities a head start – and that will count for a lot. Coursera and Udacity are making the news because they are the first on the block – but what do they have that cannot be replicated. Harvard, Stanford, Oxford and Cambridge have brands that are unique. And what the university will be providing is (a) the tuition element, and (b) the brand. On the whole, I side with the Economist, (http://www.economist.com/news/international/21568738-online-courses-are-transforming-higher-education-creating-new-opportunities-best) not you on this.
4. Agreed – competition will hot up and though I think top universities will have a distinct market advantage, they will have to keep peddling.
5. “Studying on campus will become a status symbol”. Maybe – but face-to-face contact (not necessarily on campus) is *much* more than a status symbol. But I agree that classrooms will be flipped. Maybe the richer foreign students will fly in for quarterly “summer” schools (or to your distributed “real-life meetups”).
6. “The role of the teacher will evolve”. Completely agree. The essence of the teacher should be as conversational
(i.e. responsive) tutor.
7. “The pedagogy of MOOCs will be enriched”. Agreed – I think our main difference is that I see this “pedagogical
enrichment” as (a) a necessary prerequisite for this getting off the ground, and (b) requiring market innovation – not something that the universities or even MOOC platforms can do for themselves.
8. Agreed – also at the distributed centres.
9. Also agreed. We need to develop proper data standards that can recognise
credit (and validate its worth – i.e. if institution x says that student y is
good at z – and this proves to be untrue – then the value of accreditations by
institution x is devalued on the “social network”.
10. “Online cheating will mushroom”. Low-stakes solutions will emerge from this – but some supervised exams will still be required to validate the results of e-tests. An illustration of why we should not think of going wholly online, but instead looking for blended options.
11. “Academic inflation will skyrocket”. Non-validated certificates (which includes Mozilla badges) will lack any credibility, so it won’t be so much a case of inflation, as of printing toilet paper. Not only is completion insufficient, but the predictive value of accreditation will be validated by learning analytics systems. That will be what will create real credibility in the certification business.
12. Offshoring in the UK is in retreat, owing to lack of personal contact / cultural misunderstandings (which is even more important for education than for questioning your electricity bill). Some more impersonal may be offshored.
13. Agreed corporate sector will gain from better online training. I should say that they already do and the whole kerfuffle over MOOCs was based on the claim that they were going to revolutionise HE.
14. Ditto.
15. “An instructivist approach suits novices, while an increasingly constructivist and connectivist approach suits learners as they develop their expertise”. I don’t think it always works this way: often the more experienced learner just wants to acquire a bit of sector-specific knowledge and is happy with an expositive approach. The key is discover how to handle the activity-based approach (this is currently the missing piece), after which you will be able to deliver whatever blend of the two (exposition and activity) that is appropriate.
Crispin.
Thought-provoking post, Crispin. Some of your points about MOOCs I agree with, others I don’t.
I agree that MOOCs aren’t “revolutionary” in terms of the *science* of education. A MOOC is arguably an extensible version of what universities (in particular) have been doing all along – albeit on a massive and free scale. Yet it is these latter aspects that make MOOCs revolutionary in terms of the *delivery* of education. This new paradigm (opening up content to millions of people for free) is a tectonic shift from the old model of higher education, being to restrict content to those who pay for it. By signing up to a MOOC, for example, I can study cryptography at Stanford University. Under the old model, only a privileged few could do that.
I don’t think “lectures available online” are inherently bad. Sure, the pedagogy can be improved and on-campus education is arguably superior, but if the alternative is nothing at all, I’ll take the lectures.
Regarding the business model, I think the content component of a MOOC could remain free. If the provider operates a freemium model whereby they charge for the assessment and certification, it may very well be financially viable. Then there are other intangibles to consider such as branding, goodwill, customer perception, CSR etc. How much is all that worth to the broader business?
Like you, I don’t buy The Economist’s excuse for the high dropout rates as being the lack of a proper qualification. However, I’m not sure it’s due to poor pedagogy either. I’m more inclined to believe that all the participants who sign up to a MOOC have different needs. Some are just sampling the course, while others cherry pick what’s useful for them at the time. Not everyone is looking for a formal, programmatic, accredited education, so they don’t bother “completing” it. Notwithstanding the formal intent of the MOOC, many learners will participate informally.
Will MOOCs evolve and grow or will they “pop” into oblivion? The proof of the pudding will be in the eating. It will be intriguing to witness either way.
Hi Ryan,
Thanks for the comment and sorry for my slow response – I seem to have been particularly busy over the last couple of weeks.
I agree that opening up education and making it available at scale are important objectives – I have written on the scalability issue at “Education’s coming revolution” (http://wp.me/p27xY2-8o).
I also agree that “lectures available online” are not inherently bad – they may be useful, even necessary perhaps – but not sufficient. A good video can be helpful as a quick “how to” guide to doing something fairly straightforward – but not as a way of addressing challenging learning objectives that require the acquisition of new skills or a shift in the student’s cognitive world view.
In this field (which is the field of HE) they *may* be better than nothing – but if students waste a lot of time doing something that does not deliver the goods, then they may be worse than nothing too. The key point is that without real conversational tutoring, then they are not *equivalent* to a university education and the talk of their revolutionising HE is over-the-top.
I agree with your point about drop-out rates. It is likely to be a complex picture. Our two different explanations may merge: people will have different motivations and expectations – and many of the latter will be unrealistic. So the pedagogical limitations of a predominantly expositive delivery is likely to mean that those coming along out of idle curiosity will tend to be shepherded towards the exit rather than hooked into rewarding study. So it at least represents a failur of salesmanship: although the people who wander into a shop may arrive with all sorts of different motivations, the shopkeeper should still be a bit worried if 93% of them wander straight out again.
I *suspect* that most of those who finish will be already well educated, not too busy, and will be highly motivated to pick up some extra knowledge.
What drove me to write the post was not any sort of intrinsic dislike of what MOOCs are trying to achieve – I tend to agree with you that they will probably find themselves a useful niche alongside all sorts of existing forms of online instruction. I write because I am a passionate advocate of education technology myself. I am therefore dismayed by hype that overclaims the impact of what are actually pretty simple digital pedagogies. We could do so much better if we exploited the essentially interactive and adaptive nature of IT to address the interactive and adaptive requirements of education. We might even succeed where I suspect the MOOCs will fail!
Pingback: RESOURCE: MOOCs and other ed-tech bubbles | Ed Tech Now | ArcheMedX
Pingback: Outro(s) lado(s) da(s) moeda(s) « Diálogos sobre TIC & Educação
Pingback: Key issues in OER and how we just might overcome them | learningshrew
Pingback: Arriva iVersity: siete pronti? « Oggi Scienza
Pingback: Looking into the Future – Utilizing MOOCs to Solve Grand Challenges or Wicked Problems | Education Remixed
Pingback: From cMOOCs to xMOOCs and why the difference matters | learningshrew
Pingback: New Publication: "The Rise and Stall of eLearning: Best Practices for Technology-Supported Education" - ArcheMedX