Blog

The MOOC Research Initiative conference in Arlington, Texas was held in early December. The event was memorable – and not only for the ice storm that stranded many attendees. The quality of research being conducted by MRI grantees in open learning (MOOCs specifically) is impressive. Personally, an important take away from the conference was the sophistication of research and the stark contrast with the hype-filled media view of MOOCs. While prominent media promotes grand narratives of MOOCs as disruptive, transformative, and sure to end the current model of higher education, MRI grantees, keynote speakers, and panels offered a vision of MOOCs as supplementing and enlarging the role of the university. Generally, presentations were informative, thoughtful, and indicate that a strong foundation is being built for future research around MOOCs, openness in education, learning at scale, and emerging pedagogies. Prominent during the conference was the recognition that educators need to start thinking about “what happens after MOOCs”. No clear consensus arose from the conversations that I participated in, but the general tone was one of expecting higher education to continue subsuming MOOCs under umbrellas of marketing and recruitment, alumni outreach, online/blended learning, and broadening the role of universities in a knowledge economy.

Research projects often require researchers to work on their own and then present their work to peers once research has been completed. When we were planning MRI, we (MRI Steering Committee) wanted to provide researchers an opportunity to connect with peers before research projects were nearing conclusion. Research and literature in MOOCs is still limited and by providing grantees with opportunities to hear about the research of their peers, we anticipated new research connections, conversations, and ultimately, higher quality research. Based on the feedback of attendees, the sharing of early stage research was valuable and informative.

If you want reactions on the conference, a quick search of MRI13 (or the #mri13 tag on Twitter) will provide numerous reflections from attendees. Here are a few that I found captured the event well:

Keith Devlin: The MOOC Express – Less Hype, More Hope
Jim Groom: De-Icing the MOOC Conference
Michael Feldstein: Changing the Narrative
Bonnie Stewart: the post-MOOC-hype landscape: what’s REALLY next?
Martin Weller: The Iceland of Dallas

There was some talk about a future MOOC conference. Dave Cormier and Bonnie Stewart suggested MRI14 in Charlottetown, PEI (the birthplace of Canada). Nothing has been decided, but for information on anything related to MRI13, MRI14, or MOOC research in general, please join this Google Group for updates: https://groups.google.com/forum/#!forum/mooc_research

Since early July, we’ve been in various states of MOOC paper reviews. Our first open call resulted in 266 submissions. After peer review, an invitation was extended to 80 papers for a full submission. Of these 80 invitations, we received 79 full submissions by the deadline. After extensive peer review, 28 papers were selected for funding. Invitations for funding have been sent to the researchers/projects listed below. A huge thanks for the interest expressed and effort expended by all paper submissions.

-The discursive construction of MOOCs as educational opportunity and educational threat: Neil Selwyn and Scott Bulfin
-The Life Cycle of a Million MOOC Users: Laura Perna, Alan Ruby and Robert Boruch
-Professional Learning through Massive Open Online Courses: Allison Littlejohn and Colin Milligan
-Characteristics and completion rates of distributed and centralised MOOCs: Martin Weller and Katy Jordan
-Conceptualising interaction and learning in MOOCs: Rebecca Eynon, Chris Davies, Nabeel Gillani and Taha Yasseri. 
-Peer Assessment and Academic Achievement in a Gateway MOOC: Mark Warschauer, Suhang Jiang, Adrienne Williams, Diane O’Dowd, Thurston Domina and Padhraic Smyth.
-Investigating the benefits of embedding motivational messages in online exercises: Joseph Jay Williams, John Mitchell and Neil Heffernan
-Social Network Formation and its Impact on Learning in MOOC-Eds: Shaun Kellogg, Kevin Oliver and Sherry Booth. 
-Enabling Resilient Massive Scale Open Online Learning Communities through Models of Social Emergence: Carolyn Rose
-MOOCs Personalization for Various Learning Goals: Sergiy Nesterko and Svetlana Dotsenko
-Secondary School Students and MOOC’s: A Comparison between Independent MOOC Participation and Blended Learning: Rosemary Evans, Dilip Soman, Laurie Harrison and Christopher Federico
-The Relations Between MOOC Participants’ Motivational Profiles, Engagement Profile and Persistence: Bruno Poellhuber, Terry Anderson, Jacques Raynauld, Jean Talbot and Normand Roy
-Understanding the Relationship MOOC Students Have with Traditional Institutions of Higher Education: Christopher Brooks, Stephanie Teasley and Steven Lonn. 
-Understanding Massively Open Online Courses (MOOCs) as a Pathway to Employment for Low-Income Populations: Tawanna Dillahunt and Stephanie Teasley. 
-MOOC Learner Motivation and Course Completion Rates: Yuan Wang and Ryan Baker. 
-Learning Analytics for Smarter Psychological Interventions: Daniel Greene, Carol Dweck and John Mitchell
-Beyond and Between “Traditional” MOOCs: Agile and Just-in-Time Learning: Jennifer Campbell, Alison Gibbs, Laurie Harrison and Stian Håklev
-Writing to Learn and Learning to Write across the Disciplines: Peer-to-Peer Writing in Introductory-level MOOCs: Denise Comer and Dorian Canelas
-Hatch, match, and dispatch: Examining the relationship between student intent, expectations, behaviours and outcomes in six Coursera MOOCs at the University of Toronto: Laurie Harrison, Carol Rolheiser, Stian Håklev and Chris Teplovs
-UW System College Readiness Math MOOC Study: Robert Hoar
-Mapping the Dynamics of Peer-to-Peer Interaction in Massive Open Online Courses (MOOCs): Don Huesman
-Promoting a Higher-Level Learning Experience: Investigating the Capabilities, Pedagogical Role, and Validity of Automated Essay Scoring in MOOCs: Erin Reilly, Stephanie Corliss, Cynthia Louden, Kyle Williams, Emily Cicchini, Donna Kidwell and Dawn Zimmaro
-Developing data standards and technology enablers for MOOC Data Science: Kalyan Veeramachaneni and Una-May O’Reilly
-Patterns of Persistence: What Engages Students in a Remedial English Writing MOOC?: John Whitmer, Eva Schiorring and Pat James
-Detecting and Analyzing Subpopulations within Connectivist MOOCs: Martin Hawksey and Maren Deepwell
-Finding and Developing Talent: The Role of Employers in the Future of MOOCs: Keith Whitfield, Alexandria Walton Radford and Vera Luck
-MOOC instructional design principles: Ensuring quality across scale and diversity: Martha Cleveland-Innes, Derek Briton, Mike Gismondi and Cindy Ives 
-A crowdsourced MOOC: David Cormier and Piotr Mitros

A bit about the review process

The program committee for the final stage of MRI consisted of 42 members. Each proposal was reviewed by three reviewers. As the bulk of the review activity happened during the summer months, we (the MOOC steering committee) owe a big thanks to reviewers.

Reviewing papers in a reasonably new area of research is challenging. As the list above reflects, the papers submitted covered broad territory. Papers addressed social, psychological, and technical aspects of MOOCs. Most papers focused on higher education, with some focused on K-12 and a few on professional learning. The research angles included data mining and analytics, AI/machine learning, surveys, focus groups, interviews, etc. Few people have the knowledge to effectively review across the numerous academic domains and research frameworks. Overall I am pleased with the outcome and quality of the review process.

Following notifications of our decision, I received a range of suggestions about the review process. One of the more prominent suggestions related to increasing the number of reviewers to match what happens with national research programs (i.e. up to 5 reviewers) as well as a final “selection by committee” process. These are good suggestions. The nature of this project, to quickly identify promising research projects and advance the dissemination of early stage MOOC research, required a process that was less bureaucratic. This project is not an attempt to mirror national granting processes. The timelines are too short to allow for the typical research cycle. Instead of years, MRI is focused on months. Ideally, national programs will be developed in the near future that will allow for traditional research practices. Or, from another perspective, perhaps when new areas of research arise as rapidly as MOOCs have, we need to adjust our research models. Like much of the academy, current research models seem better designed for an era where information isn’t developed as rapidly as it is today.

If I engage in similar projects in the future, I will make a few changes:
1. Spend more time on the review process, particularly in resolving reviews where differences exist between reviewers.
2. Open reviews for bidding so that reviewers can target personal areas of interest and expertise.
3. Devote additional time at the final selection process to evaluate papers based not only on the outcome of reviews, but also on a mix of criteria (institutions, themes, diversity of topics, etc). During our final process, we essentially selected the papers that were the highest ranked by reviewers. Where we had papers on the bubble, we selected those that did not duplicate research areas of papers that had already been selected.

The response to the MOOC Research Initiative has been significant. We received a total of 266 submissions, with strong international representation. Proposed research topics are broad, including topics such as K-12, higher education, corporate learning, video, automated assessment, social learning, STEM, language learning, etc.

Our initial expectation of completing reviews and informing applicants by July 20 has now been pushed back slightly so that we can ensure each submission receives three reviews. MRI reviewers have been accommodating, particularly in accepting a larger review load than many had initially agreed to take on. Our new expected date to inform applicants of the outcome of the review process is July 26.

In terms of submissions, we will provide a detailed analysis in the next few months. For now, you might be interested to know:

1. Top countries by authors were: USA:211, Canada:86, China: 71, UK:41, Spain:22
2. Countries with the largest number of submissions: USA:110, Canada:39, China:25, UK: 31, Australia:9
3. Applications were received from 37 different countries
4. Reviewers involved in conducting peer review of submissions represented 7 different countries (a total of 37 people assisted in reviews)
5. All continents were represented in submissions (except Antarctica). Submissions heavily favoured America, Europe, and Asia. The absence of significant numbers of submissions from Africa and Latin America is a topic that needs to be explored to be better understood.

The dramatic increase in online education, particularly Massive Open Online Courses (MOOCs), presents researchers, academics, administrators, learners, and policy makers with a range of questions as to the effectiveness of this format of teaching and learning. To date, the impact of MOOCs and emerging forms of digital learning has been largely disseminated through press releases and university reports, with only limited peer-reviewed research publication. The proliferation of MOOCs in higher education requires a concerted and urgent research agenda.

The MOOC Research Initiative (MRI) will fill this research gap by evaluating MOOCs and how they impact teaching, learning, and education in general.

MRI is funded by the Bill & Melinda Gates Foundation as part of a set of investments intended to explore the potential of MOOCs to extend access to postsecondary credentials through more personalized, more affordable pathways. MRI is a $400,000 investment with grants in the range of $10,000 – $25,000 each. The MRI grant program is led and administered by Athabasca University with support from an advisory committee of experts in learning design and MOOCs. The Bill & Melinda Gates Foundation supports Athabasca University and interested academic institutions through research grants to examine the efficacy of early MOOC models for various learner audiences and in a wide variety of contexts.

Due to the rapid development of online education and the pace of MOOC adoption and new technology-enabled models of learning, the grant and research, successful grantees will be announced by the end August, with research beginning immediately. An international conference on MOOC research, which will include interim reports by successful grantees, will be hosted by the University of Texas, Arlington, December 5-6, 2013.

Information on MRI, including call for proposals and timelines, is available at: www.moocresearch.com.

Questions can be directed to gsiemens@athabascau.ca