Since early July, we’ve been in various states of MOOC paper reviews. Our first open call resulted in 266 submissions. After peer review, an invitation was extended to 80 papers for a full submission. Of these 80 invitations, we received 79 full submissions by the deadline. After extensive peer review, 28 papers were selected for funding. Invitations for funding have been sent to the researchers/projects listed below. A huge thanks for the interest expressed and effort expended by all paper submissions.
-The discursive construction of MOOCs as educational opportunity and educational threat: Neil Selwyn and Scott Bulfin
-The Life Cycle of a Million MOOC Users: Laura Perna, Alan Ruby and Robert Boruch
-Professional Learning through Massive Open Online Courses: Allison Littlejohn and Colin Milligan
-Characteristics and completion rates of distributed and centralised MOOCs: Martin Weller and Katy Jordan
-Conceptualising interaction and learning in MOOCs: Rebecca Eynon, Chris Davies, Nabeel Gillani and Taha Yasseri.
-Peer Assessment and Academic Achievement in a Gateway MOOC: Mark Warschauer, Suhang Jiang, Adrienne Williams, Diane O’Dowd, Thurston Domina and Padhraic Smyth.
-Investigating the benefits of embedding motivational messages in online exercises: Joseph Jay Williams, John Mitchell and Neil Heffernan
-Social Network Formation and its Impact on Learning in MOOC-Eds: Shaun Kellogg, Kevin Oliver and Sherry Booth.
-Enabling Resilient Massive Scale Open Online Learning Communities through Models of Social Emergence: Carolyn Rose
-MOOCs Personalization for Various Learning Goals: Sergiy Nesterko and Svetlana Dotsenko
-Secondary School Students and MOOC’s: A Comparison between Independent MOOC Participation and Blended Learning: Rosemary Evans, Dilip Soman, Laurie Harrison and Christopher Federico
-The Relations Between MOOC Participants’ Motivational Profiles, Engagement Profile and Persistence: Bruno Poellhuber, Terry Anderson, Jacques Raynauld, Jean Talbot and Normand Roy
-Understanding the Relationship MOOC Students Have with Traditional Institutions of Higher Education: Christopher Brooks, Stephanie Teasley and Steven Lonn.
-Understanding Massively Open Online Courses (MOOCs) as a Pathway to Employment for Low-Income Populations: Tawanna Dillahunt and Stephanie Teasley.
-MOOC Learner Motivation and Course Completion Rates: Yuan Wang and Ryan Baker.
-Learning Analytics for Smarter Psychological Interventions: Daniel Greene, Carol Dweck and John Mitchell
-Beyond and Between “Traditional” MOOCs: Agile and Just-in-Time Learning: Jennifer Campbell, Alison Gibbs, Laurie Harrison and Stian Håklev
-Writing to Learn and Learning to Write across the Disciplines: Peer-to-Peer Writing in Introductory-level MOOCs: Denise Comer and Dorian Canelas
-Hatch, match, and dispatch: Examining the relationship between student intent, expectations, behaviours and outcomes in six Coursera MOOCs at the University of Toronto: Laurie Harrison, Carol Rolheiser, Stian Håklev and Chris Teplovs
-UW System College Readiness Math MOOC Study: Robert Hoar
-Mapping the Dynamics of Peer-to-Peer Interaction in Massive Open Online Courses (MOOCs): Don Huesman
-Promoting a Higher-Level Learning Experience: Investigating the Capabilities, Pedagogical Role, and Validity of Automated Essay Scoring in MOOCs: Erin Reilly, Stephanie Corliss, Cynthia Louden, Kyle Williams, Emily Cicchini, Donna Kidwell and Dawn Zimmaro
-Developing data standards and technology enablers for MOOC Data Science: Kalyan Veeramachaneni and Una-May O’Reilly
-Patterns of Persistence: What Engages Students in a Remedial English Writing MOOC?: John Whitmer, Eva Schiorring and Pat James
-Detecting and Analyzing Subpopulations within Connectivist MOOCs: Martin Hawksey and Maren Deepwell
-Finding and Developing Talent: The Role of Employers in the Future of MOOCs: Keith Whitfield, Alexandria Walton Radford and Vera Luck
-MOOC instructional design principles: Ensuring quality across scale and diversity: Martha Cleveland-Innes, Derek Briton, Mike Gismondi and Cindy Ives
-A crowdsourced MOOC: David Cormier and Piotr Mitros
A bit about the review process
The program committee for the final stage of MRI consisted of 42 members. Each proposal was reviewed by three reviewers. As the bulk of the review activity happened during the summer months, we (the MOOC steering committee) owe a big thanks to reviewers.
Reviewing papers in a reasonably new area of research is challenging. As the list above reflects, the papers submitted covered broad territory. Papers addressed social, psychological, and technical aspects of MOOCs. Most papers focused on higher education, with some focused on K-12 and a few on professional learning. The research angles included data mining and analytics, AI/machine learning, surveys, focus groups, interviews, etc. Few people have the knowledge to effectively review across the numerous academic domains and research frameworks. Overall I am pleased with the outcome and quality of the review process.
Following notifications of our decision, I received a range of suggestions about the review process. One of the more prominent suggestions related to increasing the number of reviewers to match what happens with national research programs (i.e. up to 5 reviewers) as well as a final “selection by committee” process. These are good suggestions. The nature of this project, to quickly identify promising research projects and advance the dissemination of early stage MOOC research, required a process that was less bureaucratic. This project is not an attempt to mirror national granting processes. The timelines are too short to allow for the typical research cycle. Instead of years, MRI is focused on months. Ideally, national programs will be developed in the near future that will allow for traditional research practices. Or, from another perspective, perhaps when new areas of research arise as rapidly as MOOCs have, we need to adjust our research models. Like much of the academy, current research models seem better designed for an era where information isn’t developed as rapidly as it is today.
If I engage in similar projects in the future, I will make a few changes:
1. Spend more time on the review process, particularly in resolving reviews where differences exist between reviewers.
2. Open reviews for bidding so that reviewers can target personal areas of interest and expertise.
3. Devote additional time at the final selection process to evaluate papers based not only on the outcome of reviews, but also on a mix of criteria (institutions, themes, diversity of topics, etc). During our final process, we essentially selected the papers that were the highest ranked by reviewers. Where we had papers on the bubble, we selected those that did not duplicate research areas of papers that had already been selected.
The response to the MOOC Research Initiative has been significant. We received a total of 266 submissions, with strong international representation. Proposed research topics are broad, including topics such as K-12, higher education, corporate learning, video, automated assessment, social learning, STEM, language learning, etc.
Our initial expectation of completing reviews and informing applicants by July 20 has now been pushed back slightly so that we can ensure each submission receives three reviews. MRI reviewers have been accommodating, particularly in accepting a larger review load than many had initially agreed to take on. Our new expected date to inform applicants of the outcome of the review process is July 26.
In terms of submissions, we will provide a detailed analysis in the next few months. For now, you might be interested to know:
1. Top countries by authors were: USA:211, Canada:86, China: 71, UK:41, Spain:22
2. Countries with the largest number of submissions: USA:110, Canada:39, China:25, UK: 31, Australia:9
3. Applications were received from 37 different countries
4. Reviewers involved in conducting peer review of submissions represented 7 different countries (a total of 37 people assisted in reviews)
5. All continents were represented in submissions (except Antarctica). Submissions heavily favoured America, Europe, and Asia. The absence of significant numbers of submissions from Africa and Latin America is a topic that needs to be explored to be better understood.
The dramatic increase in online education, particularly Massive Open Online Courses (MOOCs), presents researchers, academics, administrators, learners, and policy makers with a range of questions as to the effectiveness of this format of teaching and learning. To date, the impact of MOOCs and emerging forms of digital learning has been largely disseminated through press releases and university reports, with only limited peer-reviewed research publication. The proliferation of MOOCs in higher education requires a concerted and urgent research agenda.
The MOOC Research Initiative (MRI) will fill this research gap by evaluating MOOCs and how they impact teaching, learning, and education in general.
MRI is funded by the Bill & Melinda Gates Foundation as part of a set of investments intended to explore the potential of MOOCs to extend access to postsecondary credentials through more personalized, more affordable pathways. MRI is a $400,000 investment with grants in the range of $10,000 – $25,000 each. The MRI grant program is led and administered by Athabasca University with support from an advisory committee of experts in learning design and MOOCs. The Bill & Melinda Gates Foundation supports Athabasca University and interested academic institutions through research grants to examine the efficacy of early MOOC models for various learner audiences and in a wide variety of contexts.
Due to the rapid development of online education and the pace of MOOC adoption and new technology-enabled models of learning, the grant and research, successful grantees will be announced by the end August, with research beginning immediately. An international conference on MOOC research, which will include interim reports by successful grantees, will be hosted by the University of Texas, Arlington, December 5-6, 2013.
Information on MRI, including call for proposals and timelines, is available at: www.moocresearch.com.
Questions can be directed to firstname.lastname@example.org