[:en]Like New Year’s resolutions, people’s plans don’t always materialize into actual behavior. Plans to complete Massive Open Online Courses (MOOCs) are a good example. People often enter MOOCs with strong intentions to complete but either drop out mid-course or simply fail to complete all required course activities.
We studied professional development MOOCs for teachers and educational leaders around the globe and found that even educators themselves are not above this self-deception. Near 60% of educators who registered indicated strong intentions to earn a certificate of completion, yet less than 15% actually completed their coursework. These completion rates were very similar to those reported by MOOCs offered by providers such as Coursera and edX intended for broader audiences, despite the fact that many educators needed certificates to meet work requirements.
We wanted to understand this disconnect and ultimately work with MOOC instructors and designers to improve the design of the learning experience. We found that website data analytics complemented self-reports and provided a more complete picture of the user experience—closing the gap between intended and actual use. In addition, thinking about the analytics in a person-centric way preserved the differences between different types of users and revealed patterns that could be used to inform design.
Strong Intentions
MOOC registration forms included a survey gathering information about registrants’ goals, including intentions to complete all aspects of the multi-week course and earn a certificate. We found it was common for over half of the registrants to highly rate intentions to earn a certificate, with an average rating typically between 5 and 6 out of 7. This wasn’t surprising because many registrants needed a certificate of completion to meet work requirements for professional development.
Based on registration, we expected a large amount of online activity and high participation throughout the courses. But despite strong intentions, just over one-third of the participants became regular users, and a smaller fraction actually completed the courses. The majority of participants didn’t even enter their course more than three times. We called these participants “shoppers.” Figure 1 shows these ratios for one of the mathematics education MOOCs.
At some point between registration and the end of the course, learner motivations were changing, even though many participants had a vested interest in completing all of the course activities. However, because we didn’t know when or to what extent this change occurred, we couldn’t start designing effective improvements and interventions to support course completion.
Figure 1. Only 37% of the participants entered the mathematics education course more than three times, and only 7% actually completed the course.
Course Participation
As we dug into the data, the MOOC analytics engine tracked a variety of page views and clicks on course resources. Five key resources best represented participatory engagement in the course:
- Lecture note (text) views
- Lecture video views
- External link views
- Discussion board views
- Discussion board posts
To get a holistic look of users’ participatory engagement over time, we aggregated resource use in each week and plotted the average engagement trajectory through the courses (see Figure 2). This showed that overall, participation decreased through the weeks of the course. However, since we knew some users were completing the course, this overall average trajectory was masking the diversity of actual user experience.
Figure 2. Use of the five key course resources in the mathematics education course dropped each week, with the steepest drop-off in the first three weeks.
To parse out differences between users, we separated data in the analytics engine by the unique identifier tied to each user’s login information, and we collected engagement trajectories for each individual user. Unfortunately, the individual trajectories were difficult to inspect in such large quantities: imagine a line chart with a thousand lines. This led us to statistically cluster user trajectories using a technique known as latent class growth analysis (LCGA).
LCGA identifies groups of people that have common patterns in a variable, such as course use, over time. Groups are formed such that people within a group are as similar as possible while people in different groups are as different as possible. Thus, the resulting groups are representative of distinct experiences and condense the data into a more manageable form. Many of the leading statistical software packages, such as SAS and Mplus can perform this analysis.
Five distinct groups of users emerged from our MOOC engagement trajectories (see Figure 3), each with a different pattern of use:
- High-stable –High participation throughout the entire course
- Mid-rise – Low initial participation with a rise in the middle of the course
- Mid-decline – Intermediate initial participation with a dip in the middle of the course
- Slow-decline –Iintermediate initial participation with a gradual decline, sometimes leading to late drop-out
- Fast-decline – Intermediate initial participation with a rapid decline, often leading to drop-out in the middle of the course
Figure 3. The different styles of participation have distinctive patterns when mathematics education-course use is plotted over time.
These patterns revealed different experiences in the MOOC. Users were engaging in course resources at different intensities and different rates. Although people in the mid-decline, slow-decline, and fast-decline groups all started the course with similar participation, their engagement trajectories were very different through the course—the latter often resulting in drop-out. Notably, people in the high-stable group were consistently more active than the other groups and were most likely to receive a certificate of completion.
A follow-up survey sent to users with low participation revealed a number of external demands that directed motivation away from the MOOC and resulted in decreased participation. In some cases, unexpected emergencies could not be avoided, such as a family member being diagnosed seriously ill. However, time management was the primary factor in most situations. Many users were prone to prioritizing social, family, and work activities over participation in the MOOC as the course progressed.
At the end of this analysis, we wanted to work with MOOC instructors and designers and find ways to motivate different types of users who had the potential of staying engaged in the course. We wanted to motivate those with low participation to engage more with the course and to motivate those who started with high participation to maintain their engagement with the course.
Design Improvements
Taking different types of users into account, we worked with MOOC instructors and designers to design and implement two course improvements that targeted motivation. These changes recently went live, and initial user feedback from surveys and targeted phone interviews suggests the improvements are working.
First, instructors now send out reminders about certificate requirements periodically throughout courses. This approach appeals most to the mid-rise and mid-decline groups who remain active through the end of the course but tend to fall just short of completing all the course requirements. Early signs indicate users find the reminders helpful. Notably, there is a boost in participation in the final third of courses and more users are on track to completion.
Second, a micro-credential system now awards badges for completing activities within each course (see Figure 4). This approach appeals primarily to the slow-decline and fast-decline groups who are likely to drop out of the course. Micro-credentials provide a sense of accomplishment early on and consistently throughout the course. They also encourage users to engage in the course at an increased level of rigor. Early signs indicate that users like the micro-credential system and its presence also puts more users on track to completion.
Figure 4. Some micro-credential badges from the mathematics education MOOC. From left to right: Demonstrating Understanding, Conduct a Student Interview, Build a Plan, and Try It Today. (Credit: DIgitap Promise)
These results reinforce the idea that user insights can be converted into tangible informants of effective design. With an understanding of user goals and behavior, we have been able to modify our MOOCs to help users stay on track to accomplish their goals. As we move forward with future improvements to MOOC courses, we will continue increasing the flexibility of our system to accommodate a diverse body of learners.
Lessons Learned
Through our experience studying MOOCs, we have compiled a few lessons learned which are likely to apply to a variety of other domains as well:
- Intentions do not always predict behavior. Intentions have limited reliability, even when people have a vested interest in using a product. Consider complementing intentions to use a product with self-reports of past experiences and observations of product interactions.
- Interaction analytics can be a valuable tool for understanding users. When implemented in a person-centric way, they provide a trace of the user’s experience in a system. This helps identify individual differences and specific pain points, especially when users seem otherwise equivalent.
- Users are shaped by their interactions over time in different ways. Examining the experience for individual users over multiple points in time can be a fruitful exercise. In our case, it revealed distinct user groups that we would have missed with only information about start and finish points aggregated across all users.
- There is a balance between one-size-fits-all and individually personalized solutions. Adaptive systems are highly sought after, particularly in the educational domain. Unfortunately, the cost of design and development is often too high to be practical since users are different in so many ways. A look at patterns of use can reveal groups of users with common tendencies, and focusing on those group profiles can make the number of design challenges much more reasonable.
[:zh]通过分析研究在线课程的行为，有助于通过设计来促进参与度。研究出来的经验表明，意图并不总是能预测行为，即使人们对使用某产品存有既定兴趣。以个人为中心的分析可以帮助展示互动模式，而只关注所有用户时这会被漠视。这些模式可以揭示有共同倾向的用户群体，从而帮助设计专注于为改善低参与度和退出率而进行更改。
文章全文为英文版[:KO]분석을 통해 온라인 교육의 행동을 연구하는 것은 디자인 참여를 높일 수 있다는데 도움이 됩니다.
연구에서 얻은 교훈은 사람들이 아무리 흥미롭게 제품을 사용해도 그 사용자의 의도가 언제나 행동을 예측하지 않는 다는 점 입니다. 사람 중심의 분석은 모든 사용자만 봤을 때 놓치는 상호작용에 패턴이 있음을 보여줍니다. 이런 패턴은 공통적인 성향을 가진 사용자 그룹을 나타낼 수 있으므로 디자인은 낮은 참여율과 중도 탈락을 개선하기 위해 구체적인 변화에 초점을 둘 수 있습니다.
전체 기사는 영어로만 제공됩니다.[:pt]O estudo do comportamento em cursos on-line por meio de análise ajuda no design para promover participação. As lições da pesquisa sugerem que intenções nem sempre predizem o comportamento, mesmo quando as pessoas têm um interesse manifesto em usar um produto. A análise centrada na pessoa pode ajudar a mostrar os padrões nas interações que são perdidas quando os usuários são considerados apenas de modo geral. Esses padrões podem revelar grupos de usuários com tendências comuns, então o design pode focar em alterações específicas para aprimorar baixa participação e desistências.
O artigo completo está disponível somente em inglês.[:ja]分析を通じてオンラインコースにおける学生の行動を研究することは、参加を促進するようなデザインを生む助けとなる。この研究から得られた教訓は、意図から行動を予測することは、ユーザーがその製品を使用すれば利益があると知っている場合でさえ、必ずしも可能ではないということを示唆している。個人重視の分析は、全ユーザーを一括した分析では見えなかったインタラクションのパターンを見出すのに役立つ。これらのパターンは共通の傾向を持つユーザーの集合を明らかにするため、低参加率や中退を改善するための特定の変更に重点を置いてデザインすることが可能となる。
原文は英語だけになります[:es]El estudio de la conducta en los cursos en línea a través del análisis de estadísticas ayuda al que el diseño promueva la participación. Las lecciones de la investigación sugieren que las intenciones no siempre predicen el comportamiento, incluso cuando las personas tienen un gran interés en el uso de un producto. El análisis centrado en la persona puede ayudar a mostrar patrones de interacciones que se pierden cuando solamente se mira a la totalidad de los usuarios. Estos patrones pueden revelar grupos de usuarios con tendencias comunes, para que el diseño pueda centrarse en los cambios específicos para mejorar la baja participación y los abandonos.
La versión completa de este artículo está sólo disponible en inglés[:]
Retrieved from https://oldmagazine.uxpa.org/understanding-motivations-and-behaviors/
Comments are closed.
