ETR&D

Educational researchers have pointed to socioemotional dimensions of learning as important in gaining a more nuanced description of student engagement and learning. However, to date, research focused on the analysis of emotions has been narrow in its focus, centering on affect and sentiment analysis in isolation while neglecting how emotions potentially interact with social elements and course objectives in learning environments. In this paper, we present a case study analysis of seven asynchronous online discussions delivered as part of a blended-learning bachelor level course; we demonstrate the utility of a novel analysis workflow and visualization method which we refer to as SSEEN (social sentiment embedded epistemic networks) to uncover insights into the connection between social networks, course content, and detected student sentiment. The findings show that negative sentiment was most often associated with course content, but contrary to insights from prior research, negative sentiment served as a marker of engagement as students connected content to their own personal experiences. By simultaneously considering the social network alongside the sentiment-colored edges, we note that negative sentiment is not consistent within the discourse of particular students or between pairs indicating that sentiment did not appear to be an indicator of peer conflict or breaches in social contracts. The findings demonstrate how the proposed approach (SSEEN) can support educational researchers to gain a more nuanced understanding of the social, emotional, and epistemic dimensions of learning in asynchronous online discussions.

Technology-mediated interactions and datafication are increasingly central in contemporary social dynamics and institutions, including teaching and learning processes. In order to fully understand the complex entanglements of human and non-human actants that emerge in postdigital education, it is essential to imagine new methodological approaches that are sensitive to the multidimensional nature of education—as a socially and materially-situated phenomenon that increasingly takes place across distributed contexts. The overall goal of this paper is to propose and operationalise a new methodological approach for the study of technology in education. It draws on the notion of relational topologies to improve our understanding of educational settings and, ultimately, how learning unfolds. The proposed approach relies on a multi-paradigm enquiry strategy, based on the idea of using “topologies of digital data practices” in combination with the three dimensions that articulate design-for-learning processes according to the Activity-Centred Analysis and Design (ACAD) framework: epistemic, social and set designs. While the article focuses on presenting the elements of the approach from a theoretical perspective, we illustrate its application through the data collected in a small case study that will serve as a testbed. The topologies of relations we present in this article show uses of technology—as described by participants in their own learning experience—that involve different spaces, devices, and personal situations. In doing so, we reveal how humans and non-humans are entangled in hybrid, unstable and generative ways. The article concludes with some remarks on the value of the proposed approach for studying technology in education and its potential to explore the state-of-the-actual in this field, with the ultimate goal of helping inform educational research, practice and decision-making.

In this paper, we propose a new method for selecting cases for in situ, immediate interview research: detector-driven classroom interviewing (DDCI). Published work in educational data mining and learning analytics has yielded highly scalable measures that can detect key aspects of student interaction with computer-based learning in close to real-time. These measures detect a variety of constructs and make it possible to increase the precision and time-efficiency of this form of research. We review four examples that show how the method can be used to study why students become frustrated and how they respond, how anxiety influences how students respond to frustration, how metacognition interacts with affect, and how to improve the design of an adaptive learning system. Lastly, we compare DDCI to other mixed-methods approaches and outline opportunities for detector-driven classroom interviewing in research and practice, including research opportunities, design improvement opportunities, and pedagogical opportunities for teachers.

Analyzing qualitative data from learning processes is considered “messy” and time consuming (Chi in J Learn Sci 6(3):271–315, 1997). It is often challenging to summarize and synthesize such data in a manner that conveys the richness and complexity of learning processes in a clear and concise manner. Moreover, qualitative data often contains patterns that are not immediately apparent. Consequently, visualization can be an effective tool for representing and unpacking the complexities and multidimensions of learning processes. Additionally, visualizations provide a time-efficient approach to analyzing data and a high-level view of the learning process over time for researchers to zoom in on intriguing moments and patterns (Huang et al. in Comput Human Behav 87:480–492, 2018). In this conceptual paper, we provide a broad overview of research in the field of visualizing qualitative data and discuss two studies (1) visualizing role-changing patterns in an interdisciplinary learning environment and (2) operationalizing collaborative computational thinking practices via visualization. By leveraging these studies, we aim to demonstrate a visualization processing flow along with qualitative research and methods. Particularly, the processing flow includes three critical elements: research subjectivity, complexity of visual encoding, and purpose of visual encoding. The discussion highlights the iterative and creative nature of the visualization technique. Furthermore, we discuss the benefits, challenges, and limitations of using visualization in the context of qualitative studies.

Understanding experiences in online educational settings is crucial to improving teaching and learning. The purpose of this paper is to describe Narrative Inquiry as a research methodology that has the potential create the relational opportunities necessary to understand experiences in online learning environments. In this article, I use an example from research conducted in a special educational setting to overview narrative inquiry methodology, explain its theoretical underpinnings, and highlight its potential to enhance current knowledge of how individuals live alongside one another in online educational settings. I will also explain how narrative inquiry can support the development of new insights about time and engagement in online learning. Then I address how narrative inquiry has the potential to advance equitable research practices in these settings. Finally, I offer suggestions for future research projects that leverage the conceptual strengths and methodological tools of narrative inquiry.

Research fields related to learning (e.g., educational technology and learning sciences) have historically focused on what questions using traditional methods (e.g., comparing different learning tools and methods). New methodologies that are grounded in learning, engagement, and motivational theories are needed to additionally address the how questions. Methodologies that use learners’ process data shed more light on how learners learn and if the learning tools are effective, compared to methodologies that use just outcome data. In this paper, we discuss stealth assessment—an evidence-based methodology that can be used in technology-rich environments (e.g., games) to assess and support hard-to-measure constructs (e.g., creativity) as well as knowledge acquisition (e.g., physics). We also discuss evidence-centered design (ECD), and present specific steps to design, embed in a digital learning environment, and evaluate a stealth assessment. Additionally, we provide two examples of stealth assessment studies in the context of an educational game called Physics Playground: Study 1 illustrates a stealth assessment of creativity and Study 2 describes a stealth assessment of physics understanding and how we used it to make the game adaptive. The purpose of this paper is to provide sufficient detail about stealth assessment to help researchers in the field of educational technology and related fields to adopt this method to assess, foster, and investigate learning processes in various technology-rich environments.

Since 1983, the Instructional Theory Framework (ITF) (and its subsequent improvements) has guided instructional designers and researchers in designing and developing useful learner experiences (LX). For the past 40 years, the ITF was laser-focused on the selection of instructional methods, downplaying delivery methods (media) and management methods. The instructional design field continues to produce immature and confounded research-to-prove studies that do not provide guidance that is useful to practitioners. For more useful guidance, we suggest that researchers should embrace research-to-improve for studying immature methods and media, and research-to-prove for studying mature methods and media. In this paper, we discuss problems associated with proving versus improving, situational deficiencies, and confounding; we present a new version of the ITF that embraces media; and we then answer four key questions about (1) kinds of media knowledge, (2) forms of media knowledge, (3) research methods that deliver the knowledge, and (4) suggestions for editors and reviewers to embrace new media knowledge.

Mixed methods research is becoming more prevalent in educational technology due to its potential for addressing complex educational problems by integrating qualitative and quantitative data and findings. At the same time, a growing chorus of researchers laments the quality and rigor of research in this field. Mixed methods studies which demonstrate explicit integration in educational technology research are scarce, and even fewer apply integration strategies recommended in the literature, such as visual joint displays. Failure to address the challenge of comprehensive integration may result in missed opportunities for deeper insights. To address this methodological problem, the purpose of this paper is to shed light on the procedures, opportunities, and practical challenges associated with mixed methods integration through the use of visual joint displays as an analytical tool for data interpretation and reporting in these types of designs. Using an exploratory sequential mixed methods multiple case study design as an illustrative example, we will (1) provide step-by-step guidance on how to develop a visual joint display to conduct an integrated analysis in a complex mixed methods design; (2) demonstrate how to use a display of this type to integrate meta-inferences previously generated through a series of interconnected joint displays; and (3) illustrate the benefits of integrating at the literature review, theoretical, analysis, interpretation, and reporting levels in mixed methods studies. This methodological article aims to advance knowledge in educational technology research by addressing the integration challenge in mixed methods studies and assisting researchers in this field in achieving comprehensive integration at multiple levels.

The aim of this paper was to explore the use of autoethnography methodology, a non-traditional and reflective approach, in educational technology research. Autoethnography involves a critical analysis of personal experiences and stories being positioned within the larger cultural, political, and social context. Following an overview of the origin and development of autoethnography as empirical research, the authors discuss autoethnography in the context of educational technology research by considering its epistemological and methodological issues. In this paper, the authors also consider autoethnography and its relationship to other qualitative research approaches. Essential components and summarized evaluation criteria for novice autoethnographers are shared. Lastly, the paper reflects on the potential benefits as well as the challenges that those writing an autoethnography will inevitably face. There is a need for autoethnography research in our field to reveal voices hidden in mainstream educational technology research.

Interpersonal online interactions are key to digital learning pedagogies and student experiences. Researchers use learner log and text data collected by technologies that mediate learner interactions online to provide indicators about interpersonal interactions. However, analytical approaches used to derive these indicators face conceptual, methodological, and practical challenges. Existing analytical approaches are not well aligned with the theories of digital learning, lack rigor, and are not easily replicable. To address these challenges, we put forward a multi-level framework linking indicators of individual posting with group-level communication and emergent relational structures. We exemplify the use of the framework by analyzing twenty online and blended courses. Empirical insights demonstrate how indicators at these three levels relate to each other and to potential instructor decisions. Our conclusion highlights current gaps in the framework and the areas for future work.

Researchers tasked with understanding the effects of educational technology innovations face the challenge of providing evidence of causality. Given the complexities of studying learning in authentic contexts interwoven with technological affordances, conducting tightly-controlled randomized experiments is not always feasible nor desirable. Today, a set of tools is available that can help researchers reason about cause-and-effect, irrespective of the particular research design or approach. This theoretical paper introduces such a tool, a simple graphical formalism that can be used to reason about potential sources of bias. We further explain how causal graphs differ from structural equation models and highlight the value of explicit causal inference. The final section shows how causal graphs can be used in several stages of the research process, whether researchers plan to conduct observational or experimental research.

In the field of educational technology and instructional design, research methods are emerging that aim to curate different forms of knowledge and insights beyond traditional research studies, or what Reigeluth and An (in Reigeluth and Carr-Chellman (eds) Instructional-design theories and models: Building a common knowledge base, Lawrence Erlbaum Associates, Mahwah, 2009) refer to as “research to prove.” As a result of evolving efforts in this area, editors of research journals in the field are receiving increased submissions employing these methods but have detected some persistent confusion among authors surrounding them. This has resulted in authors submitting articles with muddled methodologies and to outlets that may not be a fit for the work an author seeks to share. It can even be unclear whether authors intentionally employed a specific design-related method prior to reporting. In this piece, we will cover four methods—instructional design cases, case studies, design-based research, and formative evaluation of designs/products—to provide clarity for both graduate students and researchers. For each of these, we will provide definitions, discuss exemplars and features of exemplars, summarize key features that should be present in such a study and its reporting, and provide guidance on front-end intentional design and planning for research studies that employ these methodologies. Additional clarity on these methods can better support scholars and emerging scholars in their roles as researchers, authors, and reviewers.

Emergent technologies present platforms for educational researchers to conduct randomized controlled trials (RCTs) and collect rich data to study students’ performance, behavior, learning processes, and outcomes in authentic learning environments. As educational research increasingly uses methods and data collection from such platforms, it is necessary to consider the most appropriate ways to analyze this data to draw causal inferences from RCTs. Here, we examine whether and how analysis results are impacted by accounting for multilevel variance in samples from RCTs with student-level randomization within one platform. We propose and demonstrate a method that leverages auxiliary non-experimental “remnant” data collected within a learning platform to inform analysis decisions. Specifically, we compare five commonly-applied analysis methods to estimate treatment effects while accounting for, or ignoring, class-level factors and observed measures of confidence and accuracy to identify best practices under real-world conditions. We find that methods that account for groups as either fixed effects or random effects consistently outperform those that ignore group-level factors, even though randomization was applied at the student level. However, we found no meaningful differences between the use of fixed or random effects as a means to account for groups. We conclude that analyses of online experiments should account for the naturally-nested structure of students within classes, despite the notion that student-level randomization may alleviate group-level differences. Further, we demonstrate how to use remnant data to identify appropriate methods for analyzing experiments. These findings provide practical guidelines for researchers conducting RCTs in similar educational technologies to make more informed decisions when approaching analyses.

The emergence of immersive digital technologies, such as shared augmented reality (shAR), virtual reality (VR) and motion capture (MC) offers promising new opportunities to advance our understanding of human cognition and design innovative technology-enhanced learning experiences. Theoretical frameworks for embodied and extended cognition can guide novel ways in which learning in these environments can be understood and analyzed. This conceptual paper explores a research method in Educational Technology—multimodal analysis for embodied technologies—and provides examples from shAR, VR, and MC projects that use this approach. This analysis involves tracking learners’ gestures, actions on physical and virtual objects, whole body movements and positions, and their talk moves, in addition to other relevant modalities (e.g., written inscriptions), over time and across space. We show how this analysis allows for new considerations to arise relating to the design of educational technology to promote collaboration, to more fully capture students’ knowledge, and to understand and leverage the perspectives of learners.

How is the academic community conceptualizing and approaching the integration of AI in education, considering its potential, complexities, and challenges? This study addresses this fundamental question by employing a multifaceted approach that combines co-occurrence network analysis, latent Dirichlet allocation (LDA), and sentiment analysis on a corpus of abstracts from academic publications from 2018 to 2024. The findings reveal key themes in the scholarly discourse, including the centrality of ethical considerations, the impact of global events on AI adoption, and the practical applications of AI in educational management and policymaking. Moreover, the study identifies the main factors discussed in literature as influencing successful AI integration, the challenges and opportunities associated with AI in education, and the evolving academic perspectives on AI’s role in educational settings. This comprehensive analysis of academic literature provides valuable insights into the current state of AI in education research, highlighting trends, challenges, and sentiments as they have evolved over time. By mapping the landscape of scholarly thought on this topic, this study aims to inform future research agendas, contribute to policy discussions, and provide a foundation for evidence-based decision-making in the development and implementation of AI technologies in educational contexts.

The increased availability of low-cost, standalone and immersive virtual reality (VR) can facilitate adoption in autism education. An immersive VR implementation of the individual work system (IWS) from the TEACCH® approach has the potential to be a safe and predictable environment for autistic learners with or without intellectual disability. This study is a multi-site usability and safety trial examining an immersive VR implementation of the IWS co-designed with autistic pupils and their teachers from three educational centers in the UK, Spain, and Turkey. Twenty-one autistic students aged between 6 and 17 years were involved in the study, six of whom had an intellectual disability. The students tested a total of 164 customized tasks. All participants were able to finish all the tasks. No significant safety issues were identified. The student’s average score on the SUS Usability Scale was 85.36 points. A linear regression analysis showed that autistic children with intellectual disability scored significantly lower on feasibility than children without intellectual disability (p < 0.01) across all locations. This study concluded that an immersive VR-IWS proved usable and safe for the 21 students. However, our findings highlight the need for further adaptations and further research on those with an intellectual disability before recommending universal use. Our findings also have implications for game design for learners with special educational needs.

This study advances the emerging research on shared metacognition through the lens of the community of inquiry framework. It seeks components and utterances of the community of inquiry and shared metacognition in online collaborative learning environments to bring an instructional design model to the fore. A three-cycle design-based research method was followed in two cases of university students by triangulating the quantitative and qualitative data sources. A coding scheme, the shared metacognition questionnaire, the community of inquiry questionnaire and one-to-one and focus-group interview protocols were used as data collection tools. Quantitative data were interpreted through descriptive, inferential statistics, and an open/selective coding process interpreted qualitative data. The findings pointed out that the community of inquiry framework presented a powerful theoretical ground to investigate and distinguish cognitive, social, and teaching presence episodes from shared metacognition episodes. Orientation-planning, monitoring, and evaluation-reflection were proved as three main components of the shared metacognition construct in online collaborative learning settings. This study further advances the specification of each shared metacognition component from the group-related regulative actions and task-related regulatory actions. Moreover, a set of six instructional design principles within an instructional design model that combines the components of shared metacognition were put forward. These guidelines are intended to aid practitioners and instructional designers in the development of online collaborative learning activities.

E-textiles provide an interesting field of research as they “blend traditional craft with modern science” (Peppler, 2016) and help learners “broaden their own perceptions of computing” (Searle et al., 2016). Despite the promising findings by primarily long-term interventions structured around e-textiles, educational curriculum reform has been slow to materialize. Educators who embrace a STEAM philosophy are more likely to endorse short workshops, integrating them in existing courses or initiatives; this could serve as a steppingstone for longer interventions and bottom-up curriculum reform. This study examines whether shorter e-textile workshops (lasting four hours) can result in significant gains in understanding. We present an investigation of e-textiles with 22 young children who have no prior experience with e-textiles or working with microprocessors. We present details of our learning design, as well as findings related to circuitry knowledge and computational making skills. We find that the children advanced their circuitry knowledge and practice a range of computational making skills. We further document a series of emerging challenges, including the children’s unwillingness to engage or lack of adeptness with software, a tension between aesthetics and construction, creativity limited by samples of previous e-textile projects, and the difficulty in grasping the materiality of e-textiles. We propose that some direct instruction and facilitation is not incompatible with the making ethos; the approach can help address these challenges, allowing young children to benefit from their participation in short-duration e-textile workshops.

Despite the efforts of instructional design (ID) to solve real-life problems, it remains challenging to adapt and be flexible in such situations. In particular, problems that require simultaneous knowledge of multiple domains and contexts are more challenging to solve because real-life problems do not reconstruct the learned experience. This is generally thought to stem from differences between learning and real-life practice, but it also stems from instructional designs that fail to reflect the problem's structure and cognitive processes. This study is based on the 4C/ID model and proposes an instructional design for developing and connecting cognitive processes across multidimensional domains and contexts. It employs a simple-to-complex method that combines emphasis manipulation sequencing with simplifying condition sequencing, exposing students to the entire domain and context from the beginning of the learning process to develop a holistic cognitive process. A quasi-experiment was conducted with 34 sophomore college students majoring in education who were asked to create a lesson plan using different teaching styles. The groups consisted of students learning using emphasis manipulation sequencing and single sequencing (emphasis + condition), and the experimental procedure consisted of a total of five sessions, with between-group and within-group analyses of the effects of cognitive strategies and structural models. In the between-group analysis, cognitive strategies and structural models using the single sequencing method were effective from sessions 2 through 5, while in the within-group analysis, the development of cognitive strategies and structural models occurred from sessions 1 through 3, when the simplifying condition sequencing principle was maintained. While the proposed instructional designs are not a foolproof way to develop cognitive processes, a combined approach that considers the nature of the task provides a starting point that can enhance real-life training.

The relationship between technology and educational processes is a complex one. At this moment, increased digitization as well as efforts to limit the use of digital tools can be observed. In view of (a) deepening our understanding of the relationship between technology and educational processes and (b) strengthening the productive educational use of technological tools, elements of a relevant and comprehensive agenda for educational technology research agenda are proposed. The agenda refers to the following elements: educational goals, learners, technological tools, a tripartite relationship learner-tool-teacher, context, and designing learning environments. Recognizing the need for diverse methods and consistent descriptive frameworks can help make the research agenda more cumulative and coherent.