ETR&D

Artificial Intelligence (AI) has brought about significant changes in our lives, making AI literacy a crucial endeavor for the future. Despite its growing importance in academia, there is limited empirical research on its impact on college students’ higher order thinking skills (HOTS). The present study systematically and comprehensively explores the relationship between college students’ AI literacy and HOTS using the 3P (Presage-process–product) model. In this model, students’ AI literacy represents the presage factors, while behavioral engagement and peer interaction serves as the process factors, and HOTS is the product factor. We gathered data from a survey of 260 college students. We utilized structural equation modeling to analyze the relationships between the 3P factors. The results showed that both AI usage and AI evaluation directly influenced HOTS and also indirectly affected HOTS through the mediating role of behavioral engagement and peer interaction. Conversely, AI awareness and AI ethics showed no direct influence on HOTS, although AI awareness impacted HOTS via peer interaction mediation. The results of the study have several theoretical and practical implications. From a theoretical perspective, this study incorporates AI literacy, behavioral engagement, peer interaction, and HOTS within the 3P model framework, shedding light on their interrelations. On a practical note, the results emphasize the need to consider AI literacy, behavioral engagement, peer interaction when designing courses to enhance HOTS in the era of AI.

Visible thinking and design thinking are two approaches that have drawn attention. This study integrated these approaches into the teaching of creativity, by which we developed and evaluated the effects of “Making Creative Thinking Visible” (MCTV) on the learning of creativity and creativity mindsets among college students. Five course modules (Introduction, Reset mindset, Unfold mind, Stimulate mind, and Cocreation and reflection) and nine thinking routines of visible thinking were included in the MCTV. The participants were 99 college students, with 54 students receiving the MCTV (the experimental group) and 45 students not receiving any training (the control group). The results suggest that the 12-week MCTV is effective in improving college students' creativity and growth mindsets (both the growth-internal control and the growth-external control mindsets) but not in decreasing fixed mindsets. Additionally, reflections on self-changes during the training show the participants' cognitive transition and the critical components for the success of MCTV. Altogether, the findings of this study provide enlightening thoughts for related educational training and implications for further research.

This article explores: (1) the barriers facing students and teachers regarding this type of assessment, and (2) a proposed framework for overcoming these barriers. In this study, the barriers and corresponding assessment techniques were determined by interviewing a sample of 17 students and five teachers. The main barrier reported by the students was the difficulty of group work, whereas the teachers found it difficult to assess manual teaching aids. Thus, a framework was proposed to address these barriers by enhancing students’ collaboration and developing their problem-solving skills through authentic activities. The effectiveness of this framework was ascertained in this study by examining the development of the students’ problem-solving skills and their perceptions of the learning design. After the experiment, there was development in some constructs of the problem-solving skills. From an analysis of the students’ responses, it was revealed that they had very positive perceptions of the framework.

This study employed speech-enabled language translation (SELT) technology in lectures presented in English as a medium of instruction (EMI). Previous research had students viewing pre-recorded lectures with incorporated content translations in controlled laboratory settings. In contrast, students in the present study experienced live lectures in actual classrooms, where the SELT system immediately translated English speech into Chinese and displayed the translation for students. The aim was to explore the potential of SELT technology in enhancing students’ understanding of the lecture content and aiding their discussions. The study used the experimental research method. Sixty university students were recruited and divided into three groups, namely, Control group 1 (students did not have any support), Control group 2 (students received transcriptions) and Experimental group (students received translations). Three lectures in EMI were carried out, and student discussion around lecture topics were arranged after each lecture. To achieve objectives of the study, students’ knowledge prior and after lectures were measured and compared, and the content of students’ discussion was analyzed. The results showed positive effects of SELT technology on learning achievement. That is, students who received SELT texts performed the best on the tests compared to their counterparts. The results also showed that discussion of students who used SELT texts was more efficient. Finally, the students had positive perceptions of SELT texts. We believe that applying SELT technology to lectures in EMI in the physical classroom settings was a feasible and useful way to facilitate students’ comprehension of lecturing content and their discussion. Therefore, it is suggested that educators and researchers can use SELT technology in lectures in foreign languages as a medium of instruction in the physical classroom to facilitate student learning.

The present paper examines possible benefits of spatial audio guides on learning outcomes in the spatial learning setting of an experimental exhibition and attempts to differentiate between different mechanisms underlying such an effect. In Experiment 1, we examined whether the spatial contiguity principle may be such a mechanism. A spatial audio guide was used that allows for perception of audio explanations either at the location of the exhibit (audio-visuo-spatially close) or within the headphones (audio-visuo-spatially distant). The results showed that with the external, exhibit-located compared to headphone-located presentation, memory of audio explanations was significantly better. No significant differences were found regarding text-picture-integration. However, these results cannot only be explained by audio-visuo-spatial contiguity but also by externalization and audio-spatial cueing. In Experiment 2, we therefore tried to disentangle these possible mechanisms and compared three conditions (audio explanations perceptually located within the exhibits, behind the visitors at the opposite wall, always at the same location in the room), thereby keeping externalization constant while varying audio-visuo-spatial contiguity and audio-spatial cueing. No significant differences between the conditions were found regarding memory of the audio explanations and text-picture integration, whereby memory of audio explanations in all conditions was at the same level as in the picture-located audio condition in the first experiment. This excludes spatial contiguity effects and audio-spatial cueing as explanations for the results in Experiment 1 and indicates a general externalization effect needing closer examination. The results are discussed regarding future studies and the use of spatial audio guides in museums and exhibitions.

This systematic review, utilizing the PRISMA framework, analyzes 248 international and 143 Iranian articles to provide an overview of studies on games in education. It examines five key themes: common terminology, methodology, type of study, variables studied, and technologies used, presenting findings in the same order of priority. Internationally, the term “serious games” is more prevalent, particularly among European scholars, while in the Americas and Asia, generic terms like “digital games” are more common. In contrast, Iranian research predominantly uses the term “computer games”. Methodologically, both international and Iranian studies primarily employ quantitative approaches, with questionnaires as the common data collection tool. However, international studies are more likely to employ experimental and quasi-experimental designs in explanatory-type research, while Iranian studies often use correlational designs to explore relationships between variables without intervention. In examining variables, both sets of literature frequently assess cognitive outcomes such as learning and motivation, with international studies showing a broader use of varied assessment tools. Technology-wise, international research shows a prevalent use of computer-based platforms and a rising interest in mobile technologies, reflecting a similar trend in Iranian studies but with a noticeable lag in adopting newer technologies. The findings of this study serve as a benchmark for scholars in various regions studying the use of games in education. Additionally, they provide new insights into how linguistic and cultural differences may influence scientific discussions. The paper concludes with key suggestions for future studies to improve research practices in the field and increase the generalizability of findings across contextual and regional boundaries.

The rapid growth in digital technologies continues to accelerate, bringing not only new opportunities, but also new challenges and needs to the field of education. As educational technologists design research to improve the implementation of learning technologies, they must adapt their research approaches to social and cultural contexts. In Participatory Action Research (PAR), teachers, students, or other members of the educational community participate as co-researchers who collaborate with researchers to build understanding and solve problems that are relevant to the school or community. This article describes the purpose, background, characteristics, and potential applications of PAR methods. It employs a meta-synthesis approach to investigate five adult-youth PAR collaborations that implement educational technology to meet needs in diverse educational and community settings. The main questions asked are:

• How can PAR advance educational technology research?

• In educational technology research, how can adult and youth collaborations in PAR benefit learning and the community?

Results show that PAR collaborations not only provide opportunities to gather and assess information, but can also increase dialogue that leads to meaningful understanding, insightful action, and positive change in the community and digital environments. Findings suggest that, in educational technology research that is focused on improving learning or addressing a community need, combining technology with adult/youth collaborative research relationships can increase insights and understanding while moving community members to actively address the issue.

Educational researchers have pointed to socioemotional dimensions of learning as important in gaining a more nuanced description of student engagement and learning. However, to date, research focused on the analysis of emotions has been narrow in its focus, centering on affect and sentiment analysis in isolation while neglecting how emotions potentially interact with social elements and course objectives in learning environments. In this paper, we present a case study analysis of seven asynchronous online discussions delivered as part of a blended-learning bachelor level course; we demonstrate the utility of a novel analysis workflow and visualization method which we refer to as SSEEN (social sentiment embedded epistemic networks) to uncover insights into the connection between social networks, course content, and detected student sentiment. The findings show that negative sentiment was most often associated with course content, but contrary to insights from prior research, negative sentiment served as a marker of engagement as students connected content to their own personal experiences. By simultaneously considering the social network alongside the sentiment-colored edges, we note that negative sentiment is not consistent within the discourse of particular students or between pairs indicating that sentiment did not appear to be an indicator of peer conflict or breaches in social contracts. The findings demonstrate how the proposed approach (SSEEN) can support educational researchers to gain a more nuanced understanding of the social, emotional, and epistemic dimensions of learning in asynchronous online discussions.

Technology-mediated interactions and datafication are increasingly central in contemporary social dynamics and institutions, including teaching and learning processes. In order to fully understand the complex entanglements of human and non-human actants that emerge in postdigital education, it is essential to imagine new methodological approaches that are sensitive to the multidimensional nature of education—as a socially and materially-situated phenomenon that increasingly takes place across distributed contexts. The overall goal of this paper is to propose and operationalise a new methodological approach for the study of technology in education. It draws on the notion of relational topologies to improve our understanding of educational settings and, ultimately, how learning unfolds. The proposed approach relies on a multi-paradigm enquiry strategy, based on the idea of using “topologies of digital data practices” in combination with the three dimensions that articulate design-for-learning processes according to the Activity-Centred Analysis and Design (ACAD) framework: epistemic, social and set designs. While the article focuses on presenting the elements of the approach from a theoretical perspective, we illustrate its application through the data collected in a small case study that will serve as a testbed. The topologies of relations we present in this article show uses of technology—as described by participants in their own learning experience—that involve different spaces, devices, and personal situations. In doing so, we reveal how humans and non-humans are entangled in hybrid, unstable and generative ways. The article concludes with some remarks on the value of the proposed approach for studying technology in education and its potential to explore the state-of-the-actual in this field, with the ultimate goal of helping inform educational research, practice and decision-making.

In this paper, we propose a new method for selecting cases for in situ, immediate interview research: detector-driven classroom interviewing (DDCI). Published work in educational data mining and learning analytics has yielded highly scalable measures that can detect key aspects of student interaction with computer-based learning in close to real-time. These measures detect a variety of constructs and make it possible to increase the precision and time-efficiency of this form of research. We review four examples that show how the method can be used to study why students become frustrated and how they respond, how anxiety influences how students respond to frustration, how metacognition interacts with affect, and how to improve the design of an adaptive learning system. Lastly, we compare DDCI to other mixed-methods approaches and outline opportunities for detector-driven classroom interviewing in research and practice, including research opportunities, design improvement opportunities, and pedagogical opportunities for teachers.

Analyzing qualitative data from learning processes is considered “messy” and time consuming (Chi in J Learn Sci 6(3):271–315, 1997). It is often challenging to summarize and synthesize such data in a manner that conveys the richness and complexity of learning processes in a clear and concise manner. Moreover, qualitative data often contains patterns that are not immediately apparent. Consequently, visualization can be an effective tool for representing and unpacking the complexities and multidimensions of learning processes. Additionally, visualizations provide a time-efficient approach to analyzing data and a high-level view of the learning process over time for researchers to zoom in on intriguing moments and patterns (Huang et al. in Comput Human Behav 87:480–492, 2018). In this conceptual paper, we provide a broad overview of research in the field of visualizing qualitative data and discuss two studies (1) visualizing role-changing patterns in an interdisciplinary learning environment and (2) operationalizing collaborative computational thinking practices via visualization. By leveraging these studies, we aim to demonstrate a visualization processing flow along with qualitative research and methods. Particularly, the processing flow includes three critical elements: research subjectivity, complexity of visual encoding, and purpose of visual encoding. The discussion highlights the iterative and creative nature of the visualization technique. Furthermore, we discuss the benefits, challenges, and limitations of using visualization in the context of qualitative studies.

Understanding experiences in online educational settings is crucial to improving teaching and learning. The purpose of this paper is to describe Narrative Inquiry as a research methodology that has the potential create the relational opportunities necessary to understand experiences in online learning environments. In this article, I use an example from research conducted in a special educational setting to overview narrative inquiry methodology, explain its theoretical underpinnings, and highlight its potential to enhance current knowledge of how individuals live alongside one another in online educational settings. I will also explain how narrative inquiry can support the development of new insights about time and engagement in online learning. Then I address how narrative inquiry has the potential to advance equitable research practices in these settings. Finally, I offer suggestions for future research projects that leverage the conceptual strengths and methodological tools of narrative inquiry.

Research fields related to learning (e.g., educational technology and learning sciences) have historically focused on what questions using traditional methods (e.g., comparing different learning tools and methods). New methodologies that are grounded in learning, engagement, and motivational theories are needed to additionally address the how questions. Methodologies that use learners’ process data shed more light on how learners learn and if the learning tools are effective, compared to methodologies that use just outcome data. In this paper, we discuss stealth assessment—an evidence-based methodology that can be used in technology-rich environments (e.g., games) to assess and support hard-to-measure constructs (e.g., creativity) as well as knowledge acquisition (e.g., physics). We also discuss evidence-centered design (ECD), and present specific steps to design, embed in a digital learning environment, and evaluate a stealth assessment. Additionally, we provide two examples of stealth assessment studies in the context of an educational game called Physics Playground: Study 1 illustrates a stealth assessment of creativity and Study 2 describes a stealth assessment of physics understanding and how we used it to make the game adaptive. The purpose of this paper is to provide sufficient detail about stealth assessment to help researchers in the field of educational technology and related fields to adopt this method to assess, foster, and investigate learning processes in various technology-rich environments.

Since 1983, the Instructional Theory Framework (ITF) (and its subsequent improvements) has guided instructional designers and researchers in designing and developing useful learner experiences (LX). For the past 40 years, the ITF was laser-focused on the selection of instructional methods, downplaying delivery methods (media) and management methods. The instructional design field continues to produce immature and confounded research-to-prove studies that do not provide guidance that is useful to practitioners. For more useful guidance, we suggest that researchers should embrace research-to-improve for studying immature methods and media, and research-to-prove for studying mature methods and media. In this paper, we discuss problems associated with proving versus improving, situational deficiencies, and confounding; we present a new version of the ITF that embraces media; and we then answer four key questions about (1) kinds of media knowledge, (2) forms of media knowledge, (3) research methods that deliver the knowledge, and (4) suggestions for editors and reviewers to embrace new media knowledge.

Mixed methods research is becoming more prevalent in educational technology due to its potential for addressing complex educational problems by integrating qualitative and quantitative data and findings. At the same time, a growing chorus of researchers laments the quality and rigor of research in this field. Mixed methods studies which demonstrate explicit integration in educational technology research are scarce, and even fewer apply integration strategies recommended in the literature, such as visual joint displays. Failure to address the challenge of comprehensive integration may result in missed opportunities for deeper insights. To address this methodological problem, the purpose of this paper is to shed light on the procedures, opportunities, and practical challenges associated with mixed methods integration through the use of visual joint displays as an analytical tool for data interpretation and reporting in these types of designs. Using an exploratory sequential mixed methods multiple case study design as an illustrative example, we will (1) provide step-by-step guidance on how to develop a visual joint display to conduct an integrated analysis in a complex mixed methods design; (2) demonstrate how to use a display of this type to integrate meta-inferences previously generated through a series of interconnected joint displays; and (3) illustrate the benefits of integrating at the literature review, theoretical, analysis, interpretation, and reporting levels in mixed methods studies. This methodological article aims to advance knowledge in educational technology research by addressing the integration challenge in mixed methods studies and assisting researchers in this field in achieving comprehensive integration at multiple levels.

The aim of this paper was to explore the use of autoethnography methodology, a non-traditional and reflective approach, in educational technology research. Autoethnography involves a critical analysis of personal experiences and stories being positioned within the larger cultural, political, and social context. Following an overview of the origin and development of autoethnography as empirical research, the authors discuss autoethnography in the context of educational technology research by considering its epistemological and methodological issues. In this paper, the authors also consider autoethnography and its relationship to other qualitative research approaches. Essential components and summarized evaluation criteria for novice autoethnographers are shared. Lastly, the paper reflects on the potential benefits as well as the challenges that those writing an autoethnography will inevitably face. There is a need for autoethnography research in our field to reveal voices hidden in mainstream educational technology research.

Interpersonal online interactions are key to digital learning pedagogies and student experiences. Researchers use learner log and text data collected by technologies that mediate learner interactions online to provide indicators about interpersonal interactions. However, analytical approaches used to derive these indicators face conceptual, methodological, and practical challenges. Existing analytical approaches are not well aligned with the theories of digital learning, lack rigor, and are not easily replicable. To address these challenges, we put forward a multi-level framework linking indicators of individual posting with group-level communication and emergent relational structures. We exemplify the use of the framework by analyzing twenty online and blended courses. Empirical insights demonstrate how indicators at these three levels relate to each other and to potential instructor decisions. Our conclusion highlights current gaps in the framework and the areas for future work.

Researchers tasked with understanding the effects of educational technology innovations face the challenge of providing evidence of causality. Given the complexities of studying learning in authentic contexts interwoven with technological affordances, conducting tightly-controlled randomized experiments is not always feasible nor desirable. Today, a set of tools is available that can help researchers reason about cause-and-effect, irrespective of the particular research design or approach. This theoretical paper introduces such a tool, a simple graphical formalism that can be used to reason about potential sources of bias. We further explain how causal graphs differ from structural equation models and highlight the value of explicit causal inference. The final section shows how causal graphs can be used in several stages of the research process, whether researchers plan to conduct observational or experimental research.

In the field of educational technology and instructional design, research methods are emerging that aim to curate different forms of knowledge and insights beyond traditional research studies, or what Reigeluth and An (in Reigeluth and Carr-Chellman (eds) Instructional-design theories and models: Building a common knowledge base, Lawrence Erlbaum Associates, Mahwah, 2009) refer to as “research to prove.” As a result of evolving efforts in this area, editors of research journals in the field are receiving increased submissions employing these methods but have detected some persistent confusion among authors surrounding them. This has resulted in authors submitting articles with muddled methodologies and to outlets that may not be a fit for the work an author seeks to share. It can even be unclear whether authors intentionally employed a specific design-related method prior to reporting. In this piece, we will cover four methods—instructional design cases, case studies, design-based research, and formative evaluation of designs/products—to provide clarity for both graduate students and researchers. For each of these, we will provide definitions, discuss exemplars and features of exemplars, summarize key features that should be present in such a study and its reporting, and provide guidance on front-end intentional design and planning for research studies that employ these methodologies. Additional clarity on these methods can better support scholars and emerging scholars in their roles as researchers, authors, and reviewers.