CHI 2025Yokohama, Japan · April 26–May 1, 2025

Exploring LLM-Powered
Role and Action-Switching

Pedagogical Agents for History Education in Virtual Reality

Zihao Zhu1, 2, * Ao Yu1 Xin Tong1 Pan Hui1, †
1 Computational Media and Arts Thrust, The Hong Kong University of Science and Technology (Guangzhou) 2 Department of Computer Science, City University of Hong Kong
Paper Video DOI arXiv Slides
Prototype architecture combining spoken conversation with LLM-powered adaptive role-switching and action-switching for pedagogical agents in virtual reality.
LLM-powered conversation, adaptive role-switching, and adaptive action-switching.View full size

Abstract

Multi-role pedagogical agents can create engaging and immersive learning experiences, helping learners better understand knowledge in history learning. However, existing pedagogical agents often struggle with multi-role interactions due to complex controls, limited feedback forms, and difficulty dynamically adapting to user inputs. In this study, we developed a VR prototype with LLM-powered adaptive role-switching and action-switching pedagogical agents to help users learn about the history of the Pavilion of Prince Teng. A 2 x 2 between-subjects study was conducted with 84 participants to assess how adaptive role-switching and action-switching affect participants’ learning outcomes and experiences. The results suggest that adaptive role-switching enhances participants’ perception of the pedagogical agent’s trustworthiness and expertise but may lead to inconsistent learning experiences. Adaptive action-switching increases participants’ perceived social presence, expertise, and humanness. The study did not uncover any effects of role-switching and action-switching on usability, learning motivation and cognitive load. Based on the findings, we proposed five design implications for incorporating adaptive role-switching and action-switching into future VR history education tools.

Video

Watch on YouTube

Published by ACM SIGCHI.

Authors

Zihao Zhu

Zihao Zhu

Ph.D. Candidate

Department of Computer Science
City University of Hong Kong

Ao Yu

Ao Yu

Postdoctoral Researcher

Computational Media and Arts Thrust
The Hong Kong University of Science and Technology (Guangzhou)

Xin Tong

Xin Tong

Assistant Professor

Computational Media and Arts Thrust
The Hong Kong University of Science and Technology (Guangzhou)

Pan Hui

Pan Hui

Chair Professor

Computational Media and Arts Thrust
The Hong Kong University of Science and Technology (Guangzhou)

* This work was conducted when Zihao Zhu was an MPhil student at The Hong Kong University of Science and Technology (Guangzhou).
† Corresponding author: Pan Hui

Acknowledgments

We sincerely thank our participants for their time in this study. This research was supported by the Guangzhou Municipal Nansha District Science and Technology Bureau (No. 2022ZD012).

Citing this work

Zihao Zhu, Ao Yu, Xin Tong, and Pan Hui. 2025. Exploring LLM-Powered Role and Action-Switching Pedagogical Agents for History Education in Virtual Reality. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25), April 26–May 1, 2025, Yokohama, Japan. ACM, New York, NY, USA, 19 pages. https://doi.org/10.1145/3706598.3713109

BibTeX
Download .bib 
@inproceedings{zhu2025exploring,
  author    = {Zhu, Zihao and Yu, Ao and Tong, Xin and Hui, Pan},
  title     = {Exploring {LLM}-Powered Role and Action-Switching Pedagogical Agents for History Education in Virtual Reality},
  booktitle = {Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems},
  series    = {CHI '25},
  year      = {2025},
  publisher = {Association for Computing Machinery},
  address   = {New York, NY, USA},
  pages     = {1--19},
  numpages  = {19},
  doi       = {10.1145/3706598.3713109},
  url       = {https://doi.org/10.1145/3706598.3713109}
}