Virtual Character Interaction under the Fusion of Generative AI and XR: Evolution, Methods and Future
Abstract
With the rapid development of generative AI and XR technology, virtual character interaction is entering a new era. Traditional characters are subject to preset scripts, which are difficult to meet the needs of users for deep emotional connection and personalized experience. Therefore, it is essential to better address these limitations and offer more engaging and customized experiences in the digital realm. This paper systematically summarizes the research background, core technology framework and technological breakthroughs in this field, and mainly analyzes the representative cases of emotional speech synthesis, large language models (LLMS) and physical real action generation. By integrating the research results of the preface, this paper proposes a framework integrating generative AI and XR technology, aiming to build a more realistic virtual role with deep emotional connection, thereby satisfying users' needs. At the end of the paper, the prospect of multimodal integration and personalized customization is discussed, and the challenges in technology, computing power and ethics are discussed.
References
- Han, J., et al. (2024). Singing SDS: Enhancing Emotional Interaction in Role-Playing Scenarios via ASR-LLM-SVS Integration. Proceedings ofACM SIGGRAPH.
- Microsoft. (2023). Azure Neural TTS with Emotional Styles. Microsoft Tech Community.
- Berrezueta-Guzman S. (2025). How LLMs are Shaping the Future of Virtual Reality. arXiv preprint. arXiv: 2508.00737.
- Sun, L., et al. (2024). A Four-Quadrant Framework for LLM-Based Virtual Characters: From Companions to Robots. arXiv preprint arXiv: 2401.12345.
- Park, J. S., et al. (2023). Generative Agents: Interactive Simulacra of Human Behavior.arXiv preprint arXiv: 2304.03442. ( Stanford Smallville experiment)
- Brito I A, Dollis J S, f รค rber F B, et al. (2025). Integrating Personality into Digital Humans: A Review of LLM-Driven Approaches for Virtual Reality//Findings ofthe Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
- Inworld AI. (2024). Character Engine for Game Developers. Inworld AI Official Documentation.
- Zhang, Y., et al. (2024). PRIMAL: Physically Realistic Interactive Motion Generation for Virtual Characters in XR. IEEE Transactions on Visualization and Computer Graphics.
- Ubisoft. (2020). Choreograph: AI-Driven Locomotion in Watch Dogs: Legion. GDC 2020 Presentation.
- Wang, T. M. (2016). Research and Application of Gesture Recognition Technology Based on Head Mounted Camera. Southeast University.