Virtual humans are artificially intelligent (AI) agents that control computer generated bodies and can interact with users through speech and gesture in virtual environments. Advanced virtual humans are able to engage in rich conversations, recognize nonverbal cues, analyze social and emotional factors and synthesize human communication and nonverbal expressions.

Building virtual humans requires fundamental advances in AI, speech recognition, natural language understanding and generation, dialog management, cognitive modeling and reasoning, virtual human architectures and computer graphics and animations. All of these technologies need to be integrated together into a single system that can work in unison, be expandable, flexible and plug-and-play with different components.

In the CNS Lab, there are three primary projects using virtual humans.

1. Grant Funded: (Co-PI with Soartech; $998,999): AVatar-Administered Neuropsychological Testing (AVANT).

  • The University of North Texas, in partnership with Soar Technology, Inc. (SoarTech), has been awarded funding for SBIR topic A13-090 – Advanced Automated Assessment of Cognitive Changes Associated with Brain Injury and Neurological Disease.
  • The work focuses on developing an automated language-based assessment system that we term AVANT (AVatar-Administered Neuropsychological Testing) that is primarily self-administering with clearly presented directions to a patient using both visual illustrations and avatar-based verbal instructions. Using speech recognition and a conversational avatar representing a virtual clinician, AVANT will enhance current approaches to neuropsychological assessment and allow for avatar-administered assessments, based on various perceptual modalities (auditory, visual, gestural), that augment a real clinician’s ability to assess patients.
  • The team aims to demonstrate the implementation of speech recognition using a virtual clinician that is sufficiently accurate to permit computer administration of verbally-based neuropsychological tasks including word-list learning, confrontation naming, and aural comprehension.

2. Virtual Standardized Patients:

  • Virtual Patients are clinically relevant virtual human agents that have been applied to the training of clinicians. At UNT, interactive agents portray a patient with a clinical or physical condition and can interact with a clinician in an effort to teach interpersonal skills. Virtual Patient (VP) technology has evolved to a point where researchers may begin developing mental health applications that make use of virtual reality patients.
  • The current work reflects lessons learned by Dr. Parsons while working as a Research Scientist at USC's ICT--application of Virtual Patient projects to both military and civilian purposes (see Parsons et al., 2008). We developed an approach that will allow novice mental health clinicians to conduct an interview with a virtual character that emulates a character with a DSM IV TR disorder (see Parsons et al., 2009). These early virtual patients were used in a number of USC Departments: Psychiatry; Psychology; and Social Work. Further, they have been implemented in studies at UCSD/SDSU; Houston: UNC; and Tripler Army Medical Center.
  • The current work at UNT, however, moves beyond this earlier work to focus more on differential diagnosis and therapeutic interaction. Further, the current format is massively deployable and uses software that enables rapid creation of virtual humans.

 

3. Social Cognition Task:

Here are some representative pubications:

Virtual Human Publications

·Parsons, T. D., Russo, N., & Schermerhorn, P. (2017, September). Avatar Administered Neuropsychological Testing (AVANT): Stroop Interference Task. In Proceedings of the Human Factors and Ergonomics Society Annual Meeting (Vol. 61, No. 1, pp. 2047-2051). Sage CA: Los Angeles, CA: SAGE Publications. (PDF)

·Parsons, T.D., Schermerhorn, P., & McMahan, T., Asbee, J., & Russo, N. (2017). An initial validation of virtual human administered neuropsychological assessments. Annual Review of CyberTherapy and Telemedicine, 15, 123-128. (PDF)

·Parsons, T.D. (2015). Virtual Standardized Patients for Assessing the Competencies of Psychologists. In Mehdi Khosrow-Pour (Ed.), Encyclopedia of Information Science and Technology, Third Edition (pp. 297- 305), Hershey: IGI Global. (PDF)

·Strimbu, N., Rooney, B., Bálint, K.E., Redmond, D., Broughan, J., Lee, C.T., Parsons., T.D. (2018). Piloting, validating, and applying a virtual eye-gaze task to measure social cognition responses. Interactive Eye Gaze, University College London, London, Feb 12-13, 2018.

·Rooney, B., Balint, K., Parsons, T.D., Burke, C., O'Leary, T., Lee, S., & Mantei, C. (2017). Attention and Social Cognition in Virtual Reality: The effect of engagement mode and character eye-gaze. Annual Review of CyberTherapy and Telemedicine, 15, 82-87. (PDF)

·Rooney, B., Balint, K., Parsons, T.D., Burke, C., O'Leary, T., Lee, S., & Mantei, C. (2017). Virtual Reality, Presence and Social Cognition: The effect of eye-gaze and narrativity on character engagement. Proceedings of International Society on Virtual Systems and MultiMedia (VSMM), October 31, 2017. Dublin, Ireland.

·Redmond, D., Strimbu, N., Rooney, B., Bálint, K.E., Parsons., T.D. (2017). Measuring social cognition in a virtual world. SIGMAC Conference, Dublin, 18/11/2017.

·Hubal, R., & Parsons, T.D. (2017). Synthetic Environments for Skills Training and Practice. In Lin L., and J.M Spector (Eds.), The Sciences of Learning and Instructional Design: Constructive Articulation Between Communities (pp. 152- 185), Elsevier, New York, NY. (PDF)

·Venturini, E., Riva, P., Serpetti, F., Romero-Lauro, L., Mantovani, F., & Parsons, T.D. (2017). Virtual environments for the assessment of social exclusion in Autism: Enhancing fidelity and anthropomorphism. Proceedings of Conference on Ostracism, Social Exclusion and Rejection, June 29th – July 2nd, 2017. Vitznau, Lake Lucerne (Switzerland).

·Venturini, E., Riva, P., Serpetti, F., Romero, L., Pallavincini, F., Mantovani, F., Cloutier, R., McMahan, T., Stonecipher, K., & Parsons, T.D. (2016). A comparison of 3D versus 2D virtual environments on the feelings of social exclusion, inclusion and over-inclusion. Annual Review of CyberTherapy and Telemedicine, 14, . 89-94 (PDF)

·Pataki, C., Pato, M., Sugar, J., Rizzo, A., St-George, C., Kenny, P., & Parsons, T.D. (2012). Virtual Patients as a Novel Teaching Tool in Psychiatry. Academic Psychiatry, 36, 398-400. (PDF)

·Parsons, T.D. (2012) Virtual Simulations and the Second Life Metaverse: Paradigm Shift in Neuropsychological Assessment. In V. Zagalo, T. Morgado. & A. Boa-Ventura (Eds.), Virtual Worlds, Second Life and Metaverse Platforms: New Communication and Identity Paradigms (pp. 234- 250). Hershey: IGI Global. (PDF)

·Parsons, T.D. (2011) Affect-sensitive Virtual Standardized Patient Interface System. In D. Surry, T. Stefurak, & R. Gray (Eds.), Technology Integration in Higher Education: Social and Organizational Aspects (pp. 201 - 221). Hershey: IGI Global. (PDF)

·Kenny, P., & Parsons, T.D. (2011; Co-lead authors) Embodied Conversational Virtual Human Patients. In C. Perez-Marin, & I. Pascual-Nieto (Eds.), Conversational Agents and Natural Language Interaction: Techniques and Effective Practices (pp. 254- 281). Hershey: IGI Global. (PDF)

·Kenny, P., & Parsons, T.D. (2010; Co-lead authors). Building Embodied Conversational Virtual Patients. Proceedings of the Interservice/Industry Training, Simulation and Education Conference (I/ITSEC), Orlando Florida, December 1-4, 2010.

·Parsons, T.D. (2010). Virtual Environments for Clinical Psychologists. The Clinical Psychologist, 64, 29-30. (PDF).

·Kenny, P., Parsons, T.D., Gratch, J., & Rizzo, A.A. (2009). Evaluation of Novice and Expert Interpersonal Interaction Skills with a Virtual Patient. Lecture Notes in Computer Science, 5773, 511-512.(PDF)

·Parsons, T.D., Kenny, P., Cosand, L., Iyer, A., Courtney, C., & Rizzo, A.A. (2009). A Virtual Human Agent for Assessing Bias in Novice Therapists. Studies in Health Technology and Informatics, 142, 253-258. (PDF)

·Kenny, P., Parsons, T.D., & Rizzo, A.A. (2009). Human Computer Interaction in Virtual Standardized Patient Systems. Lecture Notes in Computer Science, 5613, 514-523. (PDF)

·Kenny, P., Parsons, T.D., Rothbaum, B.O., Difede, J., Reger, G., & Rizzo, A.A. (2009). Optimizing Clinical Training for the Treatment of PTSD using Virtual Patients. Studies in Health Technology and Informatics. 144, 264-8. (PDF)

·Kenny, P., Parsons, T.D., & Rizzo, A.A. (2009). A Comparative Analysis between Experts and Novices Interacting with a Virtual Patient with PTSD. Studies in Health Technology and Informatics. 144, 122-124. (PDF)

·Kenny, P., Gratch, J., Parsons, T.D., & Rizzo, A.A. (2009). Evaluation of Novice and Expert Interpersonal Interaction Skills with a Virtual Patient. Proceedings of the 9th International Conference on Intelligent Virtual Agents. Amsterdam, The Netherlands, September 14-16, 2009.

·Parsons, T.D., Kenny, P., Ntuen, C., Pataki, C.S., Pato, M., Rizzo, A.A., St-George, C., & Sugar, J. (2008). Objective Structured Clinical Interview Training using a Virtual Human Patient. Studies in Health Technology and Informatics, 132, 357-362. (PDF)

·Parsons, T.D., Kenny, P., & Rizzo, A.A. (2008). Virtual Human Patients for Training of Clinical Interview and Communication Skills. Proceedings of the 2008 International Conference on Disability, Virtual Reality and Associated Technology. Porto, Portugal, September 8-11, 2008.

·Kenny, P., Parsons, T.D., Pataki, C.S., Pato, M., St-George, C., Sugar, J., & Rizzo, A.A. (2008). Virtual Justina: A PTSD Virtual Patient for Clinical Classroom Training. Annual Review of CyberTherapy and Telemedicine, 6, 113-118. (PDF)

·Kenny, P., Parsons, T.D., Gratch, J., & Rizzo, A.A. (2008). Evaluation of Justina: A Virtual Patient with PTSD. Lecture Notes in Artificial Intelligence, 5208, 394–408 (PDF).

·Macedonio, M., Parsons, T.D., & Rizzo, A.A. (2007). Immersiveness and Physiological Arousal within Panoramic Video-based Virtual Reality. Cyberpsychology and Behavior, 10, 508-516.(PDF)

·Rizzo, A.A., Lange, B., Buckwalter, J.G., Forbell, E., Kim, J., Sagae, K., Williams, J., Rothbaum, B.O., Difede, J., Reger, G., Parsons, T.D., and Kenny, P. (2011). An Intelligent Virtual Human System for Providing Healthcare Information and Support. Studies in Health Technology and Informatics, 143, 503-509. (PDF)

·Rizzo, A.A. Kenny, P., & Parsons, T.D. (2011). Intelligent Virtual Patients for Training Clinical Skills. Journal of Virtual Reality and Broadcasting, 8, 1-16. (PDF)