NLP Led VR simulation experience for Google

I delivered this pilot Natural Language Processing (NLP) led VR simulation experience designed to train de-escalation skills for NYPD officers, in collaboration with Google Jigsaw. This project posed a significant challenge, as Google’s early research indicated that factors like realism, reflection, and repetition were critical in making VR an effective tool for de-escalation training—a technical feat in terms of performance and immersion.

My team's role was to establish and execute the scenario design, UX, UI, character interactions, Unity scene development, and overall integration and optimisation. We began with extensive user and desk research, working closely with Google to determine both the user needs and real-life scenarios that would present compelling communication challenges. We then developed a unique script input pipeline tailored to the new NLP tool crafted by Google.

Throughout the project, we created low-fidelity prototypes to refine content production and implemented a scene randomiser feature after early user testing to further enhance the training impact. During the peak of the COVID lockdown, we faced the significant challenge of arranging performance capture. This led us to innovate with photogrammetric methodologies, adapting workflows traditionally used with Unreal to optimise thousands of animation files for Unity quickly.

I managed this end-to-end project, constantly driving innovation and R&D to meet the tight deadlines. The results exceeded expectations, as evidenced by behavioural analytics and user testing, and the launch received coverage in mainstream media.

The product was later handed over to select universities, who are now working to expand its application in communications training.

I managed the Content production, Unity development, API integration and QA for this VR behavioral training program pilot led by natural language processing and hyper-realistic characters for Google Jigsaw ($2M budget, 50 people staff team). Hardware: Vive Pro 2 Headset.