Spatial Reality Displays and mocopi Motion Capture System Elevate the Art of Immersive, Interactive Storytelling

June 18, 2024 - The local theater community in La Jolla, San Diego and the technology faculty at UC San Diego are re-defining audience participation, combining artistic creativity with Sony’s 27-inch 3D Spatial Reality Displays (SRD) and mocopi wireless motion capture system.

The latest version of “Without Walls” (WOW), the La Jolla Playhouse’s signature performance program, used the new technologies to merge the physical and virtual worlds by allowing real-time interaction between audiences and cast members. The theater worked with MAVERiC Studio, located in UC San Diego’s Design and Innovation Building, and Origami Air Co., a San Diego-based XR development company, to develop and build a virtual environment, featuring highly expressive motion capture-generated characters, which was published to VRChat.

The results showcased the theater experience at its best, when audiences are transported to a different place and allowed to immerse themselves in a performance.

“SRD and mocopi became part of the performance for Without Walls,” said Trisha Williams, Director of UC San Diego’s MAVERiC Studio and CEO of Origami Air. “Audiences could see a full 360-degree spatial view of actors wearing the mocopi devices to create this huge 15-foot character with large gestures and then react in real time through the SRD. It felt so real and mystical. Everyone enjoyed seeing a motion capture performance come to life in a completely different way.”

An Easy and Seamless Creative Workflow

The SRD and mocopi technologies quickly became the foundation for everything the creative teams were trying to accomplish, from giving student-performers easy, mobile access to full-body avatars to the design process for creating new characters and environments.

“We started using the SRD to show our artists and action performers who were inhabiting our avatars what they were capable of and how they could move around in a space without having to step into that space,” said Joe Unger, Chief Ecosystem Officer at Origami Air and the content manager at UC San Diego’s MAVERiC Studio.

Unger added he enjoyed working with the glasses-less 3D displays because, “I can see what I’m doing in real-time without having to put on glasses or a headset. You can see 3D objects, move them around, scale them and work on them side-by-side with my artists and with the other devices we’re using to create content. I can tell what an object is going to look like before it ever enters a scene.”

Using the combined Sony technologies, Williams created and designed the virtual scenes and worlds using mocopi and SRD Maya Plugin (a 3D computer graphics software) for real-time creation, before moving to the actual performance. Designing in Maya and seeing modifications, as they happened, on the SRD and mocopi helped the creative team decide and finalize the best approach for this project.

Williams also found the glasses-less SRD design to be a significant improvement. “Having to put a headset on to see what a character looks like in virtual reality takes you out of the workflow,” Williams said. “It’s so nice to be able to look at the SRD and see that full 3D, 360 view of a character and then go right back into the Maya workflow.”

The decision to use mocopi came down to mobility and convenience.

“It was the easiest solution for everyone to quickly jump into full body avatars and we could also send that pocket-sized device home with our actors so they could continue to perform,” Unger said. “It was up to the actors to continue to rehearse the performance on their own and mocopi enabled that. I can hand this to an artist, they can set it up at home and begin animating in real-time, right away.”

A New Language for Theater

The MAVERiC Studio and Origami Air teams worked with Hortense Gerardo, Director of UC San Diego’s Anthropology, Performance and Technology Program to create a unique form language that was optimized for mocopi and used during performances to converse with audiences Gerardo also wore the mocopi devices during the performances, getting a first-hand experience with this new motion-capture system. mocopi enabled the actors to fully express themselves with the full-body avatar within the VRChat environment, which is essential to a theatrical performance. mocopi helped to give character’s life as if they were doing a live performance.

“Once I familiarized myself with how to synchronize the system and how to move my whole body to accommodate the way that it functions, it was easy to use and allowed me to embody the character fully,” Gerardo said. “People genuinely reacted whenever the creature responded to the audiences. They sensed fear and excitement, and they could tell friendliness versus if the creature was reluctant to engage. It drove a wide range of emotions.”

Gerardo added the addition of mixed reality technologies to theater performances mirrors the shift happening across other media and entertainment platforms as audiences now expect more interactive and immersive viewing experiences.

“This will inspire artists and technologists to create in VR spaces as we enter a new era where people are increasingly going to be engaged in these new technologies,” she said. “It was also just a lot of fun!”

A “Center of Technological Learning”

Unger described MAVERiC Studio as a center of technological learning, a place where students can get hands-on time with the newest tools that are reshaping the professional workplaces they will soon enter and SRD and mocopi are now part of the technology the studio uses to showcase and teach XR students on how to create projects for virtual worlds.

“Before we started experimenting with mocopi, our emotions were centered around the front of our body,” said Aria Grossfia, Student, UC San Diego. “That didn’t show up well in VR, so with mocopi we were able to shift the movements to be larger and more ‘outside’ and around us. It made sense that these motions would be larger and a little more intimidating to the audience and it was exciting to watch the actors use mocopi to bring this new language to life.”

Jack Meir, Student, UC San Diego, also found that mocopi helped him focus on effectively communicating with the audience and translating his movements freely while controlling the character in VR, without having to think about animations or rigging of the model.

“The most useful feature of mocopi for me is its ability to stream to different outlets and devices,” he said. “I was able to use a character in VR smoothly, controlling the legs and the arms, and then easily switch over to live mocap streaming on different devices or any other type of real-time simulation in addition to virtual reality. It’s a versatile device.”

Williams is confident they are not only positively impacting the theater-going experience today but also setting the stage for meeting heightened audience expectations in coming years.

“Watching students put these technologies to use in ways we never ever imagined was really exciting,” Williams said. “This is the future of live performance, especially combining theater and technology the way we are.”