For most of us in the modern world, our touchscreen smartphones are no longer a novelty, or even just a convenience. They’re essential – embodying the 1991 prescient statement by computer scientist Mark Weiser: “The most profound technologies are those that disappear. They weave themselves into the fabric of everyday life until they are indistinguishable from it.”
Creating technology so effective that it “disappears” is what motivates Dr. Francisco R. Ortega, assistant professor in the Department of Computer Science. Since joining the CSU faculty in 2018, Ortega has built a laboratory focused on research and development of “3D user interfaces,” most notably applications of virtual and augmented reality.
His lab, called the Natural User Interaction Lab (NUILAB), is CSU’s only research group conducting what Ortega calls basic research in augmented and virtual reality technologies. He and his students and postdoctoral researchers aim to make the navigation, manipulation and gesture-centric experience of augmented and virtual reality platforms as natural to the user as the multi-touch displays on our smartphones.
“People’s use of AR or VR will continue to grow,” Ortega said. “In this lab, we are at the forefront of AR/VR basic research, and how we can improve the field and move it forward.”
Francisco Ortega, assistant professor in the Department of Computer Science, leads the Natural User Interaction Lab.
Foundational research
Since virtual and augmented reality platforms have become more common through education, gaming and other applications, they have also become increasingly complex, moving further out of reach for many people. Though scientists and engineers have for decades envisioned such 3D user interfaces to enhance or replace many aspects of life, such visions haven’t yet come to pass.
But how to get there? Ortega and the NUILAB are contributing foundational research that bridges critical gaps between the technology and end-users by solving roadblocks and problems that keep users from fully embracing all that the technology has to offer. They are running several experiments across a breadth of topics, among them general studies of user behavior in augmented reality headsets. Their goal is to expand understanding of universal gestures within these environments.
In one project, the researchers have created and tested a virtual reality-based system for assessing concussions in soccer settings. Another, funded by the Defense Advanced Research Projects Agency, looks at whether embodied avatars in virtual reality settings could be an effective tool for communication with children. The project has been a collaboration of multiple CSU computer science faculty including: Professor Bruce Draper, Professor Ross Beveridge, Assistant Professor Nikhil Krishnaswamy and Assistant Professor Nate Blanchard, as well as Associate Professor Lisa Daunhauer in the Department of Human Development and Family Studies, and Professor Anita Bundy in the Department of Occupational Therapy.
Undergraduate researcher Ethan Holen and graduate researcher Xiaoyan Zhou run a demo in the lab.
A virtual reality-based system for assessing concussions in soccer settings is one of the lab’s many projects. Madeline Watts and Aditya Raikwar demonstrate.
Ortega’s lab consists of neatly arranged rows of top-line gaming computers, and an array of cutting-edge equipment, most of which is commercially available: HoloLens 2, Oculus, HTC VIVE Pro Eye and a Varjo XR-3, among many others.
The lab is funded by several federal agencies, including the National Science Foundation, the Defense Advanced Research Projects Agency and, most recently, the Office of Naval Research.
Naval research funding
Ortega’s most recent $900,000 award from the Office of Naval Research is funding his lab’s development of a general model and guidelines for optimal augmented reality display design. In addition, ONR provided a $200,000 award for equipment to complement the research. Their proposal, which for the Navy might mean everyday deployment of VR or AR systems in training or combat environments, seeks to define tradeoffs between design principles based on human-computer interaction and currently available AR technology.
“We expect people to be wearing AR glasses all day long,” Ortega said. “We want to be sure and improve the science of how people interact with the technology.”
He used the example of notification pop-ups – something all smartphone users and office workers are familiar with. Years of user testing has iterated on the optimal ways such pop-ups appear for people.
“When you get a notification, when do you get it? How do you get it? Should I interrupt you? Am I providing enough information? Am I providing too much information? What if you’re driving?” he said.
For the Office of Naval Research work, Ortega’s lab along with Professors Ben Clegg and Chris Wickens from the Department of Psychology are finding a model to improve positioning of information, including the amount of clutter in the display.
The basic principles the team will uncover through the Office of Naval Research-funded work are broadly applicable beyond military applications. The NUILAB’s ultimate goal is to optimize augmented reality display fundamentals so profoundly that, like our phones, the technology will “disappear.”
“This vision applies to every person on the planet,” Ortega said. “Every person right now is using phones. This is the next evolution.”
Other projects in the lab seek new breakthroughs in the intersection between virtual reality applications and learning. A project led by undergraduate student Ethan Holen with the support of Ortega and Clegg, for example, observes human subjects as they wear a virtual reality headset to understand whether AR/VR can improve how users learn block-assembly tasks. By refining how the tasks and steps are presented to the user, the researchers are providing new insights into how complex tasks might be completed by the workforce of the future.
The lab regularly intersects with groups who study artificial intelligence, psychology and cognition at CSU and other universities. Undergraduate researcher Rachel Masters is running an experiment investigating virtual forest-bathing, or when subjects immerse themselves in green nature for stress relief or mental health purposes. Masters is ascertaining whether virtual reality could act as a substitute for real nature in constrained settings like nursing homes. The project includes renowned VR expert Victoria Interrante of the University of Minnesota.
In addition to building a state-of-the-art AR/VR research space, Ortega says advocacy for students, regardless of gender, race or sexual orientation, is an important cornerstone of his personal philosophy and how he runs his lab. In fact, one of his research interests is in the gendering of computer science and how these norms might be changed.
Ortega with his students and postdoctoral researchers.
Learn more
To learn more about the lab’s members, activities and projects, visit the website.
Take a 3D tour of the lab: compsci.colostate.edu/tours