In January of this year, the U.S. Military Academy began working with the Penn State Applied Research Lab to facilitate an event called the Virtual Thunderstorm Technology Demonstration. During the demo, nine vendors provided the Rapid Reaction Technology Office (RRTO) from the Office of the Secretary of Defense (OSD) with a solution to enhance small unit lethality on the battlefield. Conversations for this event began between November and December of last year with preparations for the culminating event the week of Oct 5-9.
According to Lt. Col. Chris Johnes, the director of the West Point Simulation Center, the solution is the development of a virtual reality simulation rendered by scientific research and development companies. Due to COVID-19 and travel restrictions, company representatives could not come to West Point to showcase their products live. The event had to be conducted — webinar style.
The event kick started on Oct. 5 where each company had 10 minutes to give an overview of their virtual reality product. From Oct. 7-9, three companies per-day presented 40-minute deep dives providing in-depth information on what their products were doing.
Furthermore, Johnes explained the Office of the Secretary of Defense identified different issues the Department of Defense wants to explore. For this venue, OSD focused on small unit tactics on the battlefield and reached out to vendors across the nation to solve the problem. In this case, the RRTO wanted to conduct a Rapid Response Mission Rehearsal for small tactical units and wanted to incorporate virtual reality technologies into the rehearsals.
“The RRTO wanted the companies to collect data on the training site we were using on West Point, process that data and render the environments in a matter of a couple days versus weeks or months,” Johnes said. “This company, Digital CM, sent us a Go Pro 360 camera and they asked us to take pictures outside the target building at the Aachen training area referred to as the mansion by some of the cadets.”
Johnes added four photos were shot on the outside of the target building — one on each corner. Then he and his team went into the building and set the camera in the middle of each room and took one picture per room.
In addition to the photos and videos they provided on their end, the Department of Geography, Col. Christopher Oxendine and Dr. Matthew O’banion, gathered the Light Detection and Ranging (LIDAR) data and provided most of the imagery to the companies. Usually, under a normal setting, the companies would have come to West Point and flown drones and put small bots into the nooks and crannies of the selected building or training site as a part of their imagery collection process.
“We sent them the data and they literally sent us a rendered sample of the site in two days,” Johnes said. "Subsequently, one of the team members wore a VR headset and recorded the entire walkthrough of the mansion and made a small video out of it like an Open Broadcaster Software (OBS) capture to share out with the Future Applied Systems Team."
Given this, Johnes explained what the process was like gathering information prior to the current technologies doing all the grunt work. He pointed to a rugged-looking map he drew on a whiteboard of a training site and reminisced on his tenure as a Military Police officer.
“Many years ago, I did live missions where you had to go in and get a whiteboard to draw out the plans and we’re looking at target x and then I’m breaking down how many possible windows, how many doors, how many stories was the building, etc.,” Johnes said. “That’s how we use to plan. We really had no idea what was on the other side of that wall. I really had no idea what the interior of the place looked like.”
Class of 2023 Cadet Mark Haroutunian, a member of the Future Applied Systems Team at West Point, said his team, which consists of nine cadets, worked as the test group for the VR simulation. On Oct. 6, the team ran a VR rehearsal of the mansion with a few of his teammates. During Sunday of that week, despite never physically training in that building, they used the information they gathered through the simulation to run the live training sequence in full combat-gear.
“I think that no matter how great of a job they do on the virtual simulation, it’s not going to be perfect,” Haroutunian said. “I think being able to reconcile the differences between virtual representation and what you actually see will be something that Soldiers in the future must overcome, but overall, the technology is still very effective with its conveyance of information.”
Haroutunian added it’s one thing to experience the location through VR, but it’s a whole other beast feeling the environs of the actual location. The way the stairs and corridors wind—the way they move around the doorways and through the windows was a very stimulating experience.
“The VR rehearsal allowed the cadets to tailor their plans, including how they moved from the tree line to the mansion.” Sgt. 1st Class Morgan Wallace, the Future Applied Systems Team officer-in-charge, said. “It allowed them to have a better intuitive understanding of the layout and increased spatial awareness of the mansion, which was very helpful in darkened areas.”
Wallace explained how the cadets validated that everyone knew the plan by watching them walk through the house. They knew what rooms to expect when they entered each door, and the VR simulation gave them enough sense of familiarity when exploring the building for the first time.
“I am very excited for the technology because we are able to get a rundown of the house and see what every room is like and we never had to leave the sixth floor at Washington Hall,” Haroutunian said of the VR simulation.
Dr. Gordon Cooke, an assistant professor at the West Point Simulation Center and the department’s research director for the Thunderstorm event, said this year was the first time cadets at West Point participated in a Thunderstorm event. This event gave cadets exposure to a larger military outside of the operational force they are used to, particularly, the research and acquisition communities in Futures Command and OSD.
“Cadets got to experience a little of how the military operates at an OSD level and to see how the military interacts with industry,” Cooke said. “In the next couple of years, we’re going to start seeing these technologies become fieldable products that Soldiers will actually be using that, only a few years ago, were in the realm of science fiction. You think about the Prometheus movie. There is a scene where they throw out a little drone and it goes flying around with LIDAR, mapping the surrounding space and the team, in their headquarters, gets a three-dimensional view of the terrain within that vicinity. That’s now in the realm of possibility — it can be done.”
Cooke added these products demonstrated that the technology works and there is a path for advancements that will evolve the technology further along the future.
“We will likely see these capabilities coming to the battlefield over the next several years,” Cooke said. “The technology will need to be ruggedized and improved before widespread adoption by the military, but the core capability is there.”