Unity Software Inc.
08/14/2024 | Press release | Archived content
How BMW is leveraging Unity to open up new dimensions for the driving experience
With a rich history rooted in superior engine performance, most know the BMW brand as being synonymous with an elevated driving experience. As the BMW Group has also been known for pioneering digital innovations in the automotive industry in the past decade, it should come as no surprise that in the last several years, the research team at BMW Group has been leading the charge when it comes to finding ways to bring augmented reality technology to their vehicles. We sat down with research engineers Manfred Pauli and Wolfgang Haberl to discuss their vision for the future of driving and how BMW is innovating the in-vehicle experience with augmented reality (AR).
BMW Group's AR glasses display alerts to an in-road hazard using a visual display
Which teams or business units at BMW are currently utilizing Unity?
Manfred: We are a part of the BMW Group research team. We build early prototypes, and provide a lookout for future generations of BMW vehicles. Unity is used throughout the company, especially for designing head-up display graphics or developing graphics prototypes.
How does this new prototype for a head-up display work? What are the current capabilities when it comes to using AR to build a new driver experience?
Wolfgang: In this case, it is an AR glasses display, so it has some similarities to the head-up system. We try to see what the specifics for AR glasses are and where they make sense in addition to a traditional head-up display. Our current prototype includes some driver and some passenger-related use cases to demonstrate the range of possibilities. For the driver, we have driver assistance augmentation. You see, for example, assisting graphical distance representations during parking, you have your navigation arrows that guide you exactly along the road and the intersections where you have to turn. It also highlights other things, like markers for potholes and construction sites, as well as some signage that you might overlook, like a speed limit or pedestrian crossing. All that information is brought into your central view in AR. For the passenger, we are really trying to explore what might be a nice usage of time in the car. We have implemented a gaming example that allows you to collect coins that appear along the driving route in order to gain points. We also have a feature that lets you use the glasses as your personal movie theater.
BMW Group's AR glasses display gaming dice along the route to engage passengers in a real-time experience
Can you speak about what goals you are ultimately trying to achieve by bringing AR to drivers? You touched on safety and navigation, as well as the overall driving experience and making it more immersive. What do you hope to see?
Manfred: BMW is, as you know, not a manufacturer of AR glasses. So, what we want to do is enable the customer to be able to connect their existing AR glasses that they already use outside of the car. The overall goal is to make any set of AR glasses work inside the car, as we have done with the smartphone. There are already a multitude of AR apps, but what is on the market right now doesn't work during a drive because the car and head both move. The devices have to be able to differentiate between these movements, but to our knowledge, none of the glasses manufacturers can do the necessary 6 degrees-of-freedom (6-dof) tracking without additional data from the car yet. We have developed our own glasses tracking algorithm, software, and vehicle interface so that we can enable glasses to work inside the car. That is the first step.
Wolfgang: Getting this basic system to work has taken up a lot of our time over the past years. This tracking solution enables us to place stable augmentations while the car is moving. Now we are able to put virtual assets on the dashboard that follow the movements of the car. At the same time, we can place signage or markers on the road or on a building in your surroundings that are independent of the movements of the car or the user's head.
We are prototyping different use cases to see what makes sense and what is helpful for the driver. Of course, it is always about creating content that will make driving safer. We are trying to see what we can display to reduce your cognitive load when you have the augmentation right in front of you. We test these features with test drivers in our simulator to study whether their reaction times get better when wearing the glasses compared to using a classic cockpit.
Will AR glasses be complementary to what you're seeing in the head-up display or is it basically the same information?
Wolfgang: We think that current AR glasses could be a great extension to a regular head-up system. The head-up display provides important information that is not world-locked, like vehicle speed, turn-by-turn navigation, and the current speed limit. It is always available, no matter whether you're weaning the glasses or not. The glasses provide a huge field of view, significantly larger than the image sizes of any head-up displays. This allows coverage of a good portion of the real world with augmentations, thus extending the traditional head-up.
Let's talk a bit more broadly about the auto industry and general AR adoption. As you said, this is all in early development. What do you think the current state is for most manufacturers, and what are you, as BMW, and others prioritizing as you move toward this?
Wolfgang: From what we see on the market, a lot of companies are currently integrating AR features into their existing head-up systems, which from our point of view makes a lot of sense. At the same time, these solutions are still limited to the smaller field of view of the head-up system. That's why we put a lot of effort into AR glasses to see how they could make use of the remaining field of view. We have the only system right now, which works with a small pair of AR glasses that are really ergonomic and provide a compelling form factor and design (the Xreal Air 2). We have put a lot of development effort into getting the tracking to work on a customer- friendly device.
We hope more companies that produce glasses will cooperate with us and make their glasses compatible with our vehicles. In the future, we expect glasses to be as ubiquitous as smartphones. We want to set an industry standard with our integration in order to provide a common ground for glasses manufacturers and car manufacturers to have a working platform independent of a certain combination of glasses and car brands.
Manfred: We went public with a different pair of glasses in 2014. We used ODG smart glasses, and if you compare those to the ones we're using now, a lot has changed. The field of view has almost doubled, and image quality has improved a lot. In addition, we can now use Unity. We are really glad to see this technology mature and are proud of our work.
What limitations around driver safety, consumer expectations, or hardware are you considering when you think about the next phase of this?
Wolfgang: From a consumer's point of view, we want glasses to get even smaller in order to increase wearability and comfort. Most of the current devices are targeted towards indoor use and have a very little tint. So, if you're driving outside in the sun, the tint might not be dark enough to see the displayed content. On the other hand, if you're driving at night, you don't need any tint at all. Consequently, dynamic tinting is a desirable feature. At the same time, you need a very good display that provides a broad range of brightness to achieve convincing image quality even in bright sunlight.
Many of these glasses are also battery-powered, so they have a limited runtime. Of course, we also provide the most important driving information in one of the car's built-in displays for safety reasons. Still, we hope to see runtimes increase for longer drives.
We are able to merge the separate display areas of our different systems in the car to avoid overlap. With our tracking system, we know exactly where the glasses are looking, and we also know where the area for the head-up system and other displays are. So, we can omit putting items in the glasses on top of other displays, giving us the possibility to run them in parallel.
BMW Group's AR glasses display arrows giving route guidance in real-time
What are your hopes for BMW's role in the development of this technology?
Manfred: BMW was the first to introduce the head-up display technology, inspired by the aviation industry. So what's next after head-up? We are launching a completely new user interface with the BMW Panoramic Vision in series vehicles starting in 2025. The whole width of the windshield is then used for display, pillar-to-pillar. We are always thinking about what's next. We want to point out the huge benefit of AR glasses in the car. Our main goal is creating maximum safety in combination with sheer driving pleasure. BMW drivers are not distracted but intuitively supported in their driving task, resulting in additional excitement.
Wolfgang: We believe an important step is creating a standard for car integration with more glasses manufacturers and to keep improving it. From a customer's perspective, their glasses should simply work in their car, independent of the respective manufacturers.
When it came to building this, how did you make the decision to choose Unity? How does Unity fit into your current development workflows?
Wolfgang: Most of the companies offering AR glasses on the market right now support Unity. Today, the Unity Engine is the de facto standard for AR glasses. It definitely made the most sense for us to go with Unity for this reason.
Manfred: The Unity Asset Store was another added benefit. The map plugins from Infinity Code were a huge help in developing navigation features. We had designers working on the prototype that would bring in assets from Blender, and then we integrated them into the Unity editor and deployed them through our Android devices. From everything around recording routes with our cars and simulation, Unity has been a very customizable tool that helped us out a lot.
Wolfgang: We know that a lot of times, technical research projects are hard to convey or explain to non-experts and sometimes also to decision makers. We always try to implement use cases and great visualizations to transport the idea and the vision of the final customer product. Unity was a great help with abstracting away from all the low-level problems of getting the visualizations displayed in the desired way. This made our whole vision a lot more understandable and really fascinated people. It allowed them to see how cool and immersive the future of AR glasses in vehicles could be.
Want to hear more about this exciting project? Join our deep dive session with BMW at Unite 2024 in Barcelona.
Explore the possibilities of Unity for automotive, or contact us for more information