Immersive Media & Haptics: New Standards for New Experiences

Immersive media represents the next generation of content in which the user perspective can be dynamically engaged and updated. In its simplest form, immersive media is three-dimensional and gives the user one view per eye to create a sense of depth, while encompassing user movement to encourage higher levels of immersivity.

Immersive video codecs make the delivery and consumption of data-rich video content efficient and seamless, offering the "magic" that empowers us to enjoy a diversity of 2D and increasingly 3D, volumetric and haptic-enhanced video. This piece outlines the immersive video codecs that are encouraging new depths in our multimedia ecosystem with a specific focus on haptics.

Why Standards Matter

With more than 4.1 billion devices equipped with haptic actuators and the ability to convey haptic feedback, standards will be critical in defining the mechanisms and codecs used to deliver the sensory experience. The research and innovation led at InterDigital, and the growth and evolution of this ecosystem, will graduate simple haptic-enhanced content into more complex and immersive experiences at scale.

The current haptic environment is very fragmented and difficult for content creators and product developers to navigate different proprietary methods to communicate tactile sensations. Thankfully, there is industry consensus that interoperability is crucial to success, and standards allow the industry to coalesce around defined methods for capturing, encoding, transmitting, decoding and replicating immersive haptic experiences.

Standards are vital in defining interoperability between platforms and vendors across the immersive and haptics ecosystem. From immersive content creation to distribution and the decoding characteristics recommended for haptic-enabled devices, standards establish a consistent approach to ensure that all aspects involved in the end-to-end delivery of haptics are efficiently coded and compatible. Standards, pre-standards groups, and industry forums are a critical line of defense against market fragmentation and investment stagnation that may result from distinct and proprietary haptic methods that are not interoperable.

Snapshot: Standards for Immersive and Haptic Content Haptics

Industry Forum: This industry group aims to unite the haptics ecosystem through the promotion of industry standards, practices, and metrics for haptic performance. It combines the efforts of member companies and several standards organizations to ensure they coalesce to deliver a cohesive strategy for haptics.

MPEG: In terms of content creation and service delivery, MPEG is active in describing methods for encoding, encapsulating, transmitting and decoding immersive and haptic data. In June 2021, MPEG issued a Call for Proposal to establish a reference architecture for future haptic standards, promoting haptics as a recognized media type in ISO base media file formats like 3gp and mp4 and streaming formats like DASH. This allows for the integration of haptic data alongside audio and video, including immersive content, ensuring consistent, sensory-rich experiences.

MPEG-I Standard on Haptics: In mid-2020, a working group comprised of InterDigital, immersion, Interhaptics, Apple, Lofelt, the Technical University of Munich, and many others was established to outline the requirements for a coded representation of timed haptic media. The standard should allow MPEG files to contain only haptics, haptics and audio, or other combinations of media tracks, with the goal of establishing haptics as a first-order media type in all ISO media files and distribution protocols.. It resulted in the specification of an ISO standard for the representation and coding of haptics, to be published by the end of 2024

MPEG Video-based Point Cloud Compression (V-PCC): Point clouds are a method of grouping tiny "points" together to visually represent 3D images and objects in high definition. This standard addresses the vital need for video compression mechanisms to enjoy increasingly immersive and interactive video experiences in VR, AR, and 3D imagery. Every frame of a point cloud video contains roughly 800,000 points and requires approximately 1500 MBps to deliver uncompressed, therefore V-PCC is integral to seamlessly deliver this complex and data-rich immersive video content.

MPEG Immersive Video (MIV) - The standard for pre-and post-processing immersive video formats that are interoperable with codecs like AVC, HEVC, VVC and EVC. The MIV standard helps ensure video data for immersive or other light field applications can be compressed to optimize the end user experience. It is developed to support the compression, storage and distribution of immersive video content that enables playback with six degrees of freedom for view position and orientation.

IETF (The Internet Engineering Task Force): This task force is active in describing methods for encoding, encapsulating, transmitting and decoding data. The IETF has recognized haptics as a top-level media type and is working to bringing their standards in line with other industry groups, such as ISO, MPEG and OpenXR. Their work defines haptics as a fundamental component within future internet communications and will establish agreed procedures for encapsulating haptics into IP protocols.

W3C (The World Wide Web Consortium): This consortium is working to define application programming interfaces (APIs) for internet browsers to enable haptic feedback in web pages and web-based applications. These APIs allow Chrome and Firefox browsers on Android to control vibrotactile actuators in mobile devices, offering tactile feedback through vibration patterns. W3C aims to develop more advanced and portable haptic feedback standards for consistent performance across different devices in the future.

IEEE (The Institute of Electrical and Electronics Engineers): This pre-standards group is advancing haptic experiences through two key initiatives. The IEEE 19181.1-2024 standard, published in June 2024, defines vibrotactile coding developed under the Tactile Internet in 5G applications, facilitating remote operation and touch over the internet. The IEEE P2861.3 standard is developing a high-performance haptic API standard with vendor-independent profiles to enhance experiences and haptic feedback across different devices.

ATSC (The Advanced Television Systems Committee): This committee aims to expand immersion and engagement through enabling haptic experiences across broadcast television and streaming services. The ATSC 3.0 standard aims to utilize haptic actuators in mobile devices, synchronizing tactile feedback with audiovisual content, benefiting various types of content. In April 2024, ATSC published a recommendation, A/380, that proposes including haptic content in event streams or through separate downloads, ensuring compatibility with existing mechanisms for an enriched viewing experience.

Khronos Group: Working to define operating system and platform requirements, with assistance from Haptics Industry Forum member companies, The Khronos Group is renowned for developing royalty-free open standards in 3D computer graphics. Since 2017, they have focused on standards for AR and VR, releasing OpenXR1.1, which has widely been adopted by major companies. OpenXR enables developers to reach various hardware platforms without rewriting code and enhances XR applications with haptics support, partnering with the Haptics Industry Forum to help expand capabilities.

Public link

Please use the above public link if you want to share this noodl on another website.