From 2D video conferencing to photorealistic 3D communication

Share

18 November 2024

Not a face on a screen, but a lifelike photorealistic 3D presentation: that is Social eXtended Reality, or Social XR. Computer scientist Simon Gunkel investigated how you can extend existing video conferencing tools to support holographic communication.

For Social XR, images are captured with RGBD, a camera that captures both color and depth. Gunkel and his colleagues focus on the complete end-to-end system pipeline for the RGBD format: capture, process, transmit and render.

Building blocks
The researchers divided the holographic communication pipeline into minimal and basic components. They also analyzed how each of the individual components (capture, processing, transmission and rendering) impacts performance and latency in the image. To do this, they used prototypes that consist of a complete chain of holographic 3D communication and its modular building blocks. They evaluated this system in complex simulation experiments and developed the prototypes according to various use cases.

With these results and evaluations, new and better Social XR systems can be built, and the technical characteristics and limitations of different systems can be compared more easily. The study also provides general guidelines for building and measuring Social XR systems.

New reference framework
According to Gunkel, the results of the study form a new reference framework for the building blocks of Social XR systems. In addition, it opens possibilities for controlled user evaluations, quality studies and research into technical developments around Social XR. Gunkel: 'Our findings and technical examples are applicable to education, remote assistance, healthcare, business and social gatherings and much more.'

Gunkels PhD defense takes place on December 3. His dissertation and a video in which he explains more about the use cases are available.