Microsoft has recently published a paper and patent for a new form of video streaming called ‘Gaze Based Video Stream Processing.’ It’s a simple but impressive software tool powered by AI that would bring the concept of foveated rendering to video meeting and chat applications like Microsoft Teams, Zoom, and Discord.
VIEW GALLERY – 3 IMAGES
It’s technology based on how we see as humans where objects within 5-10 degrees of our main focus are viewed with the most detail, with that quickly falling off by 20% outside of this 10 degrees.
Foveated rendering for VR gaming involves using eye-tracking to ensure that the areas or objects you’re directly looking at are rendered at the highest quality with all the effects and detail settings cranked. At the same time, areas or objects not being looked at have reduced quality to boost performance. With its new patent, Microsoft is bringing this concept to video streaming.
‘Gaze Based Video Stream Processing’ uses a system to estimate or predict where the user is looking, allowing the video stream processor to reduce the quality of the video streams that aren’t being looked at. The immediate benefit would be less bandwidth and a potentially more stable connection.
The patent offers pictures showcasing how it would look. Without the system, you’ve got three video streams of varying quality: two 1080p and one 720p. As a standard video stream, all three are presented at full quality based on the available bandwidth. With gaze-based video processing, the stream being looked at remains at full quality at 1080p 60 FPS, while the two streams outside of the user’s focus drop in quality to 360p 24 FPS.
It reduces the bandwidth requirement for video meetings and ensures that bandwidth is dedicated where needed. As AI and a neural network power gaze detection, this feature may come to Copilot+ PCs with integrated NPUs – with the first example deployed in Microsoft Teams.