Developer finds easier way to access Quest 3 cameras

A VR developer has found a smoother way to access Quest 3’s passthrough cameras.

Meta doesn’t officially allow third-party developers to access the cameras on Quest headsets, but earlier this week we reported on a clever workaround from Michael Gschwandtner, the co-founder and lead developer of hand-tracking fitness app XRWorkout.

A developer has figured out how to access Quest 3’s cameras

A developer has figured out how to access Quest 3’s passthrough cameras to run custom computer vision models.

Gschwandtner’s technique requires you to use Quest’s casting feature to cast to a WebView in his app that has loaded the URL oculus.com/casting. However, this requires the tedious process of logging into your meta account in that WebView and the user having to take action via casting.

Another VR developer, Julian Triveri, has now developed a technique that requires neither a WebView nor casting, and thus neither registration nor initiation of a system action.

Triveri realized that the Android Media Projection API, designed for screen capture for custom recording or casting setups, works on Quest headsets. On Quest, it returns 1024 x 1024 images of the central 82 degrees of field of view, slightly less than casting but still usable to run custom computer vision models.

The API requires permission granted by the user, but after that the app has full access to what the user sees.

Triveri has released an open source demo of this technique on GitHub and UploadVR has confirmed that it works as described. He even added a branch showing how to use the technique to control AprilTag tracking (a type of fiducial marker).

An interesting limitation of this method is that it cannot be used while recording or transmitting through the headset, so Triveri had to use a PC to capture footage. As with Gschwandtner’s earlier technique, the other limitation is that the captured view includes virtual objects and interfaces, not just the camera view, and this could interfere with computer vision tasks.

What happens next?

With Triveri’s discovery, any Quest app can now essentially access the cameras with minimal effort. Will Meta allow this approach to spread? Or could the company crack down and include a patch in a future version of its Horizon OS?

Last month, Meta CTO Bosworth shared his thoughts on the idea of ​​intentionally giving Quest developers access to the passthrough cameras in an interview, saying Meta will “continue to monitor” the issue.

Meta “Looking At”: Granting Quest developers access to raw cameras

Meta CTO Andrew Bosworth said the company would “continue to investigate” whether Quest developers could gain access to the camera’s raw data.

In comparison, Apple only grants raw access to its Vision Pro headset to enterprise companies and only for non-public internal apps.