Meta’s AR glasses, called Orion, stole the show at this year’s Meta Connect, but the conference featured plenty of other interesting announcements that deserve attention. While they may not be as groundbreaking as Orion, they offer valuable insights into Meta’s plans and strategy. Here’s the list.

Ray-Bans get a translate function

Meta seems to be enjoying success with its Ray-Ban smart glasses line, so much so that it extended its partnership with EssilorLuxottica, securing a ten-year deal that promises many more iterations in the future. On stage, Mark Zuckerberg highlighted new AI features, more intuitive voice commands, and fresh partnerships. However, one feature that I found especially noteworthy was the new on-the-fly translation function

As part of live demo Mark invited martial arts fighter Brandon Moreno on stage and had a real-time conversation with him. Brandon spoke in Spanish and Mark could hear an ongoing translation, similarly to how a live interpreter would function.

During the presentation both Mark and Brandon wore Smart Ray-Bans making the conversation appear seamless. It wasn’t explained what happens if only one person wears the glasses, which will likely be a much more common scenario. I expect the wearer would need to use a smartphone to translate back. Either way, this could be a very useful tool if executed properly — perfect for traveling or immersing yourself in foreign conversations. Among all the Meta Ray-Ban upgrades presented, this had the most definitive use case

Adversarial take: Meta Ray-Bans are already associated with ‘spy glasses’ and various privacy dilemmas. The ability to potentially eavesdrop on private conversations that people conduct in their native language without them knowing adds to those concerns.

Windows 11 Instant pairing

Perhaps taking cues from Apple Vision Pro’s seamless integration with the MacBook, Meta Quest will now automatically pair with a Windows 11 PC or laptop. According to Zuckerberg, the process will be frictionless: ‘You just look at the keyboard, and it’ll start pairing,’ he said, adding that this will make Quest a ‘natural extension of your PC.’

There are already a fair number of productivity apps on Quest that allow you to connect your computer to the headset, with Immersed and Fluid being two examples. However, by integrating this feature into HorizonOS, Meta will entice users to check it out, and if all goes well, increase retention rates and the average time spent in the headset.

The success of this approach was well demonstrated during the last Connect when Meta revealed Xbox Cloud Gaming

Xbox Cloud Gaming is a service that’s flat-screen but delivered inside VR. Similar to PC pairing, cloud gaming theoretically already existed via Virtual Desktop or the Xbox Android app. However, by making it an official part of the OS suite, Meta significantly increased interest in this particular solution, creating a new subset of power users who spend hours in their Quests playing Xbox games.

Meta doesn’t usually release specific numbers, but they have reiterated their commitment to Xbox Cloud Gaming this year, so it must be doing reasonably well. They also provided the audience with the slide above, which seems to support their commitment to bringing both 2D and immersive content to Quest.

Adversarial take: While it’s good that Meta is increasing support for PCs and laptops, why limit it to Windows? To truly differentiate themselves from Apple, Meta should enable pairing with Linux, Chromebooks, and any other systems that people might use for productivity. 

Instagram Reels getting AI treatment

This isn’t a VR or AR improvement, but it’s still worth pointing out. Instagram Reels has received a significant update. Again, it centers around translation, but this time the alteration goes further because the translation is done in the creator’s voice, with the video lip-synced to match the audio output.

This should help creators maximize their reach, eliminate the need for subtitles, and in turn, help Meta in attracting talent to their platform by offering higher engagement.

Adversarial take: While the altered videos mimic the original, they still represent an artificial ‘what if’ output, where neither the lip movements nor the voice are real. One could argue that Meta is easing users into accepting hallucinated AI content — a slippery slope that might eventually hurt creators.

Meta Quest API opens up camera access

At the developer segment of Meta Connect 2024, Mark Rabkin took to the stage and, in a surprise move, casually informed the audience that starting next year, Meta will allow developers to access camera data captured through passthrough.

This announcement was met with loud, cheering applause, as camera access had been the top-requested feature since the inception of passthrough on Quest 1. For years, Meta had been adamant about keeping camera access locked, and many developers had given up hope on their ideas.

Currently, the only mixed reality component available for access is the depth map, which is utilized in games like Coastermania and Mega Fireball.

It is unclear whether Meta will allow for raw camera access. More likely it will be some sort of meta-data solution. Details are not known although Mark Rabkin mentioned tracked objects, AI applications, fancy overlays, scene understanding and more.

Adversarial take: The potential risks are obvious, but it’s hard to envision quality mixed reality without some form of access solution. Interestingly Apple allows access to raw camera data for enterprise developers so perhaps Meta started to recognize changing tides

— — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — — —

These were the lesser-discussed announcements that I believe are worth highlighting. The next big event to watch for is AWE: Europe, which takes place in Vienna from October 29th to 30th. 

If you're interested in manually written (non-AI) posts regarding Tech, Virtual Reality, AR & Metaverse consider following me on Medium & X (Twitter)