A Review of the Metaverse

Abstract

The Metaverse is envisioned as a virtual shared space made up of interconnected worlds that users can access through virtual and augmented reality (VR/AR) technology. It is described as the next evolution of the internet; a place to work, socialise, and play. Meta, formerly Facebook, has invested heavily in the development of the Metaverse, but has faced criticism and financial setbacks due to poor marketing and negative public reception. Many consumers find themselves confused as to what exactly the Metaverse is and those who do grasp the concept often have little interest in pursuing it further. This article will investigate the substance behind the marketing, first exploring the history and context of VR/AR technology, including the founding of Oculus VR and its acquisition by Facebook, as well as the development of the HTC Vive by Valve, and Facebook’s later rebranding to Meta. We’ll then consider Meta’s current work, their strategic partnerships with EssilorLuxottica and Microsoft, and the future direction of their research in both hardware and software. The article will conclude with a brief look at Meta’s competitors in the area, namely Epic Games and Sony PlayStation.

Introduction

The Metaverse is conceived as a collective virtual shared space, created by the convergence of the real and the virtual. It is often pictured as a global network of interconnected virtual worlds, where users can interact with each other and access a vast range of digital services and experiences. These worlds will be experienced through VR/AR; the accompanying sense of presence offering a new frontier for human interaction and communication, offering endless possibilities for exploration, creation, and commerce.

Tech giant Meta is keen to capitalise, pouring an estimated $10bn per year into their grand vision of a shared Metaverse. However, investor scepticism is at an all-time high, and the company is now down $700bn from their peak market cap, losing more than $89bn on the 27th of October 2022 alone (Murphy, 2022). The company has not marketed the concept well to consumers and public demos have been met with almost-unanimous negative reception.

This review article will look at the major technological advances made in recent years towards achieving a Metaverse. We will look at these technologies in relation to their application in virtual spaces, but also how the research progress may prove useful elsewhere. The main body of research I will consult is Meta’s online archive of research papers. I will also look at Maynooth University Library database, Google Scholar, and the ScienceDirect database. This research work will form a solid foundation from which the Metaverse may be evaluated holistically on both its merits and deficiencies.

Discussion

History & Context:

i) Founding of Oculus VR & Acquisition by Facebook

Modern efforts in VR research can be traced back to the University of Southern California, with much research being carried out by the Mixed Reality Lab at the Institute for Creative Technologies (M. Bolas et al, 2013). One designer, Palmer Luckey, made significant progress in developing a prototype headset named the Oculus Rift, and founded a company named Oculus VR to make further developments. This prototype attracted the attention of both John Carmack of id Software and Gabe Newell of Valve. Their endorsements prompted Luckey to launch a Kickstarter for the project on August 1, 2012. The company flew past its goal of $250,000, ultimately raising $2,437,429. This gave Oculus VR the investment needed to launch a developer kit version of its headset: the Oculus Rift DK1 (Oculus, 2012).

Figure 1. Oculus Rift Development Kit 1

On March 25, 2014, Facebook announced their acquisition of Oculus VR for $2 billion. Zuckerberg was keen to position the company at the forefront of this emerging technology, stating “mobile is the platform of today, and now we’re also getting ready for the platforms of tomorrow” (Facebook, 2014). Following this acquisition, Oculus announced the first consumer-ready version of the Rift on May 6, 2015. The CV1 featured much-improved specs from the earlier development kit version, as well as a greater focus on comfort. It launched in March 2016 for €699.99. However, the CV1 still relied upon the somewhat clunky ‘constellation’ tracking system, requiring the use of external sensors and thus an extensive set-up process (Oculus VR, 2015).

Figure 2. Oculus Rift Constellation Sensor

ii) Valve’s Early Research Efforts

Valve played a key role in early VR, beginning their research in approximately 2012. They developed several technologies including tracking solutions, rendering techniques, and platforms like the SteamVR API. SteamVR allowed developers to quickly create PC VR games and apps, whilst ensuring users could easily access them, greatly easing the process of developing for VR (Vlachos, 2015).

Figure 3. SteamVR Room-Scale Tracking

Valve’s main early VR effort was a collaboration with consumer electronics company HTC in the form of a VR device named the HTC Vive. This headset allowed users to freely walk around their room using a new tracking system named ‘Lighthouse’. The Lighthouse base stations would sweep horizontal and vertical lasers across the room, allowing the headset and its controllers to determine their location based upon when its sensors receive these lasers (Dempsey, 2016). The Vive launched in 2016 at the price of €899. Much like the Oculus CV1, consumers did not readily take to the Vive due to its price and set-up process.

iii) Launch of the Oculus Quest

Around 2017, Facebook began releasing publications at an accelerated pace. Their work explored innovative solutions to problems such as sensor performance, form factor, power, and data-bandwidth. Work began in areas like hand-tracking, facial-tracking, and eye-tracking (Liu et al, 2017). Pose-tracking advances saw a new system implemented by which a headset could track its location without the use of external sensors. Cameras on the headset look outwards, building a 3D map of the environment, before using cameras & sensors to determine where the headset’s located. Evaluation of this tracking system has deemed it “more than adequate” for most virtual reality applications (Monica, 2022).

Figure 4. Oculus Quest, released May 21, 2019

The culmination of Facebook’s research saw the release of the Oculus Quest in 2019 at €399, bringing VR into the mainstream. The lack of external sensors and low-cost made the Quest highly convenient for the average consumer. Its standalone nature allowed the headset to be easily transported, without the need for a high-end PC. In the context of this success and falling social media revenue, Facebook began their transition into the Metaverse.

Today’s Metaverse:

i) Meta’s Recent Work

Facebook launched the Quest 2 on October 13, 2020, with iterative improvements over the first-generation. The headset launched to massive success and over ten million units were sold in under a year. Thus, in October 2021, Facebook announced their grand rebrand to Meta, revealing their plans for the “Metaverse”, something they position as the next evolution of the internet. The company describe the Metaverse as “an embodied internet where you’re in the experience, not just looking at it” (Meta, 2021). This would be a place for people to socialise, work, and play. To deliver on this vision, the company began to massively expand their funding of VR/AR research, with investment now reaching over $10bn a year. The Oculus brand has also been gradually phased out and the Oculus Quest 2 renamed to Meta Quest 2.

Meta’s exploration of this virtual future has covered a wide range of topics. One paper looks at developing a next-generation virtual assistant, capable of interacting with the user and observing a shared VR scene. The paper particularly focuses upon a multimodal input context, in which the input handled by the assistant includes “vision, memories of previous interactions, and the users’ utterances” (Moon et al. 2020). These next-generation assistants are seen as a key part of Meta’s vision of the future of the workplace.

Meta have also researched how to govern such a platform. Blackwell’s 2019 paper explores peoples’ experiences of harassment in a virtual space, looking at how such abuse can be worsened by the feeling of presence in a VR environment. Many users found harassment more intense in VR spaces, but also more difficult to report due to a lack of standardised controls or guidelines.

Figure 5. Horizon Worlds

Meta have released a free virtual reality game called Horizon Worlds, launched in December 2021. Players may interact with each other across user-generated worlds, taking part in various social activities. Horizon Worlds has received an almost universally negative reception. The application is rife with bugs and instability, with a small userbase. This weak application has become the face of Meta’s Metaverse and can be blamed for much negative buzz around the company as of late. Most visitors try the game once and never again return, “an empty world is a sad world” (Horwitz, 2022). Development remains steady and most recently, Meta announced the addition of legs to players’ virtual avatars. These had been simple floating torsos up until this point.

ii) Strategic Partnerships

Meta have announced several strategic partnerships. One such collaboration is their partnership with EssilorLuxottica, the parent company of Ray-Bans. The Ray-Ban Stories are first-generation smart glasses engineered by Meta to include a Snapdragon processor, touchpad, battery, and two cameras into a light glasses frame. These glasses allow the user to record up to 30 seconds of video or take a photograph. Meta sees this product as a crucial first step to a world of augmented-reality wearables that will “transform the way we connect and interact with people” (Reality Labs, 2021). The glasses have yet to catch on in mainstream markets, owing to privacy & security concerns.

Figure 6. Ray-Ban Stories

Meta’s partnership with Microsoft has seen some success. A key feature of Meta’s vision for the Metaverse is to improve productivity in the workplace. The Office 365 suite coming to Meta Quest has been a vital step towards this goal. At Meta Connect 2022, Microsoft & Meta announced their partnership, detailing their intentions to bring both Microsoft Teams and Office to the platform. (Teper, 2022).

iii) Current Issues Facing the Metaverse

Facebook, and therefore Meta, has a history of taking extreme liberties with personal data and thus, users have every right to raise questions around the vast swaths of data gathered through their use of VR devices. For example, the likes of eye-tracking data will prove immensely useful to advertisers. Despite promises from the founders of Oculus, it is now mandatory to login with a Facebook/Meta account in order to use a Quest. The risks around privacy here are unprecedented, given that the company intends for users to carry out much of life’s everyday tasks within this virtual environment. Regardless of Meta’s intentions with their users’ data, no trust has been established with consumers, providing a major roadblock in the adoption of the Metaverse as a whole (Di Pietro, 2021).

Hardware remains a challenge. The release of the Meta Quest Pro in 2022 sees a number of lingering hardware issues solved but has launched at €1,799; a price that is entirely inaccessible to the average consumer. This high price has not solved the longstanding issue of battery life, generally lasting around ninety minutes. Mobile CPU/GPU power also isn’t quite there yet; mobile SoCs unable to support the high resolution 90hz+ displays necessary for immersive VR. For many, the Metaverse will remain a dream until a device can be released which balances portability, power, and affordability. This headset must also be kept lightweight as a user’s comfort level over time is closely tied to the weight of the headset, with many reporting great discomfort from heavier headsets (Yan, 2019). Regardless of weight, many users simply don’t want to use VR for an extended period, often reporting motion sickness or a disconnect from the real world.

Meta’s marketing of the Metaverse is another large obstacle preventing its success. The company’s “Connect” presentations are excessively focus-tested and corporate-driven, often feeling completely tone-deaf and out of touch. Former consulting CTO, John Carmack, offered enthusiasts a brief reprieve from the corporate facelessness during his regular unscripted Q&A sessions. Carmack would adopt a brutally honest tone, laying bare where the company stood and where it needs to go next. This gave Meta’s otherwise sanitised tone credibility in the unforgiving marketplace of consumer attention. Carmack, however, resigned on December 17, 2022, stating he “wearied of the fight” (2022). Thus, there is little left in the way of positive marketing emanating from Meta; the company will need to work hard to reverse this image.

Future Development

i) Hardware

Whilst the current state of the Metaverse is quite bleak, the research being carried out at Reality Labs shows great promise. Meta continue to innovate in hardware, regularly showing prototypes of new headsets to the press. ‘Butterscotch’ is a high-resolution headset capable of displaying 55 pixels-per-degree; current human 20:20 vision lies around 60ppd, with the Quest 2 coming in at 20ppd. Another prototype, ‘Holocake’, focuses on comfort. It’s incredibly small & light, implementing pancake lenses to reduce the size of the headset. ‘Mirror Lake’ shows off a new type of varifocal lenses; these allow the user to switch their focus between foreground and background with ease through eye-tracking. Current headsets only allow a fixed focal plane. One particularly interesting prototype is ‘Starburst’, a high-dynamic-range display capable of 20,000 nits of brightness; Quest 2 currently stands around 100 nits. Meta carry out extensive user research for all these technologies, hoping to find a good middle-ground which balances all aspects needed for the perfect VR device (Matsuda et al, 2022).

One area of focus has been eye-tracking and face-tracking. Eye-tracking presents a number of benefits to the VR experience. Foveated rendering greatly reduces the GPU workload by using an eye-tracker to reduce image quality in the user’s peripheral vision. The user’s tracked eyes can also be simulated on their avatar, allowing for eye-contact with other users in VR. Eye-tracking data can be used to correct distortions caused by VR lenses, again heightening the sense of presence (Guan et al, 2022). The Meta Quest Pro implements a sophisticated eye-tracking system and rudimentary form of face tracking. This is accomplished using infrared sensors within the headset. A pre-trained AI model derives facial expressions from this data and can map it to a variety of avatars, both cartoon-like and realistic.

ii) Software

Figure 7. Metaverse Selfie

Perhaps the single event which has caused the most harm to the public image of the Metaverse is Mark Zuckerberg’s ill-fated Metaverse selfie. On August 16, 2022, Mark Zuckerberg announced the launch of Horizon Worlds in France and Spain. Accompanying his post was a photo of his dead-eyed avatar self in front of a low-poly Eiffel tower. The image’s lighting looks cheap and the graphics poor quality (Zuckerberg, 2022). The selfie was widely ridiculed.

Fortunately, the above image does not represent the company’s state-of-the-art work in graphics. In June 2021, Meta released a paper detailing ‘Pixel Codec Avatars’, photorealistic CG human avatars, indistinguishable from their real-life counterparts. These avatars are computationally inexpensive, with the possibility of up to five rendered in the same scene on an Oculus Quest 2. The avatar representation only decodes visible pixels on the facial model in screen-space, achieving extremely high-fidelity animation on weak hardware such as the mobile CPU/GPU seen in the Quest 2. The facial models are constructed using a deep convolutional architecture and highly sophisticated encoder/decoder set-up (Ma et al. 2021).

Figure 8. Pixel Codec Avatar

Initially, creating these facial models required an extensive process which needed to be repeated for each person using specialised equipment. However, the ability to reproduce a person’s likeliness faithfully is of course a key aspect of the Metaverse and therefore must be accessible to all; not just those in the VFX industry. To combat this, Reality Labs recently proposed a new approach wherein a user may employ a simple mobile phone capture to obtain a fully working 3D avatar. This avatar is not only extremely high-fidelity but can be animated to achieve a variety of facial expressions. The capture takes moments, before undergoing some processing at Meta’s servers. There yet remain a number of limitations: glasses are not handled correctly and the processing time for finetuning is extensive. Once this technology becomes widely available, the ability to quickly and easily create a photorealistic avatar of oneself will prove vital to the success of the Metaverse (Ma et al, 2022).

Figure 9. Authentic Volumetric Avatars from a Phone Scan

Having developed these full-body human avatars, Meta have more recently begun to pursue the simulation of photorealistic clothing. They propose a clothing model which is computationally expensive whilst training but can later be reproduced at low-cost on a mobile device. Their method achieves nigh-perfect cloth geometry simulation. Human avatars can then be dressed with this clothing, once it has been fully modelled. Purchasing clothing for one’s avatar is already common in many video games today, and one can see how a similar appeal in the Metaverse. Thus, it’s important that clothing matches its real-life counterpart as closely as possible.

Figure 10. Deep Photorealistic Appearance for Physically Simulated Clothing

iii) Alternatives to Meta’s Metaverse

Although Meta are the company with the most financial heft behind their Metaverse effort, it is not certain they will get there first. Epic Games’ Fortnite has made around $25bn since its launch in 2017 and in many ways has already achieved the Metaverse goal. Millions of kids and young adults use the game as a place to relax and socialise. You have the ability to customise your avatar in any way you’d like. Every large media property in the world has been involved; there’s nowhere else you can fight Goku as Kratos, attend a Travis Scott concert as John Wick, or equip Batman with a lightsabre and fight Dwayne ‘The Rock’ Johnson to the death. Fortnite is free of charge and available on every gaming device and phone on the planet. It has an expansive map in which you may build, fight, go fishing, and more. The game is updated at an incredible rate, ensuring players always have new content to return to. Fashion brand Balenciaga are another on the infinite list of brands who have collaborated with Epic. Perhaps the largest advantage Fortnite has over Meta’ Metaverse is that it’s simply fun, something Meta have yet to achieve despite billions of dollars invested in research and development.

PlayStation are an established brand keen to make further forays into VR technology. Their first headset, the PSVR1, launched in 2016 and has since sold five million units worldwide. In early 2021, Sony announced the PSVR2 for release in March 2023. The new headset uses a similar form of inside-out tracking as the Meta Quest 2, as well as completely redesigned controllers capable of haptic feedback. Although the headset launches at a high cost of €599.99, it will benefit from a built-in audience of over twenty million PS5 owners & hard-hitting launch titles such as Horizon: Call of the Mountain. Sony are well-positioned to overtake Meta in this area, despite their lack of a standalone headset.

Conclusion

This article looked at the historical context leading to the current state-of-the-art in Metaverse and VR technology. It covered the founding of Oculus VR, Valve’s early efforts, and the later rebranding of Facebook to Meta. We then looked at Meta’s recent work, their strategic partnerships, and issues facing the Metaverse. In terms of future direction, Meta are powering ahead with a number of innovative prototypes and some revolutionary software. Ultimately, their exact vision of the Metaverse may not come to fruition due to privacy concerns around the company and dismal marketing. However, the research carried out by Reality Labs will prove vital to the digital future; a future we will likely find ourselves in regardless. The stark reality is that Meta have lost $700bn in market cap so far and can only fund this caper for so long before it must start paying dividends. A future article could revisit this topic and examine whether Meta’s work has finally broken into the mainstream to deliver real profits and productivity improvements in the workplace. Until Meta overcomes its marketing and image problems, their vision will remain a far-away dream; one that could ultimately sink the company.

References

Blackwell et al. (2019). Harassment in Social Virtual Reality: Challenges for Platform Governance. Conference on Computer-Supported Cooperative Work and Social Computing. https://research.facebook.com/publications/harassment-in-social-virtual-reality-challenges-for-platform-governance/

Carmack, J. (2022). This is the end of my decade in VR. https://www.facebook.com/permalink.php?story_fbid=pfbid0iPixEvPJQGzNa6t2x6HUL5TYqfmKGqSgfkBg6QaTyHF5frXQi7eLGxC7uPQv5U5jl&id=100006735798590

Cao et al. (2022). Authentic Volumetric Avatars from a Phone Scan. Reality Labs. https://research.facebook.com/publications/authentic-volumetric-avatars-from-a-phone-scan/

Cipresso, P., Giglioli, I. A. C., Raya, M. A (2018, October 10). The past, present, and future of virtual and Augmented Reality Research: A network and cluster analysis of the literature. Frontiers. https://www.frontiersin.org/articles/10.3389/fpsyg.2018.02086/full

Dempsey P. (2016). The Teardown: HTC Vive VR Headset. Engineering & Technology, vol. 11, no. 7-8, pp. 80-81 doi: 10.1049/et.2016.0731.

Di Pietro R. (2021). Metaverse: Security and Privacy Issues. https://www.researchgate.net/publication/357116743_Metaverse_Security_and_Privacy_Issues

Facebook. (2014). Facebook to Acquire Oculus. https://about.fb.com/news/2014/03/facebook-to-acquire-oculus/

Guan et al. (2022). Reality Labs. https://research.facebook.com/file/561571255671080/Perceptual-Requirements-for-Eye-Tracked-Distortion-Correction-in-VR-.pdf

Horwitz J. (2022). Company Documents Show Meta’s Flagship Metaverse Falling Short. https://www.wsj.com/articles/meta-metaverse-horizon-worlds-zuckerberg-facebook-internal-documents-11665778961

Liu et al. (2017). Sensors for Future VR Applications. International Image Sensor Workshop. https://research.facebook.com/publications/international-image-sensors-workshop/

Matsuda, N., Zhao, Yang, Z., Chapiro, A., Smith, C., Lanman, D. (2022). HDR VR. Reality Labs. https://research.facebook.com/publications/hdr-vr/

Matsuda et al. (2022). Realistic Luminance in VR. Reality Labs. https://research.facebook.com/publications/realistic-luminance-in-vr/

Meta. (2021). Introducing Meta: A Social Technology Company. https://about.fb.com/news/2021/10/facebook-company-is-now-meta/

M. Bolas et al. (2013). Open virtual reality IEEE Virtual Reality (VR), 2013, pp. 183-184, doi: 10.1109/VR.2013.6549423.

Moon et al. (2020). Situated and Interactive Multimodal Conversations. International Conference on Computational Linguistics. https://research.facebook.com/publications/situated-and-interactive-multimodal-conversations/

Monica, R., Aleotti, J. (2022). Evaluation of the Oculus Rift S tracking system in room scale virtual reality. Virtual Reality 26, 1335–1345. https://doi.org/10.1007/s10055-022-00637-3

Murphy, Hannah. (2022). Meta’s value plunges by $89bn amid falling sales and rising costs. The Financial Times. https://www.ft.com/content/9e457f1a-ee9f-419d-b8aa-2b54f7809705

Oculus VR. (2012). Oculus Rift: Step Into The Game. Kickstarter.com. https://www.kickstarter.com/projects/1523379957/oculus-rift-step-into-the-game

Oculus VR. (2015). The Oculus Rift, Oculus Touch, and VR Games at E3. https://www.oculus.com/blog/the-oculus-rift-oculus-touch-and-vr-games-at-e3/

Reality Labs. (2021). Ray-Ban and Facebook introduce Ray-Ban Stories, first-generation smart glasses. https://tech.facebook.com/reality-labs/2021/9/ray-ban-and-facebook-introduce-ray-ban-stories-first-generation-smart-glasses/

Shugao et al. (2021). Pixel Codec Avatars. Reality Labs. https://research.facebook.com/publications/pixel-codec-avatars/

Singh et al. (2020). Significant applications of virtual reality for COVID-19 pandemic. Diabetes & Metabolic Syndrome: Clinical Research & Reviews. https://doi.org/10.1016/j.dsx.2020.05.011

Sparkes, Matthew. (2021). What is a metaverse. New Scientist, Volume 251, Issue 3348. https://www.sciencedirect.com/science/article/pii/S0262407921014500

Teper, J. (2022). Microsoft and Meta partner to deliver immersive experiences for the future of work and play. https://blogs.microsoft.com/blog/2022/10/11/microsoft-and-meta-partner-to-deliver-immersive-experiences-for-the-future-of-work-and-play/

Vlachos, Alex. (2015). Advanced VR Rendering. Game Developers Conference, 2015.https://media.steampowered.com/apps/valve/2015/Alex_Vlachos_Advanced_VR_Rendering_GDC2015.pdf

Wang et al. (2022). A Survey on Metaverse: Fundamentals, Security, and Privacy. IEEE Communications Surveys & Tutorials doi: 10.1109/COMST.2022.3202047.

Xiang et al. (2022). Dressing Avatars: Deep Photorealistic Appearance for Physically Simulated Clothing. Reality Labs. https://research.facebook.com/publications/dressing-avatars-deep-photorealistic-appearance-for-physically-simulated-clothing/

Xiong et al. (2021). Augmented reality and virtual reality displays: emerging technologies and future perspectives. Light: Science & Applications. https://www.nature.com/articles/s41377-021-00658-8

Yan et al. (2019). The Effects of Weight on Comfort of Virtual Reality Devices. https://www.researchgate.net/publication/325961229_The_Effects_of_Weight_on_Comfort_of_Virtual_Reality_Devices

Zuckerberg M. (2022). Metaverse Selfie. https://www.facebook.com/photo/?fbid=10114625396809351