You probably already come across and interacted with Virtual Reality’s cousin – Augmented Reality (AR) in your daily life. Sometimes without even realising it. Filters on Snapchat and Instagram, the ‘try before you buy’ trend offered by a number of retailers and even Google Maps and Waze.
Augmented reality (AR) essentially adds additional layers of digital content onto a live camera feed, making it look as if the digital content is part of the real environment. As noted, the applications of this technology could be from making your face look like a zebra to overlaying digital directions onto the physical streets around you.
VR, on the other hand, comes with much more promise – of which if it ever lives up to, will enable users to visit Mars, swim with sharks, sing on stage with the Rolling Stones and dine in royal banquet halls. These experiences can be enjoyed without ever leaving your home. Rather than adding content to, or augmenting the real-world, VR simulates entirely new and completely imaginary worlds.
(The reality of VR growth 2016-2021. Image: TechCrunch)
The vast prospect for VR across industries has caused an explosion in the VR technology market. In fact, from mobile VR, to smart glasses and console; the growth trajectory is forecast to continue its upward spell.
What is Virtual Reality?
The technology uses high-performance and sensory equipment, like headsets and gloves, to immerse users into new 3-dimensional digital environments in which they can interact with digital objects in a seemingly real way. VR as escapism at its purest form should comprise the following 5 features:
- Believable – the power of VR lays within its believability. The illusions of being, seeing, hearing and touching digital environments make users believe they are in a virtual world.
- Interactive – the vast applications of VR depend entirely on its interactive nature. As a user moves around, the VR world should move with them, they should be able to move things, change things, feel things.
- Computer-generated – the most powerful machines with realistic 3D computer-generated graphics are needed to make believable, interactive and immersive experiences of alternative worlds that change in real-time.
- Immersive – VR has the capability to engage your body and mind. When it comes to VR training, for instance, a trainee pilot could play flight simulator games on a PC and be lost in a very realistic and interactive experience – but a VR flight simulator will sit a user in a hydraulically operated mockup of a real cockpit and make them feel every force as it tilts and turns.
- Explorable – good virtual reality experiences will be ‘big’ enough or have enough scope to move around in. Exploration is key as it is the essence of escapism and provides capacity for alternative applications.
When done well and these 5 aspects are incorporated into the experience, VR can be a highly valuable tool across industries, including everyday consumer use. Before we take a look at the practically endless possibilities of virtual reality, it’s important to understand how it actually works, and how VR experiences are created.
How does Virtual Reality work?
The earliest source of today’s VR system dates back to 1957 as a head-mounted stereoscopic television device. Since then, virtual reality technology has been under continuous development. It seems to have made slow and steady progress up until the past few years. This could be down to lacking computer power in the past to make the truly immersive VR experiences we appreciate today. Once the technology came along, it was a race to make virtual reality more portable, responsive, immersive and affordable.
Today, companies like Oculus, HTC and Sony offer commercially viable VR hardware that continues to improve. These commercial options have made VR accessible to people all over the world. It’s certainly helped spread the word on virtual reality – but it’s difficult to imagine that everyone actually understands the specifics of the technology and how it works. So, how does virtual reality work?
The basics: Virtual Reality 101
Oculus Rift and even PlayStation’s VR are often referred to as head-mounted displays that include no hand-tracking. Google Cardboard – used to place a mobile screen in front of your face can be enough to get a user half-immersed in a virtual world.
The premise behind this is that such headsets create the illusion of life-size, 3 dimensional, virtual environments without the boundaries and interruptions that come with watching TV or computer screens.
Whichever way a user turns their head, the screen mounted to their face moves with them. Unlike augmented reality – which overlays digital content onto a user’s view of the real world – VR enables users to become part of an immersive new world.
Video and audio content is sent from the computer or console to the headset either through an HDMI – as is the case for HTC’s Vive – or through a smartphone that is slotted into the headset as with Samsung Gear VR.
When it comes to the field of vision, VR headsets typically use two feeds sent to one display or one LCD display per eye. VR goggles act as a lens between the users’ eyes and the pixels – these can be adjusted from person to person to ensure the optimal distance between the eyes. The purpose of these additional lenses is to reshape the picture for each eye and create a 3-dimensional image. It does so by readjusting and angling the 2-dimensional images to mimic the way each eye views things differently.
The idea behind this can be appreciated by doing the dominant eye test. For instance, with both eyes open, position an object between a circle you create with your fingers and thumb. Then, close your left eye – if the object stays centred, then your right eye (the one that stays open) is your dominant eye.
(Global VR headset market size by end device 2014-2025. Image: GVR)
High-end devices of VR headsets have been dominating the market, indicating that the industry is moving towards more sophisticated, innovative and user-experience-driven development.
The Science Behind Virtual Reality
Truly immersive virtual reality experiences call for a number of prerequisites. Beginning with a frame rate of a minimum of 60fps, an equally powerful refresh rate and while a 180-degree field of view is desirable, it should be no less than 100-degree. For context, the frame rate is the speed at which the GPU can process an image per second. Screen refresh rates’ measure the pace of the display to render images and field of view is the amount of eye and head movement the display can support.
(Frame rate per second. Image: Better Editor)
When these prerequisites are not met to minimum standards, users of VR will likely experience latency – a delay in between their actions and the response from the screen. In order to fully convince the human brain that the VR experience could be real – the response time should be less than 20 milliseconds.
‘Cybersickness’ occurs when there is any inconsistency between the frame rate and refresh rate. When the GPUs frame rate is more than the refresh rate then images become distorted. So, the frame rate should be limited to the monitor’s refresh rate using Vertical Sync (VSync).
There are a few VR headsets available on the market today that adhere to these delicate prerequisites. Among them, Vive and Rift each have a 110-degree field of vision, whereas Google Cardboard has 90, whilst the newest Google Daydream has 120-degrees. HTC Vive and Oculus Rift come with 90hz displays and PSVR offers a 60hz display.
The Technical Basis of Virtual Reality
The VR technology that exists today simply cannot provide a completely immersive experience. Glitches here and there have thus far been inevitable. Commercial VR systems are competing to provide the best user experience, of which depends on sophisticating a range of technical factors until the technology available finally catches up with the capabilities of human vision.
Audio and Sound
When combined with visuals, sound effects can seriously elevate the level of engagement in a VR experience. The user’s belief that the VR environment could possibly be real is reassured when headphones and 3D sound effects are implemented. There is a fine balance to strike here. When adding sound effects, developers should take into account the relationship between graphics and sound to create a harmonious and believable experience.
VR technology today relies on spatial audio to generate a simulated audio environment that is in sync with the visuals created by VR. It essentially helps to produce binaural audio through a set of headphones that mimics the sensation of hearing sounds coming from different directions using the following tactics:
- Volume control
- Left/right delay to convey direction
- Using head tracking to map auditory space
- Manipulating reverberation and echo to accompany environmental factors
Remember VR is all about accounting for the movement of the user. So, for a VR headset, the audio effects should be computed in real-time. There has been great development on this, but it remains to be an area of VR hardware that is still exploring possibilities.
Head and Eye Tracking
The real value of VR is less dependent on convincing visuals and sound effects, but more so, on the fact that users can move within a virtual environment that adjusts when they do. This is the very principle that separates virtual reality from video games. Eye and head tracking can be ensured using laser pointers, LED lights and mobile sensors.
Mobile VR uses the accelerometer to detect 3-dimensional movement, gyroscope for angular movement and magnetometer to locate the position relative to the Earth. A higher degree of accuracy can be achieved by installing cameras and sensors into the room – far more costly than mobile sensors.
(Stereoscopic vision illustrated. Image: MDN)
To date, there are a few options on the market. Mobile VR headsets like Samsung Gear VR, Google’s Daydream View, and Oculus Go, are all capable of rotational tracking only. This means that these headsets will be responsive to a user turning their head up, down, left and right. They won’t, however, pick up on a user moving their whole body.
By contrast, headsets that use 6DoF can track the user’s position within a room as well as their heads’ direction. These allow for full autonomous movement within a 3D environment. Depending on the platform, this level of immersion is typically achieved by using camera-based tracking in concert with infrared light beacons.
Field of View
One of the biggest differences between the human experience and even the most sophisticated virtual reality experience is that humans are capable of a much wider field of view (FOV). Typically, a human can see the space around them in a 200-220 degree arc around their head. The degree to which a human’s left and right eyes overlap is the 114-degree scope where we can see in 3D.
Typically, most VR headset developers today, focus on this 114-degree 3d vision space to deliver virtual content. Whilst no VR technology to date can match the full FOV of the average human, designers are working towards creating a 180-degree FOV to complement high performance VR simulations.
The frame rate is the speed at which the Graphics Processing Unit (GPU) can process an image per second. Since there is no real consensus on how the human eye perceives this, it’s been a hot topic in VR development. The human eye can see up to the equivalent of 1000 frames per second (FPS). However, it does not receive the information via the optic nerve.
Some studies have found that humans can discern frame rates of up to 150 FPS – beyond this point, the information gets lost in translation. That’s why VR developers typically aim for a minimum of 60 FPS – 90 FPS. Less than this causes disorientation, headaches and nausea for the user.
It’s expected that moving forward with the technology, most VR headset developers will begin to pursue a frame rate of 120 FPS or more to allow for more realistic and truly immersive experiences in most applications. Speaking of which, what are the business applications of VR?
How Does VR Work in Different Industries?
(Application of VR versus AR across sectors. Image: Digi Capital)
Aside from the obvious applications in gaming and actual first-hand participation in VR, increasingly, films, live-streamed concerts and theatre performances are being delivered through VR by capturing these events and using tools to upload and Livestream them. With added immersive factors like smells, taste and touch, these experiences will soon feel more real than ever before.
Art and design
The scope for VR in this area is truly exciting given its ability to connect people remotely. Consider creating a 3D painting whilst collaborating with people from all over the world. Further, objects on a grand scale can be built in 3D with virtual tools and projected into a digitally created replica of an environment. The new generation of 3D designers will be able to explore these tools and take product design, architecture and the arts even further.
The gaming industry was one of the first to take VR and run with it. The immersive and interactive experiences VR allows, enhanced the user experience on all levels. The developments in the technology have given game designers the freedom to create new levels of immersion. It has also played a role in reaching new audiences – those who perhaps were not previously interested in gaming. Today, VR based games attract those with an interest in innovative tech, since they can jump into a game, reach out to touch things, turn their heads to look around – instead of mastering controllers, joysticks and buttons.
VR can model the world in an incredibly visual way. From medicine and chemistry, to physics and astronomy – the human anatomy, atmospheric space, destinations far away and times of the past can all be brought into human vision using VR. What’s more, it can allow those things and places to be shrunk, expanded, turned, and tossed. VR in education can provide high quality learning experiences through participation without the risks.
Hospitality and tourism
From showcasing hotels to clients and taking travellers on tours of the most popular cities abroad to help them make their holiday making decisions, VR in the tourism space is particularly useful. The nature of VR in transporting users to another environment makes VR a powerful marketing tool. Virtual tourism could just be the next best thing to being there.
Meditation and wellbeing
Since you can go anywhere imaginable with VR – it can be the perfect environment for your mind. The ‘happy place’ you create in your head when meditating at home, can be simulated and visited to relax, think and recharge. It can also offer therapy in a safe place to explore something uncomfortable in a protective environment simulation.
As the virtual world is removed from the real world – they can be great places to explore human behaviour in a real setting without the risks. Research has also found that being immersed in a VR experience is so distracting, that it can be used as a substitute for traditional medicine.
Retail and real estate
Virtual reality is highly promising in this space since it removes the need to visit multiple stores, let’s people try before they buy, and as a result, significantly reduces returns. It effectively allows users to place life-size pieces of furniture, for example, in their homes, walk around it, move it and manipulate designs before they commit to buying. The same is true with clothes, accessories, tech and even property. In light of the Coronavirus pandemic, shopping, in particular, is likely to never be the same thanks to VR.
Telepresence and communication
We already know that we can communicate and connect meaningfully with each other remotely. Mobile phones and the internet have already done a good job in proving this. But now, VR can multiply the effects of remote communication by allowing users to inhibit a telepresence robot with cameras mounted on its body, for instance. Whilst this is still being experimented on, we know that the current possibilities to join each other in virtual reality rooms and meet up in far away lands are more enthralling than FaceTime. Mesmerise is a good example of a company that focuses on creating immersive experiences for gatherings, media and conferences.
The Future of Virtual Reality
There’s no doubt that we’ve come a long way since the inception of VR in the 1950s. As advanced as it has become, VR has a long way to go to reach its full potential. The coming years will be the most exciting. Development in VR technology should be followed by a roll-out of hardware with enhanced, more true-to-human FOV and better 3D audio match. These three elements alone, make the short term future for VR very exciting.
Current developments of VR are already set to change the course of the technology. The use of haptic feedback devices like those found in HaptX Gloves provide realistic touch sensations of objects that users interact within VR. Another development in the graphics of VR is being seen in foveated rendering. This takes advantage of the human eyes’ limited focal point and then delivers ultra-high definition images where the eye focuses. This technology significantly lowers the computing power required to create the image – which in itself has a range of speed and performance benefits.
It’s an exciting time for all emerging technologies since there have been parallel advances in machine learning technology. For the education sector, this could make immersive distance learning a reality. Surgeons and trainee doctors will benefit from advancements in VR training to improve patient outcomes since medical education and training will become more practical, safe and immersive. It will also allow for heaps of improvement in the therapy space. Those in need of treatment for PTSD, for instance, will find a safe place to heal.
Virtual reality is only just beginning to recognise its potential in reshaping the way we work, live, learn and play. As technology develops, so too will the talent of the software developers and the application of VR will become more innovative and revolutionary. We’re only just beginning to explore VR and its place in our human world. The developments to come are expected to be life-changing.