Phones, glasses, and headsets can now all either overlay information on the world we’re looking at or immerse us entirely in imaginary ones. The processes of overlaying information, termed ‘augmented reality’, can be seen when Pokémon appear on our mobile phones, directions appear for nearby restaurants, or our food’s nutritional information is displayed when we point our camera at our plate.
The full immersion into alternate worlds, termed ‘virtual reality’, can let us look around a space, play games with others in simulated cockpits, or climb mountain ranges. Both technologies present novel opportunities to experience the world while simultaneously raising questions about how our data is collected or used. Whereas Augmented Reality (AR) routinely presents information on top of our real-world environment, virtual reality (VR) replaces that environment entirely. The technologies underlying AR and VR share some commonalities but the ways in which they collect and process data are different.
Seeing the unseen – Augmented Reality (AR)
AR systems supplement our view of the real world with computer-generated information, such as location coordinates, video or picture overlays, or auditory additions. They are predominantly linked to our mobile devices, such as smartphones and tablets. We either hold up our phone and it will present something on its screen that isn’t actually visible or there, or wear headphones to ‘hear’ the world differently such as by translating what someone is saying to you in near-real time. In other cases, our glasses or eyepieces themselves might overlay information on the world, such as directions to a place we’re visiting or information about what we are looking at.
The most common types of AR systems that we encounter today use our smartphones to collect, and present, information over top of our visual environment. They rely on smartphone cameras to visualize the world and subsequently overlay information on it. These services also may use geolocation sensors to determine where the device is located, gyroscopes to detect movement, audio sensors and mics to collect sounds, and speakers to plays words or sounds. In addition to smartphone-based AR systems, there are specialized headset systems that are used in a variety of professions, such as medicine, engineering, and design, to let the users see or hear more about what they’re doing.
A series of examples makes clearer how some of the smartphone-based AR systems rely on smartphones to present an ‘augmented’ experience of reality. Yelp, a review website, uses AR on its smartphone applications to display where different places are located that meet your search criteria. It also uses the camera, the GPS, as well as the compass to identify which direction you are facing. In other cases, an AR application can use the camera so that when you point your phone at a 2D barcode such as Quick Response (QR) code an image is displayed. And commercial applications are increasingly using AR to let you “try things on” before ordering online by pointing your camera at an image of yourself and overlaying the clothing or makeup on what the camera captures, or for your home, by pointing your camera at a wall to show what a piece of art would look like on it.
Living in a dream world – Virtual Reality (VR)
VR replaces our environment with one that is artificially constructed using computer modelling. There are different degrees of interactivity in a VR environment. For some technologies, such as those that rely on putting a smartphone screen in a headset in front of your eyes, like Samsung Gear or Google Cardboard, you may be limited to looking around a three-dimensional environment but lack opportunities to walk around or interact with your surroundings. Other VR systems, especially those that are connected to an external computer, will require you to wear a helmet-like device that is connected to that computer. These more powerful systems will let you move around in your environment, interact with objects or characters in it, and generally present a more immersive experience. These latter types of VR systems include Oculus developed by Facebook, Vive developed by Valve, and PlayStation VR developed by Sony Interactive Entertainment.
The more extensive VR systems that are connected to external computing infrastructure use sensors within the headsets to show different parts of the virtual world to you as you move your head and body around the environment. Other techniques develop heat maps of wherever you’re looking. They can also, however, use sensors to monitor facial expressions or eye movements in order to detect your emotional state.
Who is watching you while you’re enjoying your new world? – Privacy Implications
AR and VR have significantly different privacy implications. Many of the privacy concerns affecting the user of a VR system are linked to the collection of sensors that are included in headsets. By detecting your emotional state based on facial movements for example an application developer or marketer could, based on what they believe you are feeling, try to nudge you into purchasing something to alleviate your stress or anxiety or improve your mood. Also, by tracking what you’re looking at, a third-party could determine what kinds of virtual advertisements were more interesting to you and try to actively exploit your attention by presenting you with stimuli to which you are susceptible. Some of the companies that provide VR systems also note that they may collect physical movements and activities for development purposes, but there may also be potential for such information to subsequently be shared with third-parties such as marketers, or other interested parties as the developers of VR systems try to find ways of expanding the profitability of their current systems.
For AR, many of the concerns are linked to how the sensors attached to a smartphone are used, and what data are collected, stored, processed, and retained, as well as whether mobile identifiers are collected by the developer. Sensor-related information can include photos of individuals with possible clothing projected over their bodies, or images of their home with potential new furniture, or locations they interested in travelling to. In addition to requesting permission to access and activate sensors, AR applications may also request information such as contacts, email, access to the telephone, or other aspects of the smartphone that are not evidently used when opening and using the application. Such requests increase the risks of disclosing individual’s personal information or data contained on their smartphone they would rather not be shared such as pictures, contact information, or other data the device owner has collected. In effect, the issues associated with smartphone applications requesting access to more data than is necessary to provide a service remains a concern with smartphone-based AR applications.
An additional concern with AR is associated with the information that it may overlay on the world through the screen or headset that you’re using. While getting directions to a coffee shop isn’t necessarily intrusive, other applications will let you take photos of landmarks or other sites and upload them to the developer of the application for photo interactive or animation services for example. In the process, you may capture images or audio recordings of persons near those landmarks and sites regardless of whether or not they have consented to the picture.
In the future, AR applications might use the camera to conduct facial recognition of other people around you to identify them, present information they have either shared with the application developer in the past or which the developer accesses via a social media application. In other situations, the AR application might present geo-located ‘public’ information, such as social media comments, that could reveal ‘notes’ left intentionally or inadvertently. These notes might include information about people’s homes and whether they have recently left on vacation, or valuation of different properties, and the person’s past travel information. Furthermore, where a developer is collecting images of where you are pointing your camera they might be able to detect what you are looking at, over time, to profile you for other purposes such as behavioral tracking or advertising.
AR has existed for some time, but its popularity is only starting to take hold now that people generally have smartphones that are powerful enough to run sophisticated applications and games that use the technology. It remains an emergent technology, however, and its full ramifications remain to be determined. VR is at an even earlier stage of development, despite having been worked on for several decades. The contemporary headsets are bulky, can cause motion sickness, and require connections to powerful computers in order to present truly immersive virtualized environments. While it is still in a relatively nascent state of development, the technology is evolving at rapid speed, and its privacy implications are sure to follow suit!
Suggested Reading
- Second Life: Privacy in Virtual Worlds
- 5 key differences between virtual worlds and virtual reality
- Real Virtuality: A Code of Ethical Conduct. Recommendations for Good Scientific Practice and the Consumers of VR-Technology
- No Escape From Reality: Security and Privacy of Augmented Reality Browsers
- The Convergence of Virtual Reality and Social Networks: Threats to Privacy and Autonomy