Augmented reality

Augmented reality (AR), also known as mixed reality (MR), is a technology that overlays real-time 3D-rendered computer graphics onto a portion of the real world through a display, such as a handheld device or head-mounted display. This experience is seamlessly interwoven with the physical world such that it is perceived as an immersive aspect of the real environment.^[1] In this way, augmented reality alters one's ongoing perception of a real-world environment, compared to virtual reality, which aims to completely replace the user's real-world environment with a simulated one.^[2][3] Augmented reality is typically visual, but can span multiple sensory modalities, including auditory, haptic, and somatosensory.^[4]

A man using an augmented reality headset to view a life-size virtual model of a building

An augmented reality mapping application

The primary value of augmented reality is the manner in which components of a digital world blend into a person's perception of the real world, through the integration of immersive sensations, which are perceived as real in the user's environment. The earliest functional AR systems that provided immersive mixed reality experiences for users were invented in the early 1990s, starting with the Virtual Fixtures system developed at the U.S. Air Force's Armstrong Laboratory in 1992.^[1][5][6] Commercial augmented reality experiences were first introduced in entertainment and gaming businesses.^[7] Subsequently, augmented reality applications have spanned industries such as education, communications, medicine, and entertainment.

Augmented reality can be used to enhance natural environments or situations and offers perceptually enriched experiences. With the help of advanced AR technologies (e.g. adding computer vision, incorporating AR cameras into smartphone applications, and object recognition) the information about the surrounding real world of the user becomes interactive and digitally manipulated.^[8] Information about the environment and its objects is overlaid on the real world. This information can be virtual or real, e.g. seeing other real sensed or measured information such as electromagnetic radio waves overlaid in exact alignment with where they actually are in space.^[9][10][11] Augmented reality also has a lot of potential in the gathering and sharing of tacit knowledge. Immersive perceptual information is sometimes combined with supplemental information like scores over a live video feed of a sporting event. This combines the benefits of both augmented reality technology and heads up display technology (HUD).

Augmented reality frameworks include ARKit and ARCore. Commercial augmented reality headsets include the Magic Leap 1 and HoloLens. A number of companies have promoted the concept of smartglasses that have augmented reality capability.

Augmented reality can be defined as a system that incorporates three basic features: a combination of real and virtual worlds, real-time interaction, and accurate 3D registration of virtual and real objects.^[12] The overlaid sensory information can be constructive (i.e. additive to the natural environment), or destructive (i.e. masking of the natural environment).^[1] As such, it is one of the key technologies in the reality-virtuality continuum.^[13] Augmented reality refers to experiences that are artificial and that add to the already existing reality.^[14][15][16]

Comparison with mixed reality/virtual reality

Augmented reality (AR) is largely synonymous with mixed reality (MR). There is also overlap in terminology with extended reality and computer-mediated reality. However, In the 2020s, the differences between AR and MR began to be emphasized.^[17][18]

Types of extended reality

Mixed reality (MR) is an advanced technology that extends beyond augmented reality (AR) by seamlessly integrating the physical and virtual worlds.^[19] In MR, users are not only able to view digital content within their real environment but can also interact with it as if it were a tangible part of the physical world.^[20] This is made possible through devices such as Meta Quest 3S and Apple Vison Pro, which utilize multiple cameras and sensors to enable real-time interaction between virtual and physical elements.^[21] Mixed reality that incorporates haptics has sometimes been referred to as visuo-haptic mixed reality.^[22][23]

In virtual reality (VR), the users' perception is completely computer-generated, whereas with augmented reality (AR), it is partially generated and partially from the real world.^[24][25] For example, in architecture, VR can be used to create a walk-through simulation of the inside of a new building; and AR can be used to show a building's structures and systems super-imposed on a real-life view. Another example is through the use of utility applications. Some AR applications, such as Augment, enable users to apply digital objects into real environments, allowing businesses to use augmented reality devices as a way to preview their products in the real world.^[26] Similarly, it can also be used to demo what products may look like in an environment for customers, as demonstrated by companies such as Mountain Equipment Co-op or Lowe's who use augmented reality to allow customers to preview what their products might look like at home.^[27]

Augmented reality (AR) differs from virtual reality (VR) in the sense that in AR, the surrounding environment is 'real' and AR is just adding virtual objects to the real environment. On the other hand, in VR, the surrounding environment is completely virtual and computer generated. A demonstration of how AR layers objects onto the real world can be seen with augmented reality games. WallaMe is an augmented reality game application that allows users to hide messages in real environments, utilizing geolocation technology in order to enable users to hide messages wherever they may wish in the world.^[28]

In a physics context, the term "interreality system" refers to a virtual reality system coupled with its real-world counterpart.^[29] A 2007 paper describes an interreality system comprising a real physical pendulum coupled to a pendulum that only exists in virtual reality.^[30] This system has two stable states of motion: a "dual reality" state in which the motion of the two pendula are uncorrelated, and a "mixed reality" state in which the pendula exhibit stable phase-locked motion, which is highly correlated. The use of the terms "mixed reality" and "interreality" is clearly defined in the context of physics and may be slightly different in other fields, however, it is generally seen as, "bridging the physical and virtual world".^[31]

History

Virtual Fixtures – first AR system, U.S. Air Force, Wright-Patterson Air Force Base (1992)

• 1901: Author L. Frank Baum, in his science-fiction novel The Master Key, first mentions the idea of an electronic display/spectacles that overlays data onto real life (in this case 'people'). It is named a 'character marker'.^[32]
• Heads-up displays (HUDs), a precursor technology to augmented reality, were first developed for pilots in the 1950s, projecting simple flight data into their line of sight, thereby enabling them to keep their "heads up" and not look down at the instruments. It is a transparent display.
• 1968: Ivan Sutherland creates the first head-mounted display that has graphics rendered by a computer.^[33]
• 1975: Myron Krueger creates Videoplace to allow users to interact with virtual objects.
• 1980: The research by Gavan Lintern of the University of Illinois is the first published work to show the value of a heads up display for teaching real-world flight skills.^[34]
• 1980: Steve Mann creates the first wearable computer, a computer vision system with text and graphical overlays on a photographically mediated scene.^[35]
• 1986: Within IBM, Ron Feigenblatt describes the most widely experienced form of AR today (viz. "magic window," e.g. smartphone-based Pokémon Go), use of a small, "smart" flat panel display positioned and oriented by hand.^[36][37]
• 1987: Douglas George and Robert Morris create a working prototype of an astronomical telescope-based "heads-up display" system (a precursor concept to augmented reality) which superimposed in the telescope eyepiece, over the actual sky images, multi-intensity star, and celestial body images, and other relevant information.^[38]
• 1990: The term augmented reality is attributed to Thomas P. Caudell, a former Boeing researcher.^[39]
• 1992: Louis Rosenberg developed one of the first functioning AR systems, called Virtual Fixtures, at the United States Air Force Research Laboratory—Armstrong, that demonstrated benefit to human perception.^[40]
• 1992: Steven Feiner, Blair MacIntyre and Doree Seligmann present an early paper on an AR system prototype, KARMA, at the Graphics Interface conference.
• 1993: Mike Abernathy, et al., report the first use of augmented reality in identifying space debris using Rockwell WorldView by overlaying satellite geographic trajectories on live telescope video.^[41]
• 1993: A widely cited version of the paper above is published in Communications of the ACM – Special issue on computer augmented environments, edited by Pierre Wellner, Wendy Mackay, and Rich Gold.^[42]
• 1993: Loral WDL, with sponsorship from STRICOM, performed the first demonstration combining live AR-equipped vehicles and manned simulators. Unpublished paper, J. Barrilleaux, "Experiences and Observations in Applying Augmented Reality to Live Training", 1999.^[43]
• 1995: S. Ravela et al. at University of Massachusetts introduce a vision-based system using monocular cameras to track objects (engine blocks) across views for augmented reality.^[44][45]
• 1996: General Electric develops system for projecting information from 3D CAD models onto real-world instances of those models.^[46]
• 1998: Spatial augmented reality introduced at University of North Carolina at Chapel Hill by Ramesh Raskar, Greg Welch, Henry Fuchs.^[47]
• 1999: Frank Delgado, Mike Abernathy et al. report successful flight test of LandForm software video map overlay from a helicopter at Army Yuma Proving Ground overlaying video with runways, taxiways, roads and road names.^[48][49]
• 1999: The US Naval Research Laboratory engages on a decade-long research program called the Battlefield Augmented Reality System (BARS) to prototype some of the early wearable systems for dismounted soldier operating in urban environment for situation awareness and training.^[50]
• 1999: NASA X-38 flown using LandForm software video map overlays at Dryden Flight Research Center.^[51]
• 2000: Rockwell International Science Center demonstrates tetherless wearable augmented reality systems receiving analog video and 3D audio over radio-frequency wireless channels. The systems incorporate outdoor navigation capabilities, with digital horizon silhouettes from a terrain database overlain in real time on the live outdoor scene, allowing visualization of terrain made invisible by clouds and fog.^[52][53]
• 2004: An outdoor helmet-mounted AR system was demonstrated by Trimble Navigation and the Human Interface Technology Laboratory (HIT lab).^[54]
• 2006: Outland Research develops AR media player that overlays virtual content onto a users view of the real world synchronously with playing music, thereby providing an immersive AR entertainment experience.^[55][56]
• 2008: Wikitude AR Travel Guide launches on 20 Oct 2008 with the G1 Android phone.^[57]
• 2009: ARToolkit was ported to Adobe Flash (FLARToolkit) by Saqoosha, bringing augmented reality to the web browser.^[58]
• 2012: Launch of Lyteshot, an interactive AR gaming platform that utilizes smart glasses for game data
• 2013: Niantic releases "Ingress", an augmented reality mobile game for iOS and Android operating systems (and a predecessor of Pokémon Go).
• 2015: Microsoft announced the HoloLens augmented reality headset, which uses various sensors and a processing unit to display virtual imagery over the real world.^[59]
• 2016: Niantic released Pokémon Go for iOS and Android in July 2016. The game quickly became one of the most popular smartphone applications and in turn spikes the popularity of augmented reality games.^[60]
• 2018: Magic Leap launched the Magic Leap One augmented reality headset.^[61] Leap Motion announced the Project North Star augmented reality headset, and later released it under an open source license.^{[62][63][64][65]}
• 2019: Microsoft announced HoloLens 2 with significant improvements in terms of field of view and ergonomics.^[66]
• 2022: Magic Leap launched the Magic Leap 2 headset.^[67]
• 2023: Meta Quest 3, a mixed reality VR headset^[68] was developed by Reality Labs, a division of Meta Platforms. In the same year, Apple Vison Pro was released.
• 2024: Meta Platforms revealed the Orion AR glasses prototype.^[69]

Hardware and displays

AR visuals appear on handheld devices (video passthrough) and head-mounted displays (optical see-through or video passthrough). Systems pair a display with sensors (e.g., cameras and IMUs) to register virtual content to the environment; research also explores near-eye optics, projection-based AR, and experimental concepts such as contact-lens or retinal-scanned displays.^[70][71]

Photograph of the head-up display of a F/A-18C

Head-mounted displays

Main article: Head-mounted display

AR HMDs place virtual imagery in the user's view using optical see-through or video passthrough and track head motion for stable registration.^[72]

Handheld

Phone and tablet AR uses the rear camera (video passthrough) plus on-device SLAM/VIO for tracking.^[73][74]

Head-up display

Main article: Head-up display

HUDs project information into the forward view; AR variants align graphics to the outside scene (e.g., lane guidance, hazards).^[75]

A user standing in the middle of a cave automatic virtual environment

Cave automatic virtual environment

Main article: Cave automatic virtual environment

Room-scale projection systems surround users with imagery for co-located, multi-user AR/VR.^[76]

Contact lenses

Main article: Bionic contact lens

Prototypes explore embedding display/antenna elements into lenses for glanceable AR; most work remains experimental.^[77][78]

Virtual retinal display

Main article: Virtual retinal display

VRD concepts scan imagery directly onto the retina for high-contrast viewing.^[79]

Projection mapping

Main article: Projection mapping

Projectors overlay graphics onto real objects/environments without head-worn displays (spatial AR).^[80]

A man wearing an augmented reality headset

AR glasses

Main article: Smartglasses

Glasses-style near-eye displays aim for lighter, hands-free AR; approaches vary in optics, tracking, and power.^[72]

Tracking and registration

Further information: Simultaneous localization and mapping, Visual odometry, Fiducial marker, and Image registration

AR systems estimate device pose and scene geometry so virtual graphics stay aligned with the real world. Common approaches include visual–inertial odometry and SLAM for markerless tracking, and fiducial markers when known patterns are available; image registration and depth cues (e.g., occlusion, shadows) maintain realism.^[71][81][82]

Software and standards

Further information: ARKit, ARCore, and Augmented Reality Markup Language

AR runtimes provide sensing, tracking, and rendering pipelines; mobile platforms expose SDKs with camera access and spatial tracking. Interchange/geospatial formats such as ARML standardize anchors and content.^[83][84][73]

Interaction and input

Further information: Human–computer interaction and Gesture recognition

Input commonly combines head/gaze with touch, controllers, voice, or hand tracking; audio and haptics can reduce visual load. Human-factors studies report performance benefits but also workload and safety trade-offs depending on task and context.^[85][82]

Design considerations

Key usability factors include stable registration, legible contrast under varied lighting, and low motion-to-photon latency. Visual design often uses depth cues (occlusion, shadows) to support spatial judgment; safety-critical uses emphasize glanceable prompts and minimal interaction.^[86][87][71]

Applications

Further information: Industrial augmented reality

Augmented reality has been explored for many uses, including gaming, medicine, and entertainment. It has also been explored for education and business.^[88] Some of the earliest cited examples include augmented reality used to support surgery by providing virtual overlays to guide medical practitioners, to AR content for astronomy and welding.^[6][89] Example application areas described below include archaeology, architecture, commerce and education.

Education and training

Overlays models and step-by-step guidance in real settings (e.g., anatomy, maintenance); systematic reviews report learning benefits alongside design and implementation caveats that vary by context and task.^[90][91][92]

Medicine

Guidance overlays and image fusion support planning and intraoperative visualization across several specialties; reviews note accuracy/registration constraints and workflow integration issues.^[93][94][95]

Industry

Hands-free work instructions, inspection, and remote assistance tied to assets; evidence highlights productivity gains alongside limits around tracking robustness, ergonomics, and change management.^[96][97][98]

An AR mobile game using a trigger image as fiducial marker

Entertainment and games

Location-based and camera-based play place virtual objects in real spaces; recent surveys cover design patterns, effectiveness, and safety/attention trade-offs.^{[99][100][101]}

Navigation and maps

Augmented reality navigation overlays route guidance or hazard cues onto the real scene, typically via smartphone "live view" or in-vehicle heads-up displays. Research finds AR can improve wayfinding and driver situation awareness, but human-factors trade-offs (distraction, cognitive load, occlusion) matter for safety-critical use.^{[102][103][104][105]}

See also: Head-up display, Automotive navigation system, Wayfinding

Architecture, engineering, and construction

In the AEC sector, AR is used for design visualization, on-site verification against BIM models, clash detection, and guided assembly/inspection. Systematic reviews report benefits for communication and error reduction, while noting limits around tracking robustness and workflow integration.^{[106][107][108]}

Archaeology

AR has been used to aid archaeological research. By augmenting archaeological features onto the modern landscape, AR allows archaeologists to formulate possible site configurations from extant structures.^[109] Computer generated models of ruins, buildings, landscapes or even ancient people have been recycled into early archaeological AR applications.^{[110][111][112]} For example, implementing a system like VITA (Visual Interaction Tool for Archaeology) will allow users to imagine and investigate instant excavation results without leaving their home. Each user can collaborate by mutually "navigating, searching, and viewing data". Hrvoje Benko, a researcher in the computer science department at Columbia University, points out that these particular systems and others like them can provide "3D panoramic images and 3D models of the site itself at different excavation stages" all the while organizing much of the data in a collaborative way that is easy to use. Collaborative AR systems supply multimodal interactions that combine the real world with virtual images of both environments.^[113]

Commerce

Main article: Commercial augmented reality

AR is used to integrate print and video marketing. Printed marketing material can be designed with certain "trigger" images that, when scanned by an AR-enabled device using image recognition, activate a video version of the promotional material. A major difference between augmented reality and straightforward image recognition is that one can overlay multiple media at the same time in the view screen, such as social media share buttons, the in-page video even audio and 3D objects. Traditional print-only publications are using augmented reality to connect different types of media.^{[114][115][116][117][118]}

AR can enhance product previews such as allowing a customer to view what's inside a product's packaging without opening it.^[119] AR can also be used as an aid in selecting products from a catalog or through a kiosk. Scanned images of products can activate views of additional content such as customization options and additional images of the product in its use.^[120]

In 2018, Apple announced Universal Scene Description (USDZ) AR file support for iPhones and iPads with iOS 12. Apple has created an AR QuickLook Gallery that allows people to experience augmented reality through their own Apple device.^[121]

In 2018, Shopify, the Canadian e-commerce company, announced AR Quick Look integration. Their merchants will be able to upload 3D models of their products and their users will be able to tap on the models inside the Safari browser on their iOS devices to view them in their real-world environments.^[122]

In 2018, Twinkl released a free AR classroom application. Pupils can see how York looked over 1,900 years ago.^[123] Twinkl launched the first ever multi-player AR game, Little Red^[124] and has over 100 free AR educational models.^[125]

Augmented reality is becoming more frequently used for online advertising. Retailers offer the ability to upload a picture on their website and "try on" various clothes which are overlaid on the picture. Even further, companies such as Bodymetrics install dressing booths in department stores that offer full-body scanning. These booths render a 3D model of the user, allowing the consumers to view different outfits on themselves without the need of physically changing clothes.^[126] For example, JC Penney and Bloomingdale's use "virtual dressing rooms" that allow customers to see themselves in clothes without trying them on.^[127] Another store that uses AR to market clothing to its customers is Neiman Marcus.^[128] Neiman Marcus offers consumers the ability to see their outfits in a 360-degree view with their "memory mirror".^[128] Makeup stores like L'Oreal, Sephora, Charlotte Tilbury, and Rimmel also have apps that utilize AR.^[129] These apps allow consumers to see how the makeup will look on them.^[129] According to Greg Jones, director of AR and VR at Google, augmented reality is going to "reconnect physical and digital retail".^[129]

AR technology is also used by furniture retailers such as IKEA, Houzz, and Wayfair.^[129][127] These retailers offer apps that allow consumers to view their products in their home prior to purchasing anything.^[129][130] In 2017, Ikea announced the Ikea Place app. It contains a catalogue of over 2,000 products—nearly the company's full collection of sofas, armchairs, coffee tables, and storage units which one can place anywhere in a room with their phone.^[131] The app made it possible to have 3D and true-to-scale models of furniture in the customer's living space. IKEA realized that their customers are not shopping in stores as often or making direct purchases anymore.^[132][133] Shopify's acquisition of Primer, an AR app aims to push small and medium-sized sellers towards interactive AR shopping with easy to use AR integration and user experience for both merchants and consumers. AR helps the retail industry reduce operating costs. Merchants upload product information to the AR system, and consumers can use mobile terminals to search and generate 3D maps.^[134]

Literature

An example of an AR code containing a QR code

The first description of AR as it is known today was in Virtual Light, the 1994 novel by William Gibson.

Fitness

AR hardware and software for use in fitness includes smart glasses made for biking and running, with performance analytics and map navigation projected onto the user's field of vision,^[135] and boxing, martial arts, and tennis, where users remain aware of their physical environment for safety.^[136] Fitness-related games and software include Pokémon Go and Jurassic World Alive.^[137]

Emergency management/search and rescue

Augmented reality systems are used in public safety situations, from super storms to suspects at large.

As early as 2009, two articles from Emergency Management discussed AR technology for emergency management. The first was "Augmented Reality—Emerging Technology for Emergency Management", by Gerald Baron.^[138] According to Adam Crow,: "Technologies like augmented reality (ex: Google Glass) and the growing expectation of the public will continue to force professional emergency managers to radically shift when, where, and how technology is deployed before, during, and after disasters."^[139]

Another early example was a search aircraft looking for a lost hiker in rugged mountain terrain. Augmented reality systems provided aerial camera operators with a geographic awareness of forest road names and locations blended with the camera video. The camera operator was better able to search for the hiker knowing the geographic context of the camera image. Once located, the operator could more efficiently direct rescuers to the hiker's location because the geographic position and reference landmarks were clearly labeled.^[140]

Social interaction

AR can be used to facilitate social interaction, however, use of an AR headset can inhibit the quality of an interaction between two people if one isn't wearing one if the headset becomes a distraction.^[141]

Augmented reality also gives users the ability to practice different forms of social interactions with other people in a safe, risk-free environment. Hannes Kauffman, Associate Professor for virtual reality at TU Vienna, says: "In collaborative augmented reality multiple users may access a shared space populated by virtual objects, while remaining grounded in the real world. This technique is particularly powerful for educational purposes when users are collocated and can use natural means of communication (speech, gestures, etc.), but can also be mixed successfully with immersive VR or remote collaboration."^{[This quote needs a citation]} Hannes cites education as a potential use of this technology.

Healthcare planning, practice and education

One of the first applications of augmented reality was in healthcare, particularly to support the planning, practice, and training of surgical procedures. As far back as 1992, enhancing human performance during surgery was a formally stated objective when building the first augmented reality systems at U.S. Air Force laboratories.^[1] AR provides surgeons with patient monitoring data in the style of a fighter pilot's heads-up display, and allows patient imaging records, including functional videos, to be accessed and overlaid. Examples include a virtual X-ray view based on prior tomography or on real-time images from ultrasound and confocal microscopy probes,^[142] visualizing the position of a tumor in the video of an endoscope,^[143] or radiation exposure risks from X-ray imaging devices.^[144][145] AR can enhance viewing a fetus inside a mother's womb.^[146] Siemens, Karl Storz and IRCAD have developed a system for laparoscopic liver surgery that uses AR to view sub-surface tumors and vessels.^[147] AR has been used for cockroach phobia treatment^[148] and to reduce the fear of spiders.^[149] Patients wearing augmented reality glasses can be reminded to take medications.^[150] Augmented reality can be very helpful in the medical field.^[151] It could be used to provide crucial information to a doctor or surgeon without having them take their eyes off the patient.

On 30 April 2015, Microsoft announced the Microsoft HoloLens, their first attempt at augmented reality. The HoloLens is capable of displaying images for image-guided surgery.^[152] As augmented reality advances, it finds increasing applications in healthcare. Augmented reality and similar computer based-utilities are being used to train medical professionals.^[153][154] In healthcare, AR can be used to provide guidance during diagnostic and therapeutic interventions e.g. during surgery. Magee et al.,^[155] for instance, describe the use of augmented reality for medical training in simulating ultrasound-guided needle placement. Recently, augmented reality began seeing adoption in neurosurgery, a field that requires heavy amounts of imaging before procedures.^[156]

Smartglasses can be incorporated into the operating room to aide in surgical procedures; possibly displaying patient data conveniently while overlaying precise visual guides for the surgeon.^[157][158] Augmented reality headsets like the Microsoft HoloLens have been theorized to allow for efficient sharing of information between doctors, in addition to providing a platform for enhanced training.^[159][158] This can, in some situations (i.e. patient infected with contagious disease), improve doctor safety and reduce PPE use.^[160] While mixed reality has lots of potential for enhancing healthcare, it does have some drawbacks too.^[158] The technology may never fully integrate into scenarios when a patient is present, as there are ethical concerns surrounding the doctor not being able to see the patient.^[158] Mixed reality is also useful for healthcare education. For example, according to a 2022 report from the World Economic Forum, 85% of first-year medical students at Case Western Reserve University reported that mixed reality for teaching anatomy was "equivalent" or "better" than the in-person class.^[161]

Spatial immersion and interaction

Augmented reality applications, running on handheld devices utilized as virtual reality headsets, can also digitize human presence in space and provide a computer generated model of them, in a virtual space where they can interact and perform various actions. Such capabilities are demonstrated by Project Anywhere, developed by a postgraduate student at ETH Zurich, which was dubbed as an "out-of-body experience".^{[162][163][164]}

Flight training

Building on decades of perceptual-motor research in experimental psychology, researchers at the Aviation Research Laboratory of the University of Illinois at Urbana–Champaign used augmented reality in the form of a flight path in the sky to teach flight students how to land an airplane using a flight simulator. An adaptive augmented schedule in which students were shown the augmentation only when they departed from the flight path proved to be a more effective training intervention than a constant schedule.^[34][165] Flight students taught to land in the simulator with the adaptive augmentation learned to land a light aircraft more quickly than students with the same amount of landing training in the simulator but with constant augmentation or without any augmentation.^[34]

Military

Augmented reality system for soldier ARC4 (U.S. Army 2017)

The first fully immersive system was the Virtual Fixtures platform, which was developed in 1992 by Louis Rosenberg at the Armstrong Laboratories of the United States Air Force.^[166] It enabled human users to control robots in real-world environments that included real physical objects and 3D virtual overlays ("fixtures") that were added enhance human performance of manipulation tasks. Published studies showed that by introducing virtual objects into the real world, significant performance increases could be achieved by human operators.^{[166][167][168]}

An interesting early application of AR occurred when Rockwell International created video map overlays of satellite and orbital debris tracks to aid in space observations at Air Force Maui Optical System. In their 1993 paper "Debris Correlation Using the Rockwell WorldView System" the authors describe the use of map overlays applied to video from space surveillance telescopes. The map overlays indicated the trajectories of various objects in geographic coordinates. This allowed telescope operators to identify satellites, and also to identify and catalog potentially dangerous space debris.^[41]

Starting in 2003 the US Army integrated the SmartCam3D augmented reality system into the Shadow Unmanned Aerial System to aid sensor operators using telescopic cameras to locate people or points of interest. The system combined fixed geographic information including street names, points of interest, airports, and railroads with live video from the camera system. The system offered a "picture in picture" mode that allows it to show a synthetic view of the area surrounding the camera's field of view. This helps solve a problem in which the field of view is so narrow that it excludes important context, as if "looking through a soda straw". The system displays real-time friend/foe/neutral location markers blended with live video, providing the operator with improved situational awareness.

Combat reality can be simulated and represented using complex, layered data and visual aides, most of which are head-mounted displays (HMD), which encompass any display technology that can be worn on the user's head.^[169] Military training solutions are often built on commercial off-the-shelf (COTS) technologies, such as Improbable's synthetic environment platform, Virtual Battlespace 3 and VirTra, with the latter two platforms used by the United States Army. As of 2018^[update], VirTra is being used by both civilian and military law enforcement to train personnel in a variety of scenarios, including active shooter, domestic violence, and military traffic stops.^[170][171] 

In 2017, the U.S. Army was developing the Synthetic Training Environment (STE), a collection of technologies for training purposes that was expected to include mixed reality. As of 2018^[update], STE was still in development without a projected completion date. Some recorded goals of STE included enhancing realism and increasing simulation training capabilities and STE availability to other systems.^[172]

It was claimed that mixed-reality environments like STE could reduce training costs,^[173][174] such as reducing the amount of ammunition expended during training.^[175] In 2018, it was reported that STE would include representation of any part of the world's terrain for training purposes.^[176] STE would offer a variety of training opportunities for squad brigade and combat teams, including Stryker, armory, and infantry teams.^[177]

Researchers at USAF Research Lab (Calhoun, Draper et al.) found an approximately two-fold increase in the speed at which UAV sensor operators found points of interest using this technology.^[178] This ability to maintain geographic awareness quantitatively enhances mission efficiency. The system is in use on the US Army RQ-7 Shadow and the MQ-1C Gray Eagle Unmanned Aerial Systems.

In combat, AR can serve as a networked communication system that renders useful battlefield data onto a soldier's goggles in real time. From the soldier's viewpoint, people and various objects can be marked with special indicators to warn of potential dangers. Virtual maps and 360° view camera imaging can also be rendered to aid a soldier's navigation and battlefield perspective, and this can be transmitted to military leaders at a remote command center.^[179] The combination of 360° view cameras visualization and AR can be used on board combat vehicles and tanks as circular review system.

AR can be an effective tool for virtually mapping out the 3D topologies of munition storages in the terrain, with the choice of the munitions combination in stacks and distances between them with a visualization of risk areas.^{[180][unreliable source?]} The scope of AR applications also includes visualization of data from embedded munitions monitoring sensors.^[180]

Navigation

See also: Automotive navigation system

LandForm video map overlay marking runways, road, and buildings during 1999 helicopter flight test

The NASA X-38 was flown using a hybrid synthetic vision system that overlaid map data on video to provide enhanced navigation for the spacecraft during flight tests from 1998 to 2002. It used the LandForm software which was useful for times of limited visibility, including an instance when the video camera window frosted over leaving astronauts to rely on the map overlays.^[48] The LandForm software was also test flown at the Army Yuma Proving Ground in 1999. In the photo at right one can see the map markers indicating runways, air traffic control tower, taxiways, and hangars overlaid on the video.^[49]

AR can augment the effectiveness of navigation devices. Information can be displayed on an automobile's windshield indicating destination directions and meter, weather, terrain, road conditions and traffic information as well as alerts to potential hazards in their path.^{[181][182][183]} Since 2012, a Swiss-based company WayRay has been developing holographic AR navigation systems that use holographic optical elements for projecting all route-related information including directions, important notifications, and points of interest right into the drivers' line of sight and far ahead of the vehicle.^[184][185] Aboard maritime vessels, AR can allow bridge watch-standers to continuously monitor important information such as a ship's heading and speed while moving throughout the bridge or performing other tasks.^[186]

Workplace

In a research project, AR was used to facilitate collaboration among distributed team members via conferences with local and virtual participants. AR tasks included brainstorming and discussion meetings utilizing common visualization via touch screen tables, interactive digital whiteboards, shared design spaces and distributed control rooms.^{[187][188][189]}

In industrial environments, augmented reality is proving to have a substantial impact with use cases emerging across all aspect of the product lifecycle, starting from product design and new product introduction (NPI) to manufacturing to service and maintenance, to material handling and distribution. For example, labels were displayed on parts of a system to clarify operating instructions for a mechanic performing maintenance on a system.^[190][191] Assembly lines benefited from the usage of AR. In addition to Boeing, BMW and Volkswagen were known for incorporating this technology into assembly lines for monitoring process improvements.^{[192][193][194]} Big machines are difficult to maintain because of their multiple layers or structures. AR permits people to look through the machine as if with an x-ray, pointing them to the problem right away.^[195]

As AR technology has progressed, the impact of AR in enterprise has grown. In the Harvard Business Review, Magid Abraham and Marco Annunziata discussed how AR devices are now being used to "boost workers' productivity on an array of tasks the first time they're used, even without prior training".^[196] They contend that "these technologies increase productivity by making workers more skilled and efficient, and thus have the potential to yield both more economic growth and better jobs".^[196]

Machine maintenance can also be executed with the help of mixed reality. Larger companies with multiple manufacturing locations and a lot of machinery can use mixed reality to educate and instruct their employees. The machines need regular checkups and have to be adjusted every now and then. These adjustments are mostly done by humans, so employees need to be informed about needed adjustments. By using mixed reality, employees from multiple locations can wear headsets and receive live instructions about the changes. Instructors can operate the representation that every employee sees, and can glide through the production area, zooming in to technical details and explaining every change needed. Employees completing a five-minute training session with such a mixed-reality program have been shown to attain the same learning results as reading a 50-page training manual.^[197] An extension to this environment is the incorporation of live data from operating machinery into the virtual collaborative space and then associated with three dimensional virtual models of the equipment. This enables training and execution of maintenance, operational and safety work processes, which would otherwise be difficult in a live setting, while making use of expertise, no matter their physical location.^[198]

Product content management

Product content management before the advent of augmented reality consisted largely of brochures and little customer-product engagement outside of this 2-dimensional realm.^[199] With augmented reality technology improvements, new forms of interactive product content management has emerged. Most notably, 3-dimensional digital renderings of normally 2-dimensional products have increased reachability and effectiveness of consumer-product interaction.^[200]

Augmented reality allows sellers to show the customers how a certain commodity will suit their demands. A seller may demonstrate how a certain product will fit into the homes of the buyer. The buyer with the assistance of the VR can virtually pick the item, spin around and place to their desired points. This improves the buyer's confidence of making a purchase and reduces the number of returns.^[201] Architectural firms can allow customers to virtually visit their desired homes.

Functional mockup

Augmented reality can be used to build mockups that combine physical and digital elements. With the use of simultaneous localization and mapping (SLAM), mockups can interact with the physical world to gain control of more realistic sensory experiences^[202] like object permanence, which would normally be infeasible or extremely difficult to track and analyze without the use of both digital and physical aides.^[203]

Broadcast and live events

Weather visualizations were the first application of augmented reality in television. It has now become common in weather casting to display full motion video of images captured in real-time from multiple cameras and other imaging devices. Coupled with 3D graphics symbols and mapped to a common virtual geospatial model, these animated visualizations constitute the first true application of AR to TV.

AR has become common in sports telecasting. Sports and entertainment venues are provided with see-through and overlay augmentation through tracked camera feeds for enhanced viewing by the audience. Examples include the yellow "first down" line seen in television broadcasts of American football games showing the line the offensive team must cross to receive a first down. AR is also used in association with football and other sporting events to show commercial advertisements overlaid onto the view of the playing area. Sections of rugby fields and cricket pitches also display sponsored images. Swimming telecasts often add a line across the lanes to indicate the position of the current record holder as a race proceeds to allow viewers to compare the current race to the best performance. Other examples include hockey puck tracking and annotations of racing car performance^[204] and snooker ball trajectories.^[205][206]

AR has been used to enhance concert and theater performances. For example, artists allow listeners to augment their listening experience by adding their performance to that of other bands/groups of users.^{[207][208][209]}

Tourism and sightseeing

Travelers may use AR to access real-time informational displays regarding a location, its features, and comments or content provided by previous visitors. Advanced AR applications include simulations of historical events, places, and objects rendered into the landscape.^{[210][211][212]}

AR applications linked to geographic locations present location information by audio, announcing features of interest at a particular site as they become visible to the user.^{[213][214][215]}

Translation

AR applications such as Word Lens can interpret the foreign text on signs and menus and, in a user's augmented view, re-display the text in the user's language. Spoken words of a foreign language can be translated and displayed in a user's view as printed subtitles.^{[216][217][218]}

Music

It has been suggested that augmented reality may be used in new methods of music production, mixing, control and visualization.^{[219][220][221][222]}

Human-in-the-loop operation of robots

Recent advances in mixed-reality technologies have renewed interest in alternative modes of communication for human-robot interaction.^[223] Human operators wearing augmented reality headsets such as HoloLens can interact with (control and monitor) e.g. robots and lifting machines^[224] on site in a digital factory setup. This use case typically requires real-time data communication between a mixed reality interface with the machine / process / system, which could be enabled by incorporating digital twin technology.^[224]

Apps

Snapchat users have access to augmented reality features. In September 2017, Snapchat announced a feature called "Sky Filters" that will be available on its app. This new feature makes use of augmented reality to alter the look of a picture taken of the sky, much like how users can apply the app's filters to other pictures. Users can choose from sky filters such as starry night, stormy clouds, beautiful sunsets, and rainbow.^[225]

Google launched an augmented reality feature for Google Maps on Pixel phones that identifies users' location and places signs and arrows on the device screen to show a user navigation directions.^[226]

MessageAR launched an augmented reality feature that allows users to send personalized video greetings which appear directly in the recipient's surroundings through their smartphone camera. The platform lets users record a short video, generate a shareable link, and have recipients view the message in augmented reality, creating the effect of the sender being present in the same space.^[227]

Concerns

Reality modifications

In a paper titled "Death by Pokémon GO", researchers at Purdue University's Krannert School of Management claim the game caused "a disproportionate increase in vehicular crashes and associated vehicular damage, personal injuries, and fatalities in the vicinity of locations, called PokéStops, where users can play the game while driving."^[228] Using data from one municipality, the paper extrapolates what that might mean nationwide and concluded "the increase in crashes attributable to the introduction of Pokémon GO is 145,632 with an associated increase in the number of injuries of 29,370 and an associated increase in the number of fatalities of 256 over the period of 6 July 2016, through 30 November 2016." The authors extrapolated the cost of those crashes and fatalities at between $2bn and $7.3 billion for the same period. Furthermore, more than one in three surveyed advanced Internet users would like to edit out disturbing elements around them, such as garbage or graffiti.^[229] They would like to even modify their surroundings by erasing street signs, billboard ads, and uninteresting shopping windows. Consumers want to use augmented reality glasses to change their surroundings into something that reflects their own personal opinions. Around two in five want to change the way their surroundings look and even how people appear to them. ^{[citation needed]}

Privacy concerns

Augmented reality devices that use cameras for 3D tracking or video passthrough depend on the ability of the device to record and analyze the environment in real time. Because of this, there are potential legal concerns over privacy.

In late 2024, Meta's collaboration with Ray-Ban on smart glasses faced heightened scrutiny due to significant privacy concerns. A notable incident involved two Harvard students who developed a program named I-XRAY, which utilized the glasses' camera in conjunction with facial recognition software to identify individuals in real-time.^[230]

According to recent studies, users are especially concerned that augmented reality smart glasses might compromise the privacy of others, potentially causing peers to become uncomfortable or less open during interactions.^[231]

While the First Amendment to the United States Constitution allows for such recording in the name of public interest, the constant recording of an AR device makes it difficult to do so without also recording outside of the public domain. Legal complications would be found in areas where a right to a certain amount of privacy is expected or where copyrighted media are displayed.

In terms of individual privacy, there exists the ease of access to information that one should not readily possess about a given person. This is accomplished through facial recognition technology. Assuming that AR automatically passes information about persons that the user sees, there could be anything seen from social media, criminal record, and marital status.^[232]

Notable researchers

• Ronald Azuma is a scientist and author of works on AR.
• Jeri Ellsworth headed a research effort for Valve on augmented reality (AR), later taking that research to her own start-up CastAR. The company, founded in 2013, eventually shuttered. Later, she created another start-up based on the same technology called Tilt Five; another AR start-up formed by her with the purpose of creating a device for digital board games.^[233]
• Steve Mann formulated an earlier concept of mediated reality in the 1970s and 1980s, using cameras, processors, and display systems to modify visual reality to help people see better (dynamic range management), building computerized welding helmets, as well as "augmediated reality" vision systems for use in everyday life. He is also an adviser to Meta.^[234]
• Dieter Schmalstieg and Daniel Wagner developed a marker tracking systems for mobile phones and PDAs in 2009.^[235]
• Ivan Sutherland invented the first VR head-mounted display at Harvard University.

See also

• ARTag – Fiducial marker system
• Augmented reality-based testing
• WebAR – Web technology
• Automotive head-up display – Advanced driver assistance system
• Bionic contact lens – Proposed device to display information
• Computer-mediated reality – Ability to manipulate one's perception of reality through the use of a computer
• Cyborg – Being with both organic and biomechatronic body parts
• Holography – Recording to reproduce a three-dimensional light field
• List of augmented reality software
• Location-based service – General class of computer program-level services that use location data to control systems
• Mixed reality games – Types of electronic games involving reality and VRPages displaying short descriptions of redirect targets
• Multimodal interaction – Form of human-machine interaction using multiple modes of input/output
• Optical head-mounted display – Type of wearable device
• Simulated reality – Concept of a false version of reality
• Visuo-haptic mixed reality
• Wearable computer – Small computing device worn on the body
• Windows Mixed Reality – Mixed reality platform