What is augmented reality?

AR devices are equipped with cameras, sensors and displays. This can include smartphones and tablets creating mobile AR experiences or ‘wearables’ like smart glasses and headsets. These devices capture the physical world and then integrate digital content (for example, 3D models, images or videos) into the scene, blending digital and virtual worlds.

Augmented reality technologies have a range of applications in commerce, manufacturing and entertainment.

Augmented reality works through the deployment of camera-equipped hardware such as smart glasses or heads-up displays.

Mobile devices like iPads or iPhones, which are already built with technologies such as GPS, accelerometers and sensors, are particularly compatible with augmented reality applications and can make the technology more accessible to the average consumer. In recent years, several technology companies have released APIs such as Apple’s ARKit and Google’s Arcore, which facilitate the development of mobile AR applications for Android and iOS.

Though the types of data and sensors a particular AR software might draw on vary, the basic steps of the AR process are:

An AR device receives a stream of video content from a user’s field of vision, sensing the environment and tracking physical objects in view. This might include the collection of data from accelerometers, gyroscopes, GPS or lasers alongside a video stream to track the user’s position and orientation.

AR software scans and processes this environment—this might mean connecting to an object’s digital twin, a 3-D copy of the object stored in the cloud. It might also mean using artificial intelligence (AI) to recognize the physical object. During this process, AR software processes the information it has received, identifying objects and environmental features that can be augmented. This might involve sensors on the physical object sending data to a digital twin, or combining tracking data with other information such as a product’s price or equipment lifecycle data.

Information streamed from the AR software is displayed on the AR device, superimposing computer-generated content onto the user’s field of vision. Digital information is rendered in the correct perspective and orientation, appearing to the user as if the object is physically present. The user follows interaction instructions, sending commands through a touchscreen, with physical gestures, or by voice. These commands are received by the software and sent to the superimposed digital object so it can be manipulated by the user.

Augmented reality, virtual reality and mixed reality are all technologies that change the user’s perception of the physical world with computer-generated content.

AR overlaps digital content onto the real world. Using augmented reality, users still see and interact with their physical environments while experiencing supplementary digital information overlaid onto their field of vision. Augmented reality can be accessed on a mobile device through augmented reality apps, or with AR glasses. Examples include:

• Mobile mapping apps that provide an augmented reality experience with directional arrow and street name overlays, helping users navigate the real world on foot

• Ikea Place, an app that allows users to virtually place furniture in their homes by using AR

• Social media filters such as Snapchat filters that allow users to alter the images seen through their phone cameras in real-time

Virtual reality immerses users in a digital environment, typically by using a VR headset, head-mounted display or VR goggles. Unlike AR, VR completely replaces the physical world, surrounding users with a 360-degree view of computer-generated environments. Examples include:

• The metaverse, an immersive virtual reality digital world developed by the company Meta

• The VR Museum of Fine Art, which allows users to virtually visit famous museum exhibits from across the world

Mixed reality, which can be considered an umbrella term for some advanced forms of AR, creates experiences where digital content interacts with the real world, blending virtual objects with the physical environment. In mixed reality, a user can interact simultaneously with physical and virtual items. Examples include:

• AR video games or platforms such as Microsoft’s HoloLens that allow users to use physical objects (such as a water bottle or a piece of furniture)

• A prototyping and design application that can project a holographic image of an object in the space where it will be placed

There are two fundamental types of augmented reality, marker-based and marker-less. While the former is less cost-intensive and more accessible, the latter provides a more immersive experience.

Marker-based AR applications overlay digital content onto a physical trigger in a real-world environment. This trigger might be a QR code, an image or another target marker. When a device’s camera detects this marker, it triggers the display of the associated AR experience. As this kind of AR can be accessed at any time from a range of devices, it is the most flexible AR model.

Marker-less AR, by contrast, doesn’t require a specific trigger. This kind of AR relies on device sensors, such as GPS, accelerometers and cameras, to understand and map a user’s environment in real time. By analyzing the user’s physical environment, often by using algorithms and computer vision, these AR systems determine where to place digital content, allowing for a more spontaneous and dynamic experience.

AR has had a significant impact on that way consumers interact with products and brands. By blending digital content with the physical shopping environment, AR technology can enhance engagement and personalize the shopping experience, creating a truly omnichannel customer experience.

Through AR-enabled apps for mobile devices or in-store displays, shoppers can virtually try on clothing, accessories or cosmetics. They can also visualize home decor in their living spaces—the furniture retailer Ikea has had particular success with its AR application. These immersive experiences blend online and offline shopping, allowing consumers to make more confident purchasing decisions.

AR seamlessly integrates with various shopping channels, including brick-and-mortar stores, ecommerce platforms and mobile apps to deliver consistent and unified shopping experiences across multiple touchpoints. For instance, a customer might use an AR-enabled mobile app to try on makeup before visiting a store to make a final purchase.

AR can enhance the user experience by providing interactive virtual environments such as maps, product locaters and contextual information overlaid onto the customer’s real-world view. Retailers might use AI to guide customers to specific products or promotions within the store or display offers based on a shopper’s location, encouraging exploration within the store.