AR – VR – MR – XR knowledge base
Frequently asked questions about Augmented Reality and other matters within the field of spatial computing.
Please note, this page is certainly not intended to be complete. Augmented Reality is a relatively young technology with countless possibilities.
It would be impossible to give a complete answer to all questions. If you have questions about AR that you cannot find online, you can call us or send us an email. This is also possible if you have any suggestions for us regarding the use of AR.
…this is what ArtiShock does with AR
Frequently Asked Questions on Augmented Reality
Augmented Reality literally means “added reality”. Augmented Reality is a method of adding digital information to the real world. This information can be anything such as text, graphic visualisations, 3D material, video and audio.
Virtual Reality (VR) simulates an environment via a computer to immerse a user in an experience through various senses.
Most VR environments are visual and auditory. This is done through special stereoscopic glasses and sound. Odor and touch effects are increasingly being added. The term VR was probably coined by internet pioneer Jaron Lanier, founder of VPL Research, in 1989. The company built early VR systems in 1980. The term is often mentioned in the same breath with cyberspace and internet. It is also the first VR company to go bankrupt;)
AR is related to Virtual Reality, in a way.
The big difference, however, is that with Virtual Reality the experience of the real world is replaced by the experience of a digitally generated environment.
Augmented Reality is an extra layer of information that is added to the perception of the real world. Virtual Reality focuses more on so-called “immersive” experiences. Augmented Reality focuses more on spatial computing and spatial communication.
Mixed Reality changes our perception of reality by adding or removing information in real time through a device such as a headset or smartphone. In other words, Augmented Reality plus diminished reality equals mediated reality. This form of reality offers an enhanced visual experience, rather than just using AR or VR.
This distinction is relatively thin. In the strictest sense, Augmented Reality displays information on top of a real view. While Mixed Reality allows you to let the content interact with the real environment. For example, a ball that bounces on the ground.
Extended Reality is purely considering digital information generated by a machine which is visualized in AR or VR. Nowadays it is more common to use the term ‘XR’ as a collection of AR, VR and MR.
Yes, you can get started yourself to create simple Augmented Reality experiences. Anyone who is a bit clever with computers can do that.
Facebook: FaceBook also has an application to make AR with Spark ARstudio (without writing code).
You can also create a channel with the ‘Augment’ app to show AR. Similar options are possible with Snapchat.
These are – in addition to many other solutions – the most easily accessible AR solutions.
Almost all output generated by computers can be used in Augmented Reality. This can be 3D models, graphic material, games, video, text and audio. The possibilities are endless.
It is applied in entertainment, education, marketing & sales, but also in a functional sense in the form of applied AR in industry.
For example, with object recognition you can recognize a machine and add a digital information layer to it, for example to give info to an engineer in order to repair something.
Several forms of Augmented Reality can be distinguished:
- Broadcast AR: With the “mirror” shape, the user looks at a screen in which he/she sees himself/herself, together with the Augmented Reality model. Both the user and the digital model are visible on the same screen.
- Screen AR: With the “lens” model there is a screen on which you can see the AR model. This screen is located between the user and the AR model. In addition to a smartphone screen, this screen can also be, for example, AR glasses (Hololens/Magic Leap), but usually these are AR apps that you can experience on your smartphone.
- Projected AR: The main feature of spatial Augmented Reality is that the object is combined with a physical object. The most famous is Projection Mapping (or 3D Video mapping). There are already many projections on buildings and other objects to come up with.
- Holograms: A holographic projection actually falls under Projected AR. As an example the projection of an artist on a stage (they do that with a ‘gauze cloth’) or a reflection of a glass screen (Hologram display/Holobox).
Augmented Commerce: Buying and selling goods online using augmented reality to virtually visualize products in the real world before purchasing. Augmented commerce is the next evolution of e-commerce. Shoppers can try products in a lifelike way with augmented reality at home before buying.
As an example the IKEA AR app …
Augmented Reality can be experienced on smartphones, tablets, via smart glasses, but also via regular desktop computers. Via an app on a smartphone, users can experience AR based on image recognition, object recognition and geo position. But things like movements, audio, thermography and color recognition can also be triggers for AR.
AR can also be spatially positioned as soon as the device “understands” the environment. For example, spatial tracking (mainly with smart glasses such as the Hololens and Magic Leap) is possible, but ground-plane tracking is also possible.
Augmented Reality technology can ensure that extra information is displayed intuitively and interactively. Augmented Reality can help users gaining a better understanding of reality. In that sense think of prototyping equipment. You can simulate gravity, collisions etcetera.
- Affordability: a solution that is within reach anywhere and anytime for the widest possible target group must be affordable or even free;
- Independence versus collaboration: on the one hand, the user expects autonomous operation of the AR application, on the other, the expectation of collaboration with other applications;
- Tracking: Real-time tracking is the most important requirement of an AR system. Image recognition should run smoothly without glitches;
- Convenience: The system should be easy to use for the average consumer. The software and hardware must be accessible, such as apps and smartphones;
- Innovation: it is important to continuously invest time and energy in devising and developing new solutions or prototypes;
- Structure: users expect practical benefits that work easily and are well designed. The user experience is important.
- AR is used in marketing (advertising);
- Augmented and Virtual Reality films are currently being made in which cinema visitors are virtually in the scenes themselves;
- In magazines and printed matter where an extra layer is placed on the printed information;
- In learning or instructional processes such as assembling or repairing a motor both simulated and in real production environments (with image or object recognition);
- In games where the game takes place in the real environment;
- In automation where stocks are managed with smart glasses by scanning so-called markers;
- In the medical industry to demonstrate the functioning of the human body;
- In home furnishings: by adding 3D furniture to your actual living room or office space to see how it is;
- and so on…
Human Augmentation explores how technology can be used to deliver cognitive and physical enhancements as an integral part of the human experience. Until the year 2030 physical and cognitive human augmentation will increase as people look for personal improvements. This creates a new ‘consumerization’ effect, in which employees take these personal improvements to the workplace.
WebAR offers the possibility to experience Augmented Reality via the web browser. It is AR using a combination of technologies including WebRTC, WebGL and WebVR.
Digital Twins are virtual replicas of the physical environment or devices and objects that data scientists and IT professionals use to run simulations before building and deploying real devices. Digital Twin technology is also developing within Augmented Reality in, for example, the merging worlds of Internet of Things (IOT), artificial intelligence (AI) and data analysis (Big-Data). Replicating real objects is therefore also possible in augmented environments.
Technical terms and AR
ARKit is Apple’s Augmented Reality (AR) development platform for iOS mobile devices.
ARKit helps us as developers to develop beautiful AR experiences for iPad and iPhone. ARkit was introduced with iOS version 11 and has become more advanced ever since. ARKit allows physical spaces to be enriched by a user and viewed by others at a later time. ARkit works on devices from iOS 11 and Core A9 processor.
Stable AR tracking
AR apps that use ARkit usually have more stable AR tracking and more detailed content than the AR apps on the iOS platform that do not. This has to do with the fact that Apple’s ARKit software makes optimal use of the underlying Apple hardware.
From the iPhone X (S), ARKit can also be used to interpret real-time facial expressions and use that data in facial expressions of virtual 3D characters.
Accelerometers and gyroscope
ARKit can use the iOS camera, accelerometers and gyroscope to map out the immediate vicinity of the device as it moves through space. By cleverly combining information from the motion sensors and camera images, a very accurate 3D map of the environment can be made.
With this map, the position of the user in space can always be defined thereby creating a basis for very stable AR experiences. In fact, the map should become more and more detailed and consume more and more memory. But Apple’s ARKit makes smart choices in which map data is most relevant and which data does not require any memory.
ARCore is Google’s platform for building Augmented Reality experiences.
Using various APIs, ARCore gives your Android phone the ability to perceive its environment.
With that skill, your phone can understand the physical world and complement it with interactive digital information.
ARCore uses 3 key skills:
The phone understands its position in relation to the world. This is the basis for a smooth Augmented Reality experience.
The phone understands the dimensions and locations of surfaces in the room, such as the floor, tables and walls. In this way we can create virtual games that play very credibly in your physical space.
The phone understands the exposure of the environment. With this information we can credibly incorporate virtual 3D objects into the environment.
Artishock uses ARCore for the type of AR experience where you want to tell a story in Augmented Reality without a prepared environment (special 2D or 3D marking in space).
Spatial computing (SC) is a collection of ideas and technologies that will change our lives as computers begin to understand the physical world. Our relationship is to communicate with that world and navigate environments. The transformational potential of spatial computing is most evident in Augmented Reality.
On the current smartphones, we use pinch, swipe spread, etc. These are screen gestures. If we define spatial as space and gestures as movements in this space. Then spatial gestures are movements that you can use to interact with objects in the spatial environment.
For example, I can pop a 3D object, such as a digitally generated balloon, with a fingertip, and push it off with an open hand.
I can grab things, push them up and down, enlarge and reduce things by moving 2 hands apart. and so on…
Hand-gestures used with the Microsoft Hololens 1:
Hand gestures with Magic Leap:
Voice control is a very important element in applied Augmented Reality solutions, whether or not in combination with spatial gestures and eye tracking. By sending information with your voice you are able to do your work hands-free. Your voice can therefore interact with the AR content. You can navigate, but also request assistance from an operator who watches your “AR space” with you, for example with video and remote assistance.
Giving orders by using your voice is extremely natural and already fairly accepted, such as Siri, Cortana and “OK Google”.
Tracking is the technique of recognizing images, objects, structures and surfaces. Image recognition (reference images) is used, for example, photos or graphics, these are also referred to as markers.
With 3D objects, we are talking about being able to recognize a machine so that you can link spatial information to it. This is then called object tracking.
Examples of tracking:
SLAM (simultaneous localization and mapping): a series of complex calculations and algorithms to construct a map of an unknown environment and at the same time keep track of the location in that space. SLAM eliminates the need for trackers or markers to anchor a product in space.
Depth perception camera: A camera with depth perception capability that is able to perceive the time it takes for light to return from surrounding objects. You can combine this with video data and thereby map a room. You can digitize the actual environment and then add data to it.
At the moment there are only a limited number of devices available that can perceive a reasonably accurate image of space in this way. Examples are the Hololens and the Magic Leap.
They call the fact of mapping the spatial environment spatial mapping
They call the fact of mapping the direct spatial environment spatial mapping. Spatial mapping is used with AR smartglasses. This is not to be confused with Ground Plane tracking although it is somewhat related.
Ground plane tracking can recognize horizontal surfaces. It is also already able to recognize vertical surfaces. This is usually done with smartphones and smart algorithms in combination with gray values. Apple was the first with ARkit to develop groundplane tracking.
Reduced Reality: Reduces parts of the physical world by removing unwanted objects from view. It is the opposite of Augmented Reality. AR improves our physical reality with digital means, while a Diminished Reality digitally removes physical objects from our view.
Occlusion is a term used in 3D to make rooms digitally visible in a surprising way, for example as a window or a door. You then look inside you can discover everything to the left and right. The entire space is of course made in 3D, but you can only see it through the opening, the rest of the 3D model that you would normally see is then ‘occluded’, meaning hidden.
Source code is a valuable part of any Artishock project. That is why it is important to use a good Source Control Management (SCM) system.
Source control is also known as version control and it is the practice of administering changes in the source code. An SCM shows a history of the development of the source code. In this way it is clear who contributed what, which code is responsible for which feature etc …
But the system also helps when a group of multiple developers has worked together on different parts of the source code and wants to bring it back together into one whole. This process is called “merging” and it can lead to “merging conflicts”. An SCM helps resolve the conflicts.
In the worst case scenario, the SCM can help bring the entire project back to the latest stable version of the source code and manually merge the changes. Within Artishock there is a lot of collaboration on the source code. We can isolate parts of the source code, edit them until a certain update is complete and then put them back and release them.
The source code version system also helps to fix bugs because it is always clear which code has been changed and by whom and shortens the time required to detect programming errors. In this way we streamline the development of the source code and we always give everyone access to the latest source code base.
Agile software development refers to a collection of software development methodologies based on iterative development, in which requirements and solutions evolve through collaborations of self-organizing multi-disciplinary teams.
The agile way of thinking fits in well with ArtiShocks SCRUM project management because it encourages to regularly draw up and adjust an interim position in the project.