AR/VR Testing Tutorial - How To Perform AR/VR Testing

Remember those days when we used to watch 3D movies and think how surreal those visuals would look. With the development in technology, we have arrived at a place where we can simulate reality using Augmented reality and Virtual Reality.

We already have them in movies, theme parks, gaming and the entertainment industry. Nowadays there is a growing demand for AR and VR in several domains and industries. Businesses are starting to discover how VR/AR can help create a more attractive, immersive product for their end-users.

As the technology and tools advance in AR/VR technology, ordinary ways of testing no longer work. In this article, we can go through how to overcome these challenges and test VR/AR software applications.

Before getting into the QA process, let’s look into what we mean by VR/AR.

What is Augmented Reality?

Augmented Reality is nothing but an enhanced version of our actual physical world, this is achieved with the help of technology. Digitally various factors such as visual elements, sound or other sensory stimuli develop this augmented reality adding elements to the existing world.

We can experience AR technology through devices such as AR glasses, smart lenses and heads-up displays, even on smartphones and tablets. Augmented Reality is used in entertainment, aviation, retail, automotive, healthcare, travel and other industries. The best example of Augmented Reality in the game Pokemon Go.

What is Virtual Reality?

Virtual Reality uses technology to create an immersive experience in an artificial environment that is created and presented to the user so that the user believes it is a real environment. We experience virtual reality on a computer through two of the five senses: sight and sound. Here we wear a head mount display or a VR headset which enables us to move among the virtual objects on the screen.

VR technologies can be accessed through smartphone connections such as Google Cardboard and Samsung Gear VR, there is another standalone VR headset like the Oculus Go. Virtual reality is used in gaming, entertainment, military, healthcare, business, education and engineering, construction and many other industries.

What is AR/ VR testing?

AR/VR testing comprises full-stack functionality, compatibility, reliability, usability, security, maintainability, performance and a new factor “Immersiveness”. As AR/VR blends the physical and virtual worlds in new ways, the experience must be immersive for the end-user. So here the goal is to prevent even a minute of bugs from disrupting the user’s experience.

As AR/VR is completely immersive, there is a possibility that the user might experience headaches, motion sickness, seizures, eye strain, and other symptoms. Our job as a QA here would be to identify such areas during the testing process. The goal of the development team would be to create a fully immersive experience, and the testing team’s goal would be to minimise the discomfort as possible and optimise the experience.

Challenges in Testing AR/VR

AR/VR testing has several challenges and many misconceptions about it.

A common misconception about AR/VR testing is that a QA team is not required, the development team can test itself. But only a Testing team can provide new perspectives and analyse various aspects of the project that may not have been considered by the internal team.
In AR/VR testing are user expectations and expertise, if there’s any flaw in the system, the user can easily point it out with their experience. For example for a flight simulator, if we have missed out on the tiniest details, an experienced pilot can point out the flaw.
AR/VR applications make use of innovative software, hardware and firmware which are not used by ordinary applications and are not part of the traditional user experience. Our test plan should cover several factors such as battery life, power consumption, firmware, Bluetooth, Wi-Fi, overheating, security and more.
AR/VR application is a complete black-box testing, the input actions and the output results might be unclear until all the other factors are built around the experience, and it would be costly to find and fix a bug after all that effort.
AR/VR testing QA engineers might experience motion sickness, vision impairment and headaches. The recommended time frame for testing is 30 minutes and must be supervised during testing to prevent injuries. With these risks in mind expecting the QA team can never keep up with the backlog with such a short testing span but we can’t risk our team’s health because of the deadline.
The AR/VR applications cannot be tested in the initial stages. QA team has to wait until the product is well underway in development. So we can discover these bugs only late in the development cycle, so without early testing, the development team won’t have enough time to fix the bugs and the testers can’t validate those fixes.
In AR/VR Testing we have to test on both the device and desktop environment. Also, review the user’s experience through tester conversation and body language during testing. Not only that, we have to gather physiological data about the tester, as captured by the wearable device or test supervisor observation.
We have to rigorously test out a wide array of devices as each device has different camera capabilities for AR and will interact differently.

How to test AR/VR Applications?

Let’s look at the step-by-step process to learn the testing in augmented and virtual reality applications:

Step 1: Understand the requirements

Here we have to collect all the data that is necessary for us to understand the functionality of the application. The goal and purpose behind the product are explored here. Then it would be converted into user stories or epics for further classification.

Then we can create a storyboard to extract use cases, it would help our QA engineers to understand all the potential scenarios for user engagement. This exercise provides a holistic view of the AR/VR application- far more thorough than a simple review of what framework can deliver.

This is also the best phase to understand various goals of the application, even a small bug can disrupt the user experience. Thus goals for the QA team would be to rigorously test and build strong test suites and leave no virtual stones unturned.

Step 2: Test Initialization

With a strong test suite with the right set of test cases, we can initiate our testing process. Analysing various conditions the software may use by the consumer can provide us with the right priority.

The device coverage, identifying the right set of devices and other products which would support the process should be included in the test plan. Understand the test environment and figure out how to set up and run the tests.

Step 3:Scope of Testing

Here we track the object in real-time and check whether the 2D/3D objects are registered by the application, thus coming under our testing scope. With multiple inputs check the performance of the application and note down the test data. Check if the AR/VR model redirects to any specific data.

Check the application’s GUI to understand and solve any errors within the product. With all this referred and documented we would have a clear picture of the scope and expectations.

Step 4: Decide the kind of Testing to be performed

When testing AR/VR applications we have to include multiple types of testing for better coverage. This list includes functional testing, usability testing, immersive testing, accessibility testing, hardware testing, configuration testing, compatibility testing, security testing, performance testing, low battery testing and much more.

Here we have to tailor our test strategy based on the project, if it is an airline domain-based VR/AR software, we have to do compliance testing to check whether the application follows all the regulations and standards set by the industry.

Step 5: Work collaboratively with Developers

As a QA team, we would have the great opportunity to collaborate with the developers who use cutting-edge technologies. It can enable us to learn new technologies and skills for the project.

We can align ourselves with the developers to understand the new technology and work partially to make the application better.

Step 6: Use your Creative side

Even though AR/VR technology has been here for a while there’s no definitive way to test these applications as AR/VR is an evolving technology and we can’t pin it down to a particular functionality.

Our best bet would be on creative solutions for the problem, we can start from the basic trial and error method. We could do pair testing with a senior tester, domain expert or any business expert.

Types of testing in AR/VR testing

In AR/VR testing we can perform various types of testing for better coverage, here is the list of testing that will execute in these applications

#1. Functional Testing

When performing functional testing in an AR/VR application we would do some exploratory testing. Then design, and implement test cases as per the practicality or business necessities, once the application under test is fully integrated.

Detailed Guide: Functional Testing

#2. Usability testing

During usability testing, we check whether the use case is aligning with the preferences and wishes of your target market. When we run the test collectively it helps us to note down viable feedback on usability and this will also provide us with insight regarding the manner of interaction of the users.

Detailed Guide: Usability Testing

#3. Immersive Testing

Here we check whether the application provides successful simulation i.e provides the user with the best user experience. We also verify the various hardware to operate and make sure that the device works well and remains compatible.

#4. Hardware Testing

AR/VR testing requires certain hardware to support its features, so testing this various hardware along with the application can help us align ourselves with the expectations in the requirements.

#5. Accessibility Testing

Accessibility testing in AR/VR applications ensures that the user can use the application regardless of age and physical condition, cut back discomfort and guarantee fast and natural navigation.

Detailed Guide: Accessibility Testing

#6. Security Testing

In AR/VR applications there are several use cases, even the military uses these simulators to train soldiers, if such information gets into the wrong hands it would breach national security. So conducting Security testing is important to help to identify risks of data loss or any privacy issues.

Detailed Guide: Security Testing

#7. Compatibility Testing

Compatibility testing methods help us to cover low to high-end configuration setup, this ensures that we accurately assess the performance of the application. This can provide us with the confidence that our product performs as expected on targeted devices and platforms.

Detailed Guide: Compatibility Testing

Test strategy for testing AR/VR application

Let’s look further into the testing approach for AR/VR applications:

#1. Tool Selection

AR/VR applications don’t have any standard tools in the industry like Selenium or Appium that are used to automate these applications. So we have to rely on native testing tools and frameworks such as XCTest, and XCUITest in iOS and for Android tools such as Espresso and Android studio can help.

#2. Testing devices

We have to collect the right set of real devices and simulators to perform the right testing.

#3. Environment setup

We have to be mindful of both the environment both the real physical environment and the test environment as well. Usually, these tests happen in various physical spaces to check various functions.

#4. Battery consumption

We have to perform low battery testing and check how the application and devices react to such situations.

#5. Expert Advice

We should have domain experts in our team to provide the right set of feedback to make improvements to the application. Let’s say we are designing dinosaur-based VR applications, having a few palaeontology experts on the team can only add benefits and improve the accuracy of the experience.

#6. Record sessions

When we record our sessions it provides us more perspective on debugging, it gives us an outside look at others and also helps us get a different look at issues. Most times issues won’t be identified directly that’s why recordings of sessions can be beneficial for us.

Example in AR/ VR use cases

#1. Augmented reality Real-time scenario

Nike – Nike enables its customers to find out their right shoe size, this process doesn’t take more than a few moments.

Sephora – Sephora is a beauty brand that created a virtual environment for its users to try shades, lipstick, and other types of makeup.

#2. Virtual Reality real-time scenario

High-risk jobs such as firefighters, police, soldiers, and pilots can provide training for their employees who will benefit from such VR training to practice routine emergency procedures to learn the dos and don’ts in a consequence-free virtual environment.

Matterport– It is a 3D platform used in the real estate industry for walkthroughs of any real-world place to help agents better interact with their clients and market real estate.

Sample Test Scenarios for AR/ VR testing

Testing the application in various devices including Oculus Quest, Oculus Go, HTC VIVE, Valve Index, Oculus Rift, Oculus Quest +, Link, and various other devices for checking the usability and intuitiveness of the application.
Verifying the on-field viewing position functionality
Check the accuracy of the data including advertising, rosters, schedules, and matchup stats.
Checking whether the images are optimal for the sellers to provide in the site placement.
Check how the virtual object reflects light
Check the voice-over functionality.

Conclusion

Augmented and Virtual Reality is rapidly growing technologies. To validate these applications successfully we should know what to test and how to test the application.

AR/VR applications can make it necessary for unique requirements for tool selection, test devices, test scope, test automation, and test scenarios. But we can address these with proper thought and planning.

Testing AR/VR applications should be a collaborative effort between various teams, from project managers, software engineers, and UX designers to brainstorm and understand the application, features it offers, outcomes we expect and limitations to create and achieve a successful testing roadmap.

Related posts: