Configuration Testing – A Detaild Guide

As per current Software Testing Trends, software applications are a huge part of our day-to-day life. We wake up to a phone alarm, check our messages, scroll through social media, and reply to important emails just 20 minutes after waking up. We have more and more users adopting their lifestyle to services provided by technology.

These technologies which are growing each day, usually consist of a combination of hardware and software systems working together. For the customer to use the applications without any issues, we need to ensure that different sets of hardware and software forming the system need are configured together correctly.

We should also be aware that hardware and software that forms various applications can have multiple versions and specifications. So we need to select the combination specification of software and hardware that performs better than any other combination.  We have to perform Configuration, to validate these combinations and figure out the best combo.

Configuration Testing

Configuration Testing deals with checking the performance of the application against different combinations of hardware and software to identify the optimal configuration in which the application can perform at its best.

Here, when we talk about different configurations of hardware and software, it refers to multiple operating system versions, various browsers, various supported drivers, distinct memory sizes, different hard drive types, various types of CPU and much more.

It aims to discover the right set of combinations in software and hardware that suits the specifications under which the application would work flawlessly. But, configuration testing is a time-consuming technique, so we must create a workable plan.

Example of Configuration Testing

Let’s look at an example for Configuration Testing in a desktop application.

Usually, desktop applications are two tires or three tiers, in this example, we can take a three-tiered one. We can take a desktop application which was built using Asp.Net and consists of a Client, Business Logic Server, and Database Server.

This application can run in the following OS:

• Windows XP
• Windows 7
• Windows 8
• Windows Vista
• Windows 10

It can be supported by other server platforms:

• Windows Server 2008 R2
• Windows Server 2008 R2,
• Windows Server 2012R2

It can be supported by the following databases platform:

• SQL Server 2008
• SQL Server 2008R2
• SQL Server 2012

To check whether the application performs properly, we have to try the application in various combinations which performs optimally. Using different combinations of Client, Server, and Database platforms and versions as mentioned above.

Configuration Testing applies to hardware as well, we would use various hardware devices such as printers, scanners, webcams, and other devices which can help us verify the application under test.

What would happen if configuration Testing is not done?

We cannot avoid the costs associated with configuration testing by not implementing it at all. Let’s look at what would happen if configuration testing was not implemented in the project:

• The team’s months of manual efforts to identify the system components would change when the requirements changed.
• The team will fail to implement according to the changes in the requirements and there’s a possibility of miscommunication between stakeholders.
• There would be unexpected outages due to modifying the system, this happens as we couldn’t accurately determine which components were impacted by a change.
• If we didn’t perform any configuration testing in the software, it may produce a bad outcome, all the configured elements may not remain connected to the software, and it’s the responsibility of the software to handle those kinds of scenarios.
• Without configuration testing, there is also a possibility of the system breaking down or completely ceasing to work, as there wouldn’t be any precautionary measures to handle the system in a different configuration.
• Sometimes the application installed might stop working when the operating system in which it is installed has changed its version. So checking platform independence is crucial for the application to function anywhere.
• When software is installed in a distributed system, the software should be designed in such a way that it collaboratively works with other nodes connected to it. Let’s take a scenario in which a node fails, our software installed in other nodes might also fail because the capability of the distributed systems might not have been synchronised with the application.

Objective of configuration Testing

Configuration testing enables our application to interact with hardware. It is more effective than other software testing in fulfilling the application’s configurable requirements. 

• The primary objective of configuration testing is to determine the optimal configuration of the application that is being tested.
• To check whether the application satisfies the configurability requirements.
• To detect defects that may not be found during testing by manually changing configurations making it fail. Let’s say by changing the regional settings of the system like language, time zone, date-time formats, etc.
• To measure the general performance of the application, by adding or changing hardware resources like Load Balancers, increasing or reducing the memory size, connecting various printer models, etc.
• To examine the application’s efficiency by checking how efficiently were the tests performed with the available resources to achieve the optimal system configuration
• To check how effectively the system performs in a geographically distributed environment.
• To analyse the reproducibility of the bug irrespective of the configuration changes
• To ensure the traceability of the application features to the documentation and maintenance which can be easily identified
• To check on how manageable the various features of the application are throughout the SDLC (software development life cycle).

Types of configuration Testing

Configuration Testing focuses on checking the application’s behaviour in various environments, this helps us to determine both long-term and short-term effects when the configuration gets changed.

So we have to install our software in as many configurations as possible, to validate the compatible configuration and highlight the incompatible ones.

We can categorise Configuration Testing into two types:

• Software Configuration Testing
• Hardware Configuration Testing

#1. Software Configuration Testing

Here we test the application under various operating systems and multiple browsers to check its compatibility with various software systems.

It can be time-consuming as we would be validating the application with various software which we have to install and uninstall, resetting the entire environment again.

Usually, software configuration Testing happens after unit testing and integration testing.

#2. Hardware Configuration Testing

Here we deal with various hardware components, usually, this takes place in a lab. Once a new build is released, it is tested in all the machines in the lab with various variations.

This can be a very time-consuming task if the number of hardware combinations has a huge range. To reduce the time and effort, usually, automation is used to run released builds against selected hardware combinations.

Usually, this process is outsourced to other companies as it is very expensive to procure various hardware and set up a lab.

The Configuration Testing types can be further classified into:

• Client Level Testing
• Server Level Testing

#1. Client-Level Testing

Usually, Client level testing is linked to usability and functionality testing. Client testing can only be executed from the user’s perspective or the client’s point of view.

#2. Server-Level Testing

Here in server-level testing, the communication between the software and the external environment is validated to ensure that integration is working fine after the release.

Example:

Let’s look at an example for Configuration Testing, here we can take an application which is connected to a money counting machine. This application gets data from the machine and saves it.

When talking about a money counting machine, it is a piece of hardware which can calculate the amount kept in the machine. These are typically used in banks and billing counters. It usually comes in several models such as Maxsell, Stok, Rolex and more.

So our challenge here is to check whether our application reads the data from the machine and stores it in the database. Here when we look at the application and the machine we have the combination of the Client Platform, Server Platform and Database.

• Here the various permutations for the client platform would be the operating system such as Windows 7, 8 or 10, MAC hardware or macOS

• For server platforms, the permutations would be Windows server platform (server platform 2016), Linux, server platform (eg.RedHat) etc.

• For Database, it would be MySQL, PostgreSQL, MSSQL, Oracle Database, & Microsoft access etc.

Pre-requisites for Configuration Testing

#1. Build a Coverage Matrix

As we have a vast variety of hardware and software combinations, it is practically impossible to test each of these combinations.

Here we have to create a coverage matrix to store various software and hardware set up. The basic configuration matrix is also another name for this coverage matrix.

Let’s look at an example, which has 3 hardware configurations, 3 Software configurations and 3 Server configurations, so now we have 3*3*3 = 27 combinations.

We can include other configurations such as 5 hard drives and 6 memory configurations, now it makes our set to 27*5*6= 810 combinations.

We have the option to include more elements to this set but time, cost and effort for such testing can make it impossible. Hence, we have to plan for the software testing effort and determine which platforms will be supported.

#2. Setup Configuration Priorities

After building up the configuration matrix, our next step here is to prioritise those configurations.

As we have a vast variety of combinations it is not possible to validate all of them, so we must identify the priority based on customer inputs or market research.

So testing out the combination that the client or user is most likely to use gives us a starting point to begin our testing.

How to perform configuration Testing

• Step 1: Based on the priority and the configuration matrix, create a test plan on what to test and how to test.
• Step 2: Once the test plan has the right set of goals, we can start identifying the key scenarios, write test cases for them and build the test suite.
• Step 3: Make sure that the hardware/ software is set up as a variable in those test cases.
  For example, let’s pick browser as the variable here, various configurations here would be Chrome, Firefox, Edge, and more
  Sample test case: Open the application URL in (Browser variable)
• Step 4: Execute the test cases for each setup separately.
• Step 5: Observe the results and document the test findings for each setup.

Sample test cases for Configuration Testing:

We can follow the same example of the money counting machine here to understand some test cases.

• Check whether the connection between the application and the model of the machine is as expected.
• Check whether the application with a model of the machine runs when the prerequisites are installed.
• Check the behaviour of the application with a model of the machine when prerequisites are NOT installed.
• Check whether the machine can detect the number mentioned in the system of application.
• Check whether the application can calculate correctly according to the entry.
• Check whether the machine counts the money and the total is exactly the actual amount of cash.

Compatibility Testing vs Configuration Testing

What is Configuration Management?

Configuration Management is a process which helps us to establish and maintain the software product’s consistently in performance, function and physical attributes with the requirements, design and operational information throughout its life.

Configuration management requires a high degree of automation to reach its goals.y

With automation, it’s easier to build in checks and redundancies. It also improves the potential for omissions because of human error and the accuracy for keeping assets in the expected state.

It is included as a key systems engineering practice because it works. It prevents incurring costs and helps IT avoid risks and battles. Moreover, good systems engineers already understand through practice and experience that efforts for configuration management pay for themselves.

Configuration management applies to various systems, but most often, we have to deal with these:

• Servers
• Databases and other storage systems
• Operating systems
• Networking
• Applications
• Software

Let’s look at some efficient and popular configuration management tools :

• Ansible
• CFEngine
• Chef
• Puppet
• Salt

Advantages of using configuration management tools include:

• It can help us to follow coding conventions that make it easier to navigate code
• It provides a state where every time we run the code, the results stay the same, consistently.
• It has a distribution design to improve managing large numbers of remote servers

What are the Best Practices in Configuration Management?

To maintain the application’s various configurations, we should follow some best practices to optimize the process.

#1. Track changes: Whenever there’s a change in the requirements, we should track it and document the details. Instead of a single file commit, we can use Changeset. Changeset is nothing but a package of commits that can track changes for packaged commits that track changes to a group of files, structure of the directory, reasons for the change, comments on unit test and environment changes in one easy to manage commit.

This process allows managers to see which related files have changed with every significant change to our assets, which makes it easier to roll back unwanted changes or revert to an earlier configuration if required.

#2. Early testing: Always test as early as possible to catch bugs that might harm the system.

#3. Performance Testing: DevOps team gets insights by running performance testing to check how the latest changes have affected the performance and system functionality.

#4. Early and Often Integration:  After testing early and performance testing, we need to integrate early and often as possible. This ensures that the new feature or the changes made work along with the rest of the environment. This can help us post problems early before they even occur in production and prove costly.

Conclusion

To wrap it up, configuration testing carries special significance as it helps us to arrive at the optimal application performance. It makes configuration testing easier even though it involves time and effort, with various software and hardware, even virtual machines.

So Configuration testing is one of the most important testing approaches that shouldn’t be ignored or considered an unnecessary wasted effort. It can help us to identify the conditions under which the application can provide optimal performance with the least flaws or problems.

Related posts:

• Compliance Testing Guide
• Banking Domain Application Testing Guide
• Product Testing Guide
• Best Product Testing Sites
• Best Crowdsourced Testing Companies