Pairwise Testing Guide: How To Perform Pairwise Testing

This blog post will cover what pairwise testing is, how to perform it and the following

Everyone thinks testing is simple, you have to give input to a system and verify the output, if the output is not as expected, you will have to raise an issue. But systems are never this simple, there is a huge variety of input and output values with unknown interdependency between different modules.

For a complex application, the outputs for the majority of the test cases may not be a single parameter but multiple factors like state transitions, input parameters, environmental factors, etc. We may have to test cross-browser, cross-platform, cross devices etc.

With Pairwise testing we can avoid such complexity, it is an effective test case design technique that combines two different parameters to create several permutations and combinations for best coverage.

What is Pairwise Testing?

Pairwise testing is a black-box testing technique in which test cases are designed in a way that we cover every possible combination of input. It relies on the observation that most defects occur by the interaction of two values. So we use input combinations to better test coverage.

Usually, we use techniques like boundary value analysis, equivalence partitioning to find out individual parameters for inputs. In Pairwise testing, we analyze the application to identify the range of value pairs to test that will help us uncover the highest percentage of defects.

It is a subset of Combinatorial Testing (t-way testing). It is also known as All-pair testing.

What is the objective of Pairwise testing?

The primary objective of pairwise testing would be to examine all possible pairs of test values and check whether the application performs as expected.
To generate a subset of input/output values to create an effective test coverage.
To save time and effort in designing test cases that help us in finding bugs.
To create an optimal data set for the automation suite.
To speed up the delivery with high quality.

How to perform All-Pairs testing?

Before getting into how to perform pairwise testing, let’s look at how we take input sets in combination.

Let’s take a system that has 3 different inputs.

System O
Input A, B, C
A= {X,Y}
B={3,5}
C={L,M}

There are 23=8 possible combinations.

We can get rid of T1,T2,T4, and T6

T1= {X,-,L} T2={X,5,L} T3={X,5,-} T4={-,5,L}

You can see how this X, L, 5 is getting repeated, so we are eliminating these Test Cases to bring in unique combinations.

Now let’s look into how to perform pairwise testing, step by step with an example of a bookstore. For context, this bookstore buys and sells books both online and in-store.

Step #1:

Create a list with all the input variables.

Using our example of a bookstore, variables would be

Enquiry
Location
Book Category
Orders Category
Order time
Order type

Step #2:

Now we have to list down all possible values for each variable.

Enquiry- {Valid, Invalid}
Location- {Delhi, Mumbai, Kolkata}
Book Category- {Fiction, Non Fiction}
Orders Category- {Buy, Sell}
Order time- {Working hours, Non-Working hours}
Order type- {Online, Instore}

Let’s say we get 1000 valid enquiries, then it would be 1000 x 3 x 2 x 2 x 2 x 2 = 48000

If we calculate invalid enquiries, the numbers might be higher than this.

It is impossible to test exhaustively in this case.

We’ll try to use smart representation samples. We can use groups and boundaries, even non-discrete data.

First we’ll start by reducing enquiry to two.

Enquiry: 1. Valid, 2. Invalid, then it would be 2 x 3 x 2 x 2 x 2 x 2 = 96

The number is still high.

Step #3:

Enter the input variables as columns in a table, each row would be a test case. Similar to the example given above, we enter variables and values in combination.

Step #4:

Now arrange the variable to create a test suite

We can start by adding Location which is the variable with a higher value repeating it twice, followed by the next high-value Book Category.

Book Category has 2 values, that’s how many times we have to insert the values.

Repeat the same process for the 3rd column, enter the values of the order category.

There’s a mistake in the above table, did you notice that Fiction books are in the order category of Buy and Non-fiction in Sell. We can swap the second set in the third column.

The same issue will happen when the combination creates repetitive sets. Here it is Fiction- Online and Nonfiction- Instore, we can switch the last rows.

We’ll repeat the same process for the following columns as well.

But here in Enquiry, we get an interesting set, we get the combination of Fiction-Valid and Non-Fiction Invalid, even if we switch it like the last ones, we might end up with another combo X- Valid and Y- Invalid

We can add two more test cases to create more possibilities. We are using ~ symbols to them as they are arbitrary.

We have got ourselves 8 unique combinations for a sample that has 96 combinations. This is how the Pairwise testing technique reduces the number of test cases without compromising the test coverage.

Real time example of Pair wise testing?

With plenty of tools in the market that companies use to implement pairwise testing in their project. The best part about Pairwise testing is the versatility, it can be used across domains, any scale, no matter the size of the company or the project.

We can take a real-time example of game testing where combinatorial testing is applied. Using pairwise techniques we can design cases for better coverage.

Let’s take an example of a popular game classic Super Mario, we have so many parameters to work with. We can start with Character Mario – Luigi, we have a positive scenario of jumping over the turtles and a negative scenario of falling by hitting the turtle.

Characters -{ Mario, Luigi} Action- { Jump, Fall}

Character	Action
Mario	Jump
Mario	Fall
Luigi	Jump
Luigi	Fall

With more input variables, we will have more scenarios to verify. Let’s add music to the input to check whether there is any issue arising with this additional input.You can see how the input set is getting constructed now.

When to use Pairwise testing?

According to a paper published on Combinatorial testing, experimental data shows that nearly 60-95% of issues arise due to interaction between two parameters. Through Pairwise testing, if we test 2-way combinations, we can detect a high percentage of error.

We can use Pairwise testing

When there are many parameters like different input variables, several configurations.
It can be used when we can’t implement exhaustive testing due to certain circumstances and the application is in a critical state.
If the application is feasible for automation.

What are the benefits of Pairwise testing?

Pairwise testing can reduce the testing time as we are working with a combination of inputs rather than a single one.
It drastically reduces the test cases count without compromising on the test coverage.
It can be used in any type of project regardless of the domain, i.e it is applicable for various fields like agriculture, healthcare, technology etc.
It helps us find bugs, which we otherwise would have missed.
It focuses on maximizing the impact and minimizing the test case design efforts.
There are several open-source and paid software tools that help in generating value set pairs, you can do it free of cost without manual intervention.

What are the challenges faced in Pairwise testing?

It can be hard to pass unknown dependencies of variables within the software.

You may set unrealistic value combinations, even ones that the user might never come across.
It can be hard to identify issues that trigger an error message into a log or other internal tracker, if it doesn’t reflect in the UI you might miss it.
It doesn’t find bugs in the back-end of the software.
If the tester does not understand the interaction between different modules, they might end up with test cases that are not effective.
Your efforts can be completely useless if you don’t find the right set of variables.

Pairwise Testing Tools

Following are some of the best All-pairs testing tools in the market.

Conclusion

In today’s complex system of multiple devices, browsers, Operating systems and so many other variables, it may take years to test an application with a single input parameter. Pairwise can help us create better test cases with optimum effort without exhausting the system.

Related posts:

Pairwise Testing Guide: How To Perform Pairwise Testing

What is Pairwise Testing?

What is the objective of Pairwise testing?