Most software testing techniques are aimed towards finding errors and defects in software or program which hamper its functioning or cause security breaches.
However, checking a system for errors is not enough; we also need to test the system for its reliability.
However, checking a system for errors is not enough; we also need to test the system for its reliability.
In order to deliver the most reliable software it is important to understand what causes errors and system failures, how frequently do errors interrupt the system, what is the program failure rate, which paths to test while performing software testing and analyzing the failures is very important.
Statistical testing is one such testing which covers all the above requirements. It helps you build and deliver reliable and robust software which work efficiently.
What is Statistical Testing?
Statistical Testing is a testing technique which is used to determine the reliability of the system. In statistical testing, we use statistical models and functions to monitor and check the overall working of the system.
The aim of statistical testing is not to find defects, but to understand how the faults in a program will affect the functioning and operations of the software.
In Statistical testing, statistical models are used to generate test data and test cases which stimulate a working environment. Operations and research models are used to solve any problems detected during the software testing process.
During statistical testing, empirical data is collected which helps testers take efficient decisions for further testing, deployment, maintenance and evolution of the software system. The statistical principles and methods are applied to software testing process with the help of the operational usage model.
What is the need for Statistical Testing?
Software testing is a lengthy process which includes the documentation and use of heavy numerical and statistical values. These values help in decision making for future testing cycles and also are seen as the results of the testing process.
However, software testing is a very complicated process and the complex statistical values are hard to handle and understand. Some issues faced during software testing are:
- There is a high cost involved in achieving 100% path coverage when the defects found are of complex nature and quality.
- The defects found by software testing need to be reworked and integrated into the system during the development life cycle.
- Without proper knowledge about statistical formulas and methods, the tester cannot conduct a bias free software test.
- Statistical functions are required to deduce an accurate solution and make efficient decisions during testing.
For this reason, statistical principles are used to facilitate and guide the testing process and generate better results.
The Statistical Testing process
The Statistical Testing process consists of the following steps:
- Operational usage modeling
- Analysis of the model and Validation
- Test planning
- Testing
Operational Usage Model
During Statistical testing, a model is needed in order to characterize and infer the population of uses of the software system. The model is used to generate a statistically accurate sample of the uses if the software.
The Usage model is one such model which characterizes the population of the possible uses in the intended environment. It does not depend on the program code but is based on the software specifications. Thus, it ensures that frequently occurring failures and bugs get detected in the early stages of software testing.
No extra time is required to develop this model as it can be created parallel to the software. This reduces the time required in completing and delivering the software.
The model is also known to improve the specifications, provides a scientific reasoning and analytical description of the specifications, quantifies testing costs, and provides bases for statistical testing which improves the system reliability.
Thus, statistical testing which is based on this model provides strong scientific foundation for quantifying the software reliability.
Usage Model generates multiple execution paths and all possible use cases for the software. It can be seen as a graph containing nodes and arcs and transition probabilities which are associated with each arc.
During operational usage modeling the following process is followed:
- Building Model Structure
- Assigning Transition Probabilities
- Generating Transition Probabilities
Step 1: You need to identify the hardware, software and human users of the software, the stimuli that these users can send, and the system boundary.
Step 2: Create a structure of the usage model by sequencing the stimuli. You need to identify the areas which can become complex and increase the cost of system development. Provide solutions and alternatives to simplify these complexities.
Step 3: Determine the number of environments to be modeled for the system and also define the environments of use which are important, e.g., malicious use, routine use, maximum capacity use, etc.
Step 4: Create transition probabilities of the usage model.
Model Analysis and Validation
A standard set of statistics exists for the usage models which have significant interpretations for safety analysis, test planning, resource allocation, and field support.
The most commonly calculated statistics include:
- Long-run occupancy of each state
- Occurrence probability
- Occurrence frequency
- First occurrence
- Expected sequence length
During the model validation process the analytical results are studied and you may find that the system functionalities which you deemed as unimportant have significant usage and those which you thought to be important may have little to no use.
Step 1: First you need to develop standard analytical results for the model. The analytical values need to be interpreted in terms of the specifications and expected usage in order to validate their accuracy.
Step 2: if values or model structure do not suit the requirements and specifications, you can change them accordingly.
Step 3: After changing the model and incorporating changes you need to repeat the first and second steps.
Step 4: Now, create test cases and check if they are realistic. If the test cases do not look realistic you need to repeat step 2.
Step 5: Use the model to facilitate activities such as performance planning, test planning, safety analysis, and correctness verification.
Test Planning
There are two types of test cases: Random test cases and Crafted/ Non-random test cases.
Non-random test cases remove uncertainties about the functioning of the system and provide control over the overall testing process whether random or crafted.
These are the non-random test cases which should be performed before random ones:
- Model Coverage Tests
- Mandatory Tests
- Regression Tests
- Critical but unlikely use
- Importance Tests
- Partition Test
Step 1: Generate the number of random test cases by using the expected test case length derived during model analysis.
Step 2: Determine the best case scenario where there are no failures during random testing. Also determine the values of product quality and process sufficiency that can be achieved by running the number of test cases generated in Step 1.
Step 3: Determine the worst case scenario by assuming that failures occur constantly and create a log of the failures.
Step 4: Study and analyze the coverage of the model stats, paths, and arcs.
Testing
After the test cases are planned, you can proceed to the testing part.
Step 1: Record the specifications and constants for each version of the software that is tested for reference.
Step 2: Apply these conditions and sustain them in the environment throughout the test.
Step 3: Execute test cases in the order that they were generated.
Step 4: Communication is the key during statistical testing. You need to communicate and discuss matters that affect test decisions with the testing team regularly.
Step 5: Record the failures and bugs that are encountered.
Step 6: You should maintain the testing chain for the current version as well as the history of testing across all the previous versions. The current testing chain will be used for certification and the historical testing chain will be used to study the development and testing process.
Step 7: If you encounter more than one error in the current version you need to take a decision of whether to discontinue testing or not. This decision depends on various factors like the nature of the failures, organizational policies, and schedules.
Guide your stopping decisions based on reliability, confidence, and convergence.
Step 8: If there are no failures during current testing you can base your stopping decision on the reliability, confidence, and convergence measures together with remaining schedules and budgets.
Advantages of Statistical testing
- It allows testers to focus on testing the parts of the software which have significant usage.
- It allows you to develop more reliable systems.
- The estimation of reliability using statistical testing is more accurate than of other methods.
Disadvantages of Statistical Testing
- Performing and implementing statistical testing properly is not easy.
- Defining operation profiles is complex and there is no way to repeat the process.
- Developing statistically significant test cases in statistical testing is very hard.
Conclusion
Statistical testing is not aimed to find defects and faults; its main aim is to increase the reliability of software.
It helps testers take informed and accurate decisions during the software testing process. This is the reason why statistical testing is important in today’s world if you want to deliver reliable and robust software applications.
Meta: Use of statistics is fathomable. What if they can be used for software testing? Have a look at this unheard testing technique.