During the recent several years, deep learning has achieved excellent results in image recognition, voice processing, and other research areas, which has set off a new upsurge of research and application. Internal defects and external malicious attacks may threaten the safe and reliable operation of a deep learning system and even cause unbearable consequences. The technology of testing deep learning systems is still in its infancy. Traditional software testing technology is not applicable to test deep learning systems. In addition, the characteristics of deep learning such as complex application scenarios, the high dimensionality of input data, and poor interpretability of operation logic bring new challenges to the testing work. This paper focuses on the problem of test case generation and points out that adversarial examples can be used as test cases. Then the paper proposes MTGAN which is a framework to generate test cases for deep learning image classifiers based on Generative Adversarial Network. Finally, this paper evaluates the effectiveness of MTGAN.

MATLAB/Simulink is the de facto standard tool for the model-based development (MBD) of control software for automotive systems. A Simulink model developed in MBD for real automotive systems involves complex computation as well as tens of thousands of blocks. In this paper, we focus on decision coverage (DC), condition coverage (CC) and modified condition/decision coverage (MC/DC) criteria, and propose a Monte-Carlo test suite generation method for large and complex Simulink models. In the method, a candidate test case is generated by assigning random values to the parameters of signal templates with specific waveforms. We try to find contributable candidates in a plausible and understandable search space, specified by a set of templates. We implemented the method as a tool, and our experimental evaluation showed that the tool was able to generate test suites for industrial implementation models with higher coverages and shorter execution times than Simulink Design Verifier. Additionally, the tool includes a fast coverage measurement engine, which demonstrated better performance than Simulink Coverage in our experiments.

The traditional tests are planned and designed at the early stages, but it is possible to execute test cases after implementing source code. Since there is a time difference between design stage and testing stage, by the time a software design error is found it will be too late. To solve this problem, this paper suggests a virtual pre-testing process. While the virtual pre-testing process can find software and testing errors before the developing stage, it can automatically generate and execute test cases with modeling and simulation (M&S) in a virtual environment. The first part of this method is to create test cases with state transition tree based on state diagram, which include state, transition, instruction pair, and all path coverage. The second part is to model and simulate a virtual target, which then pre-test the target with test cases. In other words, these generated test cases are automatically transformed into the event list. This simultaneously executes test cases to the simulated target within a virtual environment. As a result, it is possible to find the design and test error at the early stages of the development cycle and in turn can reduce development time and cost as much as possible.

Regression testing is essential for assuring the quality of a software product. Because rerunning all test cases in regression testing may be impractical under limited resources, test case prioritization is a feasible solution to optimize regression testing by reordering test cases for the current testing version. In this paper, we propose a novel test case prioritization approach that combines the clustering algorithm and the scheduling algorithm for improving the effectiveness of regression testing. By using the clustering algorithm, test cases with same or similar properties are merged into a cluster, and the scheduling algorithm helps allocate an execution priority for each test case by incorporating fault detection rates with the waiting time of test cases in candidate set. We have conducted several experiments on 12 C programs to validate the effectiveness of our proposed approach. Experimental results show that our approach is more effective than some well studied test case prioritization techniques in terms of average percentage of fault detected (APFD) values.

Combinatorial interaction testing has been well studied in recent years, and has been widely applied in practice. It generally aims at generating an effective test suite (an interaction test suite) in order to identify faults that are caused by parameter interactions. Due to some constraints in practical applications (e.g. limited testing resources), for example in combinatorial interaction regression testing, prioritized interaction test suites (called interaction test sequences) are often employed. Consequently, many strategies have been proposed to guide the interaction test suite prioritization. It is, therefore, important to be able to evaluate the different interaction test sequences that have been created by different strategies. A well-known metric is the Average Percentage of Combinatorial Coverage (shortly APCCλ), which assesses the rate of interaction coverage of a strength λ (level of interaction among parameters) covered by a given interaction test sequence S. However, APCCλ has two drawbacks: firstly, it has two requirements (that all test cases in S be executed, and that all possible λ-wise parameter value combinations be covered by S); and secondly, it can only use a single strength λ (rather than multiple strengths) to evaluate the interaction test sequence - which means that it is not a comprehensive evaluation. To overcome the first drawback, we propose an enhanced metric Normalized APCCλ (NAPCC) to replace the APCCλ Additionally, to overcome the second drawback, we propose three new metrics: the Average Percentage of Strengths Satisfied (APSS); the Average Percentage of Weighted Multiple Interaction Coverage (APWMIC); and the Normalized APWMIC (NAPWMIC). These metrics comprehensively assess a given interaction test sequence by considering different interaction coverage at different strengths. Empirical studies show that the proposed metrics can be used to distinguish different interaction test sequences, and hence can be used to compare different test prioritization strategies.

This paper proposes the test case prioritization in regression testing. The large size of a test suite to be executed in regression testing often causes large amount of testing cost. It is important to reduce the size of test cases according to prioritized test sequence. In this paper, we apply the Markov chain Monte Carlo random testing (MCMC-RT) scheme, which is a promising approach to effectively generate test cases in the framework of random testing. To apply MCMC-RT to the test case prioritization, we consider the coverage-based distance and develop the algorithm of the MCMC-RT test case prioritization using the coverage-based distance. Furthermore, the MCMC-RT test case prioritization technique is consistently comparable to coverage-based adaptive random testing (ART) prioritization techniques and involves much less time cost.

In most software development environments, time, computing and human resources needed to perform the testing of a component is strictly limited. In order to deal with such situations, this paper proposes a method of creating the best possible test suite (covering the maximum number of 3-tuples) within a fixed number of test cases.

This letter proposes a reuse method of unit test cases, which characterize internal behaviors of a called function, for enhancing capability of automatic generation of test cases. Existing test case generation tools have limits in finding solutions to the deep call structure of the source code. In our approach, the complex call structure is simplified by reusing unit test cases of called functions. As unit test cases represent the characteristics of the called function, the internal behaviors of called functions are replaced by the test cases. This approach can be applicable to existing test tools for simplifying the process of generation and enhancing their capabilities.

We try to use a computer algebra system Mathematica as a test case generation system. In test case generation, we generally need to solve equations and inequalities. The main reason why we take Mathematica is because it has a built-in function to solve equations and inequalities. In this paper, we deal with both black-box testing and white-box testing. First, we show two black-box test case generation procedures described in Mathematica. The first one is based on equivalence partitioning. Mathematica explicitly shows a case that test cases do no exist. This is an advantage in using Mathematica. The second procedure is a modification of the first one adopting boundary value analysis. For implementation of boundary value analysis, we give a formalization for it. Next, we show a white-box test case generation procedure. For this purpose, we also give a model for source programs. It is like a control flow graph model. The proposed procedure analyzes a model description of a program.

Current Web services testing techniques are unable to assure the desired level of trustworthiness, which presents a barrier to WS applications in mission and business critical environments. This paper presents a framework that assures the trustworthiness of Web services. New assurance techniques are developed within the framework, including specification verification via completeness and consistency checking, test case generation, and automated Web services testing. Traditional test case generation methods only generate positive test cases that verify the functionality of software. The proposed Swiss Cheese test case generation method is designed to generate both positive and negative test cases that also reveal the vulnerability of Web services. This integrated development process is implemented in a case study. The experimental evaluation demonstrates the effectiveness of this approach. It also reveals that the Swiss Cheese negative testing detects even more faults than positive testing and thus significantly reduces the vulnerability of Web services.

This paper proposes a test case generation method for testing concurrent programs as a black box. Typical applications are system testing for switching systems and inter-operability testing for OSI products. We adopt a two-step approach: first generate the control flow graph which represents global behaviors of a given concurrent program, and then apply conventional test case generation methods for the control flow graph. To generate a control flow graph without state space explosion, the black-box equivalence between system behaviors is introduced. The proposed algorithm generates a minimal control flow graph which consists of representatives of equivalence classes. Two practical techniques for the second step are discussed for a case study using a commercial digital PBX. The results show the feasibility of the proposed method.