Bestoun S. Ahmed

1 Software and Informatics Engineering Department, Engineering College, Salahaddin University, Kurdistan Region, Iraq.

Combinatorial interaction testing has been addressed as an important and effective software testing technique recently. The technique can significantly reduce the number of test cases. It provides an alternative method to exhaustive testing and design of experiments (DOE) by allowing a minimised set of tests to represent the actual set of test cases. The reduction of the test cases is based on the combination of input parameters for the system-under-test. This combination could be considered as input-configuration of different software families. Pairwise combinatorial test suite takes the interaction of two input parameters into consideration instead of many parameter interactions. Evidences showed that this test suite can detect most of the faults in the software-under-test as compared to higher interactions. This paper shows the adaptation and assessment of Dragon Fly (DF), a novel swarm intelligent optimisation algorithm, for pairwise combinatorial test generation. The design of the algorithm is addressed in the paper. The algorithm is evaluated extensively through different experiments and benchmarks. The evaluation shows the efficiency of the proposed technique for test suite generation and the usefulness of DF optimisation algorithm for future investigations.

[1] R. S. Pressman and B. R. Maxim, "Software Engineering: A Practitioner's Approach", 8th edition ed.: McGraw-Hill, (2015).

[2] L. Baresi and M. Pezzè, "An introduction to software testing", Electronic Notes in Theoretical Computer Science, Vol. 148, pp. 89-111. (2006).

[3] B. S. Ahmed, M. A. Sahib, and M. Y. Potrus, "Generating combinatorial test cases using Simplified Swarm Optimization (SSO) algorithm for automated GUI functional testing", Engineering Science and Technology, an International Journal, Vol. 17, pp. 218-226. (2014).

[4] D. Hoskins, R. C. Turban, and C. J. Colbourn, "Experimental designs in software engineering: d-optimal designs and covering arrays", in ACM Workshop on Interdisciplinary Software Engineering Research, Newport Beach, CA, USA, pp. 55 - 66. (2004),

[5] W. Afzal, R. Torkar, and R. Feldt, "A systematic review of search-based testing for non-functional system properties", Information and Software Technology, vol. 51, pp. 957-976. (2009).

[6] T. Mahmoud and B. S. Ahmed, "An efficient strategy for covering array construction with fuzzy logic-based adaptive swarm optimization for software testing use", Expert Systems with Applications, Vol. 42, pp. 8753-8765. (2015).

[7] B. B. Agarwal, S. P. Tayal, and M. Gupta, Software engineering and testing. Hingham; Toronto: Infinity Science Press, (2010).

[8] J. Czerwonka, "Pairwise testing in real world: practical extensions to test case generator," in 24th Pacific Northwest Software Quality Conference, Portland, Oregon, USA, pp. 419-430.( 2006).

[9] J. Czerwonka. Pairwise independent combinatorial testing (PICT) download page. Available: https://msdn.microsoft.com/en-us/library/windows/hardware/hh439673(v=vs.85).aspx (may 2008)

[10]Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, and J. Lawrence, "IPOG-IPOG-D: efficient test generation for multi-way combinatorial testing," Software Testing, Verification & Reliability, Vol. 18, pp. 125-148. (2008).

[11]B. S. Ahmed and K. Z. Zamli, "The Development of a Particle Swarm Based Optimization Strategy for Pairwise Testing," Journal of Artificial Intelligence, Vol. 4, pp. 156-165. (2011).

[12]T. Shiba, T. Tsuchiya, and T. Kikuno, "Using artificial life techniques to generate test cases for combinatorial testing," in 28th Annual International Computer Software and Applications Conference, Hong Kong, Vol. 1, pp. 72-77. (2004).

[13]K. J. Nurmela, "Upper bounds for covering arrays by tabu search," Discrete Applied Mathematics, Vol. 138, pp. 143-152. (2004).

[14]S. Mirjalili, "Dragonfly algorithm: a new meta-heuristic optimization technique for solving single-objective, discrete, and multi-objective problems," Neural Computing and Applications, Vol. 27, pp. 1053-1073. (2016).

[15]M. Harman, "The current state and future of search based software engineering," presented at the 2007 Future of Software Engineering, (2007).

[16]C. J. Colbourn, G. Kéri, P. P. R. Soriano, and J.-C. Schlage-Puchta, "Covering and radius-covering arrays: Constructions and classification," Discrete Applied Mathematics, Vol. 158, pp. 1158-1180. (2010).

[17]A. Hartman and L. Raskin, "Problems and algorithms for covering arrays," Discrete Mathematics, Vol. 284, pp. 149-156. (2004).

[18]L. Gonzalez-Hernandez, "New bounds for mixed covering arrays in t-way testing with uniform strength," Information and Software Technology, Vol. 59, pp. 17-32. (2015).

[19]R. C. Bryce, C. J. Colbourn, and M. B. Cohen, "A framework of greedy methods for constructing interaction test suites," in 27th International Conference on Software Engineering, Oxford, UK, pp. 146-155.(2005).

[20] M. F. J. Klaib, K. Z. Zamli, N. A. M. Isa, M. I. Younis, and R. Abdullah, "G2Way a backtracking strategy for pairwise test data generation," in 5th Asia-Pacific Software Engineering Conference, Beijing, Chaina, pp. 463 - 470.(2008).

[21] A. Calvagna and A. Gargantini, "IPO-s: incremental generation of combinatorial interaction test data based on symmetries of covering arrays," in IEEE International Conference on Software Testing, Verification, and Validation Workshops, Denver, Colorado, USA, pp. 10-18. (2009).

[22] C. Nie and H. Leung, "A survey of combinatorial testing," ACM Computing Surveys, Vol. 43, pp. 1-29. (2011).

[23] C. S. Cheng, "Orthogonal arrays with variable numbers of symbols," The Annals of Statistics, Vol. 8, pp. 447-453. (1980).

[24] D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, "The AETG system: an approach to testing based on combinatorial design," IEEE Transactions on Software Engineering, Vol. 23, pp. 437-444. (1997).

[25] B. Garvin, M. Cohen, and M. Dwyer, "Evaluating improvements to a meta-heuristic search for constrained interaction testing," Empirical Software Engineering, Vol. 16, pp. 61-102, 2011/02/01 2011.

[26] K. C. Tai and Y. Lie, "In-parameter-order: a test generation strategy for pairwise testing," in 3rd IEEE International Symposium on High-Assurance Systems Engineering, Washington, DC , USA, pp. 254-261. (1998).

[27] B. S. Ahmed and K. Z. Zamli, "A greedy Particle Swarm Optimization strategy for t-way software testing," in The Electrical and Electronic Postgraduate Colloquium (EEPC2011), Dusun Eco Resort, Bentong, Pahang, Malaysia, (2011).

[28] A. B. Nasser, F. Hujainah, A. A. Alsewari, and K. Z. Zamli, "Sequence and sequence-less T-way test suite generation strategy based on flower pollination algorithm," in 2015 IEEE Student Conference on Research and Development (SCOReD), pp. 676-680. (2015).

[29] B. S. Ahmed, T. S. Abdulsamad, and M. Y. Potrus, "Achievement of minimized combinatorial test suite for configuration-aware software functional testing using the Cuckoo Search algorithm," Information and Software Technology, Vol. 66, pp. 13-29. (2015).

[30] C. W. Reynolds, "Flocks, herds and schools: A distributed behavioral model," SIGGRAPH Comput. Graph., Vol. 21, pp. 25-34. (1987).

[31] X.-S. Yang, Nature-Inspired Metaheuristic Algorithms: Second Edition: Luniver Press, (2010).

[32] M. Grindal, J. Offutt, and S. F. Andler, "Combination testing strategies: a survey," Software Testing, Verification & Reliability, Vol. 15, pp. 167 – 199. (2005).

[33] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, and J. Lawrence, "IPOG: a general strategy for t-way software testing," in 4th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems, Tucson, Arizona, pp. 549-556.(2007). [34] C. Yilmaz, M. B. Cohen, and A. Porter, "Covering arrays for efficient fault characterization in complex configuration spaces," in 2004 ACM SIGSOFT International Symposium on Software Testing and Analysis, Boston, Massachusetts, USA, pp. 45-54. (2004).

^{1}1 Software and Informatics Engineering Department, Engineering College, Salahaddin University, Kurdistan Region, Iraq.

Original: 8 June 2016, Revised: 30 July 2016, Accepted: 16 October 2016, Published online: 20 March 2017

**DOI: https://doi.org/10.17656/jzs.10586**

**Abstract**

Combinatorial interaction testing has been addressed as an important and effective software testing technique recently. The technique can significantly reduce the number of test cases. It provides an alternative method to exhaustive testing and design of experiments (DOE) by allowing a minimised set of tests to represent the actual set of test cases. The reduction of the test cases is based on the combination of input parameters for the system-under-test. This combination could be considered as input-configuration of different software families. Pairwise combinatorial test suite takes the interaction of two input parameters into consideration instead of many parameter interactions. Evidences showed that this test suite can detect most of the faults in the software-under-test as compared to higher interactions. This paper shows the adaptation and assessment of Dragon Fly (DF), a novel swarm intelligent optimisation algorithm, for pairwise combinatorial test generation. The design of the algorithm is addressed in the paper. The algorithm is evaluated extensively through different experiments and benchmarks. The evaluation shows the efficiency of the proposed technique for test suite generation and the usefulness of DF optimisation algorithm for future investigations.

**Key Words:**Combinatorial interaction testing; Software testing; Test generation tools; Dragonfly optimisation; Search-based software engineering; Test case design techniques

**References**

[1] R. S. Pressman and B. R. Maxim, "Software Engineering: A Practitioner's Approach", 8th edition ed.: McGraw-Hill, (2015).

[2] L. Baresi and M. Pezzè, "An introduction to software testing", Electronic Notes in Theoretical Computer Science, Vol. 148, pp. 89-111. (2006).

[3] B. S. Ahmed, M. A. Sahib, and M. Y. Potrus, "Generating combinatorial test cases using Simplified Swarm Optimization (SSO) algorithm for automated GUI functional testing", Engineering Science and Technology, an International Journal, Vol. 17, pp. 218-226. (2014).

[4] D. Hoskins, R. C. Turban, and C. J. Colbourn, "Experimental designs in software engineering: d-optimal designs and covering arrays", in ACM Workshop on Interdisciplinary Software Engineering Research, Newport Beach, CA, USA, pp. 55 - 66. (2004),

[5] W. Afzal, R. Torkar, and R. Feldt, "A systematic review of search-based testing for non-functional system properties", Information and Software Technology, vol. 51, pp. 957-976. (2009).

[6] T. Mahmoud and B. S. Ahmed, "An efficient strategy for covering array construction with fuzzy logic-based adaptive swarm optimization for software testing use", Expert Systems with Applications, Vol. 42, pp. 8753-8765. (2015).

[7] B. B. Agarwal, S. P. Tayal, and M. Gupta, Software engineering and testing. Hingham; Toronto: Infinity Science Press, (2010).

[8] J. Czerwonka, "Pairwise testing in real world: practical extensions to test case generator," in 24th Pacific Northwest Software Quality Conference, Portland, Oregon, USA, pp. 419-430.( 2006).

[9] J. Czerwonka. Pairwise independent combinatorial testing (PICT) download page. Available: https://msdn.microsoft.com/en-us/library/windows/hardware/hh439673(v=vs.85).aspx (may 2008)

[10]Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, and J. Lawrence, "IPOG-IPOG-D: efficient test generation for multi-way combinatorial testing," Software Testing, Verification & Reliability, Vol. 18, pp. 125-148. (2008).

[11]B. S. Ahmed and K. Z. Zamli, "The Development of a Particle Swarm Based Optimization Strategy for Pairwise Testing," Journal of Artificial Intelligence, Vol. 4, pp. 156-165. (2011).

[12]T. Shiba, T. Tsuchiya, and T. Kikuno, "Using artificial life techniques to generate test cases for combinatorial testing," in 28th Annual International Computer Software and Applications Conference, Hong Kong, Vol. 1, pp. 72-77. (2004).

[13]K. J. Nurmela, "Upper bounds for covering arrays by tabu search," Discrete Applied Mathematics, Vol. 138, pp. 143-152. (2004).

[14]S. Mirjalili, "Dragonfly algorithm: a new meta-heuristic optimization technique for solving single-objective, discrete, and multi-objective problems," Neural Computing and Applications, Vol. 27, pp. 1053-1073. (2016).

[15]M. Harman, "The current state and future of search based software engineering," presented at the 2007 Future of Software Engineering, (2007).

[16]C. J. Colbourn, G. Kéri, P. P. R. Soriano, and J.-C. Schlage-Puchta, "Covering and radius-covering arrays: Constructions and classification," Discrete Applied Mathematics, Vol. 158, pp. 1158-1180. (2010).

[17]A. Hartman and L. Raskin, "Problems and algorithms for covering arrays," Discrete Mathematics, Vol. 284, pp. 149-156. (2004).

[18]L. Gonzalez-Hernandez, "New bounds for mixed covering arrays in t-way testing with uniform strength," Information and Software Technology, Vol. 59, pp. 17-32. (2015).

[19]R. C. Bryce, C. J. Colbourn, and M. B. Cohen, "A framework of greedy methods for constructing interaction test suites," in 27th International Conference on Software Engineering, Oxford, UK, pp. 146-155.(2005).

[20] M. F. J. Klaib, K. Z. Zamli, N. A. M. Isa, M. I. Younis, and R. Abdullah, "G2Way a backtracking strategy for pairwise test data generation," in 5th Asia-Pacific Software Engineering Conference, Beijing, Chaina, pp. 463 - 470.(2008).

[21] A. Calvagna and A. Gargantini, "IPO-s: incremental generation of combinatorial interaction test data based on symmetries of covering arrays," in IEEE International Conference on Software Testing, Verification, and Validation Workshops, Denver, Colorado, USA, pp. 10-18. (2009).

[22] C. Nie and H. Leung, "A survey of combinatorial testing," ACM Computing Surveys, Vol. 43, pp. 1-29. (2011).

[23] C. S. Cheng, "Orthogonal arrays with variable numbers of symbols," The Annals of Statistics, Vol. 8, pp. 447-453. (1980).

[24] D. M. Cohen, S. R. Dalal, M. L. Fredman, and G. C. Patton, "The AETG system: an approach to testing based on combinatorial design," IEEE Transactions on Software Engineering, Vol. 23, pp. 437-444. (1997).

[25] B. Garvin, M. Cohen, and M. Dwyer, "Evaluating improvements to a meta-heuristic search for constrained interaction testing," Empirical Software Engineering, Vol. 16, pp. 61-102, 2011/02/01 2011.

[26] K. C. Tai and Y. Lie, "In-parameter-order: a test generation strategy for pairwise testing," in 3rd IEEE International Symposium on High-Assurance Systems Engineering, Washington, DC , USA, pp. 254-261. (1998).

[27] B. S. Ahmed and K. Z. Zamli, "A greedy Particle Swarm Optimization strategy for t-way software testing," in The Electrical and Electronic Postgraduate Colloquium (EEPC2011), Dusun Eco Resort, Bentong, Pahang, Malaysia, (2011).

[28] A. B. Nasser, F. Hujainah, A. A. Alsewari, and K. Z. Zamli, "Sequence and sequence-less T-way test suite generation strategy based on flower pollination algorithm," in 2015 IEEE Student Conference on Research and Development (SCOReD), pp. 676-680. (2015).

[29] B. S. Ahmed, T. S. Abdulsamad, and M. Y. Potrus, "Achievement of minimized combinatorial test suite for configuration-aware software functional testing using the Cuckoo Search algorithm," Information and Software Technology, Vol. 66, pp. 13-29. (2015).

[30] C. W. Reynolds, "Flocks, herds and schools: A distributed behavioral model," SIGGRAPH Comput. Graph., Vol. 21, pp. 25-34. (1987).

[31] X.-S. Yang, Nature-Inspired Metaheuristic Algorithms: Second Edition: Luniver Press, (2010).

[32] M. Grindal, J. Offutt, and S. F. Andler, "Combination testing strategies: a survey," Software Testing, Verification & Reliability, Vol. 15, pp. 167 – 199. (2005).

[33] Y. Lei, R. Kacker, D. R. Kuhn, V. Okun, and J. Lawrence, "IPOG: a general strategy for t-way software testing," in 4th Annual IEEE International Conference and Workshops on the Engineering of Computer-Based Systems, Tucson, Arizona, pp. 549-556.(2007). [34] C. Yilmaz, M. B. Cohen, and A. Porter, "Covering arrays for efficient fault characterization in complex configuration spaces," in 2004 ACM SIGSOFT International Symposium on Software Testing and Analysis, Boston, Massachusetts, USA, pp. 45-54. (2004).