There is a lack of serious tooling for mutation analysis for Rust, a safety-focused systems programming language seeing increased adoption across the industry. As such, the testing technique has not been widely used on programs written in the language as of yet. Without robust mutation analysis, Rust developers cannot determine test thoroughness. In response to this challenge, we designed a mutation analysis pipeline for Rust, which overcomes the challenges of generating valid mutants caused by the strictness of the language. Our approach accounts for Rust’s distinction between safe and unsafe operations, ensuring that safe mutations of valid Rust programs—those with only valid unsafe code sections—can be safely evaluated within the same process, without the potential for crashes or other undefined behavior invalidating the mutation analysis. We introduce mutation batching, our novel technique for efficiently evaluating multiple mutations simultaneously, while guaranteeing they do not interact. Batching maximizes thread usage, by executing significantly more test cases in parallel. As batching is NP-hard, we present multiple fast approximation algorithms for grouping mutations. We implemented our techniques into a mutation analysis tool, mutest-rs , which we used in our empirical evaluation on a diverse set of 22 Rust libraries and programs. We found that mutation batching reduces the overall runtime of mutation analysis by up to (52.3%) and also saving 73.8 seconds in one case, and that unsafe mutations are detected at a lower rate, of up to (53.4%) . Our mutation analysis tool is available at https://mutest.rs .
@article{levai2026comprehensive,title={A Comprehensive Empirical and Theoretical Analysis of Batching Algorithms for Efficient, Safe, Parallel Mutation Analysis in Rust},author={Lévai, Zalán and Shin, Donghwan and McMinn, Phil},journal={ACM Transactions on Software Engineering and Methodology},volume={35},number={10},pages={1--54},year={2026},month=sep,doi={10.1145/3787851},}
Causal Software Engineering: A Vision and Roadmap
Roberto Pietrantuono, Luca Giamattei, Stefano Russo, and 2 more authors
In Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering, Jul 2026
@inproceedings{pietrantuono2026causal,title={Causal Software Engineering: A Vision and Roadmap},author={Pietrantuono, Roberto and Giamattei, Luca and Russo, Stefano and Siebert, Julien and Walkinshaw, Neil},booktitle={Proceedings of the 34th ACM International Conference on the Foundations of Software Engineering},publisher={ACM},pages={1346--1350},year={2026},month=jul,doi={10.1145/3803437.3805585},}
Fuzz3 : Entropy as a Third Oracle
Karine Even-Mendoza, Janine Obiri, Aidan Dakhama, and 2 more authors
@incollection{evenmendoza2026fuzz3,title={Fuzz3 : Entropy as a Third Oracle},author={Even-Mendoza, Karine and Obiri, Janine and Dakhama, Aidan and McMinn, Phil and Langdon, W. B.},booktitle={Search-Based Software Engineering},publisher={Springer Nature Switzerland},pages={107--113},year={2026},month=jul,doi={10.1007/978-3-032-30699-9_9},}
Vision-based vs. IMU-based upper-limb pose estimation in assisted dressing: a comparative study of positional accuracy and kinematic fidelity
Yasmin Rafiq, Shenglin Wang, Mohammed Al-Nuaimi, and 3 more authors
Accurate estimation of upper-limb kinematics is essential for applications such as rehabilitation assessment and assistive robotics, yet remains challenging in real-world scenarios involving occlusion and physical human interaction. While vision-based pose estimation methods have advanced significantly, their ability to recover reliable joint kinematics under such conditions remains unclear. This paper presents a systematic comparison of vision-based and wearable sensing approaches for upper-limb pose estimation during assisted dressing tasks. A monocular RGB-based convolutional neural network (CNN) and a temporally smoothed variant (CNN_temporal) are evaluated alongside a wearable IMU-based reconstruction method. All approaches are compared against an inverse kinematics (IK) reference derived from VICON motion capture data using participant-specific kinematic models. Performance is assessed using both positional error, measured via global and shoulder-centred mean per-joint position error (MPJPE), and kinematic agreement, measured via elbow flexion/extension angle error. Experiments on a real-world dataset of assisted dressing trials, involving an occupational therapist and three participants, demonstrate that IMU-based estimation provides consistently accurate and stable joint-angle reconstruction (e.g., ∼ 12 ° mean absolute error). In contrast, vision-based methods achieve reasonable positional accuracy (MPJPE ∼ 0.20 m) but exhibit substantially larger errors in joint-angle estimation (often exceeding 80 ° ), particularly under occlusion. Temporal smoothing improves positional consistency but does not preserve kinematic fidelity. These results highlight a fundamental limitation of current vision-based approaches for tasks requiring accurate joint kinematics. The findings suggest that integrating inertial sensing or incorporating biomechanical constraints may be necessary to achieve reliable pose estimation in real-world assistive scenarios.
@article{rafiq2026vision,title={Vision-based vs. IMU-based upper-limb pose estimation in assisted dressing: a comparative study of positional accuracy and kinematic fidelity},author={Rafiq, Yasmin and Wang, Shenglin and Al-Nuaimi, Mohammed and Mihaylova, Lyudmila and Hierons, Robert M. and Dogramadzi, Sanja},journal={Frontiers in Robotics and AI},volume={13},year={2026},month=jul,doi={10.3389/frobt.2026.1844439},}
There is extensive literature on automated test generation using reactive design models, where control is determined by events. In contrast, the (idealised) simulation paradigm defines control through cycles dictated by the passage of time. Within each cycle, inputs are read and processed, and outputs are provided, all instantaneously, and afterwards time progresses. To exercise a simulation using tests generated from a reactive design model requires changes to the tests to take into account this paradigm shift. This article focuses on automation of the necessary changes and of the use of the resulting tests in a simulation campaign. Based on a notion of conformance that establishes whether a simulation is correct with respect to a reactive design, we (1) identify the reactive tests that are meaningful; (2) define a process to convert those tests; (3) provide an algorithm to execute those tests and (4) prove soundness and completeness of our approach. Our work is described in the context of the RoboStar framework for model-based development of control software for robotics applications, and its process algebraic semantics. The testing approach we propose here represents a significant advancement in the current testing practices within the field of robotics, where simulations are widely used.
@article{cavalcanti2026reactive,title={Reactive Model-Based Testing of Cyclic Systems},author={Cavalcanti, Ana and Hierons, Robert M},journal={ACM Transactions on Computational Logic},volume={27},number={3},pages={1--30},year={2026},month=jun,doi={10.1145/3801960},}
Automated testing of prevalent 3D user interactions in virtual reality applications
@article{gu2026automated,title={Automated testing of prevalent 3D user interactions in virtual reality applications},author={Gu, Ruizhen and Rojas, José Miguel and Shin, Donghwan},journal={Automated Software Engineering},volume={33},number={3},year={2026},month=may,doi={10.1007/s10515-026-00620-1},}
Causality-Driven Test Case Minimisation for Cyber-Physical Systems
Michael Foster, Christopher M. Poskitt, Nicholas R. Latimer, and 3 more authors
ACM Transactions on Software Engineering and Methodology, May 2026
Cyber-physical systems allow digital control systems to interact with the physical world using sensors and actuators. They are increasingly being used to automate critical infrastructure, where software faults can have dire consequences. Due to the complex nature and unpredictability of these systems, their resilience is often tested using a technique called fuzzing, which generates quasi-random sequences of sensor and actuator manipulations with the goal of forcing a system into unsafe states. However, there is currently no way of determining which manipulations of a test case cause a failure without systematically removing each one and re-running the test, which can be extremely time-consuming and expensive. In this work, we present CausalCut , a technique that uses causal inference to estimate the causal contribution of each intervention from pre-existing runtime data, thereby reducing the number of times tests must be re-run. We evaluated CausalCut by applying it to two very different systems: an artificial pancreas and a water treatment plant. CausalCut typically managed to remove more than half of the spurious manipulations using fewer executions than the current state of the art, which represents a saving of up to 18 hours and $6300 per test case.
@article{foster2026causality,title={Causality-Driven Test Case Minimisation for Cyber-Physical Systems},author={Foster, Michael and Poskitt, Christopher M. and Latimer, Nicholas R. and Walkinshaw, Neil and Somers, Richard and Hierons, Robert M.},journal={ACM Transactions on Software Engineering and Methodology},year={2026},month=may,doi={10.1145/3816435},}
Complete FSM Testing Using Strong Separability
Robert M. Hierons and Mohammad Reza Mousavi
In Foundations of Software Science and Computation Structures, Apr 2026
@incollection{hierons2026complete,title={Complete FSM Testing Using Strong Separability},author={Hierons, Robert M. and Mousavi, Mohammad Reza},booktitle={Foundations of Software Science and Computation Structures},publisher={Springer Nature Switzerland},pages={328--349},year={2026},month=apr,doi={10.1007/978-3-032-22730-0_16},}
Test Flimsiness: Characterizing Flakiness Induced by Mutation to the Code Under Test
Owain Parry, Gregory M. Kapfhammer, Michael Hilton, and 1 more author
In Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering, Apr 2026
@inproceedings{parry2026test,title={Test Flimsiness: Characterizing Flakiness Induced by Mutation to the Code Under Test},author={Parry, Owain and Kapfhammer, Gregory M. and Hilton, Michael and McMinn, Phil},booktitle={Proceedings of the 2026 IEEE/ACM 48th International Conference on Software Engineering},publisher={ACM},pages={3195--3206},year={2026},month=apr,doi={10.1145/3744916.3773125},}
Combining sequential test cases into an equivalent set of adaptive test cases
Robert M. Hierons, Mercedes G. Merayo, and Manuel Núñez
Journal of Logical and Algebraic Methods in Programming, Jan 2026
@article{hierons2026combining,title={Combining sequential test cases into an equivalent set of adaptive test cases},author={Hierons, Robert M. and Merayo, Mercedes G. and Núñez, Manuel},journal={Journal of Logical and Algebraic Methods in Programming},volume={148},pages={101092},year={2026},month=jan,doi={10.1016/j.jlamp.2025.101092},}
Chatbot-Assisted Explainability of AI-Powered Models and Collection of User Feedback
Olesya Razuvayevskaya, Michael Foster, and Kalina Bontcheva
In Countering Disinformation in the Era of Generative AI, 2026
@incollection{razuvayevskaya2026chatbot,title={Chatbot-Assisted Explainability of AI-Powered Models and Collection of User Feedback},author={Razuvayevskaya, Olesya and Foster, Michael and Bontcheva, Kalina},booktitle={Countering Disinformation in the Era of Generative AI},publisher={Springer Nature Switzerland},pages={381--417},year={2026},doi={10.1007/978-3-032-11782-3_13},}
2025
A Test Automation Framework for User Interaction in Extended Reality Applications
@inproceedings{gu2025test,title={A Test Automation Framework for User Interaction in Extended Reality Applications},author={Gu, Ruizhen and Rojas, José Miguel},booktitle={2025 40th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW)},publisher={IEEE},pages={325--330},year={2025},month=nov,doi={10.1109/asew67777.2025.00066},}
Efficient State Identification for Finite State Machine-Based Testing
Uraz Cengiz Türker, Robert M. Hierons, Mohammad Reza Mousavi, and 1 more author
IEEE Transactions on Software Engineering, Nov 2025
@article{turker2025efficient,title={Efficient State Identification for Finite State Machine-Based Testing},author={Türker, Uraz Cengiz and Hierons, Robert M. and Mousavi, Mohammad Reza and El-Fakih, Khaled},journal={IEEE Transactions on Software Engineering},volume={51},number={11},pages={2996--3012},year={2025},month=nov,doi={10.1109/tse.2025.3604472},}
On the Promises and Challenges of AI-Powered XR Glasses as Embodied Software
@inproceedings{gu2025promises,title={On the Promises and Challenges of AI-Powered XR Glasses as Embodied Software},author={Gu, Ruizhen and Zhang, Jingqiong and Rojas, José Miguel and Shin, Donghwan},booktitle={2025 2nd IEEE/ACM International Conference on AI-powered Software (AIware)},publisher={IEEE},pages={207--212},year={2025},month=nov,doi={10.1109/aiware69974.2025.00030},}
XRintTest: An Automated Framework for User Interaction Testing in Extended Reality Applications
@inproceedings{gu2025xrinttest,title={XRintTest: An Automated Framework for User Interaction Testing in Extended Reality Applications},author={Gu, Ruizhen and Rojas, José Miguel and Shin, Donghwan},booktitle={2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE)},publisher={IEEE},pages={4013--4016},year={2025},month=nov,doi={10.1109/ase63991.2025.00363},}
deflake.rs: Detect Flaky Tests in Rust Projects using Execution Data
@article{magill2025deflake,title={deflake.rs: Detect Flaky Tests in Rust Projects using Execution Data},author={Magill, Benjamin and McMinn, Phil},journal={Journal of Open Source Software},volume={10},number={113},pages={8757},year={2025},month=sep,doi={10.21105/joss.08757},}
Symbolic Runtime Verification and Adaptive Decision-Making for Robot-Assisted Dressing
Yasmin Rafiq, Gricel Vázquez, Radu Calinescu, and 2 more authors
In Software Engineering and Advanced Applications, Sep 2025
@incollection{rafiq2025symbolic,title={Symbolic Runtime Verification and Adaptive Decision-Making for Robot-Assisted Dressing},author={Rafiq, Yasmin and Vázquez, Gricel and Calinescu, Radu and Dogramadzi, Sanja and Hierons, Robert M.},booktitle={Software Engineering and Advanced Applications},publisher={Springer Nature Switzerland},pages={290--308},year={2025},month=sep,doi={10.1007/978-3-032-04190-6_18},}
Directional Lighting-Based Deep Learning Models for Crack and Spalling Classification
External lighting is essential for autonomous inspections of concrete structures in low-light environments. However, previous studies have primarily relied on uniformly diffused lighting to illuminate images and faced challenges in detecting complex crack patterns. This paper proposes two novel algorithms that use directional lighting to classify concrete defects. The first method, named fused neural network, uses the maximum intensity pixel-level image fusion technique and selects the maximum intensity pixel values from all directional images for each pixel to generate a fused image. The second proposed method, named multi-channel neural network, generates a five-channel image, with each channel representing the grayscale version of images captured in the Right (R), Down (D), Left (L), Up (U), and Diffused (A) directions, respectively. The proposed multi-channel neural network model achieved the best performance, with accuracy, precision, recall, and F1 score of 96.6%, 96.3%, 97%, and 96.6%, respectively. It also outperformed the FusedNet and other models found in the literature, with no significant change in evaluation time. The results from this work have the potential to improve concrete crack classification in environments where external illumination is required. Future research focuses on extending the concepts of multi-channel and image fusion to white-box techniques.
@article{pennada2025directional,title={Directional Lighting-Based Deep Learning Models for Crack and Spalling Classification},author={Pennada, Sanjeetha and McAlorum, Jack and Perry, Marcus and Dow, Hamish and Dobie, Gordon},journal={Journal of Imaging},volume={11},number={9},pages={288},year={2025},month=aug,doi={10.3390/jimaging11090288},}
Empirically Evaluating the Use of Bytecode for Diversity-Based Test Case Prioritisation
@inproceedings{elgendy2025empirically,title={Empirically Evaluating the Use of Bytecode for Diversity-Based Test Case Prioritisation},author={Elgendy, Islam and Hierons, Robert and McMinn, Phil},booktitle={Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering},publisher={ACM},pages={216--226},year={2025},month=jun,doi={10.1145/3756681.3756969},}
@inproceedings{devlin2025evolving,title={Evolving Estimation Models for Causal Testing},author={Devlin, Luca and Foster, Michael},booktitle={Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering},publisher={ACM},pages={1394--1401},year={2025},month=jun,doi={10.1145/3696630.3731613},}
QAOA-PCA: Enhancing Efficiency in the Quantum Approximate Optimization Algorithm via Principal Component Analysis
@inproceedings{parry2025qaoa,title={QAOA-PCA: Enhancing Efficiency in the Quantum Approximate Optimization Algorithm via Principal Component Analysis},author={Parry, Owain and McMinn, Phil},booktitle={Proceedings of the 2025 29th International Conference on Evaluation and Assessment in Software Engineering Companion},publisher={ACM},pages={61--66},year={2025},month=jun,doi={10.1145/3727967.3756820},}
Software testing for extended reality applications: a systematic mapping study
Extended Reality (XR) is an emerging technology spanning diverse application domains and offering immersive user experiences. However, its unique characteristics, such as six degrees of freedom interactions, present significant testing challenges distinct from traditional 2D GUI applications, demanding novel testing techniques to build high-quality XR applications. This paper presents the first systematic mapping study on software testing for XR applications. We selected 34 studies focusing on techniques and empirical approaches in XR software testing for detailed examination. The studies are classified and reviewed to address the current research landscape, test facets, and evaluation methodologies in the XR testing domain. Additionally, we provide a repository summarising the mapping study, including datasets and tools referenced in the selected studies, to support future research and practical applications. Our study highlights open challenges in XR testing and proposes actionable future research directions to address the gaps and advance the field of XR software testing.
@article{gu2025software,title={Software testing for extended reality applications: a systematic mapping study},author={Gu, Ruizhen and Rojas, José Miguel and Shin, Donghwan},journal={Automated Software Engineering},volume={32},number={2},year={2025},month=jun,doi={10.1007/s10515-025-00523-7},}
Systemic Flakiness: An Empirical Analysis of Co-Occurring Flaky Test Failures
Owain Parry, Gregory Kapfhammer, Michael Hilton, and 1 more author
In Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering, Jun 2025
@inproceedings{parry2025systemic,title={Systemic Flakiness: An Empirical Analysis of Co-Occurring Flaky Test Failures},author={Parry, Owain and Kapfhammer, Gregory and Hilton, Michael and McMinn, Phil},booktitle={Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering},publisher={ACM},pages={476--487},year={2025},month=jun,doi={10.1145/3756681.3756945},}
Using Causal Inference to Test Systems with Hidden and Interacting Variables: An Evaluative Case Study
@inproceedings{foster2025causal,title={Using Causal Inference to Test Systems with Hidden and Interacting Variables: An Evaluative Case Study},author={Foster, Michael and Hierons, Robert and Shin, Donghwan and Walkinshaw, Neil and Wild, Christopher},booktitle={Proceedings of the 29th International Conference on Evaluation and Assessment in Software Engineering},publisher={ACM},pages={592--603},year={2025},month=jun,doi={10.1145/3756681.3756967},}
Comparison of Directional and Diffused Lighting for Pixel-Level Segmentation of Concrete Cracks
Hamish Dow, Marcus Perry, Jack McAlorum, and 1 more author
Visual inspections of concrete infrastructure in low-light environments require external lighting to ensure adequate visibility. Directional lighting sources, where an image scene is illuminated with an angled lighting source from one direction, can enhance the visibility of surface defects in an image. This paper compares directional and diffused scene illumination images for pixel-level concrete crack segmentation. A novel directional lighting image segmentation algorithm is proposed, which applies crack segmentation image processing techniques to each directionally lit image before combining all images into a single output, highlighting the extremities of the defect. This method was benchmarked against two diffused lighting crack detection techniques across a dataset with crack widths typically ranging from 0.07 mm to 0.4 mm. When tested on cracked and uncracked data, the directional lighting method significantly outperformed other benchmarked diffused lighting methods, attaining a 10% higher true-positive rate (TPR), 12% higher intersection over union (IoU), and 10% higher F1 score with minimal impact on precision. Further testing on only cracked data revealed that directional lighting was superior across all crack widths in the dataset. This research shows that directional lighting can enhance pixel-level crack segmentation in infrastructure requiring external illumination, such as low-light indoor spaces (e.g., tunnels and containment structures) or night-time outdoor inspections (e.g., pavement and bridges).
@article{dow2025comparison,title={Comparison of Directional and Diffused Lighting for Pixel-Level Segmentation of Concrete Cracks},author={Dow, Hamish and Perry, Marcus and McAlorum, Jack and Pennada, Sanjeetha},journal={Infrastructures},volume={10},number={6},pages={129},year={2025},month=may,doi={10.3390/infrastructures10060129},}
Can Test Generation and Program Repair Inform Automated Assessment of Programming Projects?
@inproceedings{gu2025can,title={Can Test Generation and Program Repair Inform Automated Assessment of Programming Projects?},author={Gu, Ruizhen and Rojas, José Miguel and Shin, Donghwan},booktitle={2025 IEEE Conference on Software Testing, Verification and Validation (ICST)},publisher={IEEE},pages={699--710},year={2025},month=mar,doi={10.1109/icst62969.2025.10988955},}
The Causal Testing Framework
Michael Foster, Andrew Clark, Christopher Wild, and 6 more authors
@article{foster2025causalb,title={The Causal Testing Framework},author={Foster, Michael and Clark, Andrew and Wild, Christopher and Allian, Farhad and Turner, Robert and Somers, Richard and Latimer, Nicholas and Walkinshaw, Neil and Hierons, Robert M.},journal={Journal of Open Source Software},volume={10},number={107},pages={7739},year={2025},month=mar,doi={10.21105/joss.07739},}
A Systematic Mapping Study of the Metrics, Uses and Subjects of Diversity‐Based Testing Techniques
Islam T. Elgendy, Robert M. Hierons, and Phil McMinn
Software Testing, Verification and Reliability, Jan 2025
There has been a significant amount of interest regarding the use of DBTtsfull in software testing over the past two decades. Diversity‐based testing (DBT) technique uses similarity metrics to leverage the dissimilarity between software artefacts—such as requirements, abstract models, programme structures or inputs—in order to address a software testing problem. DBT techniques have been used to assist in finding solutions to several different types of problems including generating test cases, prioritizing them and reducing very large test suites. This paper is a systematic mapping study of DBT techniques that summarizes the key aspects and trends of 167 papers that report the use of 79 different similarity metrics with 22 different types of software artefacts, which have been used by researchers to tackle 11 different types of software testing problems. We further present an analysis of the recent trends in DBT techniques and review the different application domains to which the techniques have been applied, giving an overview of the tools developed by researchers in order to do so. Finally, the paper identifies some DBT challenges that are potential topics for future work, such as exploring other diversity artefacts and measuring diversity for complex data input.
@article{elgendy2025systematic,title={A Systematic Mapping Study of the Metrics, Uses and Subjects of Diversity‐Based Testing Techniques},author={Elgendy, Islam T. and Hierons, Robert M. and McMinn, Phil},journal={Software Testing, Verification and Reliability},volume={35},number={2},year={2025},month=jan,doi={10.1002/stvr.1914},}
Configuration Testing of an Artificial Pancreas System Using a Digital Twin: An Evaluative Case Study
Richard Somers, Neil Walkinshaw, Robert Mark Hierons, and 3 more authors
Software Testing, Verification and Reliability, Jan 2025
The recent growth in popularity of wearable medical devices has improved the quality of life of people with medical conditions. Testing such devices may require users to configure these systems using physical trials, putting themselves in potentially dangerous scenarios. Misconfiguration of such devices has caused disease misdiagnoses and incorrect drug prescriptions. Digital twins have been proposed as an opportunity to reduce such risks of testing system configurations in simulated environments, decoupling the user from the system under test. In this paper, we perform an evaluative case study to assess the use of a digital twin for configuration testing of an artificial pancreas system (APS) control algorithm. These systems regulate the blood glucose levels in people with type 1 diabetes mellitus, and so misconfigurations can cause severe hypoglycaemia or hyperglycaemia, which can be life‐threatening. We tested the OpenAPS control algorithm against 156 people’s clinical data. We found that our digital twin provided an accurate simulation environment to perform configuration testing and accurately predict blood glucose–insulin behaviour. We evaluated different APS configurations, identifying a potentially unsafe configuration without the risks associated with a physical trial. We identified the challenges associated with modelling clinical data, which could lead to misinterpretations in configuration testing and the reduction of test reliability when modelling stochastic body dynamics.
@article{somers2025configuration,title={Configuration Testing of an Artificial Pancreas System Using a Digital Twin: An Evaluative Case Study},author={Somers, Richard and Walkinshaw, Neil and Mark Hierons, Robert and Elliott, Jackie and Iqbal, Ahmed and Walkinshaw, Emma},journal={Software Testing, Verification and Reliability},volume={35},number={2},year={2025},month=jan,doi={10.1002/stvr.70000},}
@article{gazda2025model,title={Model independent refusal trace testing},author={Gazda, Maciej and Hierons, Robert M.},journal={Science of Computer Programming},volume={239},pages={103173},year={2025},month=jan,doi={10.1016/j.scico.2024.103173},}
Autonomous Driving System Testing: Traffic Density Does Matter
@incollection{lou2025autonomous,title={Autonomous Driving System Testing: Traffic Density Does Matter},author={Lou, Guannan and Shin, Donghwan and Walkinshaw, Neil and Hierons, Robert M.},booktitle={Testing Software and Systems},publisher={Springer Nature Switzerland},pages={315--331},year={2025},doi={10.1007/978-3-031-80889-0_21},}
Exploratory Software Testing in Scrum: A Qualitative Study
Exploratory Testing (ET) is a dynamic software testing approach that emphasises creativity, real-time learning, and defect discovery. The integration of ET into structured frameworks like Scrum remains insufficiently explored and presents distinct challenges. This qualitative study investigates how ET is implemented in Scrum workflows and identifies key factors enabling its effective application. Interviews with 20 industry professionals highlight ET’s role in enhancing test coverage, uncovering usability issues, and addressing edge cases often missed by automated or scripted tests. The results demonstrated that the critical enablers of effective ET are the tester’s eagerness to learn about the system under test and the ability to adopt a user-centric perspective. Other key factors include testers’ curiosity, creativity, domain knowledge, and organisational support. Participants noted that ET complements Scrum’s iterative cycles, enabling teams to identify defects dynamically and improve software quality. Despite its advantages, ET faces challenges within Scrum, including time constraints and the need for traceability. Lightweight documentation practices, such as annotated mind maps and screen recordings, emerged as effective strategies to bridge these gaps. This study underscores ET’s potential to enhance Scrum workflows, providing actionable insights for optimising testing strategies in Agile environments.
@incollection{neri2025exploratory,title={Exploratory Software Testing in Scrum: A Qualitative Study},author={Neri, Giulia and Marchand, Rob and Walkinshaw, Neil},booktitle={Agile Processes in Software Engineering and Extreme Programming},publisher={Springer Nature Switzerland},pages={160--175},year={2025},doi={10.1007/978-3-031-94544-1_11},}
Where Tests Fall Short: Empirically Analyzing Oracle Gaps in Covered Code
@inproceedings{maton2025where,title={Where Tests Fall Short: Empirically Analyzing Oracle Gaps in Covered Code},author={Maton, M. and Kapfhammer, G. M. and McMinn, P.},booktitle={International Symposium on Empirical Software Engineering and Measurement (ESEM)},year={2025},}
2024
Towards a Taxonomy of Software Log Smells
Nyyti Saarimäki, Donghwan Shin, and Domenico Bianculli
Context: Logging is an important part of modern software projects; logs are used in several tasks such as debugging and testing. Due to the complex nature of logging, it remains a difficult task with several pitfalls that could have serious consequences. Several other domains of software engineering have mitigated such threats by identifying the early signs of more serious issues, i.e., ’smells’. However, this concept is not yet properly defined for logging. Objective: The goal of this study is to create a taxonomy of log smells that can help developers write better logging code. To further help the developers and to identify issues that need more attention from the research community, we also map the identified smells to existing tools repairing them. Methods: We conducted a survey of the scientific literature to identify logging issues and related tools. After extracting relevant data from 51 articles, we used open coding to define logging issues and applied card sorting to derive log smells from these issues. Finally, we classified the tools based on their reported output. Results: We present a taxonomy of nine log smells and describe several facets for each of them. We also review existing tools repairing/removing some of these facets, highlighting the lack of tools addressing some log smells and identifying future research opportunities to close this gap. Conclusions: Logging is vulnerable to log smells throughout all phases of its life cycle, and these issues can affect both logging implementation and the resulting log files. Understanding these smells - along with their causes and consequences - can help reduce their occurrence and lead to higher-quality logging.
@article{saarimaki2024taxonomy,title={Towards a Taxonomy of Software Log Smells},author={Saarimäki, Nyyti and Shin, Donghwan and Bianculli, Domenico},journal={arXiv preprint arXiv:2412.09284},year={2024},month=dec,doi={10.48550/ARXIV.2412.09284},}
3D reconstruction and measurement of concrete spalling using near-field Photometric stereo and YOLOv8
@article{dow20243d,title={3D reconstruction and measurement of concrete spalling using near-field Photometric stereo and YOLOv8},author={Dow, Hamish and Perry, Marcus and Pennada, Sanjeetha and Lunn, Rebecca and Pytharouli, Stella},journal={Automation in Construction},volume={166},pages={105633},year={2024},month=oct,doi={10.1016/j.autcon.2024.105633},}
Private-Keep Out? Understanding How Developers Account for Code Visibility in Unit Testing
@inproceedings{roslan2024private,title={Private-Keep Out? Understanding How Developers Account for Code Visibility in Unit Testing},author={Roslan, Muhammad Firhard and Rojas, José Miguel and McMinn, Phil},booktitle={2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)},publisher={IEEE},pages={312--324},year={2024},month=oct,doi={10.1109/icsme58944.2024.00037},}
Viscount: A Direct Method Call Coverage Tool for Java
@inproceedings{roslan2024viscount,title={Viscount: A Direct Method Call Coverage Tool for Java},author={Roslan, Muhammad Firhard and Rojas, José Miguel and McMinn, Phil},booktitle={2024 IEEE International Conference on Software Maintenance and Evolution (ICSME)},publisher={IEEE},pages={908--912},year={2024},month=oct,doi={10.1109/icsme58944.2024.00101},}
Subjective logic as a complementary tool to meta-analysis to explicitly address second-order uncertainty in research findings: A case from infant studies
@article{margoni2024subjective,title={Subjective logic as a complementary tool to meta-analysis to explicitly address second-order uncertainty in research findings: A case from infant studies},author={Margoni, Francesco and Walkinshaw, Neil},journal={Infant Behavior and Development},volume={76},pages={101978},year={2024},month=sep,doi={10.1016/j.infbeh.2024.101978},}
Impact of log parsing on deep learning-based anomaly detection
Zanis Ali Khan, Donghwan Shin, Domenico Bianculli, and 1 more author
Software systems log massive amounts of data, recording important runtime information. Such logs are used, for example, for log-based anomaly detection, which aims to automatically detect abnormal behaviors of the system under analysis by processing the information recorded in its logs. Many log-based anomaly detection techniques based on deep learning models include a pre-processing step called log parsing. However, understanding the impact of log parsing on the accuracy of anomaly detection techniques has received surprisingly little attention so far. Investigating what are the key properties log parsing techniques should ideally have to help anomaly detection is therefore warranted. In this paper, we report on a comprehensive empirical study on the impact of log parsing on anomaly detection accuracy, using 13 log parsing techniques, seven anomly detection techniques (five based on deep learning and two based on traditional machine learning) on three publicly available log datasets. Our empirical results show that, despite what is widely assumed, there is no strong correlation between log parsing accuracy and anomaly detection accuracy, regardless of the metric used for measuring log parsing accuracy. Moreover, we experimentally confirm existing theoretical results showing that it is a property that we refer to as distinguishability in log parsing results—as opposed to their accuracy—that plays an essential role in achieving accurate anomaly detection.
@article{khan2024impact,title={Impact of log parsing on deep learning-based anomaly detection},author={Khan, Zanis Ali and Shin, Donghwan and Bianculli, Domenico and Briand, Lionel C.},journal={Empirical Software Engineering},volume={29},number={6},year={2024},month=aug,doi={10.1007/s10664-024-10533-w},}
Bounding Random Test Set Size with Computational Learning Theory
Random testing approaches work by generating inputs at random, or by selecting inputs randomly from some pre-defined operational profile. One long-standing question that arises in this and other testing contexts is as follows: When can we stop testing? At what point can we be certain that executing further tests in this manner will not explore previously untested (and potentially buggy) software behaviors? This is analogous to the question in Machine Learning, of how many training examples are required in order to infer an accurate model. In this paper we show how probabilistic approaches to answer this question in Machine Learning (arising from Computational Learning Theory) can be applied in our testing context, to provide an upper-bound on the number of tests required to achieve a given level of adequacy. We validate this bound on a large set of Java units, and an autonomous driving system.
@article{walkinshaw2024bounding,title={Bounding Random Test Set Size with Computational Learning Theory},author={Walkinshaw, Neil and Foster, Michael and Rojas, José Miguel and Hierons, Robert M.},journal={Proceedings of the ACM on Software Engineering},volume={1},number={FSE},pages={2538--2560},year={2024},month=jul,doi={10.1145/3660819},}
Towards Simplification of Failure Scenarios for Machine Learning-Enabled Autonomous Systems
@inproceedings{shin2024simplification,title={Towards Simplification of Failure Scenarios for Machine Learning-Enabled Autonomous Systems},author={Shin, Donghwan and Pennada, Sanjeetha},booktitle={2024 IEEE 24th International Conference on Software Quality, Reliability, and Security Companion (QRS-C)},publisher={IEEE},pages={1089--1090},year={2024},month=jul,doi={10.1109/qrs-c63300.2024.00143},}
Systematic Evaluation of Deep Learning Models for Log-based Failure Prediction
Fatemeh Hadadi, Joshua H. Dawes, Donghwan Shin, and 2 more authors
With the increasing complexity and scope of software systems, their dependability is crucial. The analysis of log data recorded during system execution can enable engineers to automatically predict failures at run time. Several Machine Learning (ML) techniques, including traditional ML and Deep Learning (DL), have been proposed to automate such tasks. However, current empirical studies are limited in terms of covering all main DL types—Recurrent Neural Network (RNN), Convolutional Neural Network (CNN), and transformer—as well as examining them on a wide range of diverse datasets. In this paper, we aim to address these issues by systematically investigating the combination of log data embedding strategies and DL types for failure prediction. To that end, we propose a modular architecture to accommodate various configurations of embedding strategies and DL-based encoders. To further investigate how dataset characteristics such as dataset size and failure percentage affect model accuracy, we synthesised 360 datasets, with varying characteristics, for three distinct system behavioural models, based on a systematic and automated generation approach. Using the F1 score metric, our results show that the best overall performing configuration is a CNN-based encoder with Logkey2vec. Additionally, we provide specific dataset conditions, namely a dataset size $>350 > 350 or a failure percentage >7.5%$ > 7.5 % , under which this configuration demonstrates high accuracy for failure prediction.
@article{hadadi2024systematic,title={Systematic Evaluation of Deep Learning Models for Log-based Failure Prediction},author={Hadadi, Fatemeh and Dawes, Joshua H. and Shin, Donghwan and Bianculli, Domenico and Briand, Lionel},journal={Empirical Software Engineering},volume={29},number={5},year={2024},month=jun,doi={10.1007/s10664-024-10501-4},}
Causal Test Adequacy
Michael Foster, Christopher Wild, Robert M. Hierons, and 1 more author
In 2024 IEEE Conference on Software Testing, Verification and Validation (ICST), May 2024
@inproceedings{foster2024causal,title={Causal Test Adequacy},author={Foster, Michael and Wild, Christopher and Hierons, Robert M. and Walkinshaw, Neil},booktitle={2024 IEEE Conference on Software Testing, Verification and Validation (ICST)},publisher={IEEE},pages={161--172},year={2024},month=may,doi={10.1109/icst60714.2024.00023},}
Evaluating String Distance Metrics for Reducing Automatically Generated Test Suites
@inproceedings{elgendy2024evaluating,title={Evaluating String Distance Metrics for Reducing Automatically Generated Test Suites},author={Elgendy, Islam and Hierons, Robert and Mcminn, Phil},booktitle={Proceedings of the 5th ACM/IEEE International Conference on Automation of Software Test (AST 2024)},publisher={ACM},pages={171--181},year={2024},month=apr,doi={10.1145/3644032.3644455},}
Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMs
@inproceedings{li2024mutation,title={Mutation-based Consistency Testing for Evaluating the Code Understanding Capability of LLMs},author={Li, Ziyu and Shin, Donghwan},booktitle={Proceedings of the IEEE/ACM 3rd International Conference on AI Engineering - Software Engineering for AI},publisher={ACM},pages={150--159},year={2024},month=apr,doi={10.1145/3644815.3644946},}
Rigorous Assessment of Model Inference Accuracy using Language Cardinality
Donato Clun, Donghwan Shin, Antonio Filieri, and 1 more author
ACM Transactions on Software Engineering and Methodology, Apr 2024
Models such as finite state automata are widely used to abstract the behavior of software systems by capturing the sequences of events observable during their execution. Nevertheless, models rarely exist in practice and, when they do, get easily outdated; moreover, manually building and maintaining models is costly and error-prone. As a result, a variety of model inference methods that automatically construct models from execution traces have been proposed to address these issues. However, performing a systematic and reliable accuracy assessment of inferred models remains an open problem. Even when a reference model is given, most existing model accuracy assessment methods may return misleading and biased results. This is mainly due to their reliance on statistical estimators over a finite number of randomly generated traces, introducing avoidable uncertainty about the estimation and being sensitive to the parameters of the random trace generative process. This article addresses this problem by developing a systematic approach based on analytic combinatorics that minimizes bias and uncertainty in model accuracy assessment by replacing statistical estimation with deterministic accuracy measures. We experimentally demonstrate the consistency and applicability of our approach by assessing the accuracy of models inferred by state-of-the-art inference tools against reference models from established specification mining benchmarks.
@article{clun2024rigorous,title={Rigorous Assessment of Model Inference Accuracy using Language Cardinality},author={Clun, Donato and Shin, Donghwan and Filieri, Antonio and Bianculli, Domenico},journal={ACM Transactions on Software Engineering and Methodology},volume={33},number={4},pages={1--39},year={2024},month=apr,doi={10.1145/3640332},}
Toward Automated Compliance Checking of Fund Activities Using Runtime Verification Techniques
Marcello Ceci, Nicolas Sannier, Sallam Abualhaija, and 3 more authors
In Proceedings of the 1st IEEE/ACM Workshop on Software Engineering Challenges in Financial Firms, Apr 2024
@inproceedings{ceci2024automated,title={Toward Automated Compliance Checking of Fund Activities Using Runtime Verification Techniques},author={Ceci, Marcello and Sannier, Nicolas and Abualhaija, Sallam and Shin, Donghwan and Bianculli, Domenico and Halling, Michael},booktitle={Proceedings of the 1st IEEE/ACM Workshop on Software Engineering Challenges in Financial Firms},publisher={ACM},pages={19--20},year={2024},month=apr,doi={10.1145/3643665.3648045},}
Tuning the Feedback Controller Gains is a Simple Way to Improve Autonomous Driving Performance
Wenyu Liang, Pablo R. Baldivieso, Ross Drummond, and 1 more author
In 2024 UKACC 14th International Conference on Control (CONTROL), Apr 2024
@inproceedings{liang2024tuning,title={Tuning the Feedback Controller Gains is a Simple Way to Improve Autonomous Driving Performance},author={Liang, Wenyu and Baldivieso, Pablo R. and Drummond, Ross and Shin, Donghwan},booktitle={2024 UKACC 14th International Conference on Control (CONTROL)},publisher={IEEE},pages={72--77},year={2024},month=apr,doi={10.1109/control60310.2024.10531819},}
What’s in a Display Name? An Empirical Study on the Use of Display Names in Open-Source JUnit Tests
@inproceedings{qiao2024whats,title={What’s in a Display Name? An Empirical Study on the Use of Display Names in Open-Source JUnit Tests},author={Qiao, Yining and Rojas, José Miguel},booktitle={Proceedings of the Third ACM/IEEE International Workshop on NL-based Software Engineering},publisher={ACM},pages={17--24},year={2024},month=apr,doi={10.1145/3643787.3648037},}
Accelerating Finite State Machine-Based Testing Using Reinforcement Learning
Uraz Cengiz Türker, Robert M. Hierons, Khaled El-Fakih, and 2 more authors
IEEE Transactions on Software Engineering, Mar 2024
@article{turker2024accelerating,title={Accelerating Finite State Machine-Based Testing Using Reinforcement Learning},author={Türker, Uraz Cengiz and Hierons, Robert M. and El-Fakih, Khaled and Mousavi, Mohammad Reza and Tyukin, Ivan Y.},journal={IEEE Transactions on Software Engineering},volume={50},number={3},pages={574--597},year={2024},month=mar,doi={10.1109/tse.2024.3358416},}
Do Automatic Test Generation Tools Generate Flaky Tests?
Martin Gruber, Muhammad Firhard Roslan, Owain Parry, and 3 more authors
In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, Feb 2024
@inproceedings{gruber2024do,title={Do Automatic Test Generation Tools Generate Flaky Tests?},author={Gruber, Martin and Roslan, Muhammad Firhard and Parry, Owain and Scharnböck, Fabian and McMinn, Phil and Fraser, Gordon},booktitle={Proceedings of the IEEE/ACM 46th International Conference on Software Engineering},publisher={ACM},pages={1--12},year={2024},month=feb,doi={10.1145/3597503.3608138},}
Do maternal haemodynamics have a causal influence on treatment for gestational diabetes?
Abigail R. Anness, Michael Foster, Mohammed W. Osman, and 5 more authors
@article{anness2024do,title={Do maternal haemodynamics have a causal influence on treatment for gestational diabetes?},author={Anness, Abigail R. and Foster, Michael and Osman, Mohammed W. and Webb, David and Robinson, Thompson and Khalil, Asma and Walkinshaw, Neil and Mousa, Hatem A.},journal={Journal of Obstetrics and Gynaecology},volume={44},number={1},year={2024},month=feb,doi={10.1080/01443615.2024.2307883},}
Virtual Environment Model Generation for CPS Goal Verification using Imitation Learning
Cyber-Physical Systems (CPS) continuously interact with their physical environments through embedded software controllers that observe the environments and determine actions. Field Operational Tests (FOT) are essential to verify to what extent the CPS under analysis can achieve certain CPS goals, such as satisfying the safety and performance requirements, while interacting with the real operational environment. However, performing many FOTs to obtain statistically significant verification results is challenging due to its high cost and risk in practice. Simulation-based verification can be an alternative to address the challenge, but it still requires an accurate virtual environment model that can replace the real environment interacting with the CPS in a closed loop. In this article, we propose ENVI (ENVironment Imitation), a novel approach to automatically generate an accurate virtual environment model, enabling efficient and accurate simulation-based CPS goal verification in practice.To do this, we first formally define the problem of the virtual environment model generation and solve it by leveraging Imitation Learning (IL), which has been actively studied in machine learning to learn complex behaviors from expert demonstrations. The key idea behind the model generation is to leverage IL for training a model that imitates the interactions between the CPS controller and its real environment as recorded in (possibly very small) FOT logs. We then statistically verify the goal achievement of the CPS by simulating it with the generated model. We empirically evaluate ENVI by applying it to the verification of two popular autonomous driving assistant systems. The results show that ENVI can reduce the cost of CPS goal verification while maintaining its accuracy by generating accurate environment models from only a few FOT logs. The use of IL in virtual environment model generation opens new research directions, further discussed at the end of the article.
@article{shin2024virtual,title={Virtual Environment Model Generation for CPS Goal Verification using Imitation Learning},author={Shin, Yong-Jun and Shin, Donghwan and Bae, Doo-Hwan},journal={ACM Transactions on Embedded Computing Systems},volume={23},number={1},pages={1--29},year={2024},month=jan,doi={10.1145/3633804},}
Exploring Pseudo-Testedness: Empirically Evaluating Extreme Mutation Testing at the Statement Level
@inproceedings{maton2024exploring,title={Exploring Pseudo-Testedness: Empirically Evaluating Extreme Mutation Testing at the Statement Level},author={Maton, M. and Kapfhammer, G. M. and McMinn, P.},booktitle={Proceedings of the International Conference on Software Maintenance and Evolution (ICSME)},year={2024},}
@inproceedings{maton2024pseudosweep,title={PseudoSweep: A Pseudo-Tested Code Identifier},author={Maton, M. and Kapfhammer, G. M. and McMinn, P.},booktitle={Proceedings of the International Conference on Software Maintenance and Evolution (ICSME), Tool Track},year={2024},}
2023
Identifying the Hazard Boundary of ML-Enabled Autonomous Systems Using Cooperative Coevolutionary Search
Sepehr Sharifi, Donghwan Shin, Lionel C. Briand, and 1 more author
IEEE Transactions on Software Engineering, Dec 2023
@article{sharifi2023identifying,title={Identifying the Hazard Boundary of ML-Enabled Autonomous Systems Using Cooperative Coevolutionary Search},author={Sharifi, Sepehr and Shin, Donghwan and Briand, Lionel C. and Aschbacher, Nathan},journal={IEEE Transactions on Software Engineering},volume={49},number={12},pages={5120--5138},year={2023},month=dec,doi={10.1109/tse.2023.3327575},}
Active Inference of EFSMs Without Reset
Michael Foster, Roland Groz, Catherine Oriat, and 3 more authors
In Formal Methods and Software Engineering, Nov 2023
@incollection{foster2023active,title={Active Inference of EFSMs Without Reset},author={Foster, Michael and Groz, Roland and Oriat, Catherine and Simao, Adenilso and Vega, Germán and Walkinshaw, Neil},booktitle={Formal Methods and Software Engineering},publisher={Springer Nature Singapore},pages={29--46},year={2023},month=nov,doi={10.1007/978-981-99-7584-6_3},}
Automated Concrete Crack Inspection With Directional Lighting Platform
@article{mcalorum2023automated,title={Automated Concrete Crack Inspection With Directional Lighting Platform},author={McAlorum, Jack and Dow, Hamish and Pennada, Sanjeetha and Perry, Marcus and Dobie, Gordon},journal={IEEE Sensors Letters},volume={7},number={11},pages={1--4},year={2023},month=nov,doi={10.1109/lsens.2023.3327611},}
Testing Causality in Scientific Modelling Software
Andrew G. Clark, Michael Foster, Benedikt Prifling, and 4 more authors
ACM Transactions on Software Engineering and Methodology, Nov 2023
From simulating galaxy formation to viral transmission in a pandemic, scientific models play a pivotal role in developing scientific theories and supporting government policy decisions that affect us all. Given these critical applications, a poor modelling assumption or bug could have far-reaching consequences. However, scientific models possess several properties that make them notoriously difficult to test, including a complex input space, long execution times, and non-determinism, rendering existing testing techniques impractical. In fields such as epidemiology, where researchers seek answers to challenging causal questions, a statistical methodology known as Causal inference has addressed similar problems, enabling the inference of causal conclusions from noisy, biased, and sparse data instead of costly experiments. This article introduces the causal testing framework: a framework that uses causal inference techniques to establish causal effects from existing data, enabling users to conduct software testing activities concerning the effect of a change, such as metamorphic testing, a posteriori . We present three case studies covering real-world scientific models, demonstrating how the causal testing framework can infer metamorphic test outcomes from reused, confounded test data to provide an efficient solution for testing scientific modelling software.
@article{clark2023testing,title={Testing Causality in Scientific Modelling Software},author={Clark, Andrew G. and Foster, Michael and Prifling, Benedikt and Walkinshaw, Neil and Hierons, Robert M. and Schmidt, Volker and Turner, Robert D.},journal={ACM Transactions on Software Engineering and Methodology},volume={33},number={1},pages={1--42},year={2023},month=nov,doi={10.1145/3607184},}
Summary of the Fourth International Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest 2023)
Matteo Biagiola, Nicolás Cardozo, Donghwan Shin, and 3 more authors
Deep Learning (DL) techniques help software developers thanks to their ability to learn from historical information which is useful in several program analysis and testing tasks (e.g., malware detection, fuzz testing, bug-finding, and type-checking). DL-based software systems are also increasingly adopted in safety-critical domains, such as autonomous driving, medical diagnosis, and aircraft collision avoidance systems. In particular, testing the correctness and reliability of DL-based systems is paramount, since a failure of such systems would cause a significant safety risk for the involved people and/or environment. The 4th International Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest 2023) was co-located with the 45th International Conference on Software Engineering (ICSE), with the goal of targeting research at the intersection of software engineering and deep learning and devise novel approaches and tools to ensure the interpretability and dependability of software systems that depends on DL components.
@article{biagiola2023summary,title={Summary of the Fourth International Workshop on Deep Learning for Testing and Testing for Deep Learning (DeepTest 2023)},author={Biagiola, Matteo and Cardozo, Nicolás and Shin, Donghwan and Khomh, Foutse and Stocco, Andrea and Riccio, Vincenzo},journal={ACM SIGSOFT Software Engineering Notes},volume={48},number={4},pages={39--40},year={2023},month=oct,doi={10.1145/3617946.3617953},}
An Empirical Study on the Adoption of Scripted GUI Testing for Android Apps
@inproceedings{gu2023empirical,title={An Empirical Study on the Adoption of Scripted GUI Testing for Android Apps},author={Gu, Ruizhen and Rojas, José Miguel},booktitle={2023 38th IEEE/ACM International Conference on Automated Software Engineering Workshops (ASEW)},publisher={IEEE},pages={179--182},year={2023},month=sep,doi={10.1109/asew60602.2023.00030},}
An Extensible Modeling Method Supporting Ontology-Based Scenario Specification and Domain-Specific Extension
Young-Min Baek, Esther Cho, Donghwan Shin, and 1 more author
International Journal of Software Engineering and Knowledge Engineering, Sep 2023
Scenario-based techniques, also known as scenario methods, have been actively employed to resolve intricate problems for engineering complex software systems. Scenarios are powerful tools that allow engineers to analyze the dynamics and contexts of complex systems. Despite the widespread use, there is a lack of a well-established reference framework that systematically organizes key concepts and attributes of scenarios. This has left engineers without a systematic guidance at the method level, hindering their ability to utilize the scenario methods effectively. To address the challenges associated with scenario methods, this study aims to provide a reference framework and modeling method. By conducting a literature review and suggesting a Conceptual Scenario Framework (CSF), we establish a conceptual basis that systematically presents the core concepts and characteristics of scenarios. Additionally, we introduce the Extensible Scenario Modeling Method (ESMM) that empowers engineers to perform scenario modeling and domain-specific extensions using the framework. With the inclusion of the Extensible Scenario Modeling Language (ESML), which comprises domain-general model types and classes for scenario description and ontological analysis, ESMM facilitates flexible design of domain-specific scenario elements through language-level extensions. This study assesses the proposed method in comparison to existing scenario development methods in the automated driving system domain. Through an analysis of their ability to represent scenario data, it was established that the language constructs of ESML possess semantic expressiveness suitable for serving as a reference framework. Furthermore, the findings from the case study validate the extensibility of ESMM for specialization in creating a scenario modeling language tailored to specific domains, while also effectively supporting the ontological analysis of particular application domains.
@article{baek2023extensible,title={An Extensible Modeling Method Supporting Ontology-Based Scenario Specification and Domain-Specific Extension},author={Baek, Young-Min and Cho, Esther and Shin, Donghwan and Bae, Doo-Hwan},journal={International Journal of Software Engineering and Knowledge Engineering},volume={34},number={01},pages={91--162},year={2023},month=sep,doi={10.1142/s021819402350047x},}
Incomplete Adaptive Distinguishing Sequences for Non-Deterministic FSMs
Uraz Cengiz Türker, Robert M. Hierons, Gerassimos Barlas, and 1 more author
IEEE Transactions on Software Engineering, Sep 2023
@article{turker2023incomplete,title={Incomplete Adaptive Distinguishing Sequences for Non-Deterministic FSMs},author={Türker, Uraz Cengiz and Hierons, Robert M. and Barlas, Gerassimos and El-Fakih, Khaled},journal={IEEE Transactions on Software Engineering},volume={49},number={9},pages={4371--4389},year={2023},month=sep,doi={10.1109/tse.2023.3291137},}
Active inference of extended finite state models of software systems
R Groz, C Oriat, G Vega, and 3 more authors
In Proceedings of Machine Learning Research, Jul 2023
@inproceedings{groz2023active,title={Active inference of extended finite state models of software systems},author={Groz, R and Oriat, C and Vega, G and Simao, A and Foster, M and Walkinshaw, N},booktitle={Proceedings of Machine Learning Research},year={2023},month=jul,}
Implementation relations and testing for cyclic systems: Adding probabilities
Manuel Núñez, Robert M. Hierons, and Raluca Lefticaru
@article{nunez2023implementation,title={Implementation relations and testing for cyclic systems: Adding probabilities},author={Núñez, Manuel and Hierons, Robert M. and Lefticaru, Raluca},journal={Robotics and Autonomous Systems},volume={165},pages={104426},year={2023},month=jul,doi={10.1016/j.robot.2023.104426},}
Skeleton-based noise removal algorithm for binary concrete crack image segmentation
Hamish Dow, Marcus Perry, Jack McAlorum, and 2 more authors
@article{dow2023skeleton,title={Skeleton-based noise removal algorithm for binary concrete crack image segmentation},author={Dow, Hamish and Perry, Marcus and McAlorum, Jack and Pennada, Sanjeetha and Dobie, Gordon},journal={Automation in Construction},volume={151},pages={104867},year={2023},month=jul,doi={10.1016/j.autcon.2023.104867},}
Challenges in testing of cyclic systems
Ana Cavalcanti and Robert M. Hierons
In 2023 27th International Conference on Engineering of Complex Computer Systems (ICECCS), Jun 2023
@inproceedings{cavalcanti2023challenges,title={Challenges in testing of cyclic systems},author={Cavalcanti, Ana and Hierons, Robert M.},booktitle={2023 27th International Conference on Engineering of Complex Computer Systems (ICECCS)},publisher={IEEE},pages={1--6},year={2023},month=jun,doi={10.1109/iceccs59891.2023.00010},}
Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems
@inproceedings{ulhaq2023many,title={Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems},author={Ul Haq, Fitash and Shin, Donghwan and Briand, Lionel C.},booktitle={2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)},publisher={IEEE},pages={1814--1826},year={2023},month=may,doi={10.1109/icse48619.2023.00155},}
Modelling Second-Order Uncertainty in State Machines
@article{walkinshaw2023modelling,title={Modelling Second-Order Uncertainty in State Machines},author={Walkinshaw, Neil and Hierons, Robert M.},journal={IEEE Transactions on Software Engineering},volume={49},number={5},pages={3261--3276},year={2023},month=may,doi={10.1109/tse.2023.3250835},}
A Declarative Metamorphic Testing Framework for Autonomous Driving
Yao Deng, Xi Zheng, Tianyi Zhang, and 4 more authors
IEEE Transactions on Software Engineering, Apr 2023
@article{deng2023declarative,title={A Declarative Metamorphic Testing Framework for Autonomous Driving},author={Deng, Yao and Zheng, Xi and Zhang, Tianyi and Liu, Huai and Lou, Guannan and Kim, Miryung and Chen, Tsong Yueh},journal={IEEE Transactions on Software Engineering},volume={49},number={4},pages={1964--1982},year={2023},month=apr,doi={10.1109/tse.2022.3206427},}
A novel directional lighting algorithm for concrete crack pixel-level segmentation
Hamish Dow, Marcus Perry, Jack McAlorum, and 2 more authors
In Sensors and Smart Structures Technologies for Civil, Mechanical, and Aerospace Systems 2023, Apr 2023
@inproceedings{dow2023novel,title={A novel directional lighting algorithm for concrete crack pixel-level segmentation},author={Dow, Hamish and Perry, Marcus and McAlorum, Jack and Pennada, Sanjeetha and Dobie, Gordon},booktitle={Sensors and Smart Structures Technologies for Civil, Mechanical, and Aerospace Systems 2023},publisher={SPIE},pages={65},year={2023},month=apr,doi={10.1117/12.2657235},}
Batching Non-Conflicting Mutations for Efficient, Safe, Parallel Mutation Analysis in Rust
@inproceedings{levai2023batching,title={Batching Non-Conflicting Mutations for Efficient, Safe, Parallel Mutation Analysis in Rust},author={Lévai, Zalán and McMinn, Phil},booktitle={2023 IEEE Conference on Software Testing, Verification and Validation (ICST)},publisher={IEEE},pages={49--59},year={2023},month=apr,doi={10.1109/icst57152.2023.00014},}
Digital-twin-based testing for cyber–physical systems: A systematic literature review
Richard J. Somers, James A. Douthwaite, David J. Wagg, and 2 more authors
@article{somers2023digital,title={Digital-twin-based testing for cyber–physical systems: A systematic literature review},author={Somers, Richard J. and Douthwaite, James A. and Wagg, David J. and Walkinshaw, Neil and Hierons, Robert M.},journal={Information and Software Technology},volume={156},pages={107145},year={2023},month=apr,doi={10.1016/j.infsof.2022.107145},}
@inproceedings{clark2023metamorphic,title={Metamorphic Testing with Causal Graphs},author={Clark, Andrew G. and Foster, Michael and Walkinshaw, Neil and Hierons, Robert M.},booktitle={2023 IEEE Conference on Software Testing, Verification and Validation (ICST)},publisher={IEEE},pages={153--164},year={2023},month=apr,doi={10.1109/icst57152.2023.00023},}
Performance evaluation of an improved deep CNN-based concrete crack detection algorithm
@inproceedings{pennada2023performance,title={Performance evaluation of an improved deep CNN-based concrete crack detection algorithm},author={Pennada, Sanjeetha and Perry, Marcus and McAlorum, Jack and Dow, Hamish and Dobie, Gordon},booktitle={Sensors and Smart Structures Technologies for Civil, Mechanical, and Aerospace Systems 2023},publisher={SPIE},pages={45},year={2023},month=apr,doi={10.1117/12.2657723},}
Robotic concrete inspection with illumination-enhancement
Jack McAlorum, Marcus Perry, Hamish Dow, and 1 more author
In Sensors and Smart Structures Technologies for Civil, Mechanical, and Aerospace Systems 2023, Apr 2023
@inproceedings{mcalorum2023robotic,title={Robotic concrete inspection with illumination-enhancement},author={McAlorum, Jack and Perry, Marcus and Dow, Hamish and Pennada, Sanjeetha},booktitle={Sensors and Smart Structures Technologies for Civil, Mechanical, and Aerospace Systems 2023},publisher={SPIE},pages={23},year={2023},month=apr,doi={10.1117/12.2655938},}
Towards Log Slicing
Joshua Heneage Dawes, Donghwan Shin, and Domenico Bianculli
In Fundamental Approaches to Software Engineering, Apr 2023
This short paper takes initial steps towards developing a novel approach, called log slicing , that aims to answer a practical question in the field of log analysis: Can we automatically identify log messages related to a specific message (e.g., an error message)? The basic idea behind log slicing is that we can consider how different log messages are “computationally related” to each other by looking at the corresponding logging statements in the source code. These logging statements are identified by 1) computing a backwards program slice, using as criterion the logging statement that generated a problematic log message; and 2) extending that slice to include relevant logging statements. The paper presents a problem definition of log slicing, describes an initial approach for log slicing, and discusses a key open issue that can lead towards new research directions.
@incollection{dawes2023log,title={Towards Log Slicing},author={Dawes, Joshua Heneage and Shin, Donghwan and Bianculli, Domenico},booktitle={Fundamental Approaches to Software Engineering},publisher={Springer Nature Switzerland},pages={249--259},year={2023},month=apr,doi={10.1007/978-3-031-30826-0_14},}
Removing redundant refusals: Minimal complete test suites for failure trace semantics
@article{gazda2023removing,title={Removing redundant refusals: Minimal complete test suites for failure trace semantics},author={Gazda, Maciej and Hierons, Robert M},journal={Information and Computation},volume={291},pages={105009},year={2023},month=mar,doi={10.1016/j.ic.2023.105009},}
Trustworthy Autonomous Systems Through Verifiability
Mohammad Reza Mousavi, Ana Cavalcanti, Michael Fisher, and 8 more authors
@article{mousavi2023trustworthy,title={Trustworthy Autonomous Systems Through Verifiability},author={Mousavi, Mohammad Reza and Cavalcanti, Ana and Fisher, Michael and Dennis, Louise and Hierons, Rob and Kaddouh, Bilal and Law, Effie Lai-Chong and Richardson, Rob and Ringer, Jan Oliver and Tyukin, Ivan and Woodcock, Jim},journal={Computer},volume={56},number={2},pages={40--47},year={2023},month=feb,doi={10.1109/mc.2022.3192206},}
Testing using CSP Models: Time, Inputs, and Outputs
James Baxter, Ana Cavalcanti, Maciej Gazda, and 1 more author
The existing testing theories for CSP cater for verification of interaction patterns (traces) and deadlocks, but not time. We address here refinement and testing based on a dialect of CSP, called tock -CSP, which can capture discrete time properties. This version of CSP has been of widespread interest for decades; recently, it has been given a denotational semantics, and model checking has become possible using a well established tool. Here, we first equip tock -CSP with a novel semantics for testing, which distinguishes input and output events: the standard models of ( tock -)CSP do not differentiate them, but for testing this is essential. We then present a new testing theory for timewise refinement, based on novel definitions of test and test execution. Finally, we reconcile refinement and testing by relating timed ioco testing and refinement in tock -CSP with inputs and outputs. With these results, this paper provides, for the first time, a systematic theory that allows both timed testing and timed refinement to be expressed. An important practical consequence is that this ensures that the notion of correctness used by developers guarantees that tests pass when applied to a correct system and, in addition, faults identified during testing correspond to development mistakes.
@article{baxter2023testing,title={Testing using CSP Models: Time, Inputs, and Outputs},author={Baxter, James and Cavalcanti, Ana and Gazda, Maciej and Hierons, Robert M.},journal={ACM Transactions on Computational Logic},volume={24},number={2},pages={1--40},year={2023},month=jan,doi={10.1145/3572837},}
Implementation Relations for Distributed Testing
Robert M. Hierons, Mercedes G. Merayo, and Manuel Núñez
In Applicable Formal Methods for Safe Industrial Products, 2023
@incollection{hierons2023implementation,title={Implementation Relations for Distributed Testing},author={Hierons, Robert M. and Merayo, Mercedes G. and Núñez, Manuel},booktitle={Applicable Formal Methods for Safe Industrial Products},publisher={Springer Nature Switzerland},pages={34--48},year={2023},doi={10.1007/978-3-031-40132-9_3},}
2022
Scenario-based test reduction and prioritization for multi-module autonomous driving systems
Yao Deng, Xi Zheng, Mengshi Zhang, and 2 more authors
In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Nov 2022
@inproceedings{deng2022scenario,title={Scenario-based test reduction and prioritization for multi-module autonomous driving systems},author={Deng, Yao and Zheng, Xi and Zhang, Mengshi and Lou, Guannan and Zhang, Tianyi},booktitle={Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering},publisher={ACM},pages={82--93},year={2022},month=nov,doi={10.1145/3540250.3549152},}
Testing of autonomous driving systems: where are we and where should we go?
Guannan Lou, Yao Deng, Xi Zheng, and 2 more authors
In Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Nov 2022
@inproceedings{lou2022testing,title={Testing of autonomous driving systems: where are we and where should we go?},author={Lou, Guannan and Deng, Yao and Zheng, Xi and Zhang, Mengshi and Zhang, Tianyi},booktitle={Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering},publisher={ACM},pages={31--43},year={2022},month=nov,doi={10.1145/3540250.3549111},}
Reliable counterparts
Richard J. Somers, Andrew G. Clark, Neil Walkinshaw, and 1 more author
In Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings, Oct 2022
@inproceedings{somers2022reliable,title={Reliable counterparts},author={Somers, Richard J. and Clark, Andrew G. and Walkinshaw, Neil and Hierons, Robert M.},booktitle={Proceedings of the 25th International Conference on Model Driven Engineering Languages and Systems: Companion Proceedings},publisher={ACM},pages={468--472},year={2022},month=oct,doi={10.1145/3550356.3561589},}
Correction to: Can Offline Testing of Deep Neural Networks Replace Their Online Testing?
Fitash Ul Haq, Donghwan Shin, Shiva Nejati, and 1 more author
@article{haq2022correction,title={Correction to: Can Offline Testing of Deep Neural Networks Replace Their Online Testing?},author={Haq, Fitash Ul and Shin, Donghwan and Nejati, Shiva and Briand, Lionel},journal={Empirical Software Engineering},volume={27},number={6},year={2022},month=aug,doi={10.1007/s10664-022-10172-z},}
Maternal hemodynamics and neonatal birth weight in pregnancies complicated by gestational diabetes: new insights from novel causal inference analysis modeling
A. R. Anness, A. Clark, K. Melhuish, and 7 more authors
@article{anness2022maternal,title={Maternal hemodynamics and neonatal birth weight in pregnancies complicated by gestational diabetes: new insights from novel causal inference analysis modeling},author={Anness, A. R. and Clark, A. and Melhuish, K. and Leone, F. M. T. and Osman, M. W. and Webb, D. and Robinson, T. and Walkinshaw, N. and Khalil, A. and Mousa, H. A.},journal={Ultrasound in Obstetrics & Gynecology},volume={60},number={2},pages={215--222},year={2022},month=aug,doi={10.1002/uog.24864},}
An automated framework for verifying or refuting trace properties of extended finite state machines
Model checkers and interactive proof assistants are both used in the assurance of critical systems. Where theorem proving involves the use of axioms and inference rules to mathematically prove defined properties, model checkers can be used to provide concrete counterexamples to refute them. Thus, the two techniques can be thought of as complementary, and it is helpful to use both in tandem to take advantage of their respective strengths. However, this requires us to translate our system model and our desired properties between the two tools which is a time-consuming and error prone process if done manually. The key contribution of this work is a set of automated tools to translate between the Isabelle/HOL proof assistant and the Symbolic Analysis Laboratory (SAL) model checker. We focus on systems specified as extended finite state machines (EFSMs) and on properties specified in linear temporal logic (LTL). We present our representations in the two tools and demonstrate the applicability of our system with respect to an academic example and two realistic case studies. This is a significant contribution to broadening the applicability of these formal approaches, since it allows two powerful verification tools to be easily used in tandem without the risk of human error.
@article{taylor2022automated,title={An automated framework for verifying or refuting trace properties of extended finite state machines},author={Taylor, Ramsay G. and Foster, Michael and North, Siobhán},journal={International Journal on Software Tools for Technology Transfer},volume={24},number={6},pages={949--972},year={2022},month=jul,doi={10.1007/s10009-022-00666-y},}
Efficient online testing for DNN-enabled systems using surrogate-assisted and many-objective optimization
@inproceedings{haq2022efficient,title={Efficient online testing for DNN-enabled systems using surrogate-assisted and many-objective optimization},author={Haq, Fitash Ul and Shin, Donghwan and Briand, Lionel},booktitle={Proceedings of the 44th International Conference on Software Engineering},publisher={ACM},pages={811--822},year={2022},month=may,doi={10.1145/3510003.3510188},}
Farewell after an 11‐year journey as joint editor‐in‐chief
Robert M. Hierons
Software Testing, Verification and Reliability, May 2022
@article{hierons2022farewell,title={Farewell after an 11‐year journey as joint editor‐in‐chief},author={Hierons, Robert M.},journal={Software Testing, Verification and Reliability},volume={32},number={4},year={2022},month=may,doi={10.1002/stvr.1816},}
Guidelines for assessing the accuracy of log message template identification techniques
Zanis Ali Khan, Donghwan Shin, Domenico Bianculli, and 1 more author
In Proceedings of the 44th International Conference on Software Engineering, May 2022
@inproceedings{khan2022guidelines,title={Guidelines for assessing the accuracy of log message template identification techniques},author={Khan, Zanis Ali and Shin, Donghwan and Bianculli, Domenico and Briand, Lionel},booktitle={Proceedings of the 44th International Conference on Software Engineering},publisher={ACM},pages={1095--1106},year={2022},month=may,doi={10.1145/3510003.3510101},}
On the feasibility and challenges of synthesizing executable Espresso tests
Iván Arcuschin, Christian Ciccaroni, Juan Pablo Galeotti, and 1 more author
In Proceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test, May 2022
@inproceedings{arcuschin2022feasibility,title={On the feasibility and challenges of synthesizing executable Espresso tests},author={Arcuschin, Iván and Ciccaroni, Christian and Galeotti, Juan Pablo and Rojas, José Miguel},booktitle={Proceedings of the 3rd ACM/IEEE International Conference on Automation of Software Test},publisher={ACM},pages={92--102},year={2022},month=may,doi={10.1145/3524481.3527234},}
@incollection{foster2022reverse,title={Reverse-Engineering EFSMs with Data Dependencies},author={Foster, Michael and Derrick, John and Walkinshaw, Neil},booktitle={Testing Software and Systems},publisher={Springer International Publishing},pages={37--54},year={2022},month=may,doi={10.1007/978-3-031-04673-5_3},}
Metamorphic testing and test automation
Robert M. Hierons and Tao Xie
Software Testing, Verification and Reliability, Apr 2022
@article{hierons2022metamorphic,title={Metamorphic testing and test automation},author={Hierons, Robert M. and Xie, Tao},journal={Software Testing, Verification and Reliability},volume={32},number={3},year={2022},month=apr,doi={10.1002/stvr.1814},}
PRINS: scalable model inference for component-based system logs
Donghwan Shin, Domenico Bianculli, and Lionel Briand
Behavioral software models play a key role in many software engineering tasks; unfortunately, these models either are not available during software development or, if available, quickly become outdated as implementations evolve. Model inference techniques have been proposed as a viable solution to extract finite state models from execution logs. However, existing techniques do not scale well when processing very large logs that can be commonly found in practice. In this paper, we address the scalability problem of inferring the model of a component-based system from large system logs, without requiring any extra information. Our model inference technique, called PRINS , follows a divide-and-conquer approach. The idea is to first infer a model of each system component from the corresponding logs; then, the individual component models are merged together taking into account the flow of events across components, as reflected in the logs. We evaluated PRINS in terms of scalability and accuracy, using nine datasets composed of logs extracted from publicly available benchmarks and a personal computer running desktop business applications. The results show that PRINS can process large logs much faster than a publicly available and well-known state-of-the-art tool, without significantly compromising the accuracy of inferred models.
@article{shin2022prins,title={PRINS: scalable model inference for component-based system logs},author={Shin, Donghwan and Bianculli, Domenico and Briand, Lionel},journal={Empirical Software Engineering},volume={27},number={4},year={2022},month=apr,doi={10.1007/s10664-021-10111-4},}
An information theoretic notion of software testability
@article{patel2022information,title={An information theoretic notion of software testability},author={Patel, Krishna and Hierons, Robert M. and Clark, David},journal={Information and Software Technology},volume={143},pages={106759},year={2022},month=mar,doi={10.1016/j.infsof.2021.106759},}
Combinatorial testing and model‐based testing
Robert M. Hierons and Tao Xie
Software Testing, Verification and Reliability, Feb 2022
@article{hierons2022combinatorial,title={Combinatorial testing and model‐based testing},author={Hierons, Robert M. and Xie, Tao},journal={Software Testing, Verification and Reliability},volume={32},number={2},year={2022},month=feb,doi={10.1002/stvr.1810},}
An Empirical Comparison of EvoSuite and DSpot for Improving Developer-Written Test Suites with Respect to Mutation Score
@incollection{roslan2022empirical,title={An Empirical Comparison of EvoSuite and DSpot for Improving Developer-Written Test Suites with Respect to Mutation Score},author={Roslan, Muhammad Firhard and Rojas, José Miguel and McMinn, Phil},booktitle={Search-Based Software Engineering},publisher={Springer International Publishing},pages={19--34},year={2022},doi={10.1007/978-3-031-21251-2_2},}
2021
Deep Learning-Based Autonomous Driving Systems: A Survey of Attacks and Defenses
Yao Deng, Tiehua Zhang, Guannan Lou, and 3 more authors
IEEE Transactions on Industrial Informatics, Dec 2021
@article{deng2021deep,title={Deep Learning-Based Autonomous Driving Systems: A Survey of Attacks and Defenses},author={Deng, Yao and Zhang, Tiehua and Lou, Guannan and Zheng, Xi and Jin, Jiong and Han, Qing-Long},journal={IEEE Transactions on Industrial Informatics},volume={17},number={12},pages={7897--7912},year={2021},month=dec,doi={10.1109/tii.2021.3071405},}
Model checking, testing and debugging
Robert M. Hierons and Tao Xie
Software Testing, Verification and Reliability, Dec 2021
@article{hierons2021model,title={Model checking, testing and debugging},author={Hierons, Robert M. and Xie, Tao},journal={Software Testing, Verification and Reliability},volume={32},number={1},year={2021},month=dec,doi={10.1002/stvr.1803},}
Efficient state synchronisation in model-based testing through reinforcement learning
Uraz Cengiz Turker, Robert M. Hierons, Mohammad Reza Mousavi, and 1 more author
In 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE), Nov 2021
@inproceedings{turker2021efficient,title={Efficient state synchronisation in model-based testing through reinforcement learning},author={Turker, Uraz Cengiz and Hierons, Robert M. and Mousavi, Mohammad Reza and Tyukin, Ivan Y.},booktitle={2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE)},publisher={IEEE},pages={368--380},year={2021},month=nov,doi={10.1109/ase51524.2021.9678566},}
Property generation/verification and empirical studies
Robert M. Hierons and Tao Xie
Software Testing, Verification and Reliability, Nov 2021
@article{hierons2021property,title={Property generation/verification and empirical studies},author={Hierons, Robert M. and Xie, Tao},journal={Software Testing, Verification and Reliability},volume={31},number={8},year={2021},month=nov,doi={10.1002/stvr.1800},}
Adaptive or embedded software testing and mutation testing
Robert M. Hierons and Tao Xie
Software Testing, Verification and Reliability, Oct 2021
@article{hierons2021adaptive,title={Adaptive or embedded software testing and mutation testing},author={Hierons, Robert M. and Xie, Tao},journal={Software Testing, Verification and Reliability},volume={31},number={7},year={2021},month=oct,doi={10.1002/stvr.1798},}
VP34.05: The influence of maternal hemodynamics on neonatal birthweight in pregnancies complicated by gestational diabetes compared to low‐risk controls
A. Anness, A. Clark, K. Melhuish, and 7 more authors
@article{anness2021vp34,title={VP34.05: The influence of maternal hemodynamics on neonatal birthweight in pregnancies complicated by gestational diabetes compared to low‐risk controls},author={Anness, A. and Clark, A. and Melhuish, K. and Osman, M.W. and Leone, F. and Webb, D. and Robinson, T. and Walkinshaw, N. and Khalil, A. and Mousa, H.},journal={Ultrasound in Obstetrics & Gynecology},volume={58},number={S1},pages={89--312},year={2021},month=oct,doi={10.1002/uog.24521},}
Are 20% of Classes Responsible for 80% of Refactorings?
Steve Counsell, Robert M. Hierons, and Krishna Patel
In 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), Sep 2021
@inproceedings{counsell2021are,title={Are 20% of Classes Responsible for 80% of Refactorings?},author={Counsell, Steve and Hierons, Robert M. and Patel, Krishna},booktitle={2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA)},publisher={IEEE},pages={287--290},year={2021},month=sep,doi={10.1109/seaa53835.2021.00043},}
Editorial: Verification, reliability and performance
Robert M. Hierons and Tao Xie
Software Testing, Verification and Reliability, Aug 2021
@article{hierons2021editorial,title={Editorial: Verification, reliability and performance},author={Hierons, Robert M. and Xie, Tao},journal={Software Testing, Verification and Reliability},volume={31},number={6},year={2021},month=aug,doi={10.1002/stvr.1790},}
Minimizing Characterizing sets
Uraz Cengiz Türker, Robert M. Hierons, and Guy-Vincent Jourdan
@article{cengizturker2021minimizing,title={Minimizing Characterizing sets},author={Cengiz Türker, Uraz and Hierons, Robert M. and Jourdan, Guy-Vincent},journal={Science of Computer Programming},volume={208},pages={102645},year={2021},month=aug,doi={10.1016/j.scico.2021.102645},}
@article{hierons2021preface,title={Preface},author={Hierons, Robert M. and Mosbah, Mohamed},journal={Theoretical Computer Science},volume={880},pages={138},year={2021},month=aug,doi={10.1016/j.tcs.2021.07.001},}
Automatic test suite generation for key-points detection DNNs using many-objective search (experience paper)
Fitash Ul Haq, Donghwan Shin, Lionel C. Briand, and 2 more authors
In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, Jul 2021
@inproceedings{haq2021automatic,title={Automatic test suite generation for key-points detection DNNs using many-objective search (experience paper)},author={Haq, Fitash Ul and Shin, Donghwan and Briand, Lionel C. and Stifter, Thomas and Wang, Jun},booktitle={Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis},publisher={ACM},pages={91--102},year={2021},month=jul,doi={10.1145/3460319.3464802},}
Can Offline Testing of Deep Neural Networks Replace Their Online Testing?
Fitash Ul Haq, Donghwan Shin, Shiva Nejati, and 1 more author
We distinguish two general modes of testing for Deep Neural Networks (DNNs): Offline testing where DNNs are tested as individual units based on test datasets obtained without involving the DNNs under test, and online testing where DNNs are embedded into a specific application environment and tested in a closed-loop mode in interaction with the application environment. Typically, DNNs are subjected to both types of testing during their development life cycle where offline testing is applied immediately after DNN training and online testing follows after offline testing and once a DNN is deployed within a specific application environment. In this paper, we study the relationship between offline and online testing. Our goal is to determine how offline testing and online testing differ or complement one another and if offline testing results can be used to help reduce the cost of online testing? Though these questions are generally relevant to all autonomous systems, we study them in the context of automated driving systems where, as study subjects, we use DNNs automating end-to-end controls of steering functions of self-driving vehicles. Our results show that offline testing is less effective than online testing as many safety violations identified by online testing could not be identified by offline testing, while large prediction errors generated by offline testing always led to severe safety violations detectable by online testing. Further, we cannot exploit offline testing results to reduce the cost of online testing in practice since we are not able to identify specific situations where offline testing could be as accurate as online testing in identifying safety requirement violations.
@article{haq2021can,title={Can Offline Testing of Deep Neural Networks Replace Their Online Testing?},author={Haq, Fitash Ul and Shin, Donghwan and Nejati, Shiva and Briand, Lionel},journal={Empirical Software Engineering},volume={26},number={5},year={2021},month=jul,doi={10.1007/s10664-021-09982-4},}
Log-based slicing for system-level test cases
Salma Messaoudi, Donghwan Shin, Annibale Panichella, and 2 more authors
In Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis, Jul 2021
@inproceedings{messaoudi2021log,title={Log-based slicing for system-level test cases},author={Messaoudi, Salma and Shin, Donghwan and Panichella, Annibale and Bianculli, Domenico and Briand, Lionel C.},booktitle={Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis},publisher={ACM},pages={517--528},year={2021},month=jul,doi={10.1145/3460319.3464824},}
Test case generation for agent-based models: A systematic literature review
@article{clark2021test,title={Test case generation for agent-based models: A systematic literature review},author={Clark, Andrew G. and Walkinshaw, Neil and Hierons, Robert M.},journal={Information and Software Technology},volume={135},pages={106567},year={2021},month=jul,doi={10.1016/j.infsof.2021.106567},}
mathcal k-branching uio sequences for partially specified observable non-deterministic fsms
Khaled El-Fakih, Robert M. Hierons, and Uraz Cengiz Turker
IEEE Transactions on Software Engineering, May 2021
@article{elfakih2021mathcal,title={$mathcal k$-branching uio sequences for partially specified observable non-deterministic fsms},author={El-Fakih, Khaled and Hierons, Robert M. and Turker, Uraz Cengiz},journal={IEEE Transactions on Software Engineering},volume={47},number={5},pages={1029--1040},year={2021},month=may,doi={10.1109/tse.2019.2911076},}
Editorial: Testing, Debugging, and Defect Prediction
Robert M. Hierons and Tao Xie
Software Testing, Verification and Reliability, May 2021
@article{hierons2021editorialb,title={Editorial: Testing, Debugging, and Defect Prediction},author={Hierons, Robert M. and Xie, Tao},journal={Software Testing, Verification and Reliability},volume={31},number={5},year={2021},month=may,doi={10.1002/stvr.1775},}
Digital Twins Are Not Monozygotic – Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulators
Markus Borg, Raja Ben Abdessalem, Shiva Nejati, and 2 more authors
In 2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST), Apr 2021
@inproceedings{borg2021digital,title={Digital Twins Are Not Monozygotic – Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulators},author={Borg, Markus and Abdessalem, Raja Ben and Nejati, Shiva and Jegeden, Francois-Xavier and Shin, Donghwan},booktitle={2021 14th IEEE Conference on Software Testing, Verification and Validation (ICST)},publisher={IEEE},pages={383--393},year={2021},month=apr,doi={10.1109/icst49551.2021.00050},}
Using mutual information to test from Finite State Machines: Test suite selection
Alfredo Ibias, Manuel Núñez, and Robert M. Hierons
@article{ibias2021mutual,title={Using mutual information to test from Finite State Machines: Test suite selection},author={Ibias, Alfredo and Núñez, Manuel and Hierons, Robert M.},journal={Information and Software Technology},volume={132},pages={106498},year={2021},month=apr,doi={10.1016/j.infsof.2020.106498},}
TEA- Cloud : A Formal Framework for Testing Cloud Computing Systems
Alberto Nunez, Pablo C. Canizares, Manuel Nunez, and 1 more author
@article{nunez2021tea,title={TEA- Cloud : A Formal Framework for Testing Cloud Computing Systems},author={Nunez, Alberto and Canizares, Pablo C. and Nunez, Manuel and Hierons, Robert M.},journal={IEEE Transactions on Reliability},volume={70},number={1},pages={261--284},year={2021},month=mar,doi={10.1109/tr.2020.3011512},}
Automated visual classification of DOM‐based presentation failure reports for responsive web pages
Ibrahim Althomali, Gregory M. Kapfhammer, and Phil McMinn
Software Testing, Verification and Reliability, Feb 2021
Summary Since it is common for the users of a web page to access it through a wide variety of devices—including desktops, laptops, tablets and phones—web developers rely on responsive web design (RWD) principles and frameworks to create sites that are useful on all devices. A correctly implemented responsive web page adjusts its layout according to the viewport width of the device in use, thereby ensuring that its design suitably features the content. Since the use of complex RWD frameworks often leads to web pages with hard‐to‐detect responsive layout failures (RLFs), developers employ testing tools that generate reports of potential RLFs. Since testing tools for responsive web pages, like Re DeCheck , analyse a web page representation called the Document Object Model (DOM), they may inadvertently flag concerns that are not human visible, thereby requiring developers to manually confirm and classify each potential RLF as a true positive (TP), false positive (FP), or non‐observable issue (NOI)—a process that is time consuming and error prone. The conference version of this paper presented V iser , a tool that automatically classified three types of RLFs reported by Re DeCheck . Since V iser was not designed to automatically confirm and classify two types of RLFs that Re DeCheck ’s DOM‐based analysis could surface, this paper introduces V erve , a tool that automatically classifies all RLF types reported by Re DeCheck . Along with manipulating the opacity of HTML elements in a web page, as does V iser , the V erve tool also uses histogram‐based image comparison to classify RLFs in web pages. Incorporating both the 25 web pages used in prior experiments and 20 new pages not previously considered, this paper’s empirical study reveals that V erve ’s classification of all five types of RLFs frequently agrees with classifications produced manually by humans. The experiments also reveal that V erve took on average about 4 s to classify any of the RLFs among the 469 reported by Re DeCheck . Since this paper demonstrates that classifying an RLF as a TP, FP, or NOI with V erve , a publicly available tool, is less subjective and error prone than the same manual process done by a human web developer, we argue that it is well‐suited for supporting the testing of complex responsive web pages.
@article{althomali2021automated,title={Automated visual classification of DOM‐based presentation failure reports for responsive web pages},author={Althomali, Ibrahim and Kapfhammer, Gregory M. and McMinn, Phil},journal={Software Testing, Verification and Reliability},volume={31},number={4},year={2021},month=feb,doi={10.1002/stvr.1756},}
A Theoretical Framework for Understanding the Relationship Between Log Parsing and Anomaly Detection
Donghwan Shin, Zanis Ali Khan, Domenico Bianculli, and 1 more author
@incollection{shin2021theoretical,title={A Theoretical Framework for Understanding the Relationship Between Log Parsing and Anomaly Detection},author={Shin, Donghwan and Khan, Zanis Ali and Bianculli, Domenico and Briand, Lionel},booktitle={Runtime Verification},publisher={Springer International Publishing},pages={277--287},year={2021},doi={10.1007/978-3-030-88494-9_16},}
Software Engineering for Robotics
Ana Cavalcanti, Brijesh Dongol, Rob Hierons, and 2 more authors
@book{cavalcanti2021software,title={Software Engineering for Robotics},author={Cavalcanti, Ana and Dongol, Brijesh and Hierons, Rob and Timmis, Jon and Woodcock, Jim},publisher={Springer International Publishing},year={2021},doi={10.1007/978-3-030-66494-7},}
2020
Four-Valued Monitorability of $omega $-Regular Languages
Zhe Chen, Yunyun Chen, Robert M. Hierons, and 1 more author
In Formal Methods and Software Engineering, Dec 2020
@incollection{chen2020four,title={Four-Valued Monitorability of $$omega $$-Regular Languages},author={Chen, Zhe and Chen, Yunyun and Hierons, Robert M. and Wu, Yifan},booktitle={Formal Methods and Software Engineering},publisher={Springer International Publishing},pages={198--214},year={2020},month=dec,doi={10.1007/978-3-030-63406-3_12},}
Implementation relations and testing for cyclic systems with refusals and discrete time
Raluca Lefticaru, Robert M. Hierons, and Manuel Núñez
@article{lefticaru2020implementation,title={Implementation relations and testing for cyclic systems with refusals and discrete time},author={Lefticaru, Raluca and Hierons, Robert M. and Núñez, Manuel},journal={Journal of Systems and Software},volume={170},pages={110738},year={2020},month=dec,doi={10.1016/j.jss.2020.110738},}
Mutation Testing for RoboChart
Robert M. Hierons, Maciej Gazda, Pablo Gómez-Abajo, and 2 more authors
@incollection{hierons2020mutation,title={Mutation Testing for RoboChart},author={Hierons, Robert M. and Gazda, Maciej and Gómez-Abajo, Pablo and Lefticaru, Raluca and Merayo, Mercedes G.},booktitle={Software Engineering for Robotics},publisher={Springer International Publishing},pages={345--375},year={2020},month=dec,doi={10.1007/978-3-030-66494-7_11},}
In this short article, we discuss perspectives on issues related to peer reviewing in software engineering journals and conferences. These perspectives are based on our experiences, which have been informed by being authors, reviewers, conference organisers, members of journal editorial boards, and also co-editors-in-chief of a journal, i.e., The Journal of Software Testing, Verification and Reliability (STVR).
@article{hierons2020peer,title={Peer Reviewing in Software Engineering},author={Hierons, Robert M. and Xie, Tao},journal={ACM SIGSOFT Software Engineering Notes},volume={45},number={4},pages={18--18},year={2020},month=oct,doi={10.1145/3417564.3417568},}
Teaching Software Testing with the Code Defenders Testing Game: Experiences and Improvements
Gordon Fraser, Alessio Gambi, and Jose Miguel Rojas
In 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), Oct 2020
@inproceedings{fraser2020teaching,title={Teaching Software Testing with the Code Defenders Testing Game: Experiences and Improvements},author={Fraser, Gordon and Gambi, Alessio and Rojas, Jose Miguel},booktitle={2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW)},publisher={IEEE},pages={461--464},year={2020},month=oct,doi={10.1109/icstw50294.2020.00082},}
Effective automated repair of internationalization presentation failures in web applications using style similarity clustering and search‐based techniques
Sonal Mahajan, Abdulmajeed Alameer, Phil McMinn, and 1 more author
Software Testing, Verification and Reliability, Sep 2020
Summary Companies often employ (i18n) frameworks to provide translated text and localized media content on their websites in order to effectively communicate with a global audience. However, the varying lengths of text from different languages can cause undesired distortions in the layout of a web page. Such distortions, called Internationalization Presentation Failures (IPFs), can negatively affect the aesthetics or usability of the website. Most of the existing automated techniques developed for assisting repair of IPFs either produce fixes that are likely to significantly reduce the legibility and attractiveness of the pages or are limited to only detecting IPFs, with the actual repair itself remaining a labour intensive manual task. To address this problem, we propose a search‐based technique for automatically repairing IPFs in web applications, while ensuring a legible and attractive page. The empirical evaluation of our approach reported that our approach was able to successfully resolve 94% of the detected IPFs for 46 real‐world web pages. In a user study, participants rated the visual quality of our fixes significantly higher than the unfixed versions and also considered the repairs generated by our approach to be notably more legible and visually appealing than the repairs generated by existing techniques.
@article{mahajan2020effective,title={Effective automated repair of internationalization presentation failures in web applications using style similarity clustering and search‐based techniques},author={Mahajan, Sonal and Alameer, Abdulmajeed and McMinn, Phil and Halfond, William G. J.},journal={Software Testing, Verification and Reliability},volume={31},number={1-2},year={2020},month=sep,doi={10.1002/stvr.1746},}
Automatically identifying potential regressions in the layout of responsive web pages
Thomas A. Walsh, Gregory M. Kapfhammer, and Phil McMinn
Software Testing, Verification and Reliability, Aug 2020
Summary Providing a good user experience on the ever‐increasing number and variety of devices being used to browse the web is a difficult, yet critical, task. With responsive web design, front‐end web developers design web pages so that they dynamically resize and rearrange content to best fit the dimensions of a device’s screen. However, when making code modifications to a responsive page, developers can easily introduce regressions from the correct layout that have detrimental effects at unpredictable screen sizes. For instance, the source code change that a developer makes to improve the layout at one screen size may obscure a page’s content at other sizes. Current approaches to testing are often insufficient because they rely on limited tools and error‐prone manual inspections of web pages. As such, many unintended regressions in web page layout often go undetected and ultimately manifest in production websites. To address the challenge of detecting regressions in responsive web pages, this paper presents an automated approach that extracts the responsive layout of two versions of a page and compares them, alerting developers to the differences in layout that they may wish to investigate further. We implemented the approach and empirically evaluated it on 15 real‐world responsive web pages. Leveraging code mutations that a tool automatically injected into the pages as a systematic simulation of developer changes, the experiments show that the approach was highly effective. When compared with manual and automated baseline testing techniques, it detected 12.5% and 18.75% more injected changes, respectively. Along with identifying the best parameters for the method that extracts the responsive layout, the experiments show that the approach surpasses the baselines across changes that vary in their impact, but works particularly well for subtle, hard‐to‐detect mutants, showing the benefits of automatically identifying regressions in web page layout.
@article{walsh2020automatically,title={Automatically identifying potential regressions in the layout of responsive web pages},author={Walsh, Thomas A. and Kapfhammer, Gregory M. and McMinn, Phil},journal={Software Testing, Verification and Reliability},volume={30},number={6},year={2020},month=aug,doi={10.1002/stvr.1748},}
Improving Automated GUI Testing by Learning to Avoid Infeasible Tests
@inproceedings{walkinshaw2020improving,title={Improving Automated GUI Testing by Learning to Avoid Infeasible Tests},author={Walkinshaw, Neil},booktitle={2020 IEEE International Conference On Artificial Intelligence Testing (AITest)},publisher={IEEE},pages={107--114},year={2020},month=aug,doi={10.1109/aitest49225.2020.00023},}
On automation in software engineering
Robert M. Hierons and Tao Xie
Software Testing, Verification and Reliability, Aug 2020
@article{hierons2020automation,title={On automation in software engineering},author={Hierons, Robert M. and Xie, Tao},journal={Software Testing, Verification and Reliability},volume={30},number={6},year={2020},month=aug,doi={10.1002/stvr.1753},}
Conference Virtualization
Robert M. Hierons and Tao Xie
Software Testing, Verification and Reliability, Jun 2020
@article{hierons2020conference,title={Conference Virtualization},author={Hierons, Robert M. and Xie, Tao},journal={Software Testing, Verification and Reliability},volume={30},number={4-5},year={2020},month=jun,doi={10.1002/stvr.1749},}
Do sophisticated evolutionary algorithms perform better than simple ones?
Michael Foster, Matthew Hughes, George O’Brien, and 4 more authors
In Proceedings of the 2020 Genetic and Evolutionary Computation Conference, Jun 2020
@inproceedings{foster2020do,title={Do sophisticated evolutionary algorithms perform better than simple ones?},author={Foster, Michael and Hughes, Matthew and O'Brien, George and Oliveto, Pietro S. and Pyle, James and Sudholt, Dirk and Williams, James},booktitle={Proceedings of the 2020 Genetic and Evolutionary Computation Conference},publisher={ACM},pages={184--192},year={2020},month=jun,doi={10.1145/3377930.3389830},}
Motivating Adult Learners by Introducing Programming Concepts with Scratch
@inproceedings{krafft2020motivating,title={Motivating Adult Learners by Introducing Programming Concepts with Scratch},author={Krafft, Maren and Fraser, Gordon and Walkinshaw, Neil},booktitle={Proceedings of the 4th European Conference on Software Engineering Education},publisher={ACM},pages={22--26},year={2020},month=jun,doi={10.1145/3396802.3396818},}
Inputs and Outputs in CSP
Ana Cavalcanti, Robert M. Hierons, and Sidney Nogueira
This article addresses refinement and testing based on CSP models, when we distinguish input and output events. In a testing experiment, the tester (or the environment) controls the inputs, and the system under test controls the outputs. The standard models and refinement relations of CSP, however, do not differentiate inputs and outputs and are not, therefore, entirely suitable for testing. Here, we consider an alphabet of events partitioned into inputs and outputs, and we present a novel refusal-testing model for CSP with a notion of input-output refusal-traces refinement. We compare that with the ioco relation often used in testing, and we find that it is more widely applicable and stronger. This means that mistakes found using traditional ioco testing do indicate mistakes in the development. Finally, we provide a CSP testing theory that takes into account inputs and outputs. With our theory, it becomes feasible to develop techniques and tools for automatic generation of realistic and sound tests from CSP models. Our work reconciles the normally disparate areas of refinement and (formal) testing by identifying how ioco testing can be used to inform refinement-based results and vice-versa.
@article{cavalcanti2020inputs,title={Inputs and Outputs in CSP},author={Cavalcanti, Ana and Hierons, Robert M. and Nogueira, Sidney},journal={ACM Transactions on Computational Logic},volume={21},number={3},pages={1--53},year={2020},month=may,doi={10.1145/3379508},}
@inproceedings{walkinshaw2020reasoning,title={Reasoning about Uncertainty in Empirical Results},author={Walkinshaw, Neil and Shepperd, Martin},booktitle={Proceedings of the Evaluation and Assessment in Software Engineering},publisher={ACM},pages={140--149},year={2020},month=apr,doi={10.1145/3383219.3383234},}
Working Across Boundaries
Robert M. Hierons and Tao Xie
Software Testing, Verification and Reliability, Apr 2020
@article{hierons2020working,title={Working Across Boundaries},author={Hierons, Robert M. and Xie, Tao},journal={Software Testing, Verification and Reliability},volume={30},number={3},year={2020},month=apr,doi={10.1002/stvr.1734},}
An Analysis of Adversarial Attacks and Defenses on Autonomous Driving Models
Yao Deng, Xi Zheng, Tianyi Zhang, and 3 more authors
In 2020 IEEE International Conference on Pervasive Computing and Communications (PerCom), Mar 2020
@inproceedings{deng2020analysis,title={An Analysis of Adversarial Attacks and Defenses on Autonomous Driving Models},author={Deng, Yao and Zheng, Xi and Zhang, Tianyi and Chen, Chen and Lou, Guannan and Kim, Miryung},booktitle={2020 IEEE International Conference on Pervasive Computing and Communications (PerCom)},publisher={IEEE},pages={1--10},year={2020},month=mar,doi={10.1109/percom45495.2020.9127389},}
Guest Editorial: Special Section on ICTSS
Inmaculada Medina-Bulo, Mercedes G. Merayo, and Robert M. Hierons
@article{medinabulo2020guest,title={Guest Editorial: Special Section on ICTSS},author={Medina-Bulo, Inmaculada and Merayo, Mercedes G. and Hierons, Robert M.},journal={Information and Software Technology},volume={118},pages={106222},year={2020},month=feb,doi={10.1016/j.infsof.2019.106222},}
Many-Objective Test Suite Generation for Software Product Lines
Robert M. Hierons, Miqing Li, Xiaohui Liu, and 3 more authors
ACM Transactions on Software Engineering and Methodology, Jan 2020
A Software Product Line (SPL) is a set of products built from a number of features, the set of valid products being defined by a feature model. Typically, it does not make sense to test all products defined by an SPL and one instead chooses a set of products to test (test selection) and, ideally, derives a good order in which to test them (test prioritisation). Since one cannot know in advance which products will reveal faults, test selection and prioritisation are normally based on objective functions that are known to relate to likely effectiveness or cost. This article introduces a new technique, the grid-based evolution strategy (GrES), which considers several objective functions that assess a selection or prioritisation and aims to optimise on all of these. The problem is thus a many-objective optimisation problem. We use a new approach, in which all of the objective functions are considered but one (pairwise coverage) is seen as the most important. We also derive a novel evolution strategy based on domain knowledge. The results of the evaluation, on randomly generated and realistic feature models, were promising, with GrES outperforming previously proposed techniques and a range of many-objective optimisation algorithms.
@article{hierons2020many,title={Many-Objective Test Suite Generation for Software Product Lines},author={Hierons, Robert M. and Li, Miqing and Liu, Xiaohui and Parejo, Jose Antonio and Segura, Sergio and Yao, Xin},journal={ACM Transactions on Software Engineering and Methodology},volume={29},number={1},pages={1--46},year={2020},month=jan,doi={10.1145/3361146},}
Local Observability and Controllability Analysis and Enforcement in Distributed Testing With Time Constraints
@article{lima2020local,title={Local Observability and Controllability Analysis and Enforcement in Distributed Testing With Time Constraints},author={Lima, Bruno and Faria, Joao Pascoal and Hierons, Robert},journal={IEEE Access},volume={8},pages={167172--167191},year={2020},doi={10.1109/access.2020.3021858},}