Utilizing Process Mining for Insights on User Behavior in Expert Software

Usability is an important software quality attribute that ensures that users can execute tasks in an application without hindrance or hesitation (van Eck et al., 2019). Incorporating usability in software is achieved by implementing a user-centered design process. However, it often turns out to be rather difficult to integrate such a process into a software company as such a process involves user studies. A central problem in this step is to acquire suitable test subjects, i.e. either the end users or experts of the software tool. This leads to a passive approach to gather insights on the product usage of customers: collecting telemetry data. Telemetry includes the gathering, transmission and processing of data about user behavior in a production system. Its goal is to provide feedback that improves the ability to make decisions about this system (Riedesel, 2021; Zhang et al., 2016). At Vector Informatik GmbH, the collection of telemetry data recently has been implemented in one of their expert tools, the DaVinci Configurator Classic. The analysis of the telemetry data should provide an overview of how the customers use the tools. One approach to gain more insights on the product usage is to use process mining algorithms on this data. Process mining is used to identify and understand bottlenecks, inefficiencies, derivations and risks and provides three use cases: process discovery, conformance checking and the enhancement of process models (Theis & Darabi, 2019; W. van der Aalst, 2012; W. van der Aalst et al., 2012; W. M. P. van der Aalst, 2013). Process mining emerged from business process management, but was applied to software in the past too (Astromskis et al., 2015; Liu et al., 2018; Rubin et al., 2014). Using this approach, Astromskis et al. (2015) as well as Theis and Darabi (2019) were able to gain insights on feature usage, sequence operations, task failures or usability issues. However, it remains unclear whether similar conclusions can be drawn from process mining results of telemetry data and the conduction of the traditional usability user study.

The thesis aims to use a process mining approach on telemetry data to derive meaningful and reliable insights on the usage of an expert tool developed by Vector Informatik GmbH. This includes the investigation of workflows and the derivation of possible usability issues and task failures for this application as well as discovering the best fitting process mining algorithm. To achieve an evidence-based comparison of the process mining outcome to results of a usability study, a user study is conducted.

Literature Research & Fundamentals
• Evaluating previous work on process mining approaches applied to software
• Extracting most important workflows via expert interviews and documentation
Implementation
• Extraction of raw telemetry data
• Clustering workflows / creating models of user behavior with process mining
• Evaluation of different process mining algorithms
User Study
• Design a user study, the outcome of which can then be compared to the process mining results
• Execute the study
• Analyze the results and compare to process mining results
Thesis writing

Astromskis, S., Janes, A., & Mairegger, M. (2015). A process mining approach to measure how users interact with software: An industrial case study. Proceedings of the 2015 International Conference on Software and System Process, 137–141. https://doi.org/10.1145/2785592.2785612
Liu, C., Wang, S., Gao, S., Zhang, F., & Cheng, J. (2018). User behavior discovery from low-level software execution log. IEEJ Transactions on Electrical and Electronic Engineering, 13(11), 1624–1632. https://doi.org/10.1002/tee.22727
Riedesel, J. (2021). Software Telemetry: Reliable logging and monitoring. Simon and Schuster.
Rubin, V. A., Mitsyuk, A. A., Lomazova, I. A., & van der Aalst, W. M. P. (2014). Process mining can be applied to software too! Proceedings of the 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 1–8. https://doi.org/10.1145/2652524.2652583
Theis, J., & Darabi, H. (2019). Behavioral Petri Net Mining and Automated Analysis for Human-Computer Interaction Recommendations in Multi-Application Environments. Proc. ACM Hum.-Comput. Interact., 3(EICS), 13:1-13:16. https://doi.org/10.1145/3331155
van der Aalst, W. (2012). Process Mining: Overview and Opportunities. ACM Trans. Manage. Inf. Syst., 3(2), 7:1-7:17. https://doi.org/10.1145/2229156.2229157
van der Aalst, W., Adriansyah, A., de Medeiros, A. K. A., Arcieri, F., Baier, T., Blickle, T., Bose, J. C., van den Brand, P., Brandtjen, R., Buijs, J., Burattin, A., Carmona, J., Castellanos, M., Claes, J., Cook, J., Costantini, N., Curbera, F., Damiani, E., de Leoni, M., … Wynn, M. (2012). Process Mining Manifesto. In F. Daniel, K. Barkaoui, & S. Dustdar (Eds.), Business Process Management Workshops (pp. 169–194). Springer. https://doi.org/10.1007/978-3-642-28108-2_19
van der Aalst, W. M. P. (2013). ‘Mine your own business’: Using process mining to turn big data into real value: 21st European Conference on Information Systems (ECIS 2013). 21st European Conference on Information Systems (ECIS 2013, Utrecht, The Netherlands, June 5-8, 2013), 1–9.
van Eck, M. L., Markslag, E., Sidorova, N., Brosens-Kessels, A., & van der Aalst, W. M. P. (2019). Data-Driven Usability Test Scenario Creation. In C. Bogdan, K. Kuusinen, M. K. Lárusdóttir, P. Palanque, & M. Winckler (Eds.), Human-Centered Software Engineering (pp. 88–108). Springer International Publishing. https://doi.org/10.1007/978-3-030-05909-5_6
Zhang, X., Brown, H.-F., & Shankar, A. (2016). Data-driven Personas: Constructing Archetypal Users with Clickstreams and User Telemetry. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 5350–5359. https://doi.org/10.1145/2858036.2858523