Evaluation of Process Mining Algorithms

MACSPro'2019 - Modeling and Analysis of Complex Systems and Processes, Vienna
21 - 23 March 2019

Prof. Dr. Jan Mendling

Conference website http://macspro.club/

Website https://exactpro.com/
Linkedin https://www.linkedin.com/company/exactpro-systems-llc
Instagram https://www.instagram.com/exactpro/
Twitter https://twitter.com/exactpro
Facebook https://www.facebook.com/exactpro/
Youtube Channel https://www.youtube.com/c/exactprosystems

Transcript

1. How to Evaluate Process Mining Algorithms Prof. Dr. Jan Mendling
2. 1. What is Process Mining? 2. Current Evaluation Practices in Process Mining? 3. Methodological Frameworks for Evaluating Process Mining Algorithm 4. Epistemological Tasks for Process Mining Algorithms 5. Properties of Algorithms and Datasets 6. Research Framework for Process Mining Algorithms Agenda SEITE 2
3. 1. What is Process Mining? 2. Current Evaluation Practices in Process Mining? 3. Methodological Frameworks for Evaluating Process Mining Algorithm 4. Epistemological Tasks for Process Mining Algorithms 5. Properties of Algorithms and Datasets 6. Research Framework for Process Mining Algorithms Agenda SEITE 3
4. The BPM Lifecycle SEITE 4
5. What is Process Mining? SEITE 5
6. Process Mining 6 / event log discovered model Discovery Conformance Deviance Difference diagnostics Performance input model Enhanced model event log’
7. •Apromore •ProM Open-source •Disco Lightweight •Minit •myInvenio •QPR Process Analyzer •Signavio Process Intelligence •StereoLOGIC Discovery Analyst •Lana Labs Mid-range •ARIS Process Performance Manager •Celonis Process Mining •Perceptive Process Mining (Lexmark) •Interstage Process Discovery (Fujitsu) Heavyweight Process Mining Tools 7
8. Automated Process Discovery 8 CID Task Time Stamp … 13219 Enter Loan Application 2007-11-09 T 11:20:10 - 13219 Retrieve Applicant Data 2007-11-09 T 11:22:15 - 13220 Enter Loan Application 2007-11-09 T 11:22:40 - 13219 Compute Installments 2007-11-09 T 11:22:45 - 13219 Notify Eligibility 2007-11-09 T 11:23:00 - 13219 Approve Simple Application 2007-11-09 T 11:24:30 - 13220 Compute Installements 2007-11-09 T 11:24:35 - … … … …
9. α-miner Basic Idea: Ordering relations 9 • Direct succession: x>y iff for some case x is directly followed by y. • Causality: x→y iff x>y and not y>x. • Parallel: x||y iff x>y and y>x • Unrelated: x#y iff not x>y and not y>x. case 1 : task A case 2 : task A case 3 : task A case 3 : task B case 1 : task B case 1 : task C case 2 : task C case 4 : task A case 2 : task B ... A>B A>C B>C B>D C>B C>D E>F A→B A→C B→D C→D E→F B||C C||B ABCD ACBD EF
10. Heuristic Miner
11. Process Model discovered with Heuristics Miner
12. Process Model discovered with Inductive Miner
13. Process Model discovered with Structured Miner
14. 1. What is Process Mining? 2. Current Evaluation Practices in Process Mining? 3. Methodological Frameworks for Evaluating Process Mining Algorithm 4. Epistemological Tasks for Process Mining Algorithms 5. Properties of Algorithms and Datasets 6. Research Framework for Process Mining Algorithms Agenda SEITE 14
15. Augusto et al: Automated Discovery of Process Models from Event Logs: Review and Benchmark, TKDE (2019) Model Type Model Language Constructs Implementation Evaluation
16. • 31 of 35 algorithms evaluated with real-life logs • 11 of 35 with synthetic logs • 13 of 35 with artificial logs “... we observed a growing trend in employing publicly-available logs, as opposed to private logs which hamper the replicability of the results due to not being accessible.” “… several methods used a selection of the logs made available by the Business Process Intelligence Challenge (BPIC).” “… great majority of the methods do not have a working or available implementation.” (Augusto et al. 2017) Type of Evaluation Data
17. “Another limitation is the use of only 24 event logs, which to some extent limits the generalizability of the conclusions. However, the event logs included in the evaluation are all real-life logs of different sizes and features, including different application domains. To mitigate this limitation, we have structured the released benchmarking toolset in such a way that the benchmark can be seamlessly rerun with additional datasets.” (Augusto et al. 2017) Can we do better? Threats to Validity
18. 1. What is Process Mining? 2. Current Evaluation Practices in Process Mining? 3. Methodological Frameworks for Evaluating Algorithm 4. Epistemological Tasks for Process Mining Algorithms 5. Properties of Algorithms and Datasets 6. Research Framework for Process Mining Algorithms Agenda SEITE 18
19. Science of the Artifical SEITE 19
20. Design Science Process SEITE 20 Peffers et al. 2008
21. Artefact: sth. that is or can be transformed into matter Design theory: prescriptive -> how we should do something (Λ Knowledge) Kernel theory: descriptive -> why it works (Ω Knowledge) Grand theories are all-encompassing theories Setting the philosophical ground Gregor/Hevner 2013
22. What can DSR contribute?
23. What is the Claim? Gregor, Hevner 2013
24. Improvement: Compare with best Alternatives Gregor, Hevner 2013
25. Exaptation: Show that it works Gregor, Hevner 2013
26. Invention: Start a Business Gregor, Hevner 2013
27. Algorithm Engineering: Hypotheses and Falsification 27 Sanders 2009
28. Engineering in the Philosophy of Science 28 Staples 2014
29. 1. What is Process Mining? 2. Current Evaluation Practices in Process Mining? 3. Methodological Frameworks for Evaluating Process Mining Algorithm 4. Epistemological Tasks for Process Mining Algorithms 5. Properties of Algorithms and Datasets 6. Research Framework for Process Mining Algorithms Agenda SEITE 29
30. Staples: , ; ⊢ R(x,a) Satisfaction Question: , ; ⊢ R(x,a) with Algorithm a Input data System Design Requirements Improvement Question: , ′ ; ′ > , ; Epistemological Tasks in Process Mining
31. • Internal Validity: Causality of relationship between treatment and outcome • External Validity: Generalizibility to the scope of study • Construct Validity: Correctness of operationalization Validity of Satisfaction Wohlin et al. 2000 • Internal Validity of , ; ⊢ R(x,a) Causality of , and satisfaction • External Validity: Generalizibility of , to scope of study • Construct Validity: Correctness of operationaling satisfaction
32. • Internal Validity of , 𝑎 ; 𝑎 > , ; : Causality of relationship between , , 𝑎 and improvement • External Validity: Generalizibility of , , 𝑎 to scope of study • Construct Validity: Correctness of operationaling improvement Validity of Improvement
33. • Classes of Input data C() • System • Design • Requirements (), ; ⊢ R( C(x), a) Generalization Question
34. 1. What is Process Mining? 2. Current Evaluation Practices in Process Mining? 3. Methodological Frameworks for Evaluating Process Mining Algorithm 4. Epistemological Tasks for Process Mining Algorithms 5. Properties of Algorithms and Datasets 6. Research Framework for Process Mining Algorithms Agenda SEITE 34
35. • Using behavioural relationships • Using decomposition • Using frequencies • Using payload data • Using resource information Properties of Algorithm Design
36. • Fitness • Precision • Simplicity • Generalization Measuring Performance Requirements
37. • Size of the log • Number of variants • Size of alphabet • …more to be added Metrics for Classes of Input Data
38. 1. What is Process Mining? 2. Current Evaluation Practices in Process Mining? 3. Methodological Frameworks for Evaluating Process Mining Algorithm 4. Epistemological Tasks for Process Mining Algorithms 5. Properties of Algorithms and Datasets 6. Research Framework for Process Mining Algorithms Agenda SEITE 38
39. • , ; ⊢ R( C(x), a ) • Better understanding of - informed by demographics of input data • Better understanding of - informed by theories on design principles • Extending the spectrum of • How to find hypotheses: Tukey on exploratory data analysis or meta-heuristics Take Aways
40. Why BPM in Vienna is the best location to meet again: 1-6 September 2019 at WU Vienna Prof. Dr. Stefanie Rinderle-Ma Dean University of Vienna Prof. Dr. Jan Mendling Deputy Head of Department WU Vienna https://bpm2019.ai.wu.ac.at