Automatic Determination of Parameters Values' for Heuristics Miner++

The choice of parameters' values for noise-tolerant Process Mining algorithms is not trivial, especially for users that are not expert in Process Mining. Exhaustive exploration of all possible set of values is not feasible, since several parameters are real-valued. Selecting the "right" values, however, is important, since otherwise the control-flow network returned by the mining can be quite far from the correct one. Here we face this problem for a specific Process Mining algorithm, i.e. Heuristics Miner++. We recognize that the domain of real-valued parameters can be actually partitioned into a finite number of equivalence classes and we suggest exploring the parameters space by a local search strategy driven by a Minimum Description Length principle. We believe that the proposed approach is sufficiently general to be used for other Process Mining algorithms. Experimental results on a set of randomly generated process models show promising results.

More info: http://andrea.burattin.net/publications/2010-wcci

Transcript

1. Automatic Determination of Parameters’ Values for Heuristics Miner++ Andrea Burattin and Alessandro Sperduti Department of Pure and Applied Mathematics University of Padua, Italy July 19th, 2010 Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
2. Slide 2 of 20 Applications of Process Mining techniques This work focuses on the application of process mining algorithms When process mining is applied in real contexts many problems emerge Technical limitations, e.g.: Noise in the log Human issues, e.g.: Usage by non-technical users Difficulties in the usage of the tools Difficulties setting the parameters of the algorithms Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
3. Slide 3 of 20 Applications of Process Mining techniques Algorithms considered in the paper: Heuristics Miner (HM) “Rediscovering workflow models from event-based data using Little Thumb”. Weijters and van der Aalst, Integrated Computer 2001. Heuristics Miner++ (HM++) “Heuristics Miner for Time Intervals”. Burattin and Sperduti, ESANN 2010. The new version (HM++) can handle activities as time intervals Basic idea of the algorithms: 1 Calculate a set of measures 2 Store these values in predefined data structures 3 Use these data structures for the model generation Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
4. Slide 4 of 20 What we are going to do. . . These algorithms require a lot of parameters In these paper a technique for the automatic setting of the parameters is presented 1 Extraction of all the possible parameters’ values 2 Creation of all the possible models 3 Local search of the best model with a MDL principle Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
5. Slide 5 of 20 How HM++ works Implementation of Heuristics Miner in ProM: 1 Parameters are requested to the user 2 Measures’ values are calculated and stored into data structures 3 Final model is built, according to the data structures Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
6. Slide 6 of 20 How HM++ works Parameters of the algorithm are thresholds on: Positive observations Dependency measure AND measure Relative-to-best Length-one loop Length-two loop Long distance Calculated measures (for log W ): 1 Dependency relation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . X ⇒W Y 2 AND/XOR relations . . . . . . . . . . . . . . . . . . . . . . . X ⇒W (Y ∧ Z) 3 Long distance relation . . . . . . . . . . . . . . . . . . . . . . . . . . . X ⇒l W Y 4 Loops of length 1 and 2 . . . . . . . . . . . X ⇒W X and X ⇒2 W Y Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
7. Slide 7 of 20 Problems with HM and HM++ Problems with the setting of the parameters for HM/HM++: They are a lot Not so easy to be setted by (non expert) users: Continuous space for their values No immediate evidence on the output for a different setting of the parameters Far from an autonomous execution of the algorithm (for example for batch executions) Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
8. Slide 8 of 20 Idea for the solution Observation The log we are going to mine is finite There is only a finite number of parameters’ values that generate different process models Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
9. Slide 8 of 20 Idea for the solution Observation The log we are going to mine is finite There is only a finite number of parameters’ values that generate different process models Strategy: 1 Change the sequence of performed steps in order to 2 Calculate all the possible values for the parameters 3 We can generate all the models and choose the best one Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
10. Slide 9 of 20 Parameters’ values discretization What happens if we populate the data structures before setting the threshold parameters? We can store every calculated values into hash sets If all parameters are independent The “cartesian product” between these sets gives us all the settings that will result in different mined models v i‐1 v i v i+1 { { model i model i+1 p Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
11. Slide 10 of 20 Looking for the best model Now we have a discretization for the parameters (it’s a complete discretization, but not the smallest one) Now, a system that automatically tries all the solutions and decides the “best” one is required New problems: How can we define the notion of “best” for a model? How can we compute it? Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
12. Slide 11 of 20 Looking for the best model We opted for a solution based on the Minimum Description Length principle. The system finds the mMDL: mMDL = argmin m∈M L(m) + L(D|m) Where M is the model space (hypothesis space) L(m) is the “size” of the model L(D|m) is the number of log traces the model cannot explain Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
13. Slide 12 of 20 Looking for the best model Application of the notion of “best model” (mMDL): M: all the significant values for every parameter L(m): number of edges (vertices as “consequence” of edges) L(D|m): “continuous semantics fitness” or other conformance measures Application of the search technique: Local search (on the parameters space) Restart from different random configurations Max no. of lateral steps when in plateau Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
14. Slide 13 of 20 Looking for the best model There are some other problems . . . Heuristics Miner parameters are not independent! E.g.: consider “positive observation threshold” and “dependency threshold”: if the first is not satisfied then no values of the second will modify the output Finding all correct dependences may be difficult and computationally very expensive Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
15. Slide 14 of 20 Looking for the best model Main algorithm’s operation and the involved parameters: 1 Dependency measure . . . . . . . . . . . parameters (a), (b) and (c) 2 Length-1 loop . . . . . . . . . . . . . . . . . . . . . . parameters (d) and (b) 3 Length-2 loop . . . . . . . . . . . . . . . . . . . . . . parameters (e) and (b) 4 AND measure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . parameter (g) 5 Long distance measure . . . . . . . . . . . . . . . . . . . . . . . parameter (f) Parameters: (a) relative-to-best thr.; (b) positive obs. thr.; (c) dependency thr.; (d) length-1 loop; (e) length-2 loop; (f) long distance thr.; (g) AND-thr. It’s a very large space to explore. . . Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
16. Slide 15 of 20 Looking for the best model We can divide the space of mined models: Simple networks models with no loops and no long distance dependences (only steps no. 1 and 4) Complete networks models with at least one loop and/or one long distance dependency Search strategy: 1 Find the best model into the “simple network” space 2 Try to extend it into the “complete network” It is possible to not reach the global optimum Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
17. Slide 16 of 20 Experimental results Synthesize dataset composed of 125 process models with the following characteristics: 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 0 5 10 15 20 25 30 Activities Edges Cardoso Metric Number of processes 0 1 2 3 4 5 6 7 8 9 11 14 0 5 10 15 20 25 30 35 40 AND−split/join XOR−split/join Loop Number of processes Cardoso metric: penalize each split depending on how may substates it induces All these processes were “executed“ and their logs recorded Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
18. Slide 17 of 20 Experimental results For the generation of models: www.processmining.it/sw/plg Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
19. Slide 18 of 20 Experimental results Example of a generated model: A B C D E F G H I J K L Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
20. Slide 19 of 20 Experimental results Mining logs with 250 and 500 traces 5% of noise and 75% activities as time intervals (used by HM++) Mining result: 250 log traces 2 4 8 11 13 15 17 19 22 24 26 28 30 32 35 37 39 41 43 45 47 49 53 59 62 66 70 75 0,65 0,7 0,75 0,8 0,85 0,9 0,95 1 Cardoso metric F1 measure 500 log traces 2 4 8 11 13 15 17 19 22 24 26 28 30 32 35 37 39 41 43 45 47 49 53 59 62 66 70 75 0,65 0,7 0,75 0,8 0,85 0,9 0,95 1 Cardoso metric F1 measure F1 = 2 · p · r p + r p = tp tp + fp r = tp tp + fn Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
21. Slide 20 of 20 Conclusions and future works This paper discussed the ease of using algorithms by non-expert user with these steps: Discretization of the space of the parameters’ values for the generation of the hypothesis space Exploration of the hypothesis and selection of the best one with an MDL approach The same approach can be used by other algorithms Possible future works: Optimize the generation of the models space Introduction of machine learning techniques for learning the patterns to prefer Improvements of the “goodness measures” Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
22. Slide 19 of 20 The procedure supports lateral steps in order to avoid get stuck in plateau (local minima) Optimization steps performed (with 5, 10, 25 and 50 restarts): 250 log traces 1 2 3 4 5 6 0 10 20 30 40 50 60 70 80 90 0 lat. step 1 lat. step 5 lat.step 10 lat. Steps Number of processes 500 log traces 1 2 3 4 6 0 10 20 30 40 50 60 70 80 0 lat. step 1 lat. step 5 lat. step 10 lat. Steps Number of processes Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++
23. Slide 20 of 20 The mining of each process took from: 2 minutes 10 hours Depending on The size of the model The number of traces in the log to mine The ease of finding the best model Andrea Burattin and Alessandro Sperduti Automatic Determination of Parameters’ Values for HM++