Who is behind the Model? Classifying Modelers based on Pragmatic Model Features

Process modeling tools typically aid end users in generic, non-personalized ways. However, it is well conceivable that different types of end users may profit from different types of modeling support. In this paper, we propose an approach based on machine learning that is able to classify modelers regarding their expertise while they are creating a process model. To do so, it takes into account pragmatic features of the model under development. The proposed approach is fully automatic, unobtrusive, tool independent, and based on objective measures. An evaluation based on two data sets resulted in a prediction performance of around 90%. Our results further show that all features can be efficiently calculated, which makes the approach applicable to online settings like adaptive modeling environments. In this way, this work contributes to improving the performance of process modelers.

More info: https://andrea.burattin.net/publications/2018-bpm-2

Transcript

1. Who is behind the Model? Classifying Modelers based on Pragmatic Model Features A. Burattin1, P. Soffer 2, D. Fahland 3, J. Mendling 4, H. A. Reijers 3,5, I. Vanderfeesten 3, M. Weidlich 6, B. Weber 1,7 1 Technical University of Denmark, Kgs. Lyngby, Denmark 2 University of Haifa, Haifa, Israel 3 Eindhoven University of Technology, Eindhoven, The Netherlands 4 Vienna University of Economics and Business, Vienna, Austria 5 Vrije Universiteit Amsterdam, Amsterdam, The Netherlands 6 Humboldt-University, Berlin, Germany 7 University of Innsbruck, Innsbruck, Austria
2. Creation of process models • Creating process models is a complex cognitive design activity Soffer, P., Kaner, M., Wand, Y.: Towards Understanding the Process of Process Modeling: Theoretical and Empirical Considerations. In: Proc. ER-BPM'11. (2012) 357-369 Claes, J., Vanderfeesten, I., Pinggera, J., Reijers, H.A., Weber, B., Poels, G.: Visualizing the Process of Process Modeling with PPM Charts. In: Proc. TAProViz'12. (2013) 744-755 Wickens, C.D., Hollands, J.G.: Engineering Psychology and Human Performance. 3 edn. Pearson (1999) 2
3. Creation of process models • Creating process models is a complex cognitive design activity • To accomplish that, the modeller has to • Construct a mental representation of the problem domain • Externalize the mental model into a process model Soffer, P., Kaner, M., Wand, Y.: Towards Understanding the Process of Process Modeling: Theoretical and Empirical Considerations. In: Proc. ER-BPM'11. (2012) 357-369 Claes, J., Vanderfeesten, I., Pinggera, J., Reijers, H.A., Weber, B., Poels, G.: Visualizing the Process of Process Modeling with PPM Charts. In: Proc. TAProViz'12. (2013) 744-755 Wickens, C.D., Hollands, J.G.: Engineering Psychology and Human Performance. 3 edn. Pearson (1999) 2
4. Creation of process models • Creating process models is a complex cognitive design activity • To accomplish that, the modeller has to • Construct a mental representation of the problem domain • Externalize the mental model into a process model • Modelling is not for free: it imposes a substantial cognitive load • Cognitive load is a good predictor of task performance • Overload causes a drop in performance Soffer, P., Kaner, M., Wand, Y.: Towards Understanding the Process of Process Modeling: Theoretical and Empirical Considerations. In: Proc. ER-BPM'11. (2012) 357-369 Claes, J., Vanderfeesten, I., Pinggera, J., Reijers, H.A., Weber, B., Poels, G.: Visualizing the Process of Process Modeling with PPM Charts. In: Proc. TAProViz'12. (2013) 744-755 Wickens, C.D., Hollands, J.G.: Engineering Psychology and Human Performance. 3 edn. Pearson (1999) 2
5. Experts and novices • Experts and novices respond differently to model creation tasks Batra, D., Davis, J.G.: Conceptual data modelling in database design: similarities and differences between expert and novice designers. International journal of man machine studies 37(1) (1992) 83-101 Narasimha, B., Leung, F.S.: Assisting novice analysts in developing quality conceptual models with UML. Communications of the ACM 49(7) (2006) 108-112 3
6. Experts and novices • Experts and novices respond differently to model creation tasks • Novices • Challenged in integrating parts of the problem description • Challenged in mapping problem description into knowledge structures • Challenged in making abstractions (focus on specific functional details) Batra, D., Davis, J.G.: Conceptual data modelling in database design: similarities and differences between expert and novice designers. International journal of man machine studies 37(1) (1992) 83-101 Narasimha, B., Leung, F.S.: Assisting novice analysts in developing quality conceptual models with UML. Communications of the ACM 49(7) (2006) 108-112 3
7. Experts and novices • Experts and novices respond differently to model creation tasks • Novices • Challenged in integrating parts of the problem description • Challenged in mapping problem description into knowledge structures • Challenged in making abstractions (focus on specific functional details) • …and experts • Tend to develop a holistic understanding • Abstract from specific problem characteristics • Categorize textual descriptions before developing solutions Batra, D., Davis, J.G.: Conceptual data modelling in database design: similarities and differences between expert and novice designers. International journal of man machine studies 37(1) (1992) 83-101 Narasimha, B., Leung, F.S.: Assisting novice analysts in developing quality conceptual models with UML. Communications of the ACM 49(7) (2006) 108-112 3
8. The role of the modelling tool • Externalization of the mental model is achieved by interacting with a modelling tool 4
9. The role of the modelling tool • Externalization of the mental model is achieved by interacting with a modelling tool • Modeller performs a sequence of interactions which results into [intermediate] models 4
10. The role of the modelling tool • Externalization of the mental model is achieved by interacting with a modelling tool • Modeller performs a sequence of interactions which results into [intermediate] models • Differences between experts and novices suggest that modelling tool should provide different kinds of support and guidance 4
11. The role of the modelling tool • Externalization of the mental model is achieved by interacting with a modelling tool • Modeller performs a sequence of interactions which results into [intermediate] models • Differences between experts and novices suggest that modelling tool should provide different kinds of support and guidance Can a modelling tool distinguish between experts and novices modellers? 4
12. The role of the modelling tool • Externalization of the mental model is achieved by interacting with a modelling tool • Modeller performs a sequence of interactions which results into [intermediate] models • Differences between experts and novices suggest that modelling tool should provide different kinds of support and guidance Can a modelling tool distinguish between experts and novices modellers? 4
13. The role of the modelling tool • Externalization of the mental model is achieved by interacting with a modelling tool • Modeller performs a sequence of interactions which results into [intermediate] models • Differences between experts and novices suggest that modelling tool should provide different kinds of support and guidance Can a modelling tool distinguish between experts and novices modellers? Yes. 4
14. Table of contents • Introduction and motivation • Automatic identification of expertise levels • Formal requirements • Feature extraction • Classification technique • Case studies and evaluation • Conclusion and future work 5
15. Requirements • Requirements for expertise prediction 6
16. Requirements • Requirements for expertise prediction R1. Based on objective measures 6
17. Requirements • Requirements for expertise prediction R1. Based on objective measures R2. Unobtrusive and no additional effort required 6
18. Requirements • Requirements for expertise prediction R1. Based on objective measures R2. Unobtrusive and no additional effort required R3. Work “online” and applicable to intermediate models 6
19. Requirements • Requirements for expertise prediction R1. Based on objective measures R2. Unobtrusive and no additional effort required R3. Work “online” and applicable to intermediate models R4. Be independent of the modelling tool 6
20. Requirements • Requirements for expertise prediction R1. Based on objective measures R2. Unobtrusive and no additional effort required R3. Work “online” and applicable to intermediate models R4. Be independent of the modelling tool • Possible approaches 6
21. Requirements • Requirements for expertise prediction R1. Based on objective measures R2. Unobtrusive and no additional effort required R3. Work “online” and applicable to intermediate models R4. Be independent of the modelling tool • Possible approaches • Self-assessment of the user… violates R1 and R3 6
22. Requirements • Requirements for expertise prediction R1. Based on objective measures R2. Unobtrusive and no additional effort required R3. Work “online” and applicable to intermediate models R4. Be independent of the modelling tool • Possible approaches • Self-assessment of the user… violates R1 and R3 • Use a questionnaire to elicit expertise… violates R2 and R3 6
23. Requirements • Requirements for expertise prediction R1. Based on objective measures R2. Unobtrusive and no additional effort required R3. Work “online” and applicable to intermediate models R4. Be independent of the modelling tool • Possible approaches • Self-assessment of the user… violates R1 and R3 • Use a questionnaire to elicit expertise… violates R2 and R3 • Use neuro-physiological data (e.g., EEG)… violates R2 6
24. Requirements • Requirements for expertise prediction R1. Based on objective measures R2. Unobtrusive and no additional effort required R3. Work “online” and applicable to intermediate models R4. Be independent of the modelling tool • Possible approaches • Self-assessment of the user… violates R1 and R3 • Use a questionnaire to elicit expertise… violates R2 and R3 • Use neuro-physiological data (e.g., EEG)… violates R2 • Analyze interactions with modeling platform… violates R4 6
25. Requirements • Requirements for expertise prediction R1. Based on objective measures R2. Unobtrusive and no additional effort required R3. Work “online” and applicable to intermediate models R4. Be independent of the modelling tool • Possible approaches • Self-assessment of the user… violates R1 and R3 • Use a questionnaire to elicit expertise… violates R2 and R3 • Use neuro-physiological data (e.g., EEG)… violates R2 • Analyze interactions with modeling platform… violates R4 • Analyze pragmatic features of (intermediate) artifacts 6
26. General idea of the approach • After each interaction with the modelling tool an intermediate model is created 7
27. General idea of the approach • After each interaction with the modelling tool an intermediate model is created Time Interactions with the modeling tool 7
28. General idea of the approach • After each interaction with the modelling tool an intermediate model is created Time Interactions with the modeling tool Features vector = x1 x2 x3 xn Feature 1 Feature 2 Feature 3 Feature n Feature extraction 1 7
29. General idea of the approach • After each interaction with the modelling tool an intermediate model is created Time Interactions with the modeling tool Features vector = x1 x2 x3 xn Feature 1 Feature 2 Feature 3 Feature n Feature extraction 1 Classification model Classification 2 Novice Expert 7
30. Feature identification • Given a BPMN model we extract the following pragmatic features • Features referring to the alignment of elements Two nodes are aligned if they share at least one of the coordinates (within a threshold) 8
31. Feature identification • Given a BPMN model we extract the following pragmatic features • Features referring to the alignment of elements Two nodes are aligned if they share at least one of the coordinates (within a threshold) F1. Percentage of aligned SESE fragments F2. Percentage of activities in aligned SESE fragments F3. Percentage of activities in not-aligned SESE fragments 8
32. Feature identification • Given a BPMN model we extract the following pragmatic features • Features referring to the alignment of elements Two nodes are aligned if they share at least one of the coordinates (within a threshold) F1. Percentage of aligned SESE fragments F2. Percentage of activities in aligned SESE fragments F3. Percentage of activities in not-aligned SESE fragments vs. 8
33. Feature identification (cont.) • Features referring to the type and usage of gateways 9
34. Feature identification (cont.) • Features referring to the type and usage of gateways F4. Number of explicit gateways 9
35. Feature identification (cont.) • Features referring to the type and usage of gateways F4. Number of explicit gateways F5. Number of implicit gateways 9
36. Feature identification (cont.) • Features referring to the type and usage of gateways F4. Number of explicit gateways F5. Number of implicit gateways F6. Number of reused gateways 9
37. Feature identification (cont.) • Features referring to the style of edges 10
38. Feature identification (cont.) • Features referring to the style of edges F7. Percentage of orthogonal segments 10
39. Feature identification (cont.) • Features referring to the style of edges F7. Percentage of orthogonal segments 10
40. Feature identification (cont.) • Features referring to the style of edges F7. Percentage of orthogonal segments vs. 10
41. Feature identification (cont.) • Features referring to the style of edges F7. Percentage of orthogonal segments F8. Percentage of crossing edges vs. 10
42. Feature identification (cont.) • Feature referring to the process “as a whole” 11
43. Feature identification (cont.) • Feature referring to the process “as a whole” F9. M-BP: consistency of the flow with respect to temporal logical ordering 11
44. Feature identification (cont.) • Feature referring to the process “as a whole” F9. M-BP: consistency of the flow with respect to temporal logical ordering A B C 11
45. Feature identification (cont.) • Feature referring to the process “as a whole” F9. M-BP: consistency of the flow with respect to temporal logical ordering A B C A B C vs. 11
46. Feature identification (cont.) • Feature referring to the process “as a whole” F9. M-BP: consistency of the flow with respect to temporal logical ordering F10.Number of ending points A B C A B C vs. 11
47. Feature identification (cont.) • Feature referring to the process “as a whole” F9. M-BP: consistency of the flow with respect to temporal logical ordering F10.Number of ending points A B C A B C vs. vs. 11
48. Datasets used for validation • Two modelling datasets collected during modelling sessions in TU Eindhoven and Berlin in 2010, DOI: 10.5281/zenodo.1194779 12
49. Datasets used for validation • Two modelling datasets collected during modelling sessions in TU Eindhoven and Berlin in 2010, DOI: 10.5281/zenodo.1194779 • Cheetah as modelling platform 12
50. Datasets used for validation • Two modelling datasets collected during modelling sessions in TU Eindhoven and Berlin in 2010, DOI: 10.5281/zenodo.1194779 • Cheetah as modelling platform • Subjects were asked to model two processes 12
51. Datasets used for validation • Two modelling datasets collected during modelling sessions in TU Eindhoven and Berlin in 2010, DOI: 10.5281/zenodo.1194779 • Cheetah as modelling platform • Subjects were asked to model two processes • “pre-flight”, reference: 12
52. Datasets used for validation • Two modelling datasets collected during modelling sessions in TU Eindhoven and Berlin in 2010, DOI: 10.5281/zenodo.1194779 • Cheetah as modelling platform • Subjects were asked to model two processes • “pre-flight”, reference: • “mortgage-1”, reference: 12
53. Datasets used for validation (cont.) • Number of models and modelling sessions Experts Novices Sessions Intermediate models Sessions Intermediate models pre-flight 39 7299 118 14147 mortgage-1 31 4856 144 36141 13
54. Datasets used for validation (cont.) • Number of models and modelling sessions Experts Novices Sessions Intermediate models Sessions Intermediate models pre-flight 39 7299 118 14147 mortgage-1 31 4856 144 36141 13
55. Datasets used for validation (cont.) • Number of models and modelling sessions • Mann-Whitney U test (are features significant discriminators of expertise levels?) Experts Novices Sessions Intermediate models Sessions Intermediate models pre-flight 39 7299 118 14147 mortgage-1 31 4856 144 36141 13
56. Datasets used for validation (cont.) • Number of models and modelling sessions • Mann-Whitney U test (are features significant discriminators of expertise levels?) Experts Novices Sessions Intermediate models Sessions Intermediate models pre-flight 39 7299 118 14147 mortgage-1 31 4856 144 36141 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 pre-flight p <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 mortgage-1 p <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 <.001 13
57. Descriptive statistics (mean) Mean values mortgage-1 pre-flight Experts Novices Experts Novices F1. Alignment of fragments 0.86 > 0.81 0.82 > 0.76 F2. % acts in aligned frags 0.46 > 0.43 0.50 > 0.44 F3. % acts in not-align frags 0.09 < 0.10 0.08 < 0.10 F4. # explicit gateways 11.90 > 10.19 6.84 > 5.94 F5. # implicit gateways 1.31 < 1.58 0.37 < 0.49 F6. # reused gateways 0.34 > 0.32 0.50 > 0.47 F7. % orthogonal segments 0.71 > 0.60 0.57 > 0.49 F8. % crossing edges 0.01 < 0.02 0.012 > 0.008 F9. Flow consistency 0.95 > 0.88 0.95 > 0.91 F10. # end points 2.74 > 2.27 1.60 < 1.64 14
58. Descriptive statistics (mean) Mean values mortgage-1 pre-flight Experts Novices Experts Novices F1. Alignment of fragments 0.86 > 0.81 0.82 > 0.76 F2. % acts in aligned frags 0.46 > 0.43 0.50 > 0.44 F3. % acts in not-align frags 0.09 < 0.10 0.08 < 0.10 F4. # explicit gateways 11.90 > 10.19 6.84 > 5.94 F5. # implicit gateways 1.31 < 1.58 0.37 < 0.49 F6. # reused gateways 0.34 > 0.32 0.50 > 0.47 F7. % orthogonal segments 0.71 > 0.60 0.57 > 0.49 F8. % crossing edges 0.01 < 0.02 0.012 > 0.008 F9. Flow consistency 0.95 > 0.88 0.95 > 0.91 F10. # end points 2.74 > 2.27 1.60 < 1.64 14
59. Descriptive statistics (mean) Mean values mortgage-1 pre-flight Experts Novices Experts Novices F1. Alignment of fragments 0.86 > 0.81 0.82 > 0.76 F2. % acts in aligned frags 0.46 > 0.43 0.50 > 0.44 F3. % acts in not-align frags 0.09 < 0.10 0.08 < 0.10 F4. # explicit gateways 11.90 > 10.19 6.84 > 5.94 F5. # implicit gateways 1.31 < 1.58 0.37 < 0.49 F6. # reused gateways 0.34 > 0.32 0.50 > 0.47 F7. % orthogonal segments 0.71 > 0.60 0.57 > 0.49 F8. % crossing edges 0.01 < 0.02 0.012 > 0.008 F9. Flow consistency 0.95 > 0.88 0.95 > 0.91 F10. # end points 2.74 > 2.27 1.60 < 1.64 14
60. Descriptive statistics (mean) Mean values mortgage-1 pre-flight Experts Novices Experts Novices F1. Alignment of fragments 0.86 > 0.81 0.82 > 0.76 F2. % acts in aligned frags 0.46 > 0.43 0.50 > 0.44 F3. % acts in not-align frags 0.09 < 0.10 0.08 < 0.10 F4. # explicit gateways 11.90 > 10.19 6.84 > 5.94 F5. # implicit gateways 1.31 < 1.58 0.37 < 0.49 F6. # reused gateways 0.34 > 0.32 0.50 > 0.47 F7. % orthogonal segments 0.71 > 0.60 0.57 > 0.49 F8. % crossing edges 0.01 < 0.02 0.012 > 0.008 F9. Flow consistency 0.95 > 0.88 0.95 > 0.91 F10. # end points 2.74 > 2.27 1.60 < 1.64 14
61. Descriptive statistics (correlations) • Pearson correlation coefficient of features • Little indication of correlation: features capture complementary aspects 15
62. Problem as classification • Classification problem • Input: 10-dimensional feature space (one for each feature) 16
63. Problem as classification • Classification problem • Input: 10-dimensional feature space (one for each feature) • Each intermediate model as independent model sample • Only models from the last 70% of the modelling session (to avoid almost-empty models) 16
64. Problem as classification • Classification problem • Input: 10-dimensional feature space (one for each feature) • Each intermediate model as independent model sample • Only models from the last 70% of the modelling session (to avoid almost-empty models) • Output: likelihood of classification of each class 16
65. Problem as classification • Classification problem • Input: 10-dimensional feature space (one for each feature) • Each intermediate model as independent model sample • Only models from the last 70% of the modelling session (to avoid almost-empty models) • Output: likelihood of classification of each class • We used a feed forward neural network with a hidden layer with 50 neurons • Training with Multilayer Perceptron • Software based on Weka, available at github.com/DTU-SPE/ExpertisePredictor4BPMN 16
66. Classification performance • Tests on random datasets of (intermediate) BPMN model • Quality in terms of F1: harmonic mean between precision and recall • Results as average of each of the 10-fold cross validation 17
67. Classification performance • Tests on random datasets of (intermediate) BPMN model • Quality in terms of F1: harmonic mean between precision and recall • Results as average of each of the 10-fold cross validation 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 1000 2000 4000 8000 F-score Dataset size (number of random BPMN models used) mortgage-1 pre-flight 17
68. Classification performance • Tests on random datasets of (intermediate) BPMN model • Quality in terms of F1: harmonic mean between precision and recall • Results as average of each of the 10-fold cross validation 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 1000 2000 4000 8000 F-score Dataset size (number of random BPMN models used) mortgage-1 pre-flight 17
69. Classification performance • Tests on random datasets of (intermediate) BPMN model • Quality in terms of F1: harmonic mean between precision and recall • Results as average of each of the 10-fold cross validation 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 1000 2000 4000 8000 F-score Dataset size (number of random BPMN models used) mortgage-1 pre-flight 17
70. Time performance • Time required to compute each of the 10 features • Standard Java implementation (Cheetah) on standard laptop • Tests with typical PC usage maintained (to simulate modeller settings) • Average time over 18k samples from biggest dataset (mortgage-1) 18
71. Time performance • Time required to compute each of the 10 features • Standard Java implementation (Cheetah) on standard laptop • Tests with typical PC usage maintained (to simulate modeller settings) • Average time over 18k samples from biggest dataset (mortgage-1) 5.36 5.31 5.44 0.03 0.03 0.03 0.02 1.19 93.12 0.01 0 25 50 75 100 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 Time (ms) Features 18
72. Time performance • Time required to compute each of the 10 features • Standard Java implementation (Cheetah) on standard laptop • Tests with typical PC usage maintained (to simulate modeller settings) • Average time over 18k samples from biggest dataset (mortgage-1) 5.36 5.31 5.44 0.03 0.03 0.03 0.02 1.19 93.12 0.01 0 25 50 75 100 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 Time (ms) Features 18
73. Time performance • Time required to compute each of the 10 features • Standard Java implementation (Cheetah) on standard laptop • Tests with typical PC usage maintained (to simulate modeller settings) • Average time over 18k samples from biggest dataset (mortgage-1) 5.36 5.31 5.44 0.03 0.03 0.03 0.02 1.19 93.12 0.01 0 25 50 75 100 F1 F2 F3 F4 F5 F6 F7 F8 F9 F10 Time (ms) Features 18
74. Conclusion and limitations • We presented an approach to classify modellers • Decision is based on objective measures • Decision according to artifacts being modelled • Fast computation, applicable to intermediate models • Identified requirements are all met 19
75. Conclusion and limitations • We presented an approach to classify modellers • Decision is based on objective measures • Decision according to artifacts being modelled • Fast computation, applicable to intermediate models • Identified requirements are all met R1. Based on objective measures R2. Unobtrusive and no additional effort R3. “Online” and intermediate models R4. Independent of the modelling tool 19
76. Conclusion and limitations • We presented an approach to classify modellers • Decision is based on objective measures • Decision according to artifacts being modelled • Fast computation, applicable to intermediate models • Identified requirements are all met • Classification results as F-score • On mortgage-1: 0.94 • On pre-flight: 0.88 (the process lacks complex behavioural structures) R1. Based on objective measures R2. Unobtrusive and no additional effort R3. “Online” and intermediate models R4. Independent of the modelling tool 19
77. Conclusion and limitations • We presented an approach to classify modellers • Decision is based on objective measures • Decision according to artifacts being modelled • Fast computation, applicable to intermediate models • Identified requirements are all met • Classification results as F-score • On mortgage-1: 0.94 • On pre-flight: 0.88 (the process lacks complex behavioural structures) • Limitations • Currently only applicable to BPMN models • Big models (> 30 activities) might require more time to compute features • Same modelling task used for training and prediction R1. Based on objective measures R2. Unobtrusive and no additional effort R3. “Online” and intermediate models R4. Independent of the modelling tool 19
78. Impact and future work • Potential impact on several aspects 20
79. Impact and future work • Potential impact on several aspects • For developers: design tools that adapt themselves to the user 20
80. Impact and future work • Potential impact on several aspects • For developers: design tools that adapt themselves to the user • For educators: assess user capabilities and form groups 20
81. Impact and future work • Potential impact on several aspects • For developers: design tools that adapt themselves to the user • For educators: assess user capabilities and form groups • For practitioners: recruitment, task allocation and team formation 20
82. Impact and future work • Potential impact on several aspects • For developers: design tools that adapt themselves to the user • For educators: assess user capabilities and form groups • For practitioners: recruitment, task allocation and team formation • Future work include • Generalizing the task to predict models not used for training • Improve prediction of sessions rather than models • Continue the feature engineering process 20