Story Points Changes in Agile Iterative Development

Story Points (SP) are an effort unit that is used to represent the relative effort of a work item. In Agile software development, SP allows a development team to estimate their delivery capacity and facilitate the sprint planning activities. Although Agile embraces changes, SP changes after the sprint planning may negatively impact the sprint plan. To minimize the impact, there is a need to better understand the SP changes and an automated approach to predict the SP changes. Hence, to better understand the SP changes, we examine the prevalence, accuracy, and impact of information changes on SP changes. Through the analyses based on 19,349 work items spread across seven open-source projects, we find that on average, 10% of the work items have SP changes. These work items typically have SP value increased by 58%-100% relative to the initial SP value when they were assigned to a sprint. We also find that the unchanged SP reflect the development time better than the changed SP. Our qualitative analysis shows that the work items with changed SP often have the information changes relating to updating the scope of work. Our empirical results suggest that SP and the scope of work should be reviewed prior or during sprint planning to achieve a reliable sprint plan. Yet, it could be a tedious task to review all work items in the product (or sprint) backlog. Therefore, we develop a classifier to predict whether a work item will have SP changes after being assigned to a sprint. Our classifier achieves an AUC of 0.69-0.8, which is significantly better than the baselines. Our results suggest that to better manage and prepare for the unreliability in SP estimation, the team can leverage our insights and the classifier during the sprint planning. To facilitate future studies, we provide the replication package and the datasets, which are available online.

Transcript

1. Story Points changes in Agile Iterative Development An empirical study and a prediction approach ESEC/FSE 2022: Journal First track Patanamon (Pick)
 Thongtanunam Shanika
 Karunasekera @pamon 👈 me jpasuksmit@student.unimelb.edu.au patanamon.t@unimelb.edu.au karus@unimelb.edu.au Accepted at EMSE, 2022 Jirat
 Pasuksmit @jiratpa 1
2. Agile teams work in short sprints - Continuously work in iterations of 2-4 weeks - Easier to adapt to changes Sprint 1 Sprint 2 Sprint 3 Sprint 4 (2-4 weeks) 2
3. Agile teams work in short sprints Software 
 Increment + Planning Implementing Delivering Sprint - Continuously work in iterations of 2-4 weeks - Easier to adapt to changes Sprint 1 Sprint 2 Sprint 3 Sprint 4 - In each sprint, the teams plan, implement, 
 and deliver the software increment. (2-4 weeks) 2
4. Sprint planning Product backlog work item work item work item work item work item work item 2 SP 2 SP 4 SP 4 SP n/a n/a records list of work items (tasks) to develop the software Agile teams allocate just-enough workload for each sprint 3
5. Sprint planning Product backlog work item work item work item work item work item work item 2 SP 2 SP 4 SP 4 SP n/a n/a Add Login button Work item Agile teams allocate just-enough workload for each sprint 3
6. Sprint planning Product backlog work item work item work item work item work item work item 2 SP 2 SP 4 SP 4 SP n/a n/a Detail 
 analysis Add Login button Work item detail analysis 
 (elaborated) tests added Add login button to the Login page with unit tests and UI tests Agile teams allocate just-enough workload for each sprint 3
7. Sprint planning Product backlog work item work item work item work item work item work item 2 SP 2 SP 4 SP 4 SP n/a n/a Detail 
 analysis Estimate work items Add Login button Work item detail analysis 
 (elaborated) tests added Add login button to the Login page with unit tests and UI tests using a relative unit like Story Points Agile teams allocate just-enough workload for each sprint 3
8. Sprint planning Product backlog work item work item work item work item work item work item 2 SP 2 SP 4 SP 4 SP n/a n/a Detail 
 analysis - fine-grained & estimated 
 - ready for the next sprint - Larger size 
 - not work on soon *Prioritize Estimate work items Add Login button Work item detail analysis 
 (elaborated) tests added Add login button to the Login page with unit tests and UI tests Agile teams allocate just-enough workload for each sprint 3
9. Sprint planning Product backlog work item work item work item work item work item work item 2 SP 2 SP 4 SP 4 SP n/a n/a sprint capacity work item work item work item work item Selecting work items allocate just-enough workload for the sprint Sprint Backlog A 2 SP 2 SP 4 SP 4 SP Detail 
 analysis Estimate work items Add Login button Work item detail analysis 
 (elaborated) tests added Add login button to the Login page with unit tests and UI tests Agile teams allocate just-enough workload for each sprint 3
10. Agile teams estimate SP using a lightweight estimation method (a work item of 2 SP is twice as large as a work item of 1 SP) Story Points (SP) is a relative unit representing the size of a work item 4
11. Agile teams estimate SP using a lightweight estimation method (a work item of 2 SP is twice as large as a work item of 1 SP) Story Points (SP) is a relative unit 5 Nah (Planning Poker: each team member give a number 
 and discusses until reaches the consensus) The team relies on lightweight methods 
 (e.g., Planning poker) to estimate representing the size of a work item 4
12. Agile teams estimate SP using a lightweight estimation method Story Points can be used to estimate the delivery time (a work item of 2 SP is twice as large as a work item of 1 SP) Story Points (SP) is a relative unit 5 Nah (Planning Poker: each team member give a number 
 and discusses until reaches the consensus) The team relies on lightweight methods 
 (e.g., Planning poker) to estimate = 4 sprints = 20 SP, 1 sprints Team 
 capacity or 4 weeks (if 1 sprint = 1 week) representing the size of a work item 4
13. Achieving a reliable SP estimation helps the team create a reliable sprint plan During sprint planning Team estimates the SP and allocates just-enough workload for a sprint work item work item work item work item Sprint Backlog A 2 SP 4 SP 2 SP 4 SP 5
14. Achieving a reliable SP estimation helps the team create a reliable sprint plan During sprint planning Team estimates the SP and allocates just-enough workload for a sprint work item work item work item work item Sprint Backlog A 2 SP 4 SP 2 SP 4 SP (e.g., Planning poker: MMRE 16-61%) Lightweight estimation 
 methods are inaccurate [Dantas et al., 2018] 5
15. Achieving a reliable SP estimation helps the team create a reliable sprint plan During sprint planning Team estimates the SP and allocates just-enough workload for a sprint work item work item work item work item Sprint Backlog A 2 SP 4 SP 2 SP 4 SP The team may re-estimate (change) the Story Points [Hoda et al. 2018] 5
16. Achieving a reliable SP estimation helps the team create a reliable sprint plan During sprint planning Team estimates the SP and allocates just-enough workload for a sprint work item work item work item work item Sprint Backlog A 2 SP 4 SP 2 SP 4 SP The team may re-estimate (change) the Story Points [Hoda et al. 2018] A ft er sprint planning Sprint Backlog A work item 2 SP 4 SP 2 SP 8 SP The re-estimated effort may exceed the sprint capacity sprint capacity work item work item work item 5
17. Achieving a reliable SP estimation helps the team create a reliable sprint plan During sprint planning Sprint plan becomes unreliable Requires more e ff ort 
 in re-planning Losing customers’ trust [Rubin 2012] Team estimates the SP and allocates just-enough workload for a sprint work item work item work item work item Sprint Backlog A 2 SP 4 SP 2 SP 4 SP The team may re-estimate (change) the Story Points [Hoda et al. 2018] A ft er sprint planning Sprint Backlog A work item 2 SP 4 SP 2 SP 8 SP The re-estimated effort may exceed the sprint capacity sprint capacity work item work item work item 5
18. Achieving a reliable SP estimation helps the team create a reliable sprint plan During sprint planning Sprint plan becomes unreliable Requires more e ff ort 
 in re-planning Losing customers’ trust [Rubin 2012] Key Problem: Unstable Story Points estimation causing the sprint plan to become unreliable Team estimates the SP and allocates just-enough workload for a sprint work item work item work item work item Sprint Backlog A 2 SP 4 SP 2 SP 4 SP The team may re-estimate (change) the Story Points [Hoda et al. 2018] A ft er sprint planning Sprint Backlog A work item 2 SP 4 SP 2 SP 8 SP The re-estimated effort may exceed the sprint capacity sprint capacity work item work item work item 5
19. Research questions (RQ1) To what degree do Story Points change? (RQ2) Do changed Story Points better reflect the development time than the ones that remain unchanged? (RQ3) To what extent are Story Points changes 
 are associated with information changes? (RQ4) Can we predict whether a work item 
 will have Story Points changes? 6
20. Case Study Design Studied projects and the dataset The dataset contains 13,902 
 work items (JIRA issues) AS: Appcelerator Studio DM: Data management ME: Mesosphere MU: Mule TI: Titanium Mobile UG: Usergrid XD: Spring XD Seven open source projects that generally allows the SP to be changed 7
21. Case Study Design Identifying Story Points changes We only consider the SP changes that occurred after the work items were assigned to the sprint Work item created SP=1 SP=3** 
 Sprint A SP=1* 
 Sprint A SP=3 
 Sprint A Work item assigned to the sprint Story Points were changed from 1 to 3 Work item closed SP changes considering period Work item with changed SP 8
22. (RQ1) To what degree do Story Points change? 3% (16/636) 17% (833/4803) 8% (136/1678) 11% (316/2924) 5% (82/1630) 13% (27/201) 11% (220/2030) 0 5 10 15 AS DM ME MU TI UG XD Story points change frequency (%) of work items have SP changes 3-17% Part 1: Prevalence (#work items with SP changes in a project) 9
23. (RQ1) To what degree do Story Points change? 3% (16/636) 17% (833/4803) 8% (136/1678) 11% (316/2924) 5% (82/1630) 13% (27/201) 11% (220/2030) 0 5 10 15 AS DM ME MU TI UG XD Story points change frequency (%) of work items have SP changes 3-17% Part 1: Prevalence (#work items with SP changes in a project) Part 2: Changing frequency 88% of the work items with changed SP have only one SP change event (# of SP changes in a work item) 9
24. (RQ1) To what degree do Story Points change? 3% (16/636) 17% (833/4803) 8% (136/1678) 11% (316/2924) 5% (82/1630) 13% (27/201) 11% (220/2030) 0 5 10 15 AS DM ME MU TI UG XD Story points change frequency (%) of work items have SP changes 3-17% Part 1: Prevalence (#work items with SP changes in a project) Part 2: Changing frequency 88% of the work items with changed SP have only one SP change event (# of SP changes in a work item) Part 3: Changing size Median SP changing size 0% 25% 50% 75% 100% AS DM ME MU TI UG XD increased decreased SP are typically changed by >58% of their size when assigning to a sprint (% of change from sprint assignment to closed) 9
25. (RQ1) To what degree do Story Points change? SP are not o ft en changed, but the change can have a nega ti ve impact on the sprint plan. 3% (16/636) 17% (833/4803) 8% (136/1678) 11% (316/2924) 5% (82/1630) 13% (27/201) 11% (220/2030) 0 5 10 15 AS DM ME MU TI UG XD Story points change frequency (%) of work items have SP changes 3-17% Part 1: Prevalence (#work items with SP changes in a project) Part 2: Changing frequency 88% of the work items with changed SP have only one SP change event (# of SP changes in a work item) Part 3: Changing size Median SP changing size 0% 25% 50% 75% 100% AS DM ME MU TI UG XD increased decreased SP are typically changed by >58% of their size when assigning to a sprint (% of change from sprint assignment to closed) 9
26. Story Points Changes in Agile Iterative Development (RQ1) To what degree do Story Points change? SP are not o ft en changed, but their large changing size can have a nega ti ve impact on the sprint plan. (RQ2) Do changed Story Points better reflect the development time than the ones that remain unchanged? (RQ3) To what extent are Story Points changes 
 are associated with information changes? (RQ4) Can we predict whether a work item 
 will have Story Points changes? >58% Changing size 3-17% Prevalence 10
27. (RQ2) Do changed Story Points better reflect the development time than the ones that 
 remain unchanged? Investigate whether Story Points can be used to predict the development time (If yes, meaning that Story Points can reflect the development time) 11
28. (RQ2) Do changed Story Points better reflect the development time than the ones that 
 remain unchanged? Measure development time 
 (i.e., the period that the status 
 was set to “in progress”) status set to 
 “in progress” status set to 
 “closed” work items created Development 
 time Investigate whether Story Points can be used to predict the development time (If yes, meaning that Story Points can reflect the development time) 11
29. (RQ2) Do changed Story Points better reflect the development time than the ones that 
 remain unchanged? SP Development time measure accuracy Measure development time 
 (i.e., the period that the status 
 was set to “in progress”) status set to 
 “in progress” status set to 
 “closed” work items created Development 
 time Build mixed-effects linear regression models 
 based on Story Points(x) to predict the development time(y) 
 and measure the accuracy Investigate whether Story Points can be used to predict the development time (If yes, meaning that Story Points can reflect the development time) Actual development time Predicted development time 11
30. (RQ2) Do changed Story Points better reflect the development time than the ones that 
 remain unchanged? SP Development time measure accuracy Measure development time 
 (i.e., the period that the status 
 was set to “in progress”) status set to 
 “in progress” status set to 
 “closed” work items created Development 
 time Build mixed-effects linear regression models 
 based on Story Points(x) to predict the development time(y) 
 and measure the accuracy Changed 
 Story Points 
 last version Changed 
 Story Points 
 @sprint assignment Unchanged 
 Story Points Compare the accuracy between Unchanged and Changed story points (never changed) Investigate whether Story Points can be used to predict the development time (If yes, meaning that Story Points can reflect the development time) Actual development time Predicted development time 11
31. (RQ2) Do changed Story Points better reflect the development time than the ones that 
 remain unchanged? Unchanged SP share a stronger relationship with the development time than the changed SP Standardized Accuracy (in re fl ec ti ng the development ti me) 55 66.25 77.5 88.75 100 AS DM ME MU TI UG XD unchanged SP changed SP (sprint planning version) changed SP (last version) Investigate whether Story Points can be used to predict the development time (If yes, meaning that Story Points can reflect the development time) 12
32. (RQ2) Do changed Story Points better reflect the development time than the ones that 
 remain unchanged? Unchanged SP share a stronger relationship with the development time than the changed SP Standardized Accuracy (in re fl ec ti ng the development ti me) 55 66.25 77.5 88.75 100 AS DM ME MU TI UG XD unchanged SP changed SP (sprint planning version) changed SP (last version) Investigate whether Story Points can be used to predict the development time (If yes, meaning that Story Points can reflect the development time) Unchanged SP be tt er re fl ect the development ti me than changed SP 12
33. Story Points Changes in Agile Iterative Development (RQ1) To what degree do Story Points change? Unchanged SP be tt er re fl ect the development ti me than changed SP Changed 
 Story Points Unchanged 
 Story Points > (RQ2) Do changed Story Points better reflect the development time than the ones that remain unchanged? (RQ3) To what extent are Story Points changes 
 are associated with information changes? (RQ4) Can we predict whether a work item 
 will have Story Points changes? SP are not o ft en changed, but their large changing size can have a nega ti ve impact on the sprint plan. >58% Changing size 3-17% Prevalence 13
34. (RQ3) To what extent are Story Points changes are associated with 
 information changes? Part 1: Co-occurrences Part 2: Types of information changes 2 SP 8 SP Measures the (Fisher’s) odds ratio 
 that a work item with SP changes will also have information changes Information changes Story Points changes 14
35. (RQ3) To what extent are Story Points changes are associated with 
 information changes? Part 1: Co-occurrences Part 2: Types of information changes 2 SP 8 SP Measures the (Fisher’s) odds ratio 
 that a work item with SP changes will also have information changes Information changes Story Points changes Perform open-coding and manual classification 
 to derive the types of information changes that often occurred along with SP changes Open coding 
 (manual classification) List of codes * 250 work items 14
36. (RQ3) To what extent are Story Points changes are associated with 
 information changes? Part 1: Co-occurrences Fisher odds ra ti o 3.36 3.19 2.66 4.35 2.25 3.44 3.71 AS DM ME MU TI UG XD *Odds ratio > 1 SP changes often found along with information changes 15
37. (RQ3) To what extent are Story Points changes are associated with 
 information changes? Part 1: Co-occurrences Fisher odds ra ti o 3.36 3.19 2.66 4.35 2.25 3.44 3.71 AS DM ME MU TI UG XD *Odds ratio > 1 Changing Scope - task adding - task changing - task removing Clarification - details improving - rephrasing - text formatting SP changes often found along with information changes 15
38. (RQ3) To what extent are Story Points changes are associated with 
 information changes? Part 1: Co-occurrences Fisher odds ra ti o 3.36 3.19 2.66 4.35 2.25 3.44 3.71 AS DM ME MU TI UG XD *Odds ratio > 1 Changing Scope - task adding - task changing - task removing Clarification - details improving - rephrasing - text formatting SP changes often found along with information changes Part 2: Types of information changes 6 7 4 41 14 9 43 10 7 30 5 3 Changing scope Clarification task adding task changing task rem oving details im proving rephrasing text form atting 0 20 40 60 % work items DM 10 10 1 14 29 12 31 37 3 20 16 2 Changing scope Clarification task adding task changing task rem oving details im proving rephrasing text form atting MU 3 0 7 7 57 18 6 24 5 0 Changing scope Clarification task adding task changing details im proving rephrasing text form atting TI 1 15 4 10 20 7 24 38 6 20 22 10 Changing scope Clarification task adding task changing task rem oving details im proving rephrasing text form atting XD work items with changed SP work items with unchanged SP Information changes for changing scope often found in the work items with SP changes 15
39. (RQ3) To what extent are Story Points changes are associated with 
 information changes? Part 1: Co-occurrences Fisher odds ra ti o 3.36 3.19 2.66 4.35 2.25 3.44 3.71 AS DM ME MU TI UG XD *Odds ratio > 1 Changing Scope - task adding - task changing - task removing Clarification - details improving - rephrasing - text formatting SP changes often found along with information changes Part 2: Types of information changes 6 7 4 41 14 9 43 10 7 30 5 3 Changing scope Clarification task adding task changing task rem oving details im proving rephrasing text form atting 0 20 40 60 % work items DM 10 10 1 14 29 12 31 37 3 20 16 2 Changing scope Clarification task adding task changing task rem oving details im proving rephrasing text form atting MU 3 0 7 7 57 18 6 24 5 0 Changing scope Clarification task adding task changing details im proving rephrasing text form atting TI 1 15 4 10 20 7 24 38 6 20 22 10 Changing scope Clarification task adding task changing task rem oving details im proving rephrasing text form atting XD work items with changed SP work items with unchanged SP Information changes for changing scope often found in the work items with SP changes SP changes o ft en occurred along with Informa ti on changes (for changing scope) 15
40. Story Points Changes in Agile Iterative Development (RQ1) To what degree do Story Points change? Unchanged SP be tt er re fl ect the development ti me than changed SP Changed 
 Story Points Unchanged 
 Story Points > (RQ2) Do changed Story Points better reflect the development time than the ones that remain unchanged? (RQ3) To what extent are Story Points changes 
 are associated with information changes? (RQ4) Can we predict whether a work item 
 will have Story Points changes? SP changes o ft en occurred along with 
 Informa ti on changes (for changing scope) 2 SP 8 SP SP are not o ft en changed, but their large changing size can have a nega ti ve impact on the sprint plan. >58% Changing size 3-17% Prevalence 16
41. (RQ4) Can we predict whether a work item will have Story Points changes? Sprint Backlog A Product Backlog Selecting work items 2 SP 2 SP 4 SP 4 SP work item #1 work item #2 work item #3 work item #4 17
42. (RQ4) Can we predict whether a work item will have Story Points changes? Sprint Backlog A Product Backlog Selecting work items 2 SP 2 SP 4 SP 4 SP work item #1 work item #2 work item #3 work item #4 Work item #4 
 (Story Points changes occurred) 8 SP 17
43. (RQ4) Can we predict whether a work item will have Story Points changes? Sprint Backlog A Product Backlog Selecting work items 2 SP 2 SP 4 SP 4 SP work item #1 work item #2 work item #3 work item #4 17
44. (RQ4) Can we predict whether a work item will have Story Points changes? Sprint Backlog A Product Backlog Selecting work items The work item #4 may have SP change Helping the team manage the 
 e ff ort es ti ma ti on uncertainty 20% Predicted probability 26% 12% 73% 2 SP 2 SP 4 SP 4 SP work item #1 work item #2 work item #3 work item #4 17
45. (RQ4) Can we predict whether a work item will have Story Points changes? Sprint Backlog A Product Backlog Selecting work items The work item #4 may have SP change Helping the team manage the 
 e ff ort es ti ma ti on uncertainty 20% Predicted probability 26% 12% 73% 2 SP 2 SP 4 SP 4 SP work item #1 work item #2 work item #3 work item #4 17
46. (RQ4) Can we predict whether a work item will have Story Points changes? Sprint Backlog A Product Backlog Selecting work items The work item #4 may have SP change Helping the team manage the 
 e ff ort es ti ma ti on uncertainty What to do next? 
 - perform detail analysis and re-es ti mate the SP 
 - reserve bu ff er ti me for the uncertainty 20% Predicted probability 26% 12% 73% 2 SP 2 SP 4 SP 4 SP work item #1 work item #2 work item #3 work item #4 17
47. (RQ4) Can we predict whether a work item will have Story Points changes? Building the prediction model 41 metrics extracted from work items Input: * All inputs were extracted at sprint assignment time Developers exp Collaboration Activities Completeness Text Readability Textual content 18
48. (RQ4) Can we predict whether a work item will have Story Points changes? Building the prediction model Technique: 4 machine learning classifiers - Logistic Regression 
 - Random Forest 
 - Support vector machines 
 - Classification & Regression Trees 41 metrics extracted from work items Input: * All inputs were extracted at sprint assignment time Developers exp Collaboration Activities Completeness Text Readability Textual content 18
49. (RQ4) Can we predict whether a work item will have Story Points changes? Building the prediction model Technique: 4 machine learning classifiers - Logistic Regression 
 - Random Forest 
 - Support vector machines 
 - Classification & Regression Trees Prediction target Story Points changes that occurred after the work item was assigned to a sprint Work item created SP=1 SP=3** 
 Sprint A SP=1* 
 Sprint A SP=3 
 Sprint A Work item assigned to the sprint Story Points were changed from 1 to 3 Work item closed Predict SP changes that occurred during this period 41 metrics extracted from work items Input: * All inputs were extracted at sprint assignment time Developers exp Collaboration Activities Completeness Text Readability Textual content 18
50. (RQ4) Can we predict whether a work item will have Story Points changes? Measure the model performance 
 based on 100 bootstrap sample datasets Compare among the 4 variations and 4 baselines (using one-sided Wilcoxon signed-rank test) Model validation 19
51. (RQ4) Can we predict whether a work item will have Story Points changes? Measure the model performance 
 based on 100 bootstrap sample datasets Compare among the 4 variations and 4 baselines (using one-sided Wilcoxon signed-rank test) Model validation - Majority vote of all metrics - LDA-based classi fi er (topics generated from text) - Text-based classi fi er - OneR Comparing baselines 19
52. (RQ4) Can we predict whether a work item will have Story Points changes? Baselines - Majority vote of all metrics - LDA-based classi fi er 
 (topics generated from text) - Text-based classi fi er - OneR * based on 100 bootstrap samples LR model 
 average AUC ≈ 0.72 Prediction Performance 20
53. (RQ4) Can we predict whether a work item will have Story Points changes? LR model (average AUC ≈ 0.72) outperformed the other 3 variations and 4 baselines Baselines - Majority vote of all metrics - LDA-based classi fi er 
 (topics generated from text) - Text-based classi fi er - OneR * based on 100 bootstrap samples LR model 
 average AUC ≈ 0.72 Prediction Performance 20
54. (RQ4) Can we predict whether a work item will have Story Points changes? LR model (average AUC ≈ 0.72) outperformed the other 3 variations and 4 baselines Baselines - Majority vote of all metrics - LDA-based classi fi er 
 (topics generated from text) - Text-based classi fi er - OneR * based on 100 bootstrap samples LR model 
 average AUC ≈ 0.72 The team can use our approach to manage the uncertainty in Story Points es ti ma ti on. Prediction Performance 20
55. Story Points Changes in Agile Iterative Development (RQ1) To what degree do Story Points change? Unchanged SP be tt er re fl ect the development ti me than changed SP Changed 
 Story Points Unchanged 
 Story Points > (RQ2) Do changed Story Points better reflect the development time than the ones that remain unchanged? (RQ3) To what extent are Story Points changes 
 are associated with information changes? (RQ4) Can we predict whether a work item will have Story Points changes? 2 SP 8 SP The team can use our approach to manage the uncertainty in Story Points es ti ma ti on. (AUC ≈ 0.72 in predicting SP changes after a work item is assigned to the sprint) SP are not o ft en changed, but their large changing size can have a nega ti ve impact on the sprint plan. >58% Changing size 3-17% Prevalence SP changes o ft en occurred along with 
 Informa ti on changes (for changing scope) Jirat
 Pasuksmit @jiratpa 21