ITT 2016 - Fred George - Agile Roots: Use JIT to Go Faster

Fred George talks about how acceleration drives changes to an organization’s processes and skills, just as the original shift to Agile from waterfall. Organizations reluctant to address such changes, however, inhibit the acceleration. Indeed, their attempts to accelerate often result in lower morale and flawed delivery; these failures drive conservative (and destructive) attitudes against change. In this talk, Fred George addresses a myriad of drivers for faster delivery.

Transcript

1. Go Faster!
 Tech, Process and Organization Fred George fredgeorge@acm.org @fgeorge52 Copyright © 2015 by Fred George. All rights reserved. 1
2. Fred George Copyright © 2015 by Fred George. All rights reserved. 2 Consulting Roles • Change Agent • Disruptor • “Hand grenade I am throwing into development” • CTO describing Fred to his Vice President
3. Industry “Hot” Topics Copyright © 2015 by Fred George. All rights reserved. 3 MicroServices Cloud Docker Dev/Ops Full-stack developer Agile Lean startup Cassandra Event bus MVP (minimum viable product Programmer anarchy Go Faster No-SQL One Hacker Way
4. Why: New Problem Domains: 
 The Cynefin Framework Copyright © 2015 by Fred George. All rights reserved. 4
5. Why: New Problem Domains: 
 The Cynefin Framework Copyright © 2015 by Fred George. All rights reserved. 4 Simple Complicated Cause a Effect Cause aaa Effect
6. Why: New Problem Domains: 
 The Cynefin Framework Copyright © 2015 by Fred George. All rights reserved. 4 Simple Complicated Complex Chaotic Cause a Effect Cause aaa Effect Effect? Cause? Cause ? Effect
7. Why: New Problem Domains: 
 The Cynefin Framework Copyright © 2015 by Fred George. All rights reserved. 4 Simple Complicated Complex Chaotic Disorder
8. Why: New Problem Domains: 
 The Cynefin Framework Copyright © 2015 by Fred George. All rights reserved. 4 Simple Complicated Complex Chaotic Disorder
9. Why: Competition is coming Copyright © 2015 by Fred George. All rights reserved. 5 • Enablement through technologies • Cloud computing • Ubiquitous, high bandwidth • Languages (and supporting frameworks) • Recognition of business opportunities • Silicon Valley innovators as role models • Accelerating business needs • Few inhibitors for global competitors • Reduction of entry barriers for niche competitors • Micro-marketing to specific customer niches
10. How Fast Can You Go? Copyright © 2015 by Fred George. All rights reserved. 6
11. Iteration Length in Agile Age Copyright © 2015 by Fred George. All rights reserved. 7 1 mo. 2015 2000 2005 2010 3 wks. 2 wks. 1 wk. 1 day
12. Iteration Length in Agile Age Copyright © 2015 by Fred George. All rights reserved. 7 1 mo. 2015 2000 2005 2010 3 wks. 2 wks. 1 wk. 1 day XP SCRUM* *2-3 months
13. Iteration Length in Agile Age Copyright © 2015 by Fred George. All rights reserved. 7 1 mo. 2015 2000 2005 2010 3 wks. 2 wks. 1 wk. 1 day XP XP XP SCRUM* *2-3 months
14. Iteration Length in Agile Age Copyright © 2015 by Fred George. All rights reserved. 7 1 mo. 2015 2000 2005 2010 3 wks. 2 wks. 1 wk. 1 day XP XP XP XP SCRUM* *2-3 months
15. Project Delivery Cycles Copyright © 2015 by Fred George. All rights reserved. 8 5 yrs. 1980 1990 2000 1 mon. 1 week 1 day 2010 1 year Project Duration (log)
16. Project Delivery Cycles Copyright © 2015 by Fred George. All rights reserved. 8 5 yrs. 1980 1990 2000 1 mon. 1 week 1 day 2010 1 year Project Duration (log) Waterfall
17. Project Delivery Cycles Copyright © 2015 by Fred George. All rights reserved. 8 5 yrs. 1980 1990 2000 1 mon. 1 week 1 day 2010 1 year Project Duration (log) Waterfall Waterfall Waterfall w OO
18. Project Delivery Cycles Copyright © 2015 by Fred George. All rights reserved. 8 5 yrs. 1980 1990 2000 1 mon. 1 week 1 day 2010 1 year Project Duration (log) Waterfall Waterfall Agile Agile Waterfall w OO
19. Project Delivery Cycles Copyright © 2015 by Fred George. All rights reserved. 8 5 yrs. 1980 1990 2000 1 mon. 1 week 1 day 2010 1 year Project Duration (log) Waterfall Waterfall Agile Agile A2 A2 Waterfall w OO
20. How Fast Can You Go? Copyright © 2015 by Fred George. All rights reserved. 9 Deployment to Production Every 3.5 minutes
21. Inhibitors Copyright © 2015 by Fred George. All rights reserved. 10
22. •Cloud exploitation •Specialized databases •New programming languages •Open source frameworks (users and creators) •Continuous releases “Valley Tech” Copyright © 2015 by Fred George. All rights reserved. 11
23. Hardware Lead Times Copyright © 2015 by Fred George. All rights reserved. 12 6 mo. 1990 1995 2005 30 min. 1 week 1 day 2015 1 mo. 2000 2010 5 min. 5 sec. Order lead times (log)
24. Hardware Lead Times Copyright © 2015 by Fred George. All rights reserved. 12 6 mo. 1990 1995 2005 30 min. 1 week 1 day 2015 1 mo. 2000 2010 5 min. 5 sec. Order lead times (log) Data Center
25. Hardware Lead Times Copyright © 2015 by Fred George. All rights reserved. 12 6 mo. 1990 1995 2005 30 min. 1 week 1 day 2015 1 mo. 2000 2010 5 min. 5 sec. Order lead times (log) Data Center Virtual Machines
26. Hardware Lead Times Copyright © 2015 by Fred George. All rights reserved. 12 6 mo. 1990 1995 2005 30 min. 1 week 1 day 2015 1 mo. 2000 2010 5 min. 5 sec. Order lead times (log) Data Center Virtual Machines Commercial Cloud
27. Hardware Lead Times Copyright © 2015 by Fred George. All rights reserved. 12 6 mo. 1990 1995 2005 30 min. 1 week 1 day 2015 1 mo. 2000 2010 5 min. 5 sec. Order lead times (log) Data Center Virtual Machines Docker Commercial Cloud
28. Hardware Lead Times Copyright © 2015 by Fred George. All rights reserved. 12 6 mo. 1990 1995 2005 30 min. 1 week 1 day 2015 1 mo. 2000 2010 5 min. 5 sec. Order lead times (log) Data Center Virtual Machines Docker Commercial Cloud Capacity planning? Dedicated ops team?
29. Copyright © 2015 by Fred George. All rights reserved. 13 10 1 1000 100 100,000 1 10 100 1000 10,000 Size (loc) Number of Services Services Impact
30. Copyright © 2015 by Fred George. All rights reserved. 13 10 1 1000 100 100,000 1 10 100 1000 10,000 Size (loc) Number of Services Rails / Java Zone Services Impact
31. Copyright © 2015 by Fred George. All rights reserved. 13 10 1 1000 100 100,000 1 10 100 1000 10,000 Size (loc) Number of Services Rails / Java Zone SOA Services Impact
32. Copyright © 2015 by Fred George. All rights reserved. 13 10 1 1000 100 100,000 1 10 100 1000 10,000 Size (loc) Number of Services MicroServices Rails / Java Zone SOA Services Impact
33. Copyright © 2015 by Fred George. All rights reserved. 13 10 1 1000 100 100,000 1 10 100 1000 10,000 Size (loc) Number of Services MicroServices Rails / Java Zone SOA Services Impact
34. Copyright © 2015 by Fred George. All rights reserved. 13 10 1 1000 100 100,000 1 10 100 1000 10,000 Size (loc) Number of Services MicroServices Rails / Java Zone SOA Workshop Services Impact
35. Copyright © 2015 by Fred George. All rights reserved. 13 10 1 1000 100 100,000 1 10 100 1000 10,000 Size (loc) Number of Services MicroServices Rails / Java Zone SOA Workshop Forward Services Impact
36. Copyright © 2015 by Fred George. All rights reserved. 13 10 1 1000 100 100,000 1 10 100 1000 10,000 Size (loc) Number of Services MicroServices Rails / Java Zone SOA Workshop Forward Netflix Services Impact
37. Copyright © 2015 by Fred George. All rights reserved. 14 Kafka Emerging: Event Bus
38. 0MQ Copyright © 2015 by Fred George. All rights reserved. 14 Kafka Emerging: Event Bus
39. 0MQ 0MQ 0MQ 0MQ 0MQ 0MQ Copyright © 2015 by Fred George. All rights reserved. 14 Kafka Emerging: Event Bus
40. 0MQ 0MQ 0MQ 0MQ 0MQ 0MQ Copyright © 2015 by Fred George. All rights reserved. 14 Kafka Emerging: Event Bus
41. 0MQ 0MQ 0MQ 0MQ 0MQ 0MQ Copyright © 2015 by Fred George. All rights reserved. 14 Kafka Service Emerging: Event Bus
42. New Architecture Patterns Copyright © 2015 by Fred George. All rights reserved. 15 Bus Service
43. New Architecture Patterns Copyright © 2015 by Fred George. All rights reserved. 15 Bus Service Need
44. New Architecture Patterns Copyright © 2015 by Fred George. All rights reserved. 15 Bus Service Service Service Need Need Need
45. New Architecture Patterns Copyright © 2015 by Fred George. All rights reserved. 15 Bus Service Service Service Need Need Need Sol’n’s Sol’n Choose Sol’n
46. New Architecture Patterns Copyright © 2015 by Fred George. All rights reserved. 15 Bus Service Service Service Need Need Need Sol’n’s Sol’n •Variants easy •Graceful degradation Choose Sol’n
47. Incremental Applications Copyright © 2015 by Fred George. All rights reserved. 16 Car rent Legacy server CarRent.com Login Sally A B
48. Incremental Applications Copyright © 2015 by Fred George. All rights reserved. 16 Event Bus Rental offers Car rent Legacy server Brand offers Location offers CarRent.com Login Sally A B
49. Incremental Applications Copyright © 2015 by Fred George. All rights reserved. 16 Event Bus Rental offers Car rent Legacy server Brand offers Location offers Segmentation Membership CarRent.com Login Sally A B
50. Databases:
 Holy Grail or Ball-and-Chain? • Fortune 100 view: • Entity-oriented; consistency essential • As few as possible • MicroService view: • Shift to Event Bus, plus • DB per MicroService (if persistence needed) • Poly-glot (various NoSQL, SQL) + event buses • Few (10%) writable; even fewer transactional Copyright © 2015 by Fred George. All rights reserved. 17
51. Rental offers Car rent Legacy server Brand offers Location offers Segmentation Membership CarRent.com Login Sally A B
52. Copyright © 2015 by Fred George. All rights reserved. 18 Rental offers Car rent Legacy server Brand offers Location offers Segmentation Membership CarRent.com Login Sally A B Rental offers Event publishing Solution collecting 300ms responder Redis
53. Rental offers Car rent Legacy server Brand offers Location offers Segmentation Membership CarRent.com Login Sally A B
54. Copyright © 2015 by Fred George. All rights reserved. 19 Rental offers Car rent Legacy server Brand offers Location offers Segmentation Membership CarRent.com Login Sally A B
55. Copyright © 2015 by Fred George. All rights reserved. 19 Rental offers Car rent Legacy server Brand offers Location offers Segmentation Membership CarRent.com Login Sally A B Membership enrichment ETL Key-value
56. Open Source: Netflix Copyright © 2015 by Fred George. All rights reserved. 20
57. Open Source: Docker Copyright © 2015 by Fred George. All rights reserved. 21
58. Disruptive Technology: Functional Languages Copyright © 2015 by Fred George. All rights reserved. 22 •Middleware exploiting: • Kafka written in Scala • RabbitMQ written in Erlang •Case study: MailOnline page rendering • Legacy system: 130K loc Java/JSP’s • Replacement: 4,000 loc Clojure
59. Mitigate Process Inhibitors Copyright © 2015 by Fred George. All rights reserved. 23
60. Understand Your Problem Copyright © 2015 by Fred George. All rights reserved. 24 Complex Simple Complicated Chaotic Cause a Effect Cause aaa Effect Effect? Cause? Cause ? Effect Disorder
61. Copyright © 2015 by Fred George. All rights reserved. 25 Requirements New Vision of “What” to Build
62. Copyright © 2015 by Fred George. All rights reserved. 25 Requirements Trying ideas New Vision of “What” to Build
63. Copyright © 2015 by Fred George. All rights reserved. 25 Requirements Trying ideas “Experimentation drives Innovation” New Vision of “What” to Build
64. Copyright © 2015 by Fred George. All rights reserved. 26 Task Story Co-Credit: Greg Reiser, ThoughtWorks Re-think Interactions
65. Copyright © 2015 by Fred George. All rights reserved. 26 Task Story Feature Project Initiative Co-Credit: Greg Reiser, ThoughtWorks Re-think Interactions
66. Copyright © 2015 by Fred George. All rights reserved. 26 Task Story Feature Project Initiative Co-Credit: Greg Reiser, ThoughtWorks Re-think Interactions
67. Copyright © 2015 by Fred George. All rights reserved. 26 Task Story Feature Project Initiative Co-Credit: Greg Reiser, ThoughtWorks Re-think Interactions
68. Copyright © 2015 by Fred George. All rights reserved. 26 Task Story Feature Project Initiative Co-Credit: Greg Reiser, ThoughtWorks Re-think Interactions
69. • Bad metrics: • Lines of code • Development stages and milestones met • Business success metrics: • Sales, clicks, registrations, customer retention, etc. Measure What Matters Copyright © 2015 by Fred George. All rights reserved. 27
70. Mitigate Organization Inhibitors Copyright © 2015 by Fred George. All rights reserved. 28
71. Over-Specialization Copyright © 2015 by Fred George. All rights reserved. •Theory: • Specialist are more productive •Practice: • Overhead of communication is under-estimated • Unbalanced workload creates delays 29
72. Specialization Institutionalized with Titles Copyright © 2015 by Fred George. All rights reserved. 30 Case Study:
73. Specialization Institutionalized with Titles Copyright © 2015 by Fred George. All rights reserved. 30 Case Study: •50 IT professionals
74. Specialization Institutionalized with Titles Copyright © 2015 by Fred George. All rights reserved. 30 Case Study: •50 IT professionals •25+ titles
75. Specialization Institutionalized with Titles Copyright © 2015 by Fred George. All rights reserved. 30 Case Study: •50 IT professionals •25+ titles •0 people understanding projects
76. Solution: Fix the Titles Copyright © 2015 by Fred George. All rights reserved. 31 Step 1: Define competence in key technologies
77. Solution: Fix the Titles Copyright © 2015 by Fred George. All rights reserved. 31 Step 1: Define competence in key technologies Master Journeyman Apprentice
78. Solution: Fix the Titles Copyright © 2015 by Fred George. All rights reserved. 31 Step 1: Define competence in key technologies Master Journeyman Apprentice Strategic Technologies • Ruby • Java • iOS • Android • DB (SQL and non-SQL) • Testing • OO Design • Node.js • HTML/CSS/JavaScript • …
79. New Titles for New Projects Copyright © 2015 by Fred George. All rights reserved. 32 Developer Competent in a key technology
80. New Titles for New Projects Copyright © 2015 by Fred George. All rights reserved. 32 Graduate Developer Not competent in a key technology Developer
81. New Titles for New Projects Copyright © 2015 by Fred George. All rights reserved. 32 Graduate Developer Developer Senior Developer Expert in a key technology
82. New Titles for New Projects Copyright © 2015 by Fred George. All rights reserved. 32 Graduate Developer Developer Senior Developer System Developer Competent in 5-7 key technologies
83. New Titles for New Projects Copyright © 2015 by Fred George. All rights reserved. 32 Graduate Developer Developer Senior Developer System Developer Master Developer Expert in 3-5 key technologies
84. New Titles for New Projects Copyright © 2015 by Fred George. All rights reserved. 32 Graduate Developer Developer Senior Developer System Developer Master Developer
85. “Fix the Furniture”* Copyright © 2015 by Fred George. All rights reserved. 33 *Kent Beck, XP Explained
86. Non-dedicated Leadership Copyright © 2015 by Fred George. All rights reserved. 34
87. Bring Work to the Team… Copyright © 2015 by Fred George. All rights reserved. 35 Not musical chairs for each small project
88. Copyright © 2015 by Fred George. All rights reserved. 36 MicroServices Cloud Docker Dev/Ops Full-stack developer Agile Lean startup Cassandra Event bus MVP (minimum viable product Programmer anarchy Go Faster No-SQL Managerless process
89. MicroServices Cloud Docker Dev/Ops Full-stack developer Agile Lean startup Cassandra Event bus MVP (minimum viable product Programmer anarchy Go Faster No-SQL Managerless process
90. Copyright © 2015 by Fred George. All rights reserved. 37 MicroServices Cloud Docker Dev/Ops Full-stack developer Agile Lean startup Cassandra Event bus MVP (minimum viable product Programmer anarchy Go Faster No-SQL Managerless process
91. Project Delivery Cycles Copyright © 2015 by Fred George. All rights reserved. 38 5 yrs. 1980 1990 2000 1 mon. 1 week 1 day 2010 1 year Project Duration (log) Waterfall Waterfall Agile Agile Waterfall w OO
92. Project Delivery Cycles Copyright © 2015 by Fred George. All rights reserved. 38 5 yrs. 1980 1990 2000 1 mon. 1 week 1 day 2010 1 year Project Duration (log) Waterfall Waterfall Agile Agile Waterfall w OO
93. Project Delivery Cycles Copyright © 2015 by Fred George. All rights reserved. 38 NRK NRK 5 yrs. 1980 1990 2000 1 mon. 1 week 1 day 2010 1 year Project Duration (log) Waterfall Waterfall Agile Agile Waterfall w OO
94. Acknowledgements Copyright © 2015 by Fred George. All rights reserved. 39 •Neil Hutchinson – Founder of Forward • Adopting a “Fail Fast” culture •Carl Gaywood • Former programmer who wanted to Go Faster
95. Acknowledgements Copyright © 2015 by Fred George. All rights reserved. 39 •Neil Hutchinson – Founder of Forward • Adopting a “Fail Fast” culture •Carl Gaywood • Former programmer who wanted to Go Faster •Paul Ingles and Mike Jones • Implementers of “Experimentation drives innovation”
96. More Information… •Google: • MicroService Architecture for videos • MicroXchg 2015 conference and TODAY! • Programmer Anarchy for managerless teams • Docker for latest container deployment Copyright © 2015 by Fred George. All rights reserved. 40
97. Go Faster:
 Tech, Process, & Organization Fred George fredgeorge@acm.org @fgeorge52 Copyright © 2015 by Fred George. All rights reserved. 41