Avoiding Technical Bankruptcy

1. A story of reducing technical debt Founder + Digital Craftsman Markus Kobler Avoiding Technical Bankruptcy
2. Overview The technical debt build up Five Techniques that Worked 1) Getting a Canary 2) Rejecting Scrum for Kanban 3) Planning Poker 4) Automating Deployments with ‘Munge’ 5) Sharing Ownership with Fez & Pirate Hats! Dealing with Collapse : A Success Story
3. Taglab did a bit of everything web related... Strategy Transactional Sites Design Email (Even built/hosted our own campaign system!?!) Banners Micro-Sites Brochure-ware Social Media Hosting Ad Campaigns 24/7/365 Support J2EE Web Apps Messaging Middleware Workflow Systems
4. Notable Clients Included Betfair Times Online Starbucks The Economist Disney Channel Turner Global Switch Channel Five NBC Universal Skandia STA Travel Yell Online The Spectator British Gas Avis One Water Liverpool Victoria
5. My First three years at Taglab had been exciting - Six fold growth - Wide range of big and small projects - Rounded skillset from architectural & development to system administration *The E-Myth 2001 After the initial exciting growth the company started to hit a celling - J2EE was loosing its perceived competitive advantage - Common in small business, 80% fail in their first 5 years*
6. Over time technical debt started to build up Its around this time Taglab asked me to rejoin the company as CTO...
7. Hosted Apps had 45+ external integration points - RESTful & SOAP API’s - But many XML(sometimes)/RPC Designed by Architects with Capital A... ...but built by developers (with a lowercase d) Supporting 40-50 live production web apps - 100+ Environments in total (UAT, Integration, etc) - Key required 24/7/365 SLA’s Pain stemmed from multiple points
8. Legacy and modern Codebase - 7+ year old code, Singleton galore with poor test coverage - Some code even needing decompiling from bytecode... - To modern lightweight clean GWT, Hibernate, Spring Hosting Mix - Mix of our own rack in Docklands to clients hardware and the cloud - Although our rack provided many benefits some hardware dated back 5-7 years Pain stemmed from multiple points
9. Projects were increasingly Death Marches
10. Death March Types Happiness Chances of Success Kamikaze Mission Impossible Ugly Suicide Team are sacrificial lambs at the slaughter for successful delivery Team dreams of fame, glory and riches ‘if’ they succeed Projects and Team are doomed. Only alternative is to be fired Projects are doomed, but everyone agrees it will be glorious failure! Death March - Edward Yourdon
11. At its core this is a story about Happiness Chances of Success Positively and creatively changing peoples perspective to improve delivery
12. Problem 1 : Notification Hell Nagios monitoring was already in place - but needed updating Were getting 4-5 alerts a day! - Regular Database/Hardware failures - Deployment, Code Quality problems - Subtle networking issues (ISP or self inflicted) You dreaded your phone Make things worse we needed more checks! - CI needed and much better test coverage
13. Getting a Canary (or Nabaztag) Flickr: julianbleecker
14. Flickr: krissatmontreal Could differentiate personal from work txt messages Problems started feeling a little less ‘catastrophic’ Now was not the right time for ‘one more copy change...’ Over time major alerts dropped to 1 every few weeks Getting a Canary (or Nabaztag)
15. Problem 2 : Juggling Delivery + Support We where hungry for work so new projects varied wildly in size and complexity - Some were multi year builds - Others ‘OMG we need it done by the weekend’ Mixing both functions can be efficient and satisfying Support’s unpredictability impacted new build work - More common in-practice with teams impacted by 2nd/3rd line support issues
16. 'Known Knowns' - requirements captured in the specification 'Known Unknowns' - requirements flagged as ‘assumptions’ 'Unknown Knowns' - requirements the client had not yet told you about 'Unknowns Unknowns' - requirements neither you or client knew about Successful Delivery is about Accepting Uncertainty Flickr : ooocha
17. Sounds like Agile Right? Many competitors and clients were touting SCRUM We needed something less rigid How would unpredictable support be incorporated? - A ‘batman’ would have been wasteful or not enough resource Work/Budget would often not fit neatly into an iteration(s) Clients liked ability to embrace continuous change but still needed fixed delivery costs “Don’t you know the difference between the Chicken and Pig!” “I need to know what is being delivered before I signing off costs!”
18. Rejecting Scrum for Kanban To-Do Doing Done Task Task Task Task Task Task Task Task Task Task Task Task Task So we started simply...
19. added columns that made sense to us... Rejecting Scrum for Kanban To-Do Doing Task Task Task Task Task Task Task Task Task Task Task Task Task UAT CAT Live Task Task Task Task Task
20. matching our internal processes... Rejecting Scrum for Kanban Backlog In Progress Task Task Task Task Task Task Task Task Task Task Task Task Task UAT CAT Live Task Task Task Task Task Planned Dev Internal Test Task Task Task Task Task Task Task Task
21. We introduced swim lanes... Rejecting Scrum for Kanban Backlog In Progress Task Task Task Task Task Task Task Task Task Task Task UAT CAT Live Task Task Task Task Planned Dev Internal Test Task Task Task Task Task Task Task M ajor Project 1 M ajor Project 2 Other Projects Internal Task Task Task Task
22. and then Work In Progress (WIP) limits and SLA’s Rejecting Scrum for Kanban Backlog In Progress Task Task Task Task Task Task Task Task Task Task Task UAT CAT Live Task Task Task Task Planned Dev Internal Test Task Task Task Task Task Task Task M ajor Project 1 M ajor Project 2 Other Projects Internal Task Task Task Task 5 Day SLA 15 4 12
23. added ability to visually distinguish types of tasks Issue Improv- ement Improv- ement Rejecting Scrum for Kanban Backlog In Progress Task Task Task Task Task Task Task Task Task Task Task UAT CAT Live Task Task Task Task Planned Dev Internal Test Task Task Task Task Task Task Task M ajor Project 1 M ajor Project 2 Other Projects Internal Task Task Task Task 5 Day SLA 15 4 12 Improv- ement Improv- ement Issue Issue
24. Rejecting Scrum for Kanban Work is continuously pulled through system Visually provides a Micro & Macro view of work Support issues easily absorbed WIP and SLA’s provided key indicators of quality and resource planning
25. Problem 3 : Estimation + Tracking Velocity Flickr : kozumel
26. Planning Poker Cards going up in value User Story discussed Once happy, everyone estimates in private Reveal at same time Hi & Lo estimates discuss differences Repeat estimate again if no consensus Flickr: lejoe
27. Shares ownership of problem solving Could start analytically looking back at projects past progress Enabled plotting projects into future based on previous tasks/estimates As a PM you can often get team to develop simplest solution and feel good about it! Planning Poker
28. Problem 4 : Deployment Deployment should be the last problem developers need to worry about - Needed to re-instil confidence back in the team Reducing time to deploy was key lowering costs - Worst cases deployments took 5 hours! Needed to adapt to different environments - Our Rack, Client’s hardware, The cloud Had to work well with Designers and Developers
29. Designers Needed Something Simple Flickr : thunderchild5
30. Developers Demanded More Rigour Flickr : shutupyourface
31. Automating Deployments with ‘Munge’ + Version Control Clean Build Environment + = Predictable Process
32. Automating Deployments with ‘Munge’ Mix of automation and strict project conventions - Minor changes could be released in seconds - Major releases, tested and deployed in 5-20mins (depending on changes) Every step could be manually overridden Live environment metrics used to determine deployment hosts
33. Problem 5 : Shared Ownership of Prod Issues Problems often only needed 1 person’s attention Whole team needed to share ownership of production issues Taking ownership should be visual minimising disruption to rest of team
34. So we got some hats of responsibility Flickr : doctorow
35. So we got some hats of responsibility Flickr : c_r_i_s
36. So we got some hats of responsibility Flickr : striatic
37. So we got some hats of responsibility Flickr : blackcountrymuseums
38. So we got some hats of responsibility
39. Finally a Success Story Flickr : dybarber
40. 2009, Setanta Sports was on verge of collapse - A key client dependant on their sports coverage - Unsure if someone would step in with finance - Or if someone else would buy the rights Finally a Success Story Sign off received 3 days before changes needed to be live Particularly complex CRM/Sales intergration - Estimated >5 weeks worth of effort - Deadline was immovable and very damaging for all parties involved if failed to deliver
41. Success Story Work broken into 5 interleaved dev streams We hit the first key 3 day deadline - 4 major releases followed over next 7 days - More followed in the subsequent weeks/months Sales in the days that followed dwarfed anything before with 0 downtime One of the more satisfying projects have been lucky enough to be involved in All thanks to prior hard work put in by the team
42. Final Thoughts Bleak situations can often be turned to opportunities - It just might need a creative solution - Taking educated technical risks are often needed An informative workspace is key - Although is largely about ‘Psychological Manipulation’ Embrace change and unpredictability - but always make sure the team has a clear guiding project vision Software delivery is an art-form - backed by scientific approaches
43. Credits Death March (Yourdon Press Series) - http://www.amazon.co.uk/Death-March-Yourdon-Press-Edward/dp/ 013143635X/ref=sr_1_1?ie=UTF8&qid=1290856412&sr=8-1 Nabaztag - julianbleecker - http://www.flickr.com/photos/julianbleecker/156303245/ Nabaztag - krissatmontreal - http://www.flickr.com/photos/krissatmontreal/3027612184/ 061108-F-5586B-137 - ooocha - http://www.flickr.com/photos/ooocha/3051551020/ Poker Phase - kozumel - http://www.flickr.com/photos/kozumel/4063207100/ Planning poker! I've a straight flush! - lejoe - http://www.flickr.com/photos/lejoe/4553607341/ The Big Easy - thunderchild5 - http://www.flickr.com/photos/thunderchild5/2397161370/ Factory - Robotic Arms - shutupyourface - http://www.flickr.com/photos/shutupyourface/105779625/ 'Brookie Bond' - the happy chimp - http://www.flickr.com/photos/lwr/4018943800/ Me in Ada's pirate hat, monorail queue, Walt Disney World, Orlando, Florida - doctorow - http://www.flickr.com/ photos/doctorow/2137686/ Fez / 87.365 - c_r_i_s - http://www.flickr.com/photos/c_r_i_s/2852508138/ past tense - striatic - http://www.flickr.com/photos/striatic/2418853122/ 1940s flying helmet , LC1995_15_1 - blackcountrymuseums - http://www.flickr.com/photos/ blackcountrymuseums/4970602229/ Fremantle Bridge Collapse, 22 July 1926 - dybarber - http://www.flickr.com/photos/dybarber/3461384691/
44. Thank you. distinctive.co @markuskobler