Value Driven Development - Maximum Impact, Maximum Speed - Dave Thomas - Agile SG 2016

Value Driven Development - Maximum Impact, Maximum Speed - Dave Thomas - Agile SG 2016

Transcript

1. 10/18/16 1 © 2014 Bedarra Research Labs. All rights reserved. Value Driven Development Maximum Value, Maximum Speed – Targeted Innovation Trumps Systemic Transformation Dave Thomas www.davethomas.net Chief Scientist, Kx Systems. Chairman YOW! Conferences, Carleton University, Canada Outline • A Brief Message from your CEO • Process Driven Development • Value Driven Development • Techniques for Reduced Time/Cost to Value
2. 10/18/16 2 “We have great confidence in your abilities to meet our competitive challenges. Hence we have given you Objects, Agile, Tools, Hardware, New Offices, Mentors because we believe you need these to do your best. Here is what the business needs from you: 1. We need to be much faster in selecting and executing high value projects. This means much better estimates of $ and time to value based on our capacity. 2. We need to turn up the dials on productivity while maintaining quality and features. CEO – All Hands Challenge As long as we keep doing software the same way as everyone else were have no competitive advantage. …. THIS IS YOUR MISSION…THIS MESSAGE WILL SELF DESTRUCT …. Better, Faster, Cheaper – A New Road? We must adapt and improve our way of building software to meet the challenge. 1.Focus on Value – Target resources and innovations to where they will make a difference. 2.Advance our practices – use alternative techniques to better communicate, design, estimate, build, test and deploy. 3.Refactor our organization to streamline and enable more concurrency and reduce cycle time without reducing quality. We need leverage what works and not be constrained by current best practices. If it is slow it has to go! 4.Explore and Experiment – we need to envision alternatives and evaluate them quickly before betting too much on any approach. We need to fail fast to maximize ROI and time. © Dave Thomas Bedarra Corp
3. 10/18/16 3 Lean and Agile Values and Principles Product Development Activities At-A-Glance Envisioning Product Owner Team Common Work Practices Definition Development Release Engineering Prototypes/Models Requirements Backlog Risk Backlog Team… Team… GUI Guidelines Architecture Product Backlog Release Backlogs Team… Team… Team… Team… Potentially Shippable Product Team Release Backlog Shippable Code Increments Sprint Release Backlogs CI&T CI&T ©2006-2007 Bedarra Research Labs and Object Mentor Process Driven Development Lean and Agile in the Large works but ….10-15% improvement in productivity, 30%+ quality Seldom has the Impact or Agility expected by executives Encourages micro improvements, rather than high impact change Biased to tools and technologies of the day Systemic change expensive to implement and sustain A generic solution is very seldom as effective as a strong specific solution
4. 10/18/16 4 Leadership Matters! Has integrity/stands by values, Leads by example, Executes with transparency, Has the wisdom of experience , Articulates the vision, Says when they are wrong, Judges fairly, promotes trust, Delegates authority, Coaches versus directs, Promotes constructive diversity, Makes timely consistent decisions, Sense of humour, Positively motivates and educates, Always tries to find the best wrong answer, Has time to listen ©2011 Bedarra Research Labs. All rights reserved. “Agile Leadership is Fragile” Nigel Dalton, GM Lonley Planet, now CIO Realestate.com Accelerate Your Generic Process Move from Projects to Products (Do Less Better) Make requirements tangible. Envisions Automate Everything and Measure It. Understand that process always lags best practice often by 5+ years! © 2014 Bedarra Research Labs. All rights reserved.
5. 10/18/16 5 AIL Portfolio (Backlogs) Program Feature Team P1 F1 Blue F2 Blue F3 Red F4 Red F5 Red F6 Red P2 F7 Yellow F8 Green F9 Green F10 Purple F11 Purple P3 F12 White F13 White F14 White F15 White Component F16 Orange F17 Orange F18 Orange Company Backlog Program Backlogs Team Backlogs Program Feature Epic Story Task MRI Mechanical Control Table Movement Horizontal Movement Medical Imaging Position w/ Joy Sticks Tangible Requirements • easily expressed by Business • easily understood by IT • acceptance easily understood by Business • acceptance easily understood by IT • acceptance criteria => acceptance test (AT) ©2004 Bedarra Research Labs. All rights reserved. Story Competitive Delta Analysis Customer Field Studies & Interviews Technology Evaluations Market & Product Analysis Brainstorming & Visioning QFD House of Quality Prototyping Acceptance Criteria Story Acceptance Criteria
6. 10/18/16 6 Envisioning – Requirements Through Design The Waterfall Pitfall Plan ‘everything’ before you do ‘anything’ until… It’s 2 years late, 173% over budget, kinda buggy, & costs $32,345.99… But “it does everything you’d ever want to do!” The ‘Potential’ Agile Pitfall “Planning, shmanning, I need to add 1+1, so let’s just build it!” On time, on budget, and we’re giving it away as a beta. “It’s bug-free, it works, and it adds 1+1!” Darn… now how do we add all the features we never had time to think about? Envisioning gives Agile some breathing room… Allows us to understand enough of the vision of ‘tomorrow’… Low-Fidelity Envisioning Prototype Example Low-fidelity prototype § Initially rough and then later refined drawings § Interactive branching allowed walkthrough § User model, task model, task flows § 3 structure and navigation alternatives § 2 visual form alternatives Concept iterations § 6 iterations (expanding from 8 to 48 screens) § 3 sprints § 3 internal / 2 external customer sessions Detail iterations § 3 iterations (148 screens) § 8 sprints § 3 internal / 1 external customer sessions Investment § Less than 2% of overall effort
7. 10/18/16 7 Value Driven Development Identify a high value bottleneck which if accelerated will make a large impact on the business. Quickly envision a solution and experiment • Think Out of the Box (Innovate) • Use the appropriate HW • Use appropriate SW and practices Validate at scale Complete the work in 3 – 4 months © 2014 Bedarra Research Labs. All rights reserved. Management Buy In – ROI and Risk Mitigation Business • Clear and tangible measureable goal • ROI model shows significant business value (5x, >15%) • Strong Senior Experienced Business Sponsor • Implementation Timeline of 3 months, max 5 • Minimal Impact on Business Operations Technical • Single small team tech plus business with track record • Localized code changes • Proof of Concept validation in weeks • Proof of Scale validation in weeks • SLA easy to monitor • Life cycle costs...e.g. special skills, dependency partner? • Straight forward DevOps deployment • Minimum dependency • Independent Acceptance Testing
8. 10/18/16 8 Leverage Innovations Improved Business Practices • Simplification, Partnering, Regulatory … Improved Hardware • Performance, Capacity, Latency, Functionality – atomic clock, mobility, voice, cloud … Improved Software • Usability, Productivity, Flexibility – NoSQL, JSON, REST; Fault Tolerance – isolation, BASE vs ACID, Languages, Frameworks… Improved Practices • Agile, Micro Services, Continuous Deployment, Functional Programming, Algorithms, Self Service … ©2009 Bedarra Research Labs. All rights reserved. Innovation Opportunities/Insertion Points ©2009 Bedarra Research Labs. All rights reserved. Data and Flow Interfaces • Database Interface • File Interface • Sterilization Interface • Messaging Interface • Functional Transformers - ETL; Map Reduce; GPU … • Query Interface • Disk/San Interface • Shared Memory Interface • Log Interface – Event Sourcing; Historical DB (Batch) + Real-time DB (Stream) • Sync Replicate Interface - CDN leverage; Mobile Occasionally Disconnected; BASE Fault Tolerant • Reactive MVC Interface • Small computational bottle necks • Highly structured rules/calculations • Points of high variability/constant change • Points with large numbers of defects Selective Code Focus
9. 10/18/16 9 Fast Software FAST Software = FAST Hardware + Simplicity Simplicity – Software Sympathy* • Simplicity in Code • Simplicity in Architecture • The flexibility of Data versus Code • Functional Thinking * Mechanical Sympathy – Martin Thompson © 2014 Bedarra Research Labs. All rights reserved. Simplicity! - The Road Not Taken? © 2012 Bedarra Research Labs Complexity Simplicity
10. 10/18/16 10 Let the Hardware Do The Work! $20,000 buys a computer 1 TB RAM with 40 TB disk and 64 cores! • Automated Build and Test • All interesting data is in memory! • Inexpensive Data Conversion/Translation • Data Compression and Encryption is “free” on multi- core • Data Base is an oxymoron • Speed and Memory enable Simpler Algorithms • Enable End User Computing e.g. Excel and R versus C#, Java ©2009 Bedarra Research Labs. All rights reserved. Ship Less Code! © 2012 Bedarra Research Labs. All rights reserved. KLOCS Kill! • The more code you ship the more code you need test and fix! Dependencies Strangle!! • Increase Integration Testing • Increase Build and Deploy Time • Often pull in Much than required Abstractions Bloat! • Bloat Costs • Development Time • Execution Time and Space • Frameworks inject dependencies into your code!
11. 10/18/16 11 Use Less Objects and Less Code ! Object Refactoring harder than Changing Data and Functions © 2012 Bedarra Research Labs. All rights reserved. Table Driven Programming Rules Decision Table Calculation Spreadsheet Data Validation Domain and Range Table Mapping Lookup Table Flow Data, Work Flow, Message Events, Matches State Table Process, Reports Input-Output Table Acceptance Criteria BDD Domain Models Entity-Attribute Dictionary Enable Customer Self Service Atom feeds versus APIs Query versus custom Expose Scriptable Services Self Described Data Table Driven Programming A picture is 1000 words, a table 400 and a diagram 50 Advantages • Easily understood by Business, BA, Dev and QA • Easy to create, refactor and extend using Excel • Modularity through structured tables • Consistency /Completeness Checking • Easy to version and Diff • Efficient Automated Data Driven implementation • Data Driven means changes can be “hot deployed” to a running application Applications Insurance, Banking, Taxation, Healthcare, ATC, Real-time... ©2009 Bedarra Research Labs. All rights reserved.
12. 10/18/16 12 Insurance Legacy + SOA + EAP Challenge Legacy L1, L2, L3,L4; Vendor EAP V1, V2,V3,V4,V5; Integration Services I5, I6; Commercial Insurance V1, L1 Personal Insurance L2; Vehicle Insurance V2 Rating Engine L2, L3, V3; Billing System V4 Policy System of Record L4 + I5 + V5 Enterprise Software Bus I6 Solution 1. Outstanding BAs define all products in tables 2. Agile experts generate applications from BA tables 3. SI experts build simple interfaces to ESB + Interface Acceptance Tests 4. All vendors required to deliver acceptance test HR System Bottleneck Challenge • calculations very complex. • analysts capture in Excel. • 100 Devs in COBOL too slow • Agile OO gave only 15% in productivity Solution 1. Retain 2 Agile OO experts 2. Excel rule checker in java 3. Spreadsheet in Java on mainframe 4. Legacy team deployed on other systems
13. 10/18/16 13 Product at Risk – Massive Data Migration Challenge Legacy Database Migration required due to DB Vendor Risk Product Vendor dependent on DB vendor Customers want improved query and reporting Customers have massive data locked in DB vendor at risk Solution 1. Product Team implements ODBC interface to legacy data 2. Product Team migrates product to ODBC and provides improved query and reporting . . . BUT Customers and Product locked to legacy physical data! and Product needs to change schema! 3. Expert SI retained by Product to perform high performance bulk conversion to modern database. (Agile DB Refactoring) 4. Independently developed data comparison program used for acceptance testing © 2010 - 2011 Bedarra Research Labs. All rights reserved. Reduce Integration Time and $$$ Avoid Slow and Complex APIs • Leverage stable physical data formats – pages; logs • Make R/O SQL replicas for fast reporting • Put ATOM/RSS feeds on our legacy/partner systems – journal files, events … • REST and JSONify your services • Use ODBC as a simple interface to complex server systems • Use a simple MashUp tool against to deliver a integrated application view ©2004 Bedarra Research Labs. All rights reserved.
14. 10/18/16 14 Modernize Legacy Factory Automation Challenge $$$ investment in proprietary automation system Must to integrate new systems and new GUI-based control rooms Solution 1. Modify interface to control units using TCP/IP 2. Implement HTTP and ATOM 3. Client devices are Web 2.0 appliances or PCs Why Binary Data Formats are Important 1. Textual data formats such as XML, JSON, CSV are very expensive to parse 2. Textual data formats are bulky to store 10x 3. A trivial print library will allow easy inspection of binary logs 4. Make sure your format has a version id for every format (aka schema change) else working with schemaless data is problematic © 2014 Bedarra Research Labs. All rights reserved.
15. 10/18/16 15 Batching , RDB + HDB, Event Source, Lambda Architecture © 2013 Bedarra Research Labs. All rights reserved. Real-time DB + Historical DB Real-time DB + Batch DB (Twitter Storm) Enabling Loose Coupling • All APIs are value based and where possible stateless • Isolation of services in separate processes/machines • Simple Pipes and Filters when possible • RSS/ATOM feeds from events/updates/logs • Query interface such as ODBC • Occasionally Disconnected – replication and sync; event source.. • Simple efficient implementations using co-routines.. • Orchestration/Composition using Scripting Process • Service Bus, Messaging.. ZeroMq, Rabbit • Node.js e.g. Azure Mobile Services • Erlang, Actors, Scala Akka • FP thinking encourages value orientation and composition ©2003 Bedarra Research Labs. All rights reserved.
16. 10/18/16 16 ©2003 Bedarra Research Labs. All rights reserved. Service Architecture and Design Testing is still 30 – 50% of Development Test Soon – TDD, Inspections, Acceptance Tests It all starts with acceptance criteria (including ilities) SBE/Cucumber Good|Bad News? Side by Side Comparison of Two Full or Partial Implementations Property Based Testing • QuickCheck is currently the best examplar • Generates random tests from a spec • Collapses False Positives to minimum fail test sequence • RIAK, Level DB experience • Dog Fooding Still Best for UI Testing • Every time a significant amount of code has changes, Freeze the code for 2 – 3 days • Have developers test and log bugs on the code of others. • Fix the bugs for, usually 2 – 3 days. © 2014 Bedarra Research Labs. All rights reserved.
17. 10/18/16 17 Collection Oriented Programming • Programs written with a functional style of map, filter … are • More concise and easier to read. • Reduce side effects (leverage immutability) • More amendable to optimization • Examples • Underscore.js; CoffeeSript, Elm.., ES6 • LINQ, Rx, Reative Streams • Map Reduce • Clojure, Scala, Haskell, Erlang, Ocaml, q, Java 8+, C# 4.0 … © 2014 Bedarra Research Labs. All rights reserved. Cyber Analytics 1. Order of magnitude improvement in productivity § Interactive, Incremental, Exploratory with rapid deploy to production § Scripting/Domain Oriented Programming by Specialist Analyst § Analyst/End User Visual Query, Transformation and Visualization at scale § Provide full provenance for all artefacts and analysis § Granular Cross Jurisdictional Collaboration 2. Fast Data § Efficient computation over large volumes of data (at least 10s – 100s TB) § Minimal Impedance between persistent (historical) and live data (real-time) § Enable Analytics from Sensors to Cloud § Interoperable with existing data formats and tools
18. 10/18/16 18 Analyst and Expert Programming Models • 100:1 ratio analysts to experts • Experts surface new functionality to analysts as DSL library extensions Expert Programming Model • Wide Spectrum Functional Vector Language • Fully Interoperable with current technologies – C to OO to Web Analysts Application Programming Model • Wide Spectrum DSLs ( SQL, Spreadsheets...) • Narrow Domain Specific DSL (IP packets, finance, geo, cultural ...) • Leverage existing standards and user models • Interoperable with R, MatLab, MS Office... Core VEE Platform – High Performance Vector Functional Runtime • Data Store Features • Interoperability Features • Core Libraries New Functionalit y Go Forth and Simplify! Thanks! © 2014 Bedarra Research Labs. All rights reserved.
19. 10/18/16 19 A Conference for Developers by Developers September 11, 2017