Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Value Driven Development - Maximum Impact, Maximum Speed - Dave Thomas - Agile SG 2016

Value Driven Development - Maximum Impact, Maximum Speed - Dave Thomas - Agile SG 2016

Value Driven Development - Maximum Impact, Maximum Speed - Dave Thomas - Agile SG 2016

Agile Singapore

October 19, 2016
Tweet

More Decks by Agile Singapore

Other Decks in Technology

Transcript

  1. 10/18/16
    1
    © 2014 Bedarra Research Labs. All rights reserved.
    Value Driven Development
    Maximum Value, Maximum Speed –
    Targeted Innovation Trumps Systemic Transformation
    Dave Thomas
    www.davethomas.net
    Chief Scientist, Kx Systems.
    Chairman YOW! Conferences,
    Carleton University, Canada
    Outline
    • A Brief Message from your CEO
    • Process Driven Development
    • Value Driven Development
    • Techniques for Reduced Time/Cost to Value

    View full-size slide

  2. 10/18/16
    2
    “We have great confidence in your abilities to meet our competitive
    challenges. Hence we have given you Objects, Agile, Tools, Hardware,
    New Offices, Mentors because we believe you need these to do your
    best. Here is what the business needs from you:
    1. We need to be much faster in selecting and executing high
    value projects. This means much better estimates of $ and time
    to value based on our capacity.
    2. We need to turn up the dials on productivity while maintaining
    quality and features.
    CEO – All Hands Challenge
    As long as we keep doing software the same way as everyone else
    were have no competitive advantage.
    …. THIS IS YOUR MISSION…THIS MESSAGE WILL SELF DESTRUCT ….
    Better, Faster, Cheaper – A New Road?
    We must adapt and improve our way of building software to
    meet the challenge.
    1.Focus on Value – Target resources and innovations to
    where they will make a difference.
    2.Advance our practices – use alternative techniques to
    better communicate, design, estimate, build, test and
    deploy.
    3.Refactor our organization to streamline and enable more
    concurrency and reduce cycle time without reducing
    quality. We need leverage what works and not be
    constrained by current best practices. If it is slow it has
    to go!
    4.Explore and Experiment – we need to envision
    alternatives and evaluate them quickly before betting
    too much on any approach. We need to fail fast to
    maximize ROI and time.
    © Dave Thomas Bedarra Corp

    View full-size slide

  3. 10/18/16
    3
    Lean and Agile Values and Principles
    Product Development Activities At-A-Glance
    Envisioning
    Product Owner Team Common Work Practices
    Definition Development Release
    Engineering
    Prototypes/Models
    Requirements
    Backlog
    Risk Backlog
    Team… Team…
    GUI Guidelines
    Architecture
    Product Backlog
    Release Backlogs
    Team…
    Team…
    Team…
    Team…
    Potentially
    Shippable
    Product
    Team
    Release
    Backlog
    Shippable
    Code
    Increments
    Sprint
    Release
    Backlogs
    CI&T
    CI&T
    ©2006-2007 Bedarra Research Labs and Object Mentor
    Process Driven Development
    Lean and Agile in the Large works but
    ….10-15% improvement in productivity, 30%+
    quality
    Seldom has the Impact or Agility expected by
    executives
    Encourages micro improvements, rather than high
    impact change
    Biased to tools and technologies of the day
    Systemic change expensive to implement and sustain
    A generic solution is very seldom as
    effective as a strong specific solution

    View full-size slide

  4. 10/18/16
    4
    Leadership Matters!
    Has integrity/stands by values, Leads by example,
    Executes with transparency, Has the wisdom of
    experience , Articulates the vision, Says when they
    are wrong, Judges fairly, promotes trust, Delegates
    authority, Coaches versus directs, Promotes
    constructive diversity, Makes timely consistent
    decisions, Sense of humour, Positively motivates and
    educates, Always tries to find the best wrong answer,
    Has time to listen
    ©2011 Bedarra Research Labs. All rights reserved.
    “Agile Leadership is Fragile” Nigel Dalton, GM Lonley
    Planet, now CIO Realestate.com
    Accelerate Your Generic Process
    Move from Projects to Products (Do Less
    Better)
    Make requirements tangible. Envisions
    Automate Everything and Measure It.
    Understand that process always lags best
    practice often by 5+ years!
    © 2014 Bedarra Research Labs. All rights reserved.

    View full-size slide

  5. 10/18/16
    5
    AIL Portfolio (Backlogs)
    Program Feature Team
    P1 F1 Blue
    F2 Blue
    F3 Red
    F4 Red
    F5 Red
    F6 Red
    P2 F7 Yellow
    F8 Green
    F9 Green
    F10 Purple
    F11 Purple
    P3 F12 White
    F13 White
    F14 White
    F15 White
    Component F16 Orange
    F17 Orange
    F18 Orange
    Company Backlog
    Program Backlogs
    Team Backlogs
    Program
    Feature
    Epic
    Story
    Task
    MRI Mechanical Control
    Table Movement
    Horizontal Movement
    Medical Imaging
    Position w/ Joy Sticks
    Tangible Requirements
    • easily expressed by Business
    • easily understood by IT
    • acceptance easily understood by Business
    • acceptance easily understood by IT
    • acceptance criteria => acceptance test (AT)
    ©2004 Bedarra Research Labs. All rights reserved.
    Story
    Competitive Delta
    Analysis
    Customer Field
    Studies & Interviews
    Technology
    Evaluations
    Market & Product
    Analysis Brainstorming
    & Visioning
    QFD
    House of Quality
    Prototyping
    Acceptance
    Criteria
    Story Acceptance
    Criteria

    View full-size slide

  6. 10/18/16
    6
    Envisioning – Requirements Through Design
    The Waterfall Pitfall
    Plan ‘everything’ before you do ‘anything’ until…
    It’s 2 years late, 173% over budget, kinda
    buggy, & costs $32,345.99…
    But “it does everything you’d ever want to do!”
    The ‘Potential’ Agile Pitfall
    “Planning, shmanning, I need to add 1+1, so let’s just
    build it!” On time, on budget, and we’re giving it
    away as a beta. “It’s bug-free, it works, and it adds
    1+1!” Darn… now how do we add all the features
    we never had time to think about?
    Envisioning gives Agile some breathing room…
    Allows us to understand enough of the vision of ‘tomorrow’…
    Low-Fidelity Envisioning Prototype Example
    Low-fidelity prototype
    § Initially rough and then later refined drawings
    § Interactive branching allowed walkthrough
    § User model, task model, task flows
    § 3 structure and navigation alternatives
    § 2 visual form alternatives
    Concept iterations
    § 6 iterations (expanding from 8 to 48 screens)
    § 3 sprints
    § 3 internal / 2 external customer sessions
    Detail iterations
    § 3 iterations (148 screens)
    § 8 sprints
    § 3 internal / 1 external customer sessions
    Investment
    § Less than 2% of overall effort

    View full-size slide

  7. 10/18/16
    7
    Value Driven Development
    Identify a high value bottleneck which if
    accelerated will make a large impact on the
    business.
    Quickly envision a solution and experiment
    • Think Out of the Box (Innovate)
    • Use the appropriate HW
    • Use appropriate SW and practices
    Validate at scale
    Complete the work in 3 – 4 months
    © 2014 Bedarra Research Labs. All rights reserved.
    Management Buy In – ROI and Risk Mitigation
    Business
    • Clear and tangible measureable goal
    • ROI model shows significant business value (5x, >15%)
    • Strong Senior Experienced Business Sponsor
    • Implementation Timeline of 3 months, max 5
    • Minimal Impact on Business Operations
    Technical
    • Single small team tech plus business with track record
    • Localized code changes
    • Proof of Concept validation in weeks
    • Proof of Scale validation in weeks
    • SLA easy to monitor
    • Life cycle costs...e.g. special skills, dependency partner?
    • Straight forward DevOps deployment
    • Minimum dependency
    • Independent Acceptance Testing

    View full-size slide

  8. 10/18/16
    8
    Leverage Innovations
    Improved Business Practices
    • Simplification, Partnering, Regulatory …
    Improved Hardware
    • Performance, Capacity, Latency, Functionality – atomic
    clock, mobility, voice, cloud …
    Improved Software
    • Usability, Productivity, Flexibility – NoSQL, JSON, REST;
    Fault Tolerance – isolation, BASE vs ACID, Languages,
    Frameworks…
    Improved Practices
    • Agile, Micro Services, Continuous Deployment,
    Functional Programming, Algorithms, Self Service …
    ©2009 Bedarra Research Labs. All rights reserved.
    Innovation Opportunities/Insertion Points
    ©2009 Bedarra Research Labs. All rights reserved.
    Data and Flow Interfaces
    • Database Interface
    • File Interface
    • Sterilization Interface
    • Messaging Interface
    • Functional Transformers - ETL; Map
    Reduce; GPU …
    • Query Interface
    • Disk/San Interface
    • Shared Memory Interface
    • Log Interface – Event Sourcing; Historical
    DB (Batch) + Real-time DB (Stream)
    • Sync Replicate Interface - CDN leverage;
    Mobile Occasionally Disconnected; BASE
    Fault Tolerant
    • Reactive MVC Interface
    • Small computational bottle necks
    • Highly structured rules/calculations
    • Points of high variability/constant
    change
    • Points with large numbers of defects
    Selective Code Focus

    View full-size slide

  9. 10/18/16
    9
    Fast Software
    FAST Software = FAST Hardware + Simplicity
    Simplicity – Software Sympathy*
    • Simplicity in Code
    • Simplicity in Architecture
    • The flexibility of Data versus Code
    • Functional Thinking
    * Mechanical Sympathy – Martin Thompson
    © 2014 Bedarra Research Labs. All rights reserved.
    Simplicity! - The Road Not Taken?
    © 2012 Bedarra Research Labs
    Complexity
    Simplicity

    View full-size slide

  10. 10/18/16
    10
    Let the Hardware Do The Work!
    $20,000 buys a computer 1 TB RAM with 40 TB disk and 64 cores!
    • Automated Build and Test
    • All interesting data is in memory!
    • Inexpensive Data Conversion/Translation
    • Data Compression and Encryption is “free” on multi-
    core
    • Data Base is an oxymoron
    • Speed and Memory enable Simpler Algorithms
    • Enable End User Computing e.g. Excel and R versus
    C#, Java
    ©2009 Bedarra Research Labs. All rights reserved.
    Ship Less Code!
    © 2012 Bedarra Research Labs. All rights reserved.
    KLOCS Kill!
    • The more code you ship the more code you need test
    and fix!
    Dependencies Strangle!!
    • Increase Integration Testing
    • Increase Build and Deploy Time
    • Often pull in Much than required
    Abstractions Bloat!
    • Bloat Costs
    • Development Time
    • Execution Time and Space
    • Frameworks inject dependencies into your code!

    View full-size slide

  11. 10/18/16
    11
    Use Less Objects and Less Code !
    Object Refactoring harder than Changing Data and
    Functions
    © 2012 Bedarra Research Labs. All rights reserved.
    Table Driven Programming
    Rules Decision Table
    Calculation Spreadsheet
    Data Validation Domain and Range Table
    Mapping Lookup Table
    Flow Data, Work Flow, Message
    Events, Matches State Table
    Process, Reports Input-Output Table
    Acceptance Criteria BDD
    Domain Models Entity-Attribute Dictionary
    Enable Customer Self Service
    Atom feeds versus APIs
    Query versus custom
    Expose Scriptable Services
    Self Described Data
    Table Driven Programming
    A picture is 1000 words, a table 400 and a diagram 50
    Advantages
    • Easily understood by Business, BA, Dev and QA
    • Easy to create, refactor and extend using Excel
    • Modularity through structured tables
    • Consistency /Completeness Checking
    • Easy to version and Diff
    • Efficient Automated Data Driven implementation
    • Data Driven means changes can be “hot deployed” to a
    running application
    Applications
    Insurance, Banking, Taxation, Healthcare, ATC, Real-time...
    ©2009 Bedarra Research Labs. All rights reserved.

    View full-size slide

  12. 10/18/16
    12
    Insurance Legacy + SOA + EAP
    Challenge
    Legacy L1, L2, L3,L4; Vendor EAP V1, V2,V3,V4,V5;
    Integration Services I5, I6; Commercial Insurance V1, L1
    Personal Insurance L2; Vehicle Insurance V2
    Rating Engine L2, L3, V3; Billing System V4
    Policy System of Record L4 + I5 + V5
    Enterprise Software Bus I6
    Solution
    1. Outstanding BAs define all products in tables
    2. Agile experts generate applications from BA tables
    3. SI experts build simple interfaces to ESB + Interface
    Acceptance Tests
    4. All vendors required to deliver acceptance test
    HR System Bottleneck
    Challenge
    • calculations very complex.
    • analysts capture in Excel.
    • 100 Devs in COBOL too slow
    • Agile OO gave only 15% in productivity
    Solution
    1. Retain 2 Agile OO experts
    2. Excel rule checker in java
    3. Spreadsheet in Java on mainframe
    4. Legacy team deployed on other systems

    View full-size slide

  13. 10/18/16
    13
    Product at Risk – Massive Data Migration
    Challenge
    Legacy Database Migration required due to DB Vendor Risk
    Product Vendor dependent on DB vendor
    Customers want improved query and reporting
    Customers have massive data locked in DB vendor at risk
    Solution
    1. Product Team implements ODBC interface to legacy data
    2. Product Team migrates product to ODBC and provides
    improved query and reporting . . .
    BUT Customers and Product locked to legacy physical data!
    and Product needs to change schema!
    3. Expert SI retained by Product to perform high performance
    bulk conversion to modern database. (Agile DB Refactoring)
    4. Independently developed data comparison program used for
    acceptance testing
    © 2010 - 2011 Bedarra Research Labs. All rights reserved.
    Reduce Integration Time and $$$
    Avoid Slow and Complex APIs
    • Leverage stable physical data formats – pages; logs
    • Make R/O SQL replicas for fast reporting
    • Put ATOM/RSS feeds on our legacy/partner systems – journal
    files, events …
    • REST and JSONify your services
    • Use ODBC as a simple interface to complex server systems
    • Use a simple MashUp tool against to deliver a integrated
    application view
    ©2004 Bedarra Research Labs. All rights reserved.

    View full-size slide

  14. 10/18/16
    14
    Modernize Legacy Factory Automation
    Challenge
    $$$ investment in proprietary
    automation system
    Must to integrate new systems and new
    GUI-based control rooms
    Solution
    1. Modify interface to control units
    using TCP/IP
    2. Implement HTTP and ATOM
    3. Client devices are Web 2.0
    appliances or PCs
    Why Binary Data Formats are Important
    1. Textual data formats such as XML, JSON,
    CSV are very expensive to parse
    2. Textual data formats are bulky to store 10x
    3. A trivial print library will allow easy
    inspection of binary logs
    4. Make sure your format has a version id for
    every format (aka schema change) else
    working with schemaless data is
    problematic
    © 2014 Bedarra Research Labs. All rights reserved.

    View full-size slide

  15. 10/18/16
    15
    Batching , RDB + HDB, Event Source, Lambda
    Architecture
    © 2013 Bedarra Research Labs. All rights reserved.
    Real-time DB + Historical DB
    Real-time DB + Batch DB (Twitter Storm)
    Enabling Loose Coupling
    • All APIs are value based and where possible stateless
    • Isolation of services in separate processes/machines
    • Simple Pipes and Filters when possible
    • RSS/ATOM feeds from events/updates/logs
    • Query interface such as ODBC
    • Occasionally Disconnected – replication and sync; event
    source..
    • Simple efficient implementations using co-routines..
    • Orchestration/Composition using Scripting Process
    • Service Bus, Messaging.. ZeroMq, Rabbit
    • Node.js e.g. Azure Mobile Services
    • Erlang, Actors, Scala Akka
    • FP thinking encourages value orientation and composition
    ©2003 Bedarra Research Labs. All rights reserved.

    View full-size slide

  16. 10/18/16
    16
    ©2003 Bedarra Research Labs. All rights reserved.
    Service Architecture and Design
    Testing is still 30 – 50% of Development
    Test Soon – TDD, Inspections, Acceptance Tests
    It all starts with acceptance criteria (including ilities)
    SBE/Cucumber Good|Bad News?
    Side by Side Comparison of Two Full or Partial Implementations
    Property Based Testing
    • QuickCheck is currently the best examplar
    • Generates random tests from a spec
    • Collapses False Positives to minimum fail test sequence
    • RIAK, Level DB experience
    • Dog Fooding Still Best for UI Testing
    • Every time a significant amount of code has changes, Freeze the
    code for 2 – 3 days
    • Have developers test and log bugs on the code of others.
    • Fix the bugs for, usually 2 – 3 days.
    © 2014 Bedarra Research Labs. All rights reserved.

    View full-size slide

  17. 10/18/16
    17
    Collection Oriented Programming
    • Programs written with a functional style of map, filter
    … are
    • More concise and easier to read.
    • Reduce side effects (leverage immutability)
    • More amendable to optimization
    • Examples
    • Underscore.js; CoffeeSript, Elm.., ES6
    • LINQ, Rx, Reative Streams
    • Map Reduce
    • Clojure, Scala, Haskell, Erlang, Ocaml, q, Java 8+, C# 4.0 …
    © 2014 Bedarra Research Labs. All rights reserved.
    Cyber Analytics
    1. Order of magnitude improvement in productivity
    § Interactive, Incremental, Exploratory with rapid deploy to
    production
    § Scripting/Domain Oriented Programming by Specialist
    Analyst
    § Analyst/End User Visual Query, Transformation and
    Visualization at scale
    § Provide full provenance for all artefacts and analysis
    § Granular Cross Jurisdictional Collaboration
    2. Fast Data
    § Efficient computation over large volumes of data (at least
    10s – 100s TB)
    § Minimal Impedance between persistent (historical) and
    live data (real-time)
    § Enable Analytics from Sensors to Cloud
    § Interoperable with existing data formats and tools

    View full-size slide

  18. 10/18/16
    18
    Analyst and Expert Programming Models
    • 100:1 ratio analysts to experts
    • Experts surface new functionality to analysts as DSL library extensions
    Expert Programming Model
    • Wide Spectrum Functional Vector Language
    • Fully Interoperable with current technologies – C to OO to Web
    Analysts Application Programming Model
    • Wide Spectrum DSLs ( SQL, Spreadsheets...)
    • Narrow Domain Specific DSL (IP packets, finance, geo, cultural ...)
    • Leverage existing standards and user models
    • Interoperable with R, MatLab, MS Office...
    Core VEE Platform – High Performance Vector Functional Runtime
    • Data Store Features
    • Interoperability Features
    • Core Libraries
    New
    Functionalit
    y
    Go Forth and Simplify!
    Thanks!
    © 2014 Bedarra Research Labs. All rights reserved.

    View full-size slide

  19. 10/18/16
    19
    A Conference for Developers by Developers
    September 11, 2017

    View full-size slide