Value Driven Development - Maximum Impact, Maximum Speed - Dave Thomas - Agile SG 2016

10/18/16
1
© 2014 Bedarra Research Labs. All rights reserved.
Value Driven Development
Maximum Value, Maximum Speed –
Targeted Innovation Trumps Systemic Transformation
Dave Thomas
www.davethomas.net
Chief Scientist, Kx Systems.
Chairman YOW! Conferences,
Carleton University, Canada
Outline
• A Brief Message from your CEO
• Process Driven Development
• Value Driven Development
• Techniques for Reduced Time/Cost to Value

View Slide

10/18/16
2
“We have great confidence in your abilities to meet our competitive
challenges. Hence we have given you Objects, Agile, Tools, Hardware,
New Offices, Mentors because we believe you need these to do your
best. Here is what the business needs from you:
1. We need to be much faster in selecting and executing high
value projects. This means much better estimates of $ and time
to value based on our capacity.
2. We need to turn up the dials on productivity while maintaining
quality and features.
CEO – All Hands Challenge
As long as we keep doing software the same way as everyone else
were have no competitive advantage.
…. THIS IS YOUR MISSION…THIS MESSAGE WILL SELF DESTRUCT ….
Better, Faster, Cheaper – A New Road?
We must adapt and improve our way of building software to
meet the challenge.
1.Focus on Value – Target resources and innovations to
where they will make a difference.
2.Advance our practices – use alternative techniques to
better communicate, design, estimate, build, test and
deploy.
3.Refactor our organization to streamline and enable more
concurrency and reduce cycle time without reducing
quality. We need leverage what works and not be
constrained by current best practices. If it is slow it has
to go!
4.Explore and Experiment – we need to envision
alternatives and evaluate them quickly before betting
too much on any approach. We need to fail fast to
maximize ROI and time.
© Dave Thomas Bedarra Corp

View Slide

10/18/16
3
Lean and Agile Values and Principles
Product Development Activities At-A-Glance
Envisioning
Product Owner Team Common Work Practices
Definition Development Release
Engineering
Prototypes/Models
Requirements
Backlog
Risk Backlog
Team… Team…
GUI Guidelines
Architecture
Product Backlog
Release Backlogs
Team…
Team…
Team…
Team…
Potentially
Shippable
Product
Team
Release
Backlog
Shippable
Code
Increments
Sprint
Release
Backlogs
CI&T
CI&T
©2006-2007 Bedarra Research Labs and Object Mentor
Process Driven Development
Lean and Agile in the Large works but
….10-15% improvement in productivity, 30%+
quality
Seldom has the Impact or Agility expected by
executives
Encourages micro improvements, rather than high
impact change
Biased to tools and technologies of the day
Systemic change expensive to implement and sustain
A generic solution is very seldom as
effective as a strong specific solution

View Slide

10/18/16
4
Leadership Matters!
Has integrity/stands by values, Leads by example,
Executes with transparency, Has the wisdom of
experience , Articulates the vision, Says when they
are wrong, Judges fairly, promotes trust, Delegates
authority, Coaches versus directs, Promotes
constructive diversity, Makes timely consistent
decisions, Sense of humour, Positively motivates and
educates, Always tries to find the best wrong answer,
Has time to listen
©2011 Bedarra Research Labs. All rights reserved.
“Agile Leadership is Fragile” Nigel Dalton, GM Lonley
Planet, now CIO Realestate.com
Accelerate Your Generic Process
Move from Projects to Products (Do Less
Better)
Make requirements tangible. Envisions
Automate Everything and Measure It.
Understand that process always lags best
practice often by 5+ years!

View Slide

10/18/16
5
AIL Portfolio (Backlogs)
Program Feature Team
P1 F1 Blue
F2 Blue
F3 Red
F4 Red
F5 Red
F6 Red
P2 F7 Yellow
F8 Green
F9 Green
F10 Purple
F11 Purple
P3 F12 White
F13 White
F14 White
F15 White
Component F16 Orange
F17 Orange
F18 Orange
Company Backlog
Program Backlogs
Team Backlogs
Program
Feature
Epic
Story
Task
MRI Mechanical Control
Table Movement
Horizontal Movement
Medical Imaging
Position w/ Joy Sticks
Tangible Requirements
• easily expressed by Business
• easily understood by IT
• acceptance easily understood by Business
• acceptance easily understood by IT
• acceptance criteria => acceptance test (AT)
Story
Competitive Delta
Analysis
Customer Field
Studies & Interviews
Technology
Evaluations
Market & Product
Analysis Brainstorming
& Visioning
QFD
House of Quality
Prototyping
Acceptance
Criteria
Story Acceptance
Criteria

View Slide

10/18/16
6
Envisioning – Requirements Through Design
The Waterfall Pitfall
Plan ‘everything’ before you do ‘anything’ until…
It’s 2 years late, 173% over budget, kinda
buggy, & costs $32,345.99…
But “it does everything you’d ever want to do!”
The ‘Potential’ Agile Pitfall
“Planning, shmanning, I need to add 1+1, so let’s just
build it!” On time, on budget, and we’re giving it
away as a beta. “It’s bug-free, it works, and it adds
1+1!” Darn… now how do we add all the features
we never had time to think about?
Envisioning gives Agile some breathing room…
Allows us to understand enough of the vision of ‘tomorrow’…
Low-Fidelity Envisioning Prototype Example
Low-fidelity prototype
§ Initially rough and then later refined drawings
§ Interactive branching allowed walkthrough
§ User model, task model, task flows
§ 3 structure and navigation alternatives
§ 2 visual form alternatives
Concept iterations
§ 6 iterations (expanding from 8 to 48 screens)
§ 3 sprints
§ 3 internal / 2 external customer sessions
Detail iterations
§ 3 iterations (148 screens)
§ 8 sprints
§ 3 internal / 1 external customer sessions
Investment
§ Less than 2% of overall effort

View Slide

10/18/16
7
Value Driven Development
Identify a high value bottleneck which if
accelerated will make a large impact on the
business.
Quickly envision a solution and experiment
• Think Out of the Box (Innovate)
• Use the appropriate HW
• Use appropriate SW and practices
Validate at scale
Complete the work in 3 – 4 months
Management Buy In – ROI and Risk Mitigation
Business
• Clear and tangible measureable goal
• ROI model shows significant business value (5x, >15%)
• Strong Senior Experienced Business Sponsor
• Implementation Timeline of 3 months, max 5
• Minimal Impact on Business Operations
Technical
• Single small team tech plus business with track record
• Localized code changes
• Proof of Concept validation in weeks
• Proof of Scale validation in weeks
• SLA easy to monitor
• Life cycle costs...e.g. special skills, dependency partner?
• Straight forward DevOps deployment
• Minimum dependency
• Independent Acceptance Testing

View Slide

10/18/16
8
Leverage Innovations
Improved Business Practices
• Simplification, Partnering, Regulatory …
Improved Hardware
• Performance, Capacity, Latency, Functionality – atomic
clock, mobility, voice, cloud …
Improved Software
• Usability, Productivity, Flexibility – NoSQL, JSON, REST;
Fault Tolerance – isolation, BASE vs ACID, Languages,
Frameworks…
Improved Practices
• Agile, Micro Services, Continuous Deployment,
Functional Programming, Algorithms, Self Service …
Innovation Opportunities/Insertion Points
Data and Flow Interfaces
• Database Interface
• File Interface
• Sterilization Interface
• Messaging Interface
• Functional Transformers - ETL; Map
Reduce; GPU …
• Query Interface
• Disk/San Interface
• Shared Memory Interface
• Log Interface – Event Sourcing; Historical
DB (Batch) + Real-time DB (Stream)
• Sync Replicate Interface - CDN leverage;
Mobile Occasionally Disconnected; BASE
Fault Tolerant
• Reactive MVC Interface
• Small computational bottle necks
• Highly structured rules/calculations
• Points of high variability/constant
change
• Points with large numbers of defects
Selective Code Focus

View Slide

10/18/16
9
Fast Software
FAST Software = FAST Hardware + Simplicity
Simplicity – Software Sympathy*
• Simplicity in Code
• Simplicity in Architecture
• The flexibility of Data versus Code
• Functional Thinking
* Mechanical Sympathy – Martin Thompson
Simplicity! - The Road Not Taken?
© 2012 Bedarra Research Labs
Complexity
Simplicity

View Slide

10/18/16
10
Let the Hardware Do The Work!
$20,000 buys a computer 1 TB RAM with 40 TB disk and 64 cores!
• Automated Build and Test
• All interesting data is in memory!
• Inexpensive Data Conversion/Translation
• Data Compression and Encryption is “free” on multi-
core
• Data Base is an oxymoron
• Speed and Memory enable Simpler Algorithms
• Enable End User Computing e.g. Excel and R versus
C#, Java
Ship Less Code!
KLOCS Kill!
• The more code you ship the more code you need test
and fix!
Dependencies Strangle!!
• Increase Integration Testing
• Increase Build and Deploy Time
• Often pull in Much than required
Abstractions Bloat!
• Bloat Costs
• Development Time
• Execution Time and Space
• Frameworks inject dependencies into your code!

View Slide

10/18/16
11
Use Less Objects and Less Code !
Object Refactoring harder than Changing Data and
Functions
Table Driven Programming
Rules Decision Table
Calculation Spreadsheet
Data Validation Domain and Range Table
Mapping Lookup Table
Flow Data, Work Flow, Message
Events, Matches State Table
Process, Reports Input-Output Table
Acceptance Criteria BDD
Domain Models Entity-Attribute Dictionary
Enable Customer Self Service
Atom feeds versus APIs
Query versus custom
Expose Scriptable Services
Self Described Data
Table Driven Programming
A picture is 1000 words, a table 400 and a diagram 50
Advantages
• Easily understood by Business, BA, Dev and QA
• Easy to create, refactor and extend using Excel
• Modularity through structured tables
• Consistency /Completeness Checking
• Easy to version and Diff
• Efficient Automated Data Driven implementation
• Data Driven means changes can be “hot deployed” to a
running application
Applications
Insurance, Banking, Taxation, Healthcare, ATC, Real-time...

View Slide

10/18/16
12
Insurance Legacy + SOA + EAP
Challenge
Legacy L1, L2, L3,L4; Vendor EAP V1, V2,V3,V4,V5;
Integration Services I5, I6; Commercial Insurance V1, L1
Personal Insurance L2; Vehicle Insurance V2
Rating Engine L2, L3, V3; Billing System V4
Policy System of Record L4 + I5 + V5
Enterprise Software Bus I6
Solution
1. Outstanding BAs define all products in tables
2. Agile experts generate applications from BA tables
3. SI experts build simple interfaces to ESB + Interface
Acceptance Tests
4. All vendors required to deliver acceptance test
HR System Bottleneck
Challenge
• calculations very complex.
• analysts capture in Excel.
• 100 Devs in COBOL too slow
• Agile OO gave only 15% in productivity
Solution
1. Retain 2 Agile OO experts
2. Excel rule checker in java
3. Spreadsheet in Java on mainframe
4. Legacy team deployed on other systems

View Slide

10/18/16
13
Product at Risk – Massive Data Migration
Challenge
Legacy Database Migration required due to DB Vendor Risk
Product Vendor dependent on DB vendor
Customers want improved query and reporting
Customers have massive data locked in DB vendor at risk
Solution
1. Product Team implements ODBC interface to legacy data
2. Product Team migrates product to ODBC and provides
improved query and reporting . . .
BUT Customers and Product locked to legacy physical data!
and Product needs to change schema!
3. Expert SI retained by Product to perform high performance
bulk conversion to modern database. (Agile DB Refactoring)
4. Independently developed data comparison program used for
acceptance testing
© 2010 - 2011 Bedarra Research Labs. All rights reserved.
Reduce Integration Time and $$$
Avoid Slow and Complex APIs
• Leverage stable physical data formats – pages; logs
• Make R/O SQL replicas for fast reporting
• Put ATOM/RSS feeds on our legacy/partner systems – journal
files, events …
• REST and JSONify your services
• Use ODBC as a simple interface to complex server systems
• Use a simple MashUp tool against to deliver a integrated
application view

View Slide

10/18/16
14
Modernize Legacy Factory Automation
Challenge
$$$ investment in proprietary
automation system
Must to integrate new systems and new
GUI-based control rooms
Solution
1. Modify interface to control units
using TCP/IP
2. Implement HTTP and ATOM
3. Client devices are Web 2.0
appliances or PCs
Why Binary Data Formats are Important
1. Textual data formats such as XML, JSON,
CSV are very expensive to parse
2. Textual data formats are bulky to store 10x
3. A trivial print library will allow easy
inspection of binary logs
4. Make sure your format has a version id for
every format (aka schema change) else
working with schemaless data is
problematic

View Slide

10/18/16
15
Batching , RDB + HDB, Event Source, Lambda
Architecture
Real-time DB + Historical DB
Real-time DB + Batch DB (Twitter Storm)
Enabling Loose Coupling
• All APIs are value based and where possible stateless
• Isolation of services in separate processes/machines
• Simple Pipes and Filters when possible
• RSS/ATOM feeds from events/updates/logs
• Query interface such as ODBC
• Occasionally Disconnected – replication and sync; event
source..
• Simple efficient implementations using co-routines..
• Orchestration/Composition using Scripting Process
• Service Bus, Messaging.. ZeroMq, Rabbit
• Node.js e.g. Azure Mobile Services
• Erlang, Actors, Scala Akka
• FP thinking encourages value orientation and composition

View Slide

10/18/16
16
Service Architecture and Design
Testing is still 30 – 50% of Development
Test Soon – TDD, Inspections, Acceptance Tests
It all starts with acceptance criteria (including ilities)
SBE/Cucumber Good|Bad News?
Side by Side Comparison of Two Full or Partial Implementations
Property Based Testing
• QuickCheck is currently the best examplar
• Generates random tests from a spec
• Collapses False Positives to minimum fail test sequence
• RIAK, Level DB experience
• Dog Fooding Still Best for UI Testing
• Every time a significant amount of code has changes, Freeze the
code for 2 – 3 days
• Have developers test and log bugs on the code of others.
• Fix the bugs for, usually 2 – 3 days.

View Slide

10/18/16
17
Collection Oriented Programming
• Programs written with a functional style of map, filter
… are
• More concise and easier to read.
• Reduce side effects (leverage immutability)
• More amendable to optimization
• Examples
• Underscore.js; CoffeeSript, Elm.., ES6
• LINQ, Rx, Reative Streams
• Map Reduce
• Clojure, Scala, Haskell, Erlang, Ocaml, q, Java 8+, C# 4.0 …
Cyber Analytics
1. Order of magnitude improvement in productivity
§ Interactive, Incremental, Exploratory with rapid deploy to
production
§ Scripting/Domain Oriented Programming by Specialist
Analyst
§ Analyst/End User Visual Query, Transformation and
Visualization at scale
§ Provide full provenance for all artefacts and analysis
§ Granular Cross Jurisdictional Collaboration
2. Fast Data
§ Efficient computation over large volumes of data (at least
10s – 100s TB)
§ Minimal Impedance between persistent (historical) and
live data (real-time)
§ Enable Analytics from Sensors to Cloud
§ Interoperable with existing data formats and tools

View Slide

10/18/16
18
Analyst and Expert Programming Models
• 100:1 ratio analysts to experts
• Experts surface new functionality to analysts as DSL library extensions
Expert Programming Model
• Wide Spectrum Functional Vector Language
• Fully Interoperable with current technologies – C to OO to Web
Analysts Application Programming Model
• Wide Spectrum DSLs ( SQL, Spreadsheets...)
• Narrow Domain Specific DSL (IP packets, finance, geo, cultural ...)
• Leverage existing standards and user models
• Interoperable with R, MatLab, MS Office...
Core VEE Platform – High Performance Vector Functional Runtime
• Data Store Features
• Interoperability Features
• Core Libraries
New
Functionalit
y
Go Forth and Simplify!
Thanks!

View Slide

10/18/16
19
A Conference for Developers by Developers
September 11, 2017

View Slide

Value Driven Development - Maximum Impact, Maximum Speed - Dave Thomas - Agile SG 2016

Value Driven Development - Maximum Impact, Maximum Speed - Dave Thomas - Agile SG 2016

Agile Singapore

More Decks by Agile Singapore

Other Decks in Technology

Featured

Transcript