From Coders to Builders of the Intelligent World

Transcript

1. From coders to builders of the intelligent world
   software engineering for AI: invariants and unsolved problems
   Jez Humble
   CTO, DevOps Research and Assessment LLC
   

   View Slide

2. @jezhumble
   abstract
   • a very short history of devops
   • a brief introduction to ml delivery lifecycle
   • what stays the same
   • what changes
   • goals and unsolved problems
   

   View Slide

3. devops
   Idea Business Delivery Operation Market
   Agile
   addresses this
   DevOps
   addresses this
   

   View Slide

4. @jezhumble
   devops movement
   a cross-functional community of practice dedicated to the study of
   building, evolving and operating rapidly changing, secure, resilient
   systems at scale
   

   View Slide

5. @jezhumble
   the three ways
   

   View Slide

6. @jezhumble
   time to restore service
   lead time for changes (checkin to release)
   deploy frequency
   change fail rate
   first way: metrics for software delivery performance
   

   View Slide

7. @jezhumble
   feedback loops
   “so much of creation is discovery, and you can’t discover anything if
   you can’t see what you’re doing.”
   — Bret Victor
   Bret Victor, Inventing on Principle, http://vimeo.com/36579366
   

   View Slide

8. @jezhumble
   feedback loops
   

   View Slide

9. @jezhumble
   machine learning development lifecycle
   Data ETL
   Select
   algorithm
   Train model
   Validate model
   Data ETL
   Production
   

   View Slide

10. @jezhumble
    mldlc vs sdlc
    sdlc mldlc
    materials code algorithms, scenarios, data
    production package model
    configuration management
    code versioning, infrastructure-as-code, api
    versioning
    data & data dictionary management, model
    & platform versioning, api versioning
    continuous integration
    constantly validating behavior of code
    against tests
    constantly validating model against
    scenarios and data
    continuous delivery
    always ready to deploy to production and
    smart devices (ios / android)
    always ready to deploy to production and
    smart devices (edge)
    observability /
    care and feeding
    instrument code, monitoring & alerting
    infrastructure
    collect model accuracy, continuous model
    training - data feeds back into training
    thanks to 勇张
    

    View Slide

11. @jezhumble
    what’s the same?
    • toolchain for deployment pipeline
    • platform for testing, training, and production deployment
    • optimize for short lead times / tight feedback loops
    • tdd
    

    View Slide

12. @jezhumble
    what’s different?
    • training lead times
    • data management - only possible with teams
    • data pipeline as well as delivery pipeline
    • allocate R&D time for algorithm selection and model training
    • edge: hardware heterogeneity - interface hell
    

    View Slide

13. @jezhumble
    real systems
    

    View Slide

14. @jezhumble
    edge: lessons from microservices
    • each model independently verifiable
    • how to avoid big up-front design for data schemas / apis?
    • how to avoid chatty, fine-grained communication?
    • security is an emergent property
    

    View Slide

15. @jezhumble
    the goal
    exploit and elevate the constraints: hardware and r&d time
    • invest in ml infrastructure: toolchains and platforms
    • for developers, training and validating models, data
    pipelines / ETL, deployment to the cloud and edge,
    instrumentation and ongoing training
    • comprehensive data and configuration management
    • get visibility into - and optimize for - lead times
    

    View Slide

16. @jezhumble
    unsolved problems
    • development and training lead times
    • large-scale development
    • understanding and debugging ml models
    • building and operating highly distributed systems
    • ml hype
    

    View Slide

17. thank you!
    © 2016-18 Jez Humble& Associates LLC
    https://continuous-delivery.com/
    To receive the following:
    • A copy of this presentation
    • The link to the 2018 Accelerate State of DevOps Report (and previous years)
    • A 100 page excerpt from Lean Enterprise
    • Excerpts from the DevOps Handbook and Accelerate
    • 30% off my video workshop: creating high performance organizations
    • A 20m preview of my Continuous Delivery video workshop
    • Discount code for CD video + interviews with Eric Ries & more
    Just pick up your phone and send an email
    To: jezhumble@sendyourslides.com
    Subject: devops
    

    View Slide