Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Machine Learning Pipelines in Production - ML Galway Meetup

Emir Muñoz
November 11, 2019

Machine Learning Pipelines in Production - ML Galway Meetup

Talk at the Machine Learning Galway Meetup
https://www.meetup.com/machine-learning-galway/events/266142136/

While most of us have done Machine Learning over well-known/clean/small datasets, there is a big difference when dealing with noisy/large data, which is used to train models and support near-real-time services available to customers. First, there are challenges collecting and cleaning data for training ML models. Then, once the models are trained and ready for deployment, there are monitoring tasks that go beyond DevOps and that can have a huge impact on the performance of products and services. In this talk, Emir will speak about how to standardise the process of data collection for ML, and how to deal with the challenges that appear after models are deployed in production.

Emir Muñoz

November 11, 2019
Tweet

More Decks by Emir Muñoz

Other Decks in Technology

Transcript

  1. Emir Muñoz
    Machine Learning
    Pipelines in Production
    Mon, November 11, 2019

    View full-size slide

  2. …But it can be done 2

    View full-size slide

  3. • Senior ML Engineer and Team Lead at Genesys
    • CS Researcher, SW Developer & Engineer
    • Finishing a PhD @ NUIG in Machine Learning
    using Knowledge Graphs
    • Interested in ML algorithms and their applications
    in real-world problems
    About
    • Genesys is a leader for omnichannel CX & contact
    centre solutions
    • 10,000+ customers in over 100 countries
    • 5,935 employees
    • Genesys AI centre for excellence in Galway with
    80+ people
    • … and growing --- we are hiring ;)

    View full-size slide

  4. “It works in my local machine”
    “It was working 10 minutes ago”
    The gap
    From months to years!

    View full-size slide

  5. Genesys confidential and proprietary information. Unauthorized disclosure is prohibited.
    Machine learning in production
    vs
    Jumping from Blackrock diving
    tower
    • Seems simple, but not everyone can
    do it, or needs to do it
    • You require time, courage, and practice
    (a lot)
    • You require training as well
    • Once you nailed it, you can do it with
    your eyes closed (not really :P)
    Photo by Joe Hyland, https://www.flickr.com/photos/36224551@N04/7267942282

    View full-size slide

  6. • The term artificial intelligence (AI) is applied when
    a machine mimics "cognitive" functions that
    humans associate with other human minds, such
    as "learning" and "problem solving" [Wikipedia]
    o As humans, we can walk, drive cars, read and
    understand a text, create art, etc
    • Machine learning (ML) is a subset of AI that
    focuses on more narrow problems. ML is the
    study of algorithms and statistical models that
    give the ability to computer systems to learn and
    perform a specific task without using explicit
    instructions, relying on patterns and inference
    instead [Wikipedia]
    AI ≠ ML Marketing’s confusion
    Artificial Intelligence
    Machine Learning
    Deep
    Learning
    Focus of
    this talk

    View full-size slide

  7. • What is ML?
    o "A computer program is said to learn from
    experience E with respect to some class of
    tasks T and performance measure P if its
    performance at tasks in T, as measured by P,
    improves with experience E.” --- Mitchell, T.
    (1997). Machine Learning. McGraw Hill.
    • What should I consider?
    o Data is messy, noisy, inconsistent, incomplete, etc
    o You need robustness: training and testing with
    clean data
    o Good pipelines: data processing, scheduled
    trainings, scalability
    o Collaboration with DevOps
    o Keep customers in the loop
    Machine learning in production You need to know what you are doing!
    What is needed
    to build a
    supervised
    learning model
    What is needed to
    build a machine
    learning product

    View full-size slide

  8. You require many people in your organisation to do Machine
    Learning in production
    Tip: Do not underestimate any of these blocks
    (Sculley et al., Hidden Technical Debt in Machine Learning Systems, NIPS 2015)
    Machine learning in production

    View full-size slide

  9. Some people started calling this MLOps
    Machine learning life cycle
    (Amershi et al., Software engineering for machine learning: a case study, ICSE (SEIP) 2019)
    Business case
    Data sources
    KPIs
    etc
    GDPR
    Anonymisation
    Data dictionary
    Joins
    etc
    Cardinality
    Schema
    Seasonality
    etc
    Black magic
    Enrichment
    Selection
    Scaling
    etc
    Algorithm
    Hyperparameters
    Constraints
    Distributed sys.
    etc
    Eval. metrics
    KPIs
    Plots
    etc
    Reproducibility
    Repository
    CI/CD
    etc
    Eval. Metrics
    Logging
    Reports
    A/B testing
    etc

    View full-size slide

  10. • What does this feature mean?
    • From where came this feature?
    • Why is this feature a string/int?
    • Why there are so many null values?
    • Which version of the data is this?
    Data pipelines Ensure you feed quality data to your ML algorithms
    • We need
    o Professional Services or Customers in the loop
    o Data Engineers and data pipelines
    o Stable and scalable technologies
    Inputs Outputs

    View full-size slide

  11. “A ML model is only as good as the data it is fed”
    Model training
    Dataset
    (features + labels)
    Train data
    (features + labels)
    Test data
    (features)
    +
    Model
    Train (fit)
    model
    Model evaluation
    Test data
    (labels)
    Model
    predictions

    View full-size slide

  12. • The metrics depend on your type of model, e.g., classification, regression
    • Share the interpretation of your metrics with everyone
    Tip: Important to obtain and track your metrics
    Model evaluation

    View full-size slide

  13. • Track your experiments: code, data,
    configurations, and results
    • Specially if you are doing hyperparameter
    tuning or model selection
    • Keep track of which models you push to
    production
    • They usually allow you to deploy models to
    production
    Model management
    Sacred

    View full-size slide

  14. Photo by Emir Munoz J

    View full-size slide

  15. • Machine learning models can fail silently!
    • They do not fail as traditional software
    • You should have a post-deployment monitoring
    • Define metrics that make sense for your product
    • You probably need A/B testing methodologies to
    prove business value
    The work after deployment You cannot just relax yet, stay alert!
    • Be proactive
    o Review your unit testing and code in lower
    environments
    o Always have a backup model or dummy method in
    case your fancy model fails
    o Run tests in your production models during off-pick
    hours
    • Check every assumption
    o Your model is not getting the correct number of
    features
    o A feature changed format, e.g., date_timestamp
    o Data doesn’t look like the one you saw during
    training

    View full-size slide

  16. Genesys confidential and proprietary information. Unauthorized disclosure is prohibited.
    Trained using
    adults
    Tested with
    kids
    Different
    demographics

    View full-size slide

  17. Trained using
    bright images
    Tested with
    different lighting

    View full-size slide

  18. • Data may change since the moment you train
    your models
    • Covariate shift: ℙ"#$%&() ≠ ℙ+#,-()
    o Shift in the independent variables
    • Prior probability shift: ℙ"#$%&
    () ≠ ℙ+#,-
    ()
    o Shift in the target variable (labels)
    • Concept shift: ℙ"#$%&(|) ≠ ℙ+#,-(|)
    o Shift in the relationship between the independent
    and the target variables
    Dataset Shift Also called bias by some people
    Formally, ℙ"#$%&(, ) ≠ ℙ+#,-(, )
    (Dawid Kopczyk’s blog post)
    (Krawczyk and Cano, Online Ensemble Learning with Abstaining Classifiers for
    Drifting and Noisy Data Streams, Applied Soft Computing 2018)

    View full-size slide

  19. Takeaways
    Data pipelines
    • Keep data dictionaries
    • Validate that data follows a given
    schema
    • Validate your data types and joins
    Model training
    • Do feature analysis
    • Ensure all features are available for
    prediction
    • Tune hyperparameters only when
    needed
    • Ensure you deal with data imbalance
    Dataset shift
    • There are different ways that things
    could go wrong and sideways
    • Monitor your models’ performance
    • Read about how to identify shifts in
    your data
    Model evaluation and
    management
    • Select models that benefit your
    products
    • Keep track of your experiments and
    models. Specially, for those you push
    to production

    View full-size slide

  20. Thanks
    Ping me if you
    would like to join
    our team
    Emir Muñoz
    first.last [at] genesys.com
    • Remember that ML models are not like traditional software and can
    fail silently
    • Research in ML is evolving pretty fast, try keeping up to date by
    reading regularly
    • Attend conferences and events like ML Galway meetup
    • Test tools like mlflow and Amazon SageMaker to manage your
    models
    • Ensure you monitor your models and empirically demonstrate how
    they affect your business

    View full-size slide