Validating your machine learning models with Minimal Effort using DeepChecks

A Talk at MLOps: Machine Learning in production New York City

https://mlopsworld.com/newyork/

What is Deepchecks?
When should I use Deepchecks?
Deepchecks validation structure.
When should I validate?
Use case.
Conclusion.

Transcript

3. Validating your Machine learning models & Data with minimal effort using Deepchecks
4. Who am I? - Data Scientist. - Founder of The African Data Community Newsletter. - Technical Writer, Public Speaker, Social Media Content creator, Machine Learning Thought Leader (Global AI Hub), Open- source & Community Advocate. Gift Ojeabulu CBBAnalytics @GiftOjeabulu_
5. • what active learning is • various data sampling (querying) strategies used with active learning • benefits of, and... • ...disadvantages to active learning Learning Objective
6. What is Active Learning?
7. Active learning is a semi-supervised learning with a human-in-the-loop. The main difference to standard supervised learning is that rather than having humans label all training data, an algorithm or model chooses which data should be labeled.
8. What is DeepChecks?
9. Deep Checks is the leading tool for validating your machine learning models and data, and it enables doing so with minimal effort. Deep Checks accompany you through various validation needs such as: - Verifying your Data Integrity. - Inspect its distribution. - Validating data splits. - Evaluating your model & comparing different models.
10. • Building the ML Model • Data: Collection, Preparation, Processing • Model: Evaluation, Analysis > 80% < 20%
11. When should I validate?
13. Typical Validation Scenarios 1. When you start working with a new dataset: Validate New Data. 2. When you split the data (before training / various cross-validation split / hold-out test set/ …): Validate the Split. 3. When you evaluate a model: Validate Model Performance.
14. 12 Types of Issues – What Can Go Wrong? Data Integrity Methodological Flaws Model Performance Fairness & Bias Data Distribution 9
15. Deepchecks Validation Structure
16. Validation Structure Test Suites Check s Built-in or Custom Display and Result 13 Condition s Pass / Fail / Warning
17. Validation with Deepchecks Checks Condition s Pass / Fail / Warning Test Suites https://github.com/deepchecks/deepchecs
18. Examples of Checks in Deepchecks Integrity Evaluation Methodology Distribution https://docs.deepchecks.com /
19. Check Each check enables inspecting a specific aspect of your data and models, such as data drift, duplicate values, etc. Each check can have 2 types of results: - A visual result meant for display (e.g. a figure or a table) - A return value that can be used for validating the expected check results
20. Condition A condition is a function that can be added to a Check, which returns a pass ✓, fail ✖ or warning ! result, intended for validating the Check’s return value.
21. Suite A suite is an ordered collection of checks that can have conditions added to them. The Suite enables displaying a concluding report for all of the Checks that ran.
22. Use-case
23. How to install Deepchecks Deep Checks requires Python 3 and can be installed using pip or conda, depending on the package manager you’re working with for most of your packages. - Using Pip - PIp Install Deepchecks - Using Conda - conda install -c conda-forge deepchecks - Using Google Colab or Kaggle Kernel - !pip install deepchecks --user
24. All you need to run your first Deepchecks suite
25. In order to run your first Deepchecks Suite all you need to have is the data and model that you wish to validate. More specifically, you need: • Your train and test data (in Pandas DataFrames or Numpy Arrays) • (optional) A supported model (including XGBoost, scikit-learn models, and many more). Required for running checks that need the model’s predictions for running.
27. Load Data, Split Train-Val, and Train a Simple Model
29. Define a Dataset Object
31. Run a DeepChecks suite
34. Run a check
37. Conclusion
38. Deepchecks is fantastic and will only serve to make machine learning model valuation an easier experience. Running all of those tests by hand would take hundreds of lines of code.
39. Questions?
40. Thank you! Gift Ojeabulu CBB Analytics @GiftOjeabulu_