Machine Learning System Dynamics: Beyond Model Development

November 2022 - Company conference talk for an audience of diverse backgrounds (Data Practitioners, Product Owners, Software Engineers, Machine Learning Engineers, Engineering Managers, and more).

Transcript

1. Machine Learning System Dynamics: Beyond Model Development Omayma S.
3. “Magic!”
4. How many of you have “interacted” with at least one ML-system? 🙋🙋
5. And more…
6. FLDSMDFR
7. FLDSMDFR
8. Flint Lockwood Diatonic Super Mutating Dynamic Food Replicator
13. I’ve got this under control and it’s not gonna end in disaster
16. FLDSMDFR Post-mortem!
17. LINEAR THINKING LINEAR WORLD IN A NON
18. Source
19. Source help doctors
20. 232 Algorithms! Source
21. Dataset Curation Model Training Metrics Calculation Metrics Reporting Monitoring Model Serving …… How would you describe an ML-System?
22. Dataflow VertexAI Job BigQuery Cloud Workflow ……. CloudRun/ API …… How would you describe an ML-System?
23. Different Areas of Expertise = Different Perspectives Relativity M.C.Escher, 1953
24. Different Areas of Expertise = Different Mental Models Relativity M.C.Escher, 1953
25. Research Scientist Data Engineer Data Scientist MLOps Macine Learning Engineer Data Analyst Data Product Manager Social Scientists Legal Practitioners And more….. Linguists UX Designers Software Engineer
26. An INTERCONNECTED set of COMPONENTS, organized in a way that achieves a certain OBJECTIVE. SYSTEM
27. An INTERCONNECTED set of COMPONENTS, organized in a way that achieves a certain OBJECTIVE. SYSTEM
28. 1 | COMPONENTS
29. 1 | COMPONENTS • Identify. • Describe characteristics. • Understand limitations.
30. Think about electronic components!
31. source
32. source
33. source 22 PAGES
34. Datasheets for Datasets Model Cards
35. Datasheets for Datasets Model Cards https://arxiv.org/abs/1803.09010 https://arxiv.org/abs/1810.03993
36. Datasheets for Datasets Model Cards
37. https://huggingface.co/datasets/imagenet-1k
38. https://huggingface.co/datasets/imagenet-1k Tasks Annotation Size Data Splits
39. Personal and Sensitive Information The 1,000 categories selected for this subset contain only 3 people categories (scuba diver, bridegroom, and baseball player) while the full ImageNet contains 2,832 people categories under the person subtree (accounting for roughly 8.3% of the total images). This subset does contain the images of people without their consent. Though, the study in [1] on obfuscating faces of the people in the ImageNet 2012 subset shows that blurring people's faces causes a very minor decrease in accuracy (~0.6%) suggesting that privacy-aware models can be trained on ImageNet. On larger ImageNet, there has been an attempt at filtering and balancing the people subtree in the larger ImageNet.
40. Personal and Sensitive Information The 1,000 categories selected for this subset contain only 3 people categories (scuba diver, bridegroom, and baseball player) while the full ImageNet contains 2,832 people categories under the person subtree (accounting for roughly 8.3% of the total images). This subset does contain the images of people without their consent. Though, the study in [1] on obfuscating faces of the people in the ImageNet 2012 subset shows that blurring people's faces causes a very minor decrease in accuracy (~0.6%) suggesting that privacy-aware models can be trained on ImageNet. On larger ImageNet, there has been an attempt at filtering and balancing the people subtree in the larger ImageNet.
41. Source
42. Datasheets for Datasets Model Cards
43. Toxicity Classifier https://arxiv.org/abs/1810.03993
44. “Toxic” Text Classifier https://arxiv.org/abs/1810.03993
45. "a photograph of an astronaut riding a horse" https://huggingface.co/blog/stable_diffusion Stable Diffusion
46. Stable Diffusion https://huggingface.co/CompVis/stable-diffusion-v1-4
47. Stable Diffusion h<ps://huggingface.co/CompVis/stable-diffusion-v1-4
48. Stable Diffusion https://huggingface.co/CompVis/stable-diffusion-v1-4
49. Stable Diffusion h<ps://huggingface.co/CompVis/stable-diffusion-v1-4
50. 2 | INTERCONNECTIONS
51. 2 | INTERCONNECTIONS • Flow of Data. • ML System Archetypes. • Feedback Loops.
52. > THE SYSTEM P+A+R+T+S
53. Open-loop vs. Closed-loop System Behavior
54. ML System Archetypes Explore-Exploit Dilemma e.g. Ranking
55. ML System Archetypes Bias Amplififcation e.g. Text Generation, Image Classification
56. ML System Archetypes Echo Chamber Generation e.g. Recommender Systems
57. ML System Archetypes Explore-Exploit Dilemma Bias Amplififcation Echo Chamber Generation …..
59. Machine learning is used extensively in recommender systems deployed in products. The decisions made by these systems can influence user beliefs and preferences which in turn affect the feedback the learning system receives - thus creating a feedback loop. This phenomenon can give rise to the so-called "echo chambers" or "filter bubbles" that have user and societal implications. In this paper, we provide a novel theoretical analysis that examines both the role of user dynamics and the behavior of recommender systems, disentangling the echo chamber from the filter bubble effect. In addition, we offer practical solutions to slow down system degeneracy. Our study contributes toward understanding and developing solutions to commonly cited issues in the complex temporal scenario, an area that is still largely unexplored.
60. Machine learning is used extensively in recommender systems deployed in products. The decisions made by these systems can influence user beliefs and preferences which in turn affect the feedback the learning system receives - thus creating a feedback loop. This phenomenon can give rise to the so-called "echo chambers" or "filter bubbles" that have user and societal implications. In this paper, we provide a novel theoretical analysis that examines both the role of user dynamics and the behavior of recommender systems, disentangling the echo chamber from the filter bubble effect. In addition, we offer practical solutions to slow down system degeneracy. Our study contributes toward understanding and developing solutions to commonly cited issues in the complex temporal scenario, an area that is still largely unexplored.
61. 3 | OBJECTIVE
62. 3 | OBJECTIVE Or Should We Say OBJECTIVE S?
63. Should We Say OBJECTIVE S?
64. OBJECTIVE S? Performance Compression Interpretability Fairness …… ……………. Icons from: https://www.flaticon.com/
65. Performance Compression Interpretability Fairness …… ……………. ROI Regulatory Compliance Icons from: https://www.flaticon.com/ OBJECTIVE S?
66. Performance Compression Interpretability Fairness …… ……………. ROI Regulatory Compliance Icons from: https://www.flaticon.com/ OBJECTIVE S?
67. How Do You Define/Report Your Metrics? Graduated in the top half of the bottom half of my class
68. Performance Compression How Do You Evaluate Tradeoffs? AGGREGATED METRICS ?
69. https://arxiv.org/abs/1911.05248 Model Compression vs Performance Impact on different groups
70. https://arxiv.org/abs/1911.05248 Pruning Identified Exemplars (PIEs)
71. Challenges! Incentive System
72. Gate Keeping Challenges!
73. Dynamic Systems Complexity Challenges!
74. “You have no idea, Felice, what havoc literature creates inside certain heads. It is like monkeys leaping about in the treetops, instead of staying firmly on the ground.” Franz Kafka May 9, 1913
75. “You have no idea, Felice, what havoc literature ML-Systems create inside certain heads. It is like monkeys leaping about in the treetops, instead of staying firmly on the ground.” Franz Kafka May 9, 1913