Machine Learning System Dynamics: Beyond Model Development

Slide 1

Slide 1 text

Machine Learning System Dynamics: Beyond Model Development Omayma S.

Slide 2

Slide 2 text

No content

Slide 3

Slide 3 text

“Magic!”

Slide 4

Slide 4 text

How many of you have “interacted” with at least one ML-system? 🙋🙋

Slide 5

Slide 5 text

And more…

Slide 6

Slide 6 text

FLDSMDFR

Slide 7

Slide 7 text

FLDSMDFR

Slide 8

Slide 8 text

Flint Lockwood Diatonic Super Mutating Dynamic Food Replicator

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

I’ve got this under control and it’s not gonna end in disaster

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

FLDSMDFR Post-mortem!

Slide 17

Slide 17 text

LINEAR THINKING LINEAR WORLD IN A NON

Slide 18

Slide 18 text

Source

Slide 19

Slide 19 text

Source help doctors

Slide 20

Slide 20 text

232 Algorithms! Source

Slide 21

Slide 21 text

Dataset Curation Model Training Metrics Calculation Metrics Reporting Monitoring Model Serving …… How would you describe an ML-System?

Slide 22

Slide 22 text

Dataflow VertexAI Job BigQuery Cloud Workflow ……. CloudRun/ API …… How would you describe an ML-System?

Slide 23

Slide 23 text

Different Areas of Expertise = Different Perspectives Relativity M.C.Escher, 1953

Slide 24

Slide 24 text

Different Areas of Expertise = Different Mental Models Relativity M.C.Escher, 1953

Slide 25

Slide 25 text

Research Scientist Data Engineer Data Scientist MLOps Macine Learning Engineer Data Analyst Data Product Manager Social Scientists Legal Practitioners And more….. Linguists UX Designers Software Engineer

Slide 26

Slide 26 text

An INTERCONNECTED set of COMPONENTS, organized in a way that achieves a certain OBJECTIVE. SYSTEM

Slide 27

Slide 27 text

An INTERCONNECTED set of COMPONENTS, organized in a way that achieves a certain OBJECTIVE. SYSTEM

Slide 28

Slide 28 text

1 | COMPONENTS

Slide 29

Slide 29 text

1 | COMPONENTS • Identify. • Describe characteristics. • Understand limitations.

Slide 30

Slide 30 text

Think about electronic components!

Slide 31

Slide 31 text

source

Slide 32

Slide 32 text

source

Slide 33

Slide 33 text

source 22 PAGES

Slide 34

Slide 34 text

Datasheets for Datasets Model Cards

Slide 35

Slide 35 text

Datasheets for Datasets Model Cards https://arxiv.org/abs/1803.09010 https://arxiv.org/abs/1810.03993

Slide 36

Slide 36 text

Datasheets for Datasets Model Cards

Slide 37

Slide 37 text

https://huggingface.co/datasets/imagenet-1k

Slide 38

Slide 38 text

https://huggingface.co/datasets/imagenet-1k Tasks Annotation Size Data Splits

Slide 39

Slide 39 text

Personal and Sensitive Information The 1,000 categories selected for this subset contain only 3 people categories (scuba diver, bridegroom, and baseball player) while the full ImageNet contains 2,832 people categories under the person subtree (accounting for roughly 8.3% of the total images). This subset does contain the images of people without their consent. Though, the study in [1] on obfuscating faces of the people in the ImageNet 2012 subset shows that blurring people's faces causes a very minor decrease in accuracy (~0.6%) suggesting that privacy-aware models can be trained on ImageNet. On larger ImageNet, there has been an attempt at filtering and balancing the people subtree in the larger ImageNet.

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Source

Slide 42

Slide 42 text

Datasheets for Datasets Model Cards

Slide 43

Slide 43 text

Toxicity Classifier https://arxiv.org/abs/1810.03993

Slide 44

Slide 44 text

“Toxic” Text Classiﬁer https://arxiv.org/abs/1810.03993

Slide 45

Slide 45 text

"a photograph of an astronaut riding a horse" https://huggingface.co/blog/stable_diffusion Stable Diffusion

Slide 46

Slide 46 text

Stable Diffusion https://huggingface.co/CompVis/stable-diffusion-v1-4

Slide 47

Slide 47 text

Stable Diffusion h

Slide 48

Slide 48 text

Stable Diffusion https://huggingface.co/CompVis/stable-diffusion-v1-4

Slide 49

Slide 49 text

Stable Diffusion h

Slide 50

Slide 50 text

2 | INTERCONNECTIONS

Slide 51

Slide 51 text

2 | INTERCONNECTIONS • Flow of Data. • ML System Archetypes. • Feedback Loops.

Slide 52

Slide 52 text

> THE SYSTEM P+A+R+T+S

Slide 53

Slide 53 text

Open-loop vs. Closed-loop System Behavior

Slide 54

Slide 54 text

ML System Archetypes Explore-Exploit Dilemma e.g. Ranking

Slide 55

Slide 55 text

ML System Archetypes Bias Ampliﬁfcation e.g. Text Generation, Image Classiﬁcation

Slide 56

Slide 56 text

ML System Archetypes Echo Chamber Generation e.g. Recommender Systems

Slide 57

Slide 57 text

ML System Archetypes Explore-Exploit Dilemma Bias Amplififcation Echo Chamber Generation …..

Slide 58

Slide 58 text

No content

Slide 59

Slide 59 text

Machine learning is used extensively in recommender systems deployed in products. The decisions made by these systems can influence user beliefs and preferences which in turn affect the feedback the learning system receives - thus creating a feedback loop. This phenomenon can give rise to the so-called "echo chambers" or "filter bubbles" that have user and societal implications. In this paper, we provide a novel theoretical analysis that examines both the role of user dynamics and the behavior of recommender systems, disentangling the echo chamber from the filter bubble effect. In addition, we offer practical solutions to slow down system degeneracy. Our study contributes toward understanding and developing solutions to commonly cited issues in the complex temporal scenario, an area that is still largely unexplored.

Slide 60

Slide 60 text

Slide 61

Slide 61 text

3 | OBJECTIVE

Slide 62

Slide 62 text

3 | OBJECTIVE Or Should We Say OBJECTIVE S?

Slide 63

Slide 63 text

Should We Say OBJECTIVE S?

Slide 64

Slide 64 text

OBJECTIVE S? Performance Compression Interpretability Fairness …… ……………. Icons from: https://www.ﬂaticon.com/

Slide 65

Slide 65 text

Performance Compression Interpretability Fairness …… ……………. ROI Regulatory Compliance Icons from: https://www.ﬂaticon.com/ OBJECTIVE S?

Slide 66

Slide 66 text

Performance Compression Interpretability Fairness …… ……………. ROI Regulatory Compliance Icons from: https://www.flaticon.com/ OBJECTIVE S?

Slide 67

Slide 67 text

How Do You Deﬁne/Report Your Metrics? Graduated in the top half of the bottom half of my class

Slide 68

Slide 68 text

Performance Compression How Do You Evaluate Tradeoffs? AGGREGATED METRICS ?

Slide 69

Slide 69 text

https://arxiv.org/abs/1911.05248 Model Compression vs Performance Impact on different groups

Slide 70

Slide 70 text

https://arxiv.org/abs/1911.05248 Pruning Identified Exemplars (PIEs)

Slide 71

Slide 71 text

Challenges! Incentive System

Slide 72

Slide 72 text

Gate Keeping Challenges!

Slide 73

Slide 73 text

Dynamic Systems Complexity Challenges!

Slide 74

Slide 74 text

“You have no idea, Felice, what havoc literature creates inside certain heads. It is like monkeys leaping about in the treetops, instead of staying firmly on the ground.” Franz Kafka May 9, 1913

Slide 75

Slide 75 text

“You have no idea, Felice, what havoc literature ML-Systems create inside certain heads. It is like monkeys leaping about in the treetops, instead of staying ﬁrmly on the ground.” Franz Kafka May 9, 1913