pycon2023-BERT.pdf

Transcript

1. BIDIRECTIONAL ENCODER REPRESENTIONS FROM TRANSFORMERS For Sentiment Analysis *Images generated by stable-diffusion Prompt: Bert from sesame street dressed as a synthesizer FINE TUNING
2. Why are you here? Me? Prompt: Sesame street character sitting at a desk with a laptop
3. I want to specialize in NLP! Chris Mccormick Prompt: A polyglot sesame street character sitting at a desk with a laptop Jay Alammar Andrew Ng
4. Queen – woman + 7*man = ?
5. Queen – woman + 7*man = snow white
6. 1. https://www.who.int/news/item/17-06-2021-one-in-100-deaths-is-by-suicide 2. https://www.ijcai.org/proceedings/2022/0704.pdf “a code-mixed text has a dominant language or matrix language (here, Hindi) and an inserted language or embedded language (here, English)”
7. Sentiment analysis is on our roadmap for Q3. Prompt: A sesame street character dressed as a manager sitting at a desk with a laptop https://arxiv.org/pdf/1908.10063.pdf https://github.com/ProsusAI/finBERT https://medium.com/prosus-ai-tech-blog/finbert- financial-sentiment-analysis-with-bert-b277a3607101
10. I want to apply what I learn today to improve society Prompt: A superhero sesame street character sitting at a desk with a laptop
11. Detecting Potentially Harmful and Protective Suicide-related Content on Twitter: A Machine Learning Approach Metzler, H., Baginski, H., Niederkrotenthaler, T., & Garcia, D. (2022). Journal of medical Internet research, 24(8), e34705. https://doi.org/10.2196/34705 1. https://www.who.int/news/item/17-06-2021-one-in-100-deaths-is-by-suicide 2. https://arxiv.org/pdf/2112.04796.pdf
12. [2]
14. I’m a software engineer and I want to create ML apps with vision and generative models too. Prompt: A sesame street character who is a software engineer sitting at a desk 👩🎓 Train Sentence Transformers with little labeled data using a technique called SetFit 🐭 Compress models with knowledge distillation 🏎 Accelerate inference with quantization and 🤗 Optimum and Intel® Neural Compressor 🐳 Deploy models with Inference Endpoints For more details about SetFit, check out the library 👉: https://github.com/huggingface/setfit
15. Transfer Learning Prompt: sesame street character amazed by understanding a new concept Ultimately, we all follow tutorials hoping to transfer what we learn to use-cases we care about
16. From Wikipedia Transfer learning (TL) is a research problem in machine learning (ML) that focuses on applying knowledge gained while solving one task to a related task.[1] For example, knowledge gained while learning to recognize cars could be applied when trying to recognize trucks. This topic is related to the psychological literature on transfer of learning, although practical ties between the two fields are limited. Reusing/transferring information from previously learned tasks to new tasks has the potential to significantly improve learning efficiency.[2]
17. Machines From Wikipedia Transfer learning (TL) is a research problem in machine learning (ML) that focuses on applying knowledge gained while solving one task to a related task.[1] From Wikipedia Transfer of learning occurs when people apply information, strategies, and skills they have learned to a new situation or context. Humans
18. BERT YOU Embeddings Intuition Tokenizer Gut Masked Language Modeling and Next Sentence Prediction Finetuning Objective: How to classify the sentiment of a piece of text Fine Tuning Objective: How to finetune a BERT based model
19. Manager Polyglot Superhero Software Engineer https://huggingface.co/models Human base models
20. ["Jack", "Jack", "Jack", "Jack", "There's a boat, Jack"] Interpreting Sentiment Positive or Negative?
21. Interpreting Sentiment (Advanced!) Plutchik, 1982
22. Interpreting Sentiment (Advanced!)
23. Interpreting Sentiment (Advanced!) ["Jack", "Jack", "Jack", "Jack", "There's a boat”, “Jack”, “Jack”, “JACK!”, “Jack”, “There’s a boat, Jack”, “Jack”] "Jack… Jack…Jack…Jack…There's a boat…Jack…Jack…JACK! Jack? There’s a boat, Jack. Jack” Relief Despair Ecstatic Confusion Guilt Fear Anger Time
24. Interpreting Sentiment (Advanced!) ["Jack", "Jack", "Jack", "Jack", "There's a boat”, “Jack”, “Jack”, “JACK!”, “Jack”, “There’s a boat, Jack”, “Jack”] Relief Ecstatic Confusion Despair Anger Guilt Fear Time [ .95, .00, .00, .00, .00, .00, .00] [ .95, .95, .00, .00, .00, .00, .00] [ .80, .50, .50, .00, .00, .00, .00] [ .00, .00, .20, .99, .00, .00, .00] [ .00, .00, .00, .95, .30, .00, .00] [ .00, .00, .00, .95, .02, .99, .00] [ .00, .00, .00, .00, .00, .00, .99]
25. Emotion Embeddings https://arxiv.org/pdf/2211.00171.pdf https://github.com/gchochla/Demux-MEmo