How to apply the NLP model to advertising

By Bronn

Transcript

1. How to apply the NLP model to advertising
2. Goal • How to apply the NLP model to advertising • Why BERT
3. Agenda • NLP models • Transformer • BERT • Sequence analyze vs Human analyze • Advertising by analyzed human
4. Who am I? • Used Collaborative-Filtering for Recommendation • Predict correct/incorrect answer for TOEIC • Learn “Transformer” from NLP experts • seq-to-seq • LSTM • encoder/decoder • RNN • query/key/value
5. • Because of their internal memory, RNN’s can remember important things about the input they received, which allows them to be very precise in predicting what’s coming next. This is why they’re the preferred algorithm for sequential data like time series, speech, text, fi nancial data, audio, video, weather and much more Recurrent Neural Network
6. Recurrent Neural Network
7. Recurrent Neural Network
8. Recurrent Neural Network
9. sequence-to-sequence աח ೟Үী р׮ I go to school seq2seq
10. sequence-to-sequence աח ೟Үী р׮ I go to school seq2seq Encoder Decoder Context
11. sequence-to-sequence աח ೟Үী р׮ I go to school seq2seq Encoder Decoder Context RNN RNN RNN RNN I h1 go h2 to h3 school h4 Contex Vector Contex Vector RNN RNN RNN աח ೟Үী р׮
12. Vanishing Gradients Problem
13. Long short-term memory
14. sequence-to-sequence(LSTM) աח ೟Үী р׮ I go to school seq2seq Encoder Decoder Context LSTM LSTM LSTM LSTM I h1 go h2 to h3 school h4 Contex Vector Contex Vector LSTM LSTM LSTM աח ೟Үী р׮
15. Long sentence problem
17. StateOfTheArt
18. Transformer
19. Transformer(Attention is all you need) • Abstract
 “The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models are connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely. …”
20. Transformer Architecture
21. Attention
22. Top-down / Bottom-up
23. Top-down attention
24. Attention map
25. Transformer Architecture I go to school աח ೟Үী р׮
26. Transformer Architecture ࠺о ৡ׮ it is raining
28. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding • Abstract
 “… Unlike recent language representation models, BERT is designed to pre-train deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. …”
29. BERT input representation
30. Fine-tuning tasks
31. Q&A about NLP models
32. Sequence vs Human
33. Human by meta data
34. Mutable vs Immutable
35. Handle Immutable object
36. Behavior
37. Intention
38. Use case
39. References • VIT • CLIP • BERT • GPT-3 • Transformer • RNN • Sequence to Sequence • https://arxiv.org/pdf/1409.0473.pdf • https://arxiv.org/pdf/1905.10949.pdf • https://www.youtube.com/watch?v=I1wJ_-kEvNQ
40. Q&A