Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
text_mining_slides_20180512
Search
Sponsored
·
Your Podcast. Everywhere. Effortlessly.
Share. Educate. Inspire. Entertain. You do you. We'll handle the rest.
→
Leo Lu
May 12, 2018
Technology
110
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
text_mining_slides_20180512
Leo Lu
May 12, 2018
Transcript
29 slides
Text Mining and Data Viz 2018-05-12 leoluyi@iii Slides http://pcse.pw/6WHWJ ©
leoluyi, 2018 1
橕ෝ౯ 4 㸎瓽 Leo Lu 4 ݣय़ૡᓕ 4 ፓ獮ෝᰂᣟ禂๐率 4
Build data products 4 ETL 4 Models 4 Text mining 4 Viz 4 ... © leoluyi, 2018 2
Text Minning 窕纷 膏 ૡٍ㮉 © leoluyi, 2018 3
膑碻դጱૡٍ vs. 碝Ӯդጱૡٍ © leoluyi, 2018 4
犥獮౯㮉᮷አक़㾴Ո䌃ጱ䩚ᥜ tm + tmcn Rwordseg © leoluyi, 2018 5
֕ฎ蝡犚ॺկஃஃࣁӾ 䨝磪๚Ꭳጱ襊 © leoluyi, 2018 6
犡ॠ౯㮉ᥝአӞ犚碝ጱૡٍ © leoluyi, 2018 7
窕纷 Get data ➜ Tokenize ➜ Embedding ➜ Viz ➜
Model © leoluyi, 2018 8
Get data Get data ➜ Tokenize ➜ Embedding ➜ Viz
➜ Model 9
PTT ฎ疌疌ጱঅ๏ Get data ➜ Tokenize ➜ Embedding ➜ Viz
➜ Model 10
ྯॠ᮷磪盄ग़盄ग़ጱ䔂承碘 © leoluyi, 2018 11
ᛔ૩ጱ粖恝ᛔ૩䌃 devtools::install_packages( "leoluyi/PTTr") © leoluyi, 2018 12
Cleaning and preprocessing text ኸӥ虻懱牧݄ധ褾懱 © leoluyi, 2018 13
Tokenize Transform whole text into parts Get data ➜ Tokenize
➜ Embedding ➜ Viz ➜ Model 14
For English 4 normalization 4 stemming (扃䓄玲) 4 lemmatization (扃ࣳ螭ܻ)
4 POS tagging 4 ... Get data ➜ Tokenize ➜ Embedding ➜ Viz ➜ Model 15
Ӿ犲Ԓ穉斃墋㻌 4 䥁扃 4 犋䥁扃 4 POS tagging 4 ...
Get data ➜ Tokenize ➜ Embedding ➜ Viz ➜ Model 16
Semantic Parsing vs. Bag-of-Words © leoluyi, 2018 17
R tools 4 stringr 4 jiebaR Get data ➜ Tokenize
➜ Embedding ➜ Viz ➜ Model 18
Embedding (Encode, Feature Extraction) Get data ➜ Tokenize ➜ Embedding
➜ Viz ➜ Model 19
Embedding In a nutshell, Word Embedding turns text into numbers.
4 Embedding Layer1 4 Word2Vec 4 GloVe 4 doc2vec 4 sense2vec 1 https://machinelearningmastery.com/what-are-word-embeddings/ Get data ➜ Tokenize ➜ Embedding ➜ Viz ➜ Model 20
© leoluyi, 2018 21
Demo Information Retrieval Get data ➜ Tokenize ➜ Embedding ➜
Viz ➜ Model 22
Visualize 4 Dimension Reduction 4 t-sne 4 PCA 4 Clustering
4 Interactive or static plots Get data ➜ Tokenize ➜ Embedding ➜ Viz ➜ Model 23
Visualize 4 tsne::tsne() 4 prcomp() Get data ➜ Tokenize ➜
Embedding ➜ Viz ➜ Model 24
Model Get data ➜ Tokenize ➜ Embedding ➜ Viz ➜
Model 25
Tasks 4 Classification 4 獤觊 4 Clustering 4 ತ疨ፘ犲 4
Generative models 4 ᛔ㵕ኞ౮ Get data ➜ Tokenize ➜ Embedding ➜ Viz ➜ Model 26
አک磧盅᮷䨝మᥝ䌃ᛔ૩ጱ toolkit 4 Sparse Matrix manipulation 4 Informaiton retrieval tools
4 ... © leoluyi, 2018 27
Summary 1. Problem definition & specific goal: Get Curious About
Text 2. Finding Your Data 3. Preprocessing Your Data 4 Removing stopwords, Stemming, Segmentation, ... 4. Feature Extraction 4 Document-Term Matrix: tm, text2vec 4 Named Entity Recognition, POS tagging 4 Word embeddings: word2vec, GloVe 5. More Text Mining Skills 4 sentiment analysis 4 topicmodels, LDAViz: LDA 6. More Than Words - Visualizing Your Results © leoluyi, 2018 28
碍硁ᑀ䋊 㸎瓽 leoluyi@github https://leoluyi.github.io © leoluyi, 2018 29
More Decks by Leo Lu
See All by Leo Lu
R from Data Analysis to Production
leoluyi
1
180
2018-07-28_viz_talk
leoluyi
0
110
Other Decks in Technology
See All in Technology
Context parameters で作る!はじめての型クラス
kworkdev
PRO
1
170
「個社」ではなく「業界」の課題を解く AI 対話開発 ー 価値を広げる設計と、価値を守るエンジニアリング ー
ivry_presentationmaterials
4
830
20261113_JAWS_Osaka
takuyay0ne
1
600
Test Impact Analysis を武器に立ち向かう、AI 時代のテスト増加に対するヌーラボのアプローチ / getting-started-datadog-test-impact-analysis
k6s4i53rx
3
310
Re:ゼロから始めるシェアハウスのペットな鯖
silmin_
0
260
Terraform 0 to 100:100個以上のリポジトリを常に最新にし続けるための自動化
sue445
3
1.3k
認知負荷の減らし方 ~開発にも日常業務にも活かせるノウハウ~
minorun365
PRO
15
9.6k
みん強Lightning! データ基盤ひとりでできるかな
endonanana
0
170
SORACOM × AWS IoT Core でSREする
butadora
0
120
その修正で本当に速くなったのか? k6とOpenTelemetryで確かめる性能改善 / Performance review with k6 and OpenTelemetry
ymotongpoo
3
420
設計手法を「道具」として使いこなすために
a_suenami
7
1.6k
おうち環境でも「なるべく止めたくない!」 ― おうち環境の機器冗長化の理想と現実
yuutosi
1
640
Featured
See All Featured
AI Search: Implications for SEO and How to Move Forward - #ShenzhenSEOConference
aleyda
1
1.4k
Imperfection Machines: The Place of Print at Facebook
scottboms
270
14k
Embracing the Ebb and Flow
colly
88
5.2k
Why You Should Never Use an ORM
jnunemaker
PRO
61
10k
[RailsConf 2023] Rails as a piece of cake
palkan
59
7.2k
The Pragmatic Product Professional
lauravandoore
37
7.5k
The Power of CSS Pseudo Elements
geoffreycrofte
82
6.6k
Keith and Marios Guide to Fast Websites
keithpitt
413
23k
The Art of Programming - Codeland 2020
erikaheidi
57
15k
コードの90%をAIが書く世界で何が待っているのか / What awaits us in a world where 90% of the code is written by AI
rkaga
63
48k
The Cult of Friendly URLs
andyhume
79
7.1k
My Coaching Mixtape
mlcsv
0
390