Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Text Mining: Exploratory Data Analysis to Machi...
Search
Julia Silge
March 04, 2019
Technology
1
240
Text Mining: Exploratory Data Analysis to Machine Learning
March 2019 talk at WiDS Salt Lake City regional event
Julia Silge
March 04, 2019
Tweet
Share
More Decks by Julia Silge
See All by Julia Silge
Introducing Positron
juliasilge
1
290
The right tool for the job
juliasilge
0
44
Good practices for applied machine learning
juliasilge
0
200
Applied machine learning with tidymodels
juliasilge
0
120
Maintaining an R Package
juliasilge
0
370
Publishing the Stack Overflow Developer Survey
juliasilge
2
70
Text Mining Using Tidy Data Principles
juliasilge
0
140
North American Developer Hiring Landscape
juliasilge
0
51
Understanding Principal Component Analysis Using Stack Overflow Data
juliasilge
13
4.5k
Other Decks in Technology
See All in Technology
Snowflake のアーキテクチャは本当に筋がよかったのか / Data Engineering Study #30
indigo13love
0
280
Wasmで社内ツールを作って配布しよう
askua
0
150
AI駆動開発 with MixLeap Study【大阪支部 #3】
lycorptech_jp
PRO
0
270
From Live Coding to Vibe Coding with Firebase Studio
firebasethailand
1
320
MCPと認可まわりの話 / mcp_and_authorization
convto
2
290
マルチモーダル基盤モデルに基づく動画と音の解析技術
lycorptech_jp
PRO
2
260
robocopy の怖い話/scary-story-about-robocopy
emiki
0
410
機械学習を「社会実装」するということ 2025年夏版 / Social Implementation of Machine Learning July 2025 Version
moepy_stats
1
1.4k
経験がないことを言い訳にしない、 AI時代の他領域への染み出し方
parayama0625
0
260
大規模イベントを支える ABEMA の アーキテクチャ 変遷 2025
nagapad
5
510
LLM開発を支えるエヌビディアの生成AIエコシステム
acceleratedmu3n
0
340
モバイルゲームの開発を支える基盤の歩み ~再現性のある開発ラインを量産する秘訣~
qualiarts
0
610
Featured
See All Featured
Optimizing for Happiness
mojombo
379
70k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.3k
The Invisible Side of Design
smashingmag
301
51k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
110
19k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
For a Future-Friendly Web
brad_frost
179
9.8k
A better future with KSS
kneath
238
17k
RailsConf 2023
tenderlove
30
1.2k
YesSQL, Process and Tooling at Scale
rocio
173
14k
Balancing Empowerment & Direction
lara
1
510
Why Our Code Smells
bkeepers
PRO
337
57k
Transcript
T E X T M I N I N G
EXPLORATORY DATA ANALYSIS TO MACHINE LEARNING
HELLO T I D Y T E X T Data
Scientist at Stack Overflow @juliasilge https://juliasilge.com/ I’m Julia Silge
T I D Y T E X T TEXT DATA
IS INCREASINGLY IMPORTANT
T I D Y T E X T TEXT DATA
IS INCREASINGLY IMPORTANT NLP TRAINING IS SCARCE ON THE GROUND
TIDY DATA PRINCIPLES + COUNT-BASED METHODS = T I D
Y T E X T
https://github.com/juliasilge/tidytext
https://github.com/juliasilge/tidytext
http://tidytextmining.com/
T I D Y T E X T EXPLORATORY DATA
ANALYSIS N-GRAMS AND MORE WORDS MACHINE LEARNING
EXPLORATORY DATA ANALYSIS T I D Y T E X
T
from the Washington Post’s Wonkblog
from the Washington Post’s Wonkblog
D3 visualization on Glitch
WHAT IS A DOCUMENT ABOUT? T I D Y T
E X T TERM FREQUENCY INVERSE DOCUMENT FREQUENCY
None
None
• As part of the NASA Datanauts program, I worked
on a project to understand NASA datasets • Metadata includes title, description, keywords, etc
None
T A K I N G T I D Y
T E X T T O T H E N E X T L E V E L N-GRAMS, NETWORKS, & NEGATION
None
None
None
None
None
T A K I N G T I D Y
T E X T T O T H E N E X T L E V E L TOPIC MODELING
TOPIC MODELING T I D Y T E X T
•Each DOCUMENT = mixture of topics •Each TOPIC = mixture of words
None
None
None
None
T A K I N G T I D Y
T E X T T O T H E N E X T L E V E L TEXT CLASSIFICATION
TRAIN A GLMNET MODEL T I D Y T E
X T
TEXT CLASSIFICATION T I D Y T E X T
> library(glmnet) > library(doMC) > registerDoMC(cores = 8) > > is_jane <- books_joined$title == "Pride and Prejudice" > > model <- cv.glmnet(sparse_words, is_jane, family = "binomial", + parallel = TRUE, keep = TRUE)
None
None
THANK YOU T I D Y T E X T
@juliasilge https://juliasilge.com JULIA SILGE
THANK YOU T I D Y T E X T
@juliasilge https://juliasilge.com Author portraits from Wikimedia Photos by Glen Noble and Kimberly Farmer on Unsplash JULIA SILGE