Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
NLTK Intro for PUGS
Search
Victor Neo
March 27, 2012
Programming
7
550
NLTK Intro for PUGS
Slides for the NLTK talk given on March 2012 for Python User Group SG Meetup.
Victor Neo
March 27, 2012
Tweet
Share
More Decks by Victor Neo
See All by Victor Neo
Django - The Next Steps
victorneo
5
600
DevOps: Python tools to get started
victorneo
9
13k
Git and Python workshop
victorneo
2
780
Other Decks in Programming
See All in Programming
Package Traits
ikesyo
1
180
競技プログラミングへのお誘い@阪大BOOSTセミナー
kotamanegi
0
400
週次リリースを実現するための グローバルアプリ開発
tera_ny
1
930
Findy Team+ Awardを受賞したかった!ベストプラクティス応募内容をふりかえり、開発生産性向上もふりかえる / Findy Team Plus Award BestPractice and DPE Retrospective 2024
honyanya
0
130
.NETでOBS Studio操作してみたけど…… / Operating OBS Studio by .NET
skasweb
0
110
Lookerは可視化だけじゃない。UIコンポーネントもあるんだ!
ymd65536
1
110
rails newと同時に型を書く
aki19035vc
5
680
Оптимизируем производительность блока Казначейство
lamodatech
0
910
歴史と現在から考えるスケーラブルなソフトウェア開発のプラクティス
i10416
0
280
PHPUnitしか使ってこなかった 一般PHPerがPestに乗り換えた実録
mashirou1234
0
400
PHPとAPI Platformで作る本格的なWeb APIアプリケーション(入門編) / phpcon 2024 Intro to API Platform
ttskch
0
370
LLM Supervised Fine-tuningの理論と実践
datanalyticslabo
8
1.8k
Featured
See All Featured
Designing Experiences People Love
moore
139
23k
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k
Learning to Love Humans: Emotional Interface Design
aarron
274
40k
Imperfection Machines: The Place of Print at Facebook
scottboms
266
13k
Building a Scalable Design System with Sketch
lauravandoore
460
33k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
666
120k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
44
9.4k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
230
52k
Fontdeck: Realign not Redesign
paulrobertlloyd
82
5.3k
4 Signs Your Business is Dying
shpigford
182
21k
Rebuilding a faster, lazier Slack
samanthasiow
79
8.8k
Improving Core Web Vitals using Speculation Rules API
sergeychernyshev
2
160
Transcript
Natural Language Toolkit @victorneo
Natural Language Processing
"the process of a computer extracting meaningful information from natural
language input and/or producing natural language output"
None
Getting started with NLTK
Open source Python modules, linguistic data and documentation for research
and development in natural language processing and text analytics, with distributions for Windows, Mac OSX and Linux. NLTK
None
installatio n # you might need numpy pip install nltk
# enter Python shell import nltk nltk.download()
None
packages # For Part of Speech tagging maxent_treebank_pos_tagger # Get
a list of stopwords stopwords # Brown corpus to play around brown
Preparing data / corpus
tokens NLTK works on Tokens, for example, "Hello World!" will
be tokenized to: ['Hello', 'World', '!'] The built-in tokenizer for most use cases: nltk.word_tokenize("Hello World!")
text processing HTML text: raw = nltk.clean_html(html_text) tokens = nltk.word_tokenize(raw)
text = nltk.Text(tokens) Use BeautifulSoup for preprocessing of the HTML text to discard unnecessary data.
Part-of-speech tagging
pos tagging text = "Run away!" nltk.word_tokenize(text) nltk.pos_tag(tokens) [('Run', 'NNP'),
('away', 'RB'), ('!', '.')]
pos tagging [('Run', 'NNP'), ('away', 'RB'), ('!', '.')] NNP: Proper
Noun, Singular RB : Adverb http://www.ling.upenn.edu/courses/Fall_2003/ling001/penn_treebank_pos. html
pos tagging "The sailor dogs the barmaid." [('The', 'DT'), ('sailor',
'NN'), ('dogs', 'NNS'), ('the', 'DT'), ('barmaid', 'NN'), ('.', '.')]
Sentiment Analysis Code: http://bit.ly/GLu2Q9
Differentiate between "happy" and "sad" tweets. Teach the classifier the
"features" of happy & sad tweets and test how good it is.
Happy: "Looking through old pics and realizing everything happens for
a reason. So happy with where I am right now" Sad: "So sad I have 8 AM class tomorrow"
Process data (tweets) Extract Features Train classifier Test classifer accuracy
Tokenize tweets extract_features Naive Bayes Classifier
Process data (tweets) Extract Features Train classifier Test classifer accuracy
Tokenize tweets extract_features Naive Bayes Classifier
happy.txt sad.txt happy_test.txt sad_test.txt } training data } testing data
Tweets obtained from Twitter Search API
Process data (tweets) Extract Features Train classifier Test classifer accuracy
Tokenize tweets extract_features Naive Bayes Classifier
Happy tweets usually contain the following words: "am happy", "great
day" etc. Sad tweets usually contain the following: "not happy", "am sad" etc. features
{'contains(not)': False, 'contains(view)': False, 'contains(best)': False, 'contains(excited)': False, 'contains(morning)': False,
'contains(about)': False, 'contains(horrible)': True, 'contains(like)': False, ... } output of extract_features()
Process data (tweets) Extract Features Train classifier Test classifer accuracy
Tokenize tweets extract_features Naive Bayes Classifier
training_set = \ nltk.classify.util.\ apply_features(extract_features, tweets) classifier = \ NaiveBayesClassifier.train
(training_set) training the classifer training classifer
Process data (tweets) Extract Features Train classifier Test classifer accuracy
Tokenize tweets extract_features Naive Bayes Classifier
def classify_tweet(tweet): return \ classifier.classify(extract_features (tweet)) testing classifer
$ python classification.py Total accuracy: 90.00% (18/20) 18 tweets got
classified correctly.
Where to go from here.
http://www.nltk.org/book
https://class.coursera.org/nlp/auth/welcome
http://www.slideshare.net/shanbady/nltk-boston-text-analytics
[('Thank', 'NNP'), ('you', 'PRP'), ('.', '.')] @victorneo