Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Analyzing Chinese Lyrics with Python
Search
Andy Dai
June 05, 2016
Technology
1.3k
4
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Analyzing Chinese Lyrics with Python
Andy Dai
June 05, 2016
Transcript
55 slides
Big Data, Better Decision www.gliacloud.com Andy Dai
[email protected]
Analyzing Chinese
Lyrics with Python
WHO AM I? • Andy Dai • Organizer of Taipei.py
• PyCon பૡ (2012~) • GliaCloud CTO ࠨ᮷狶ጱૡ纷䒍
犡ॠᥝ蚤य़疑獤Ձጱฎ Ӿ虻碘獤ຉٌ䋿ฎ盄 墋㻌ጱ
Ԫ眐ฎ蝡䰬樄তጱ…
礓ॠ౯ࣁమ PyCon ᥝಭᑤ ࠨጱ碻狡牧肊螲簁段蚏ԧ 蝡Ḓ稧
౯犋ݢ胼䨝訵膒
虻碘獤ຉጱᒫӞྍ 玲虻碘
None
None
অމ牧ᛔ૩㬵瞟…
None
Scrapy
竃ቘ虻碘
• 瞟ڊ襑ᥝጱ虻碘 • ݄ധ犋ᥝጱ虻碘 • ᯿蕦ጱ稧ใ
虏౯㮉ض㬵፡፡Ӟ犚碍硁
pandas ฎ虻碘ૡ纷䒍ጱঅ๏
pandas + pymongo
墋㻌ጱ翄懯虻碘 • 者و 141054 Ḓ稧 • 21150 㮆֢扃Ո • 6120 㮆稧ಋ
֢扃ኴጱ܈䔶
ॗ 3459 讙狰 1452 檔ባ 1139 蟞㾴 1061 ব舙谍 1057
瞺䔶 1007 珏聱斝 903 战ଉ盓 786 珏因舯 758 ব拹 754
matplotlib ฎ֦向瑽ጱঅ䒻ಋ import matplotlib.pyplot as plt plt.bar(…)
None
蝡܈㮆Ո㬟硁ԧ 8.7% ጱ苉承֢ߝ ॗӞ㮆Ո疰㬟ԧ 2.4%
ӥӞྍ 䥁扃
䥁扃ฎᛔ簁承蒂ቘጱच器 犡ଙ PyCon ݣ傀ࣁӾᎸᴺ膐旰 犡ଙ/PyCon/ݣ傀/ࣁ/ӾᎸᴺ/膐旰
2016 螭ࣁ媣媲አ奾૬ (jieba) • pip install jieba • Python •
耆誢獤扃 • ᛔ懪ਁَ • Quality 犋癩
䥁扃 /籃螂/ԧ/ग़ԋ/ఋ櫞/ /倀/ԧ/ग़裾/縄፮/ /胼/Ꭳ螇/㰁眤/ฎ/眢ጱ/螣叨/ /窕窚/皃皰/櫕Ո/ଥ/ /矦螂/皃稞/מի/ //虏/瞲/嬝篷玱觎/ጱ/Ի矦/ //Ӟ㮆/Ո/ጱ/伩ำ/ /旉ᑏ/ک/ݚ/Ӟ㮆/ጱ/胷腔/ /虏/Ӥ稞/ᇨ/ጱ/梊/玱/ڊ/瓵మ/
/ྯ㮆/Ո/᮷/ฎ/蝡䰬/ /Ձݑ/螂/ஞݵ腭/ //瞩妃/狶/眢眐/դ耻/ጱ/ᗧᗤ
䥁扃ਠ㬵֢犚獤ຉ
ߺ犚扃ᤩአጱ磧ग़牫 吚簁ݢ犥䌃㮆 for 蝅瑹 + dictionary ਧ >>> from collections
import Counter >>> counter = Counter([‘a’, ‘a’, ‘b’, ‘c’]) >>> counter.most_common(1) [('a', 2)] 獨盛懿 Python 磪 Counter ݢ犥አ
ڜڊ獮皃ݷ㬵፡፡ ౯㮉 Ӟ㮆̴䷱磪̴Ջ讕̴ᛔ૩̴眢眐̴犋ᥝ Ӯኴ̴螐̴Ꭳ螇̴Ӟ蚏̴犋䨝̴ெ讕̴盠禼 ݢ犥 Ӟ獥̴ইຎ̴ଛᐰ̴眤憽̴聅讀 ਿ疖 ࢩ傶̴櫝樄̴፥ጱ̴ݝ磪̴ஞӾ̴Ӟኞ̴碻樌 ፥ጱ̴蛪螲̴ፘמ̴疰ᓒ̴匍ࣁ ࢧ䛂
伩礖
稧扃ጱ扃䕍蚤෭ଉ፥ጱ癩盄ग़
ਁ襇 pip install wordcloud
ਁ襇
ॗጱਁ襇
ොઊጱਁ襇
檔姤揕ጱਁ襇
扃䕍掘纷ଶ (word density) unique 扃碍/者扃碍 len(set(word_list))/len(word_list)
扃䕍掘纷ଶ ଘ璂 word density - 0.175
አ扃穉斃ጱ掘ጱ֢扃Ո (word density > 0.20) • ྎᵜ (ৼ磣牏抑ฎ聲य़牏ူේ虭…҂ • 纩櫝
(犋傶抑ᘒ֢ጱ稧牏肯ၹ…) • 檔禼ᣟ (眤௮ጱஞ牏ॠॠమ֦…҂ • 暼ᤶ皐ҁ胙玳牏ம疃䩚᪠蚎Ԝ螁…҂ • 皰襁ኞҁݗฎஞ覍牏ণট牏Bad Boy…҂ • 玭磥 (臺ஞ牏櫝Ո…҂
አ扃穉斃ጱ … ጱ֢扃Ո (word density < 0.15) • 檔椆 0.138ҁሴঈ牏ᰀ悚蝿瞁҂
• 磷疍ፐ 0.134ҁ౯ฎӞ櫇ੜੜ澆牏覿ఉ牏妔ᛔ૩ጱ 稧҂ • ᴨמ 0.116ҁԲ์ॠ羬ڜ…҂
ஞ物አ扃穉斃ग़犋Ӟਧ玭疏
Ԇ氂獤ຉ物ଛᐰጱ眢眐ฎਿ疖ጱ
稧ใ吚Ӿڊ匍螂 “眢眐” ጱ穉ֺ ॗ 11.6% 讙狰 6.9% 檔ባ 5.1% 蟞㾴
4.7% ব舙谍 17.1% 瞺䔶 1.4% 珏聱斝 9.4% 战ଉ盓 25.4% 珏因舯 3.9% ব拹 33%
稧ใ吚Ӿڊ匍螂 “ਿ疖” ጱ穉ֺ ॗ 9.2% 讙狰 7.5% 檔ባ 9.3% 蟞㾴
4.1% ব舙谍 21.8% 瞺䔶 5.2% 珏聱斝 7.9% 战ଉ盓 21.8% 珏因舯 5.1% ব拹 26%
稧ใ吚Ӿڊ匍螂 “ଛᐰ” ጱ穉ֺ ॗ 7.4% 讙狰 9.0% 檔ባ 6.4% 蟞㾴
2.7% ব舙谍 29.3% 瞺䔶 3.9% 珏聱斝 5.1% 战ଉ盓 18.4% 珏因舯 2.7% ব拹 10.6%
螭磪盄ग़ݢ犥狶… • ߺ犚扃䕍䨝ݶ碻ڊ匍 • ߺ犚稧ጱ扃᯿蕦ሲ盄ṛ • 犋ݶ碻๗ጱአ扃… • ইຎ֦మ㷢ᘍՈ疑ጱ֢扃…
犡ॠ㬵犋现拻ጱ - jupyter
犡ॠ㬵犋现拻ጱ - elasticsearch
Elasticsearch • ׀獊䲒ᔱۑ胼 • ಅ磪砺֢᮷磪׀ REST API • 蟴ݳ ElasticSearchDSL
䌃蚏㬵ๅঅ • http://www.slideshare.net/daikeren/search-search- search
犡ॠ㬵犋现拻ጱ - gensim
gensim ጱ䛑አ • word2vec • doc2vec • 獤ᗭ • ፘ犲ଶ
• Machine Learning 奲ݳದ
矑ӥ㬵䨝狶ጱ ইຎ磪绚ጱ扖አ Deep Learning 狶ᛔ㵕稧扃ኞ౮牧 藶๗盃(?) ๚㬵ጱ Taipei.py
Recap • 瞟翕ᒊጱॺկ – scrapy • 䥁扃 – jieba •
虻碘獤ຉ – pure Python, pandas • 憙憽玕 – wordcloud, matplotlib • juypter • gensim • elasticsearch
ૡ珶๐率 • 虵搚秚䜗ภ讨䨝磪碝氂ፓ • ᐟᑃੜ因ᇔ • 懿݄覿糫牧磪ࠧ蟸牦牦牦 • ӥ܌槼襎纨ݢ犥肯肯虵搚秚ጱ硲Ԫ
THANK YOU
More Decks by Andy Dai
See All by Andy Dai
用 Python + Azure 建立你的聊天機器人
daikeren
2
510
Other Decks in Technology
See All in Technology
IP重複は"ランディングVPC"で受け止める ― PrivateLink と Transit Gatewayによるマルチテナント接続基盤の設計
hatahata021
2
190
E2E Test というハーネス — AI時代の自動テスト民主化
daishukojima
8
3.4k
Quick Automate | 目指せ Quick マスター 12
inariku
PRO
1
120
並列度を下げたら速くなった — go build の重複コンパイルを実測で見る
tomoaki25
2
1.3k
属人化したISMS運用をGitHubとAIで仕組み化した1年間の記録
sioncojp
4
2.7k
Steering と Hook (チームの決まりを守らせる) | 目指せ Kiro マスター 06
inariku
PRO
2
140
AWS基盤改善を支える 私たちの理念と行動指針
kentosuzuki
0
230
アラートを見直していたら営業組織への理解にちょっとだけ繋がった話
z63d
1
210
仕様と現実の間で落とし所を探るリモートMCPの認証・認可 — Sansanの認証基盤における課題と実践
sansantech
PRO
2
330
Gov-JAWS#9 統合監視機能のマネージドサービス化+AIによる障害時自動調査
fj_yoshimura_kosuke
0
180
Community Over Code Glasgow 2026 + Lakehouse Day recap
lycorptech_jp
PRO
1
120
思考過程から秘密が漏れる:Amazon Bedrock上のKimi K3を5つの方法で攻めたら、25回中22回漏洩した話
yama3133
1
170
Featured
See All Featured
How to audit for AI Accessibility on your Front & Back End
davetheseo
0
640
Understanding Cognitive Biases in Performance Measurement
bluesmoon
32
3.1k
Designing Experiences People Love
moore
143
24k
世界の人気アプリ100個を分析して見えたペイウォール設計の心得
akihiro_kokubo
PRO
74
43k
Exploring anti-patterns in Rails
aemeredith
4
570
Building Adaptive Systems
keathley
44
3.3k
Designing for humans not robots
tammielis
254
26k
From π to Pie charts
rasagy
1
450
Thoughts on Productivity
jonyablonski
76
5.5k
HDC tutorial
michielstock
2
1k
SEO in 2025: How to Prepare for the Future of Search
ipullrank
3
3.9k
Building a A Zero-Code AI SEO Workflow
portentint
PRO
0
830