Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
身近に潜むtokenize 2016
Search
Sponsored
·
Ship Features Fearlessly
Turn features on and off without deploys. Used by thousands of Ruby developers.
→
moznion
July 03, 2016
Technology
0
4.1k
身近に潜むtokenize 2016
YAPC Hachioji 2016 LT 資料です
moznion
July 03, 2016
Tweet
Share
More Decks by moznion
See All by moznion
履歴テーブル、今回はこう作りました 〜 Delegated Types編 〜 / How We Built Our History Table This Time — With Delegated Types
moznion
16
11k
「データ無い! 腹立つ! 推論する!」から 「データ無い! 腹立つ! データを作る」へ チームでデータを作り、育てられるようにするまで / How can we create, use, and maintain data ourselves?
moznion
10
7k
避けられないI/O待ちに対処する: Rails アプリにおけるSSEとasync gemの活用 / Tackling Inevitable I/O Latency in Rails Apps with SSE and the async gem
moznion
3
5.3k
RubyKaigi Hack Space in Tokyo & 函館最速 "予習" 会 / RubyKaigi Hack Space in Tokyo & The Fastest Briefing of RubyKaigi 2026 in Hakodate
moznion
1
320
地に足の付いた現実的な技術選定から魔力のある体験を得る『AIレシート読み取り機能』のケーススタディ / From Grounded Tech Choices to Magical UX: A Case Study of AI Receipt Scanning
moznion
6
4.7k
Chrome Extension Techniques from Hell
moznion
1
270
Simple組み合わせ村から大都会Railsにやってきた俺は / Coming to Rails from the Simple
moznion
4
8.5k
AIレシート読み取り機能をRuby on Rails on AWSで実現するLLMにまつわるアレコレ / AI-based receipt reading function powered by LLM on Ruby on Rails on AWS
moznion
3
1.1k
Develop to Survive - YAPC::Hakodate 2024 Keynote
moznion
11
21k
Other Decks in Technology
See All in Technology
Kiro IDEのドキュメントを全部読んだので地味だけどちょっと嬉しい機能を紹介する
khmoryz
0
200
Cosmos World Foundation Model Platform for Physical AI
takmin
0
940
15 years with Rails and DDD (AI Edition)
andrzejkrzywda
0
200
usermode linux without MMU - fosdem2026 kernel devroom
thehajime
0
240
OWASP Top 10:2025 リリースと 少しの日本語化にまつわる裏話
okdt
PRO
3
820
Amazon Bedrock Knowledge Basesチャンキング解説!
aoinoguchi
0
150
SREチームをどう作り、どう育てるか ― Findy横断SREのマネジメント
rvirus0817
0
320
Ruby版 JSXのRuxが気になる
sansantech
PRO
0
160
SREが向き合う大規模リアーキテクチャ 〜信頼性とアジリティの両立〜
zepprix
0
460
日本の85%が使う公共SaaSは、どう育ったのか
taketakekaho
1
230
コミュニティが変えるキャリアの地平線:コロナ禍新卒入社のエンジニアがAWSコミュニティで見つけた成長の羅針盤
kentosuzuki
0
130
[CV勉強会@関東 World Model 読み会] Orbis: Overcoming Challenges of Long-Horizon Prediction in Driving World Models (Mousakhan+, NeurIPS 2025)
abemii
0
140
Featured
See All Featured
Prompt Engineering for Job Search
mfonobong
0
160
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
27k
Paper Plane (Part 1)
katiecoart
PRO
0
4.3k
Building a Scalable Design System with Sketch
lauravandoore
463
34k
Skip the Path - Find Your Career Trail
mkilby
0
57
Helping Users Find Their Own Way: Creating Modern Search Experiences
danielanewman
31
3.1k
Leading Effective Engineering Teams in the AI Era
addyosmani
9
1.6k
How to train your dragon (web standard)
notwaldorf
97
6.5k
HDC tutorial
michielstock
1
390
End of SEO as We Know It (SMX Advanced Version)
ipullrank
3
3.9k
Done Done
chrislema
186
16k
Marketing to machines
jonoalderson
1
4.6k
Transcript
ۙʹજΉ tokenize 2016 @moznion
@moznion
͜͜Ͱݴ͏ tokenize - ͳΜ͔͍ҙͷจࣈྻ͕͋ͬͯ - ͦΕΛίϯϐϡʔλ͕ղऍ͢͠ ͍୯Ґ (token) ʹͿͬͨΔ
Έͳ͞Μ tokenize ͯ͠·͔͢
༷ʑͳͷΛ token ʹ ͚͍͖ͯ·͠ΐ͏ ͚Δඞཁ͕͋Γ·͢
ྫ͑ʁ
༣ศ൪߸
༣ศ൪߸ ken_all.csv
ॅॴ
ॅॴ ૯লࢿྉ (pdf)
ి൪߸
ి൪߸ ૯লࢿྉ (doc, pdf)
ςϯγϣϯ্͕ͬͯ ͖·͔ͨ͠ʁ
͍ɼྑ͍Ͱ͢ ͘͢͝ྑ͍Ͱ͢
ࢢ֎ہ൪ͷΛ͠·͠ΐ͏
03-5321-1111
03-5321-1111
Φοɼ؆୯ͦ͏Ͱ͢Ͷ
ࢢ֎ہ൪
ࢢ֎ہ൪ ૯লࢿྉ (doc, pdf)
ࢢ֎ہ൪ ૯লࢿྉ (doc, pdf)
None
ͱΓ͋͑ͣதݟͯΈ·͠ΐ͏
൪߸۠ըίʔυ ൪߸۠ը ࢢ֎ہ൪ ࢢہ൪ 1 ւಓߐผࢢɺࡳຈࢢɺ ౡࢢɺۭ܊ೆຈொ 11 CDE
OKOK
൪߸۠ըίʔυ ൪߸۠ը ࢢ֎ہ൪ ࢢہ൪ 3 ւಓ༦ுࢢʢΛআ ͘ɻʣ 123 DE
OKOK
൪߸۠ըίʔυ ൪߸۠ը ࢢ֎ہ൪ ࢢہ൪ 32 ւಓҏୡࢢɺѾా܊ ʢಎ但ބொٴͼ๛Ӝொʹ ݶΔɻʣɺ༗च܊ 142 DE
OKOK
൪߸۠ըίʔυ ൪߸۠ը ࢢ֎ہ൪ ࢢہ൪ 93 ੨ݝेాࢢɺࡾ ࢢɺ্܊ʢ౦ொʢѴ ɺѴೆɺ্ɺେӜɺ ্ɺ্ೆٴ ͼ৽ؗʹݶΔɻʣɺࣣށ
ொٴͼށொʹݶΔɻʣ 176 DE
Γ্͕ͬͯ·͍Γ·ͨ͠
൪߸۠ըίʔυ ൪߸۠ը ࢢ֎ہ൪ ࢢہ൪ 363 େࡕాࢢۭߓɺେࡕࢢʢ౦ॅ٢۠ాࣣஸٴͼฏ۠٢ล࢛ஸΛআ ͘ɻʣɺਅࢢʢੴݪொɺઘொɺҰ൪ொɺେொɺ֞ொɺ܂࠽৽ொɺொɺणொɺӫ ொɺ খ࿏ொɺ৽ڮொɺொɺ݄ग़ொɺಊࢁொɺ఼ౡொɺதொɺொɺݟொɺ౦ాொɺਂా ொɺݹொɺຊொɺদੜொɺদ༿ொɺޚಊொɺౡொɺݩொɺ༄ాொٴͼ༄ொʹݶ
Δɻʣɺ ਧాࢢɺઁࢢʢผொɺ৽ࡏՈɺਖ਼ɺਖ਼ຊொɺঙɺઍཬٰɺઍཬٰ৽ொɺઍཬ ٰ౦࢛ஸٴͼޒஸɺҰɺொɺ౦ਖ਼ɺ౦Ұɺ౦ผɺҰɺผɺ ࡾౡɺೆઍཬٰฒͼʹೆผொʹݶΔɻʣɺ๛தࢢɺ౦େࡕࢢʢѴொɺౡொɺ೭ ொɺग़ӢҪொɺग़ӢҪຊொɺҴ༿ɺࠓถɺؠాொʢࡾஸΛআ͘ɻʣɺӝੜಊҰஸɺՃ ೲɺ্ੴொɺ্࢛ொɺ্ສࣉொɺதɺాɺՏொɺਆాொɺتཬொɺੴ ொɺߵொɺ٬ொɺԼொɺޒொɺߵொɺߵಙ҇ொɺߵຊொɺߵݩொɺݹ ຳྠɺࡩொɺ࢛ொɺౡ೭ɺԼສࣉொɺতொɺ৽ౡொɺ৽ߵொɺ৽ொɺ৽ঙɺ ொɺ֯ాɺળࠜࣉொɺୋ఼ொɺๅொɺཱՖொɺۄ۲ொɺۄ۲ொ౦ɺۄ۲ݩொɺ๛Ӝ ொɺௗډொɺதੴொɺத৽։ɺதɺதߵொɺೆொɺੴொɺؠాҰஸɺ ߵொɺֹాொɺࢢொɺശ఼ொɺՖԂொɺՖԂ౦ொɺՖԂຊொɺ౦ੴொɺ౦ߵ ொɺ౦๛Ӝொɺ౦ࢁொɺඛߐɺඛ౦ɺතࢁொɺຊঙதҰஸɺຊொɺদݪɺদݪೆɺ ਫɺೆߵொɺೆ࢛ொɺຳྠɺޚொɺݩொɺࢁखொɺੜொɺԣখ࿏ொɺԣປɺԣ ປɺԣປೆɺ٢ాɺ٢ాຊொɺ٢ాԼౡɺ٢ݪɺສࣉொٴͼएொΛআ͘ɻʣɺकޱ ࢢɺീඌࢢʢᔹɺᔹٴͼᔹ౦ʹݶΔɻʣɺฌݿݝೌ࡚ࢢ 6 BCDE
େࡕాࢢۭߓɺେࡕࢢʢ౦ॅ٢۠ాࣣஸٴͼฏ۠٢ล࢛ஸΛআ ͘ɻʣɺਅࢢʢੴݪொɺઘொɺҰ൪ொɺେொɺ֞ொɺ܂࠽৽ொɺொɺणொɺӫொɺ খ࿏ொɺ৽ڮொɺொɺ݄ग़ொɺಊࢁொɺ఼ౡொɺதொɺொɺݟொɺ౦ాொɺਂా ொɺݹொɺຊொɺদੜொɺদ༿ொɺޚಊொɺౡொɺݩொɺ༄ాொٴͼ༄ொʹݶΔɻʣɺ ਧాࢢɺઁࢢʢผொɺ৽ࡏՈɺਖ਼ɺਖ਼ຊொɺঙɺઍཬٰɺઍཬٰ৽ொɺઍཬ ٰ౦࢛ஸٴͼޒஸɺҰɺொɺ౦ਖ਼ɺ౦Ұɺ౦ผɺҰɺผɺ ࡾౡɺೆઍཬٰฒͼʹೆผொʹݶΔɻʣɺ๛தࢢɺ౦େࡕࢢʢѴொɺౡொɺ೭ ொɺग़ӢҪொɺग़ӢҪຊொɺҴ༿ɺࠓถɺؠాொʢࡾஸΛআ͘ɻʣɺӝੜಊҰஸɺՃ ೲɺ্ੴொɺ্࢛ொɺ্ສࣉொɺதɺాɺՏொɺਆాொɺتཬொɺੴ ொɺߵொɺ٬ொɺԼொɺޒொɺߵொɺߵಙ҇ொɺߵຊொɺߵݩொɺݹ
ຳྠɺࡩொɺ࢛ொɺౡ೭ɺԼສࣉொɺতொɺ৽ౡொɺ৽ߵொɺ৽ொɺ৽ঙɺ ொɺ֯ాɺળࠜࣉொɺୋ఼ொɺๅொɺཱՖொɺۄ۲ொɺۄ۲ொ౦ɺۄ۲ݩொɺ๛Ӝ ொɺௗډொɺதੴொɺத৽։ɺதɺதߵொɺೆொɺੴொɺؠాҰஸɺ ߵொɺֹాொɺࢢொɺശ఼ொɺՖԂொɺՖԂ౦ொɺՖԂຊொɺ౦ੴொɺ౦ߵ ொɺ౦๛Ӝொɺ౦ࢁொɺඛߐɺඛ౦ɺතࢁொɺຊঙதҰஸɺຊொɺদݪɺদݪೆɺ ਫɺೆߵொɺೆ࢛ொɺຳྠɺޚொɺݩொɺࢁखொɺੜொɺԣখ࿏ொɺԣປɺԣ ປɺԣປೆɺ٢ాɺ٢ాຊொɺ٢ాԼౡɺ٢ݪɺສࣉொٴͼएொΛআ͘ɻʣɺकޱ ࢢɺീඌࢢʢᔹɺᔹٴͼᔹ౦ʹݶΔɻʣɺฌݿݝೌ࡚ࢢ
͍
ݟ·͢
੨ݝेాࢢɺࡾࢢɺ্܊ʢ౦ொʢѴɺ Ѵೆɺ্ɺେӜɺ্ɺ্ೆٴ ͼ৽ؗʹݶΔɻʣɺࣣށொٴͼށொʹݶΔɻʣ
ϋϋʔϯ
શ֯ಡͰ۟ΒΕͨ શׅ֯ހͷSࣜͩͳʁ
੨ݝेాࢢɺࡾࢢɺ্܊ʢ౦ொʢѴɺ Ѵೆɺ্ɺେӜɺ্ɺ্ೆٴ ͼ৽ؗʹݶΔɻʣɺࣣށொٴͼށொʹݶΔɻʣ ʢ੨ݝेాࢢɺࡾࢢɺ্܊ʢ౦ொʢѴ ɺѴೆɺ্ɺେӜɺ্ɺ্ೆٴ ͼ৽ؗʹݶΔɻʣɺࣣށொٴͼށொʹݶΔɻʣʣ
ʢ੨ݝेాࢢɺࡾࢢɺ্܊ʢ౦ொʢѴ ɺѴೆɺ্ɺେӜɺ্ɺ্ೆٴ ͼ৽ؗʹݶΔɻʣɺࣣށொٴͼށொʹݶΔɻʣʣ ୯७ͳϦετ
ʢ੨ݝेాࢢɺࡾࢢɺ্܊ʢ౦ொʢѴ ɺѴೆɺ্ɺେӜɺ্ɺ্ೆٴ ͼ৽ؗʹݶΔɻʣɺࣣށொٴͼށொʹݶ Δɻʣʣ ͜Εവ
ʢ੨ݝेాࢢɺࡾࢢɺ্܊ʢ౦ொʢѴ ɺѴೆɺ্ɺେӜɺ্ɺ্ೆٴ ͼ৽ؗʹݶΔɻʣɺࣣށொٴͼށொʹݶ Δɻʣʣ ߴ֊വ
ʢ੨ݝेాࢢɺࡾࢢɺ্܊ʢ౦ொʢѴ ɺѴೆɺ্ɺେӜɺ্ɺ্ೆٴ ͼ৽ؗʹݶΔɻʣɺࣣށொٴͼށொʹݶ Δɻʣʣ bool (cond)
φϧϗσΟε
͍͖ͬͯ·͢
ࠓ·Ͱͷઆ໌ͱؔͷͳ͍࣮ https://github.com/moznion/Number-Phone- JP-AreaCode
ͦͷ͏ͪ SࣜΛཧղ͢Δ࣮ʹͳΓ·͢
࠷ޙʹ
ެி mysqldump ఏڙͯ͘͠Ε