Upgrade to Pro — share decks privately, control downloads, hide ads and more …

the world of characters

Sponsored · Your Podcast. Everywhere. Effortlessly. Share. Educate. Inspire. Entertain. You do you. We'll handle the rest. →
Avatar for orisano orisano
September 13, 2018
8
1.5k

the world of characters

Avatar for orisano

orisano

September 13, 2018
Tweet

More Decks by orisano

See All by orisano
OSS Performance Tuning Tips
Avatar for orisano orisano
8
6.1k
Docker-Compose & BuildKit
Avatar for orisano orisano
4
1k
Container Build Talk
Avatar for orisano orisano
3
2.6k
dockerignore talk
Avatar for orisano orisano
2
7.3k
Better docker image+
Avatar for orisano orisano
6
6.4k
Socket.IO Introduction
Avatar for orisano orisano
0
3.3k
Profiling Go Application
Avatar for orisano orisano
11
8k
Multi-stage Builds Patterns & Practice
Avatar for orisano orisano
6
5.3k
better docker image
Avatar for orisano orisano
22
30k

Featured

See All Featured
Practical Tips for Bootstrapping Information Extraction Pipelines
Avatar for Matthew Honnibal honnibal
25
1.8k
From π to Pie charts
Avatar for Rasagy Sharma rasagy
0
140
The MySQL Ecosystem @ GitHub 2015
Avatar for Sam Lambert samlambert
251
13k
The Director’s Chair: Orchestrating AI for Truly Effective Learning
Avatar for Mike Taylor tmiket
1
120
Taking LLMs out of the black box: A practical guide to human-in-the-loop distillation
Avatar for Ines Montani inesmontani
PRO
3
2.1k
Odyssey Design
Avatar for Ryan Kendrick rkendrick25
PRO
2
530
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
Avatar for Marc Duiker marcduiker
35
2.4k
Understanding Cognitive Biases in Performance Measurement
Avatar for Philip Tellis bluesmoon
32
2.8k
How to optimise 3,500 product descriptions for ecommerce in one day using ChatGPT
Avatar for Katarina Dahlin katarinadahlin
PRO
1
3.5k
How to Build an AI Search Optimization Roadmap - Criteria and Steps to Take #SEOIRL
Avatar for Aleyda Solis aleyda
1
1.9k
A Guide to Academic Writing Using Generative AI - A Workshop
Avatar for Kenji Saito ks91
PRO
0
230
The Illustrated Children's Guide to Kubernetes
Avatar for Chris Short chrisshort
51
52k

Transcript

  1. 1จࣈͷੈք @orisano

  2. Έͳ͞Μ จࣈΛ਺͑ΒΕ·͢ΑͶʁ

  3. a

  4. a => 1

  5. ͋

  6. ͋ => 1

  7. 佛

  8. 佛 => 1

  9. None

  10. => 1

  11. None

  12. => 1

  13. Z͑ͫ̓ͪ̂ͫ̽ ̴̙̤̞͉͚̯̞̠͍A̴̵̜̰͔ ͫ͗ ͢ L̠ͨͧͩ͘ G̴̻͈͍͔̹ ̑͗̎̅͛ ́ Ǫ̵̹̻̝̳ ͂̌

    ̌͘! ͖̬̰̙̗ ̿̋ ͥ ͥ̂ͣ̐́́͜͞

  14. Z͑ͫ̓ͪ̂ͫ̽ ̴̙̤̞͉͚̯̞̠͍A̴̵̜̰͔ ͫ͗ ͢ L̠ͨͧͩ͘ G̴̻͈͍͔̹ ̑͗̎̅͛ ́ Ǫ̵̹̻̝̳ ͂̌

    ̌͘! ͖̬̰̙̗ ̿̋ ͥ ͥ̂ͣ̐́́͜͞ => 6

  15. Έͳ͞Μ όΠτ਺Λ਺͑ΒΕ·͔͢ʁ (UTF-8)

  16. a

  17. a => 1

  18. ͋

  19. ͋ => 3

  20. 佛

  21. 佛 => 4

  22. None

  23. => 4

  24. None

  25. => 18

  26. Z͑ͫ̓ͪ̂ͫ̽ ̴̙̤̞͉͚̯̞̠͍A̴̵̜̰͔ ͫ͗ ͢ L̠ͨͧͩ͘ G̴̻͈͍͔̹ ̑͗̎̅͛ ́ Ǫ̵̹̻̝̳ ͂̌

    ̌͘! ͖̬̰̙̗ ̿̋ ͥ ͥ̂ͣ̐́́͜͞

  27. Z͑ͫ̓ͪ̂ͫ̽ ̴̙̤̞͉͚̯̞̠͍A̴̵̜̰͔ ͫ͗ ͢ L̠ͨͧͩ͘ G̴̻͈͍͔̹ ̑͗̎̅͛ ́ Ǫ̵̹̻̝̳ ͂̌

    ̌͘! ͖̬̰̙̗ ̿̋ ͥ ͥ̂ͣ̐́́͜͞ => 143

  28. ͋ͳ͕ͨࢥ͏1จࣈ͸ Ͳ͏਺͑Δ΂͖͔ʁ

  29. byte਺Ͱ͸਺͑ΒΕͳ͍

  30. Unicode͸จࣈू߹ จࣈͱ਺஋͕ରԠ͢Δ

  31. ͋ => 3042

  32. => 1F914

  33. ͜ͷ਺஋ͷ͜ͱΛ ίʔυϙΠϯτ ͱݺͿ

  34. ͜ͷίʔυϙΠϯτΛ byteྻͰදݱ͢Δํ๏Λ ΤϯίʔσΟϯάͱ͍͏

  35. UTF-8ͱ͔UTF-16ͱ͔͸ ΤϯίʔσΟϯάͷҰछ

  36. ͱΓ͋͑ͣ ίʔυϙΠϯτΛ਺͑Ε͹ ղܾʁ

  37. ͍͍͑

  38. => 1F468 + 200D + 1F469 + 200D + 1F466

  39. ࣮͸ෳ਺ͷίʔυϙΠϯτͰ ҰͭͷจࣈʹͳͬͨΓ͢Δ

  40. ਓ͕ؒೝ͍ࣝͯ͠Δ̍จࣈ͸ ॻهૉ(Grapheme cluster) ͱݺ͹Ε͍ͯΔ

  41. Ͳ͏΍Ε͹ ίʔυϙΠϯτͷྻ͔Β ॻهૉΛऔΓग़ͤΔ͔

  42. ίʔυϙΠϯτ͕ؒ ॻهૉڥքʹͳΔ͔Ͳ͏͔ͷ ݫີͳϧʔϧ͕͋Δ

  43. UAX #29 Unicode Text Segmentation

  44. None

  45. ͜ΕΛJSͰ࣮૷ͯ͠·ͨ͠ github.com/orisano/graphemesplit

  46. ৄ͘͠͸ UAX #29 Λݟͯ http://unicode.org/reports/tr29/

  47. ݟ஌Β͵ਓʹʓจࣈͱ ݴΘΕͨͱ͖ʹ͸ ͪΌΜͱ֬ೝ͠Α͏ʂ

  48. 1 byte? 1 codepoint? 1 grapheme cluster?