the world of characters

6247c099ad62bf727a4f9df10b6c6f23?s=47 orisano
September 13, 2018
1k

the world of characters

6247c099ad62bf727a4f9df10b6c6f23?s=128

orisano

September 13, 2018
Tweet

Transcript

  1. 1จࣈͷੈք @orisano

  2. Έͳ͞Μ จࣈΛ਺͑ΒΕ·͢ΑͶʁ

  3. a

  4. a => 1

  5. ͋

  6. ͋ => 1

  7. 佛 => 1

  8. None
  9. => 1

  10. None
  11. => 1

  12. Z͑ͫ̓ͪ̂ͫ̽ ̴̙̤̞͉͚̯̞̠͍A̴̵̜̰͔ ͫ͗ ͢ L̠ͨͧͩ͘ G̴̻͈͍͔̹ ̑͗̎̅͛ ́ Ǫ̵̹̻̝̳ ͂̌

    ̌͘! ͖̬̰̙̗ ̿̋ ͥ ͥ̂ͣ̐́́͜͞
  13. Z͑ͫ̓ͪ̂ͫ̽ ̴̙̤̞͉͚̯̞̠͍A̴̵̜̰͔ ͫ͗ ͢ L̠ͨͧͩ͘ G̴̻͈͍͔̹ ̑͗̎̅͛ ́ Ǫ̵̹̻̝̳ ͂̌

    ̌͘! ͖̬̰̙̗ ̿̋ ͥ ͥ̂ͣ̐́́͜͞ => 6
  14. Έͳ͞Μ όΠτ਺Λ਺͑ΒΕ·͔͢ʁ (UTF-8)

  15. a

  16. a => 1

  17. ͋

  18. ͋ => 3

  19. 佛 => 4

  20. None
  21. => 4

  22. None
  23. => 18

  24. Z͑ͫ̓ͪ̂ͫ̽ ̴̙̤̞͉͚̯̞̠͍A̴̵̜̰͔ ͫ͗ ͢ L̠ͨͧͩ͘ G̴̻͈͍͔̹ ̑͗̎̅͛ ́ Ǫ̵̹̻̝̳ ͂̌

    ̌͘! ͖̬̰̙̗ ̿̋ ͥ ͥ̂ͣ̐́́͜͞
  25. Z͑ͫ̓ͪ̂ͫ̽ ̴̙̤̞͉͚̯̞̠͍A̴̵̜̰͔ ͫ͗ ͢ L̠ͨͧͩ͘ G̴̻͈͍͔̹ ̑͗̎̅͛ ́ Ǫ̵̹̻̝̳ ͂̌

    ̌͘! ͖̬̰̙̗ ̿̋ ͥ ͥ̂ͣ̐́́͜͞ => 143
  26. ͋ͳ͕ͨࢥ͏1จࣈ͸ Ͳ͏਺͑Δ΂͖͔ʁ

  27. byte਺Ͱ͸਺͑ΒΕͳ͍

  28. Unicode͸จࣈू߹ จࣈͱ਺஋͕ରԠ͢Δ

  29. ͋ => 3042

  30. => 1F914

  31. ͜ͷ਺஋ͷ͜ͱΛ ίʔυϙΠϯτ ͱݺͿ

  32. ͜ͷίʔυϙΠϯτΛ byteྻͰදݱ͢Δํ๏Λ ΤϯίʔσΟϯάͱ͍͏

  33. UTF-8ͱ͔UTF-16ͱ͔͸ ΤϯίʔσΟϯάͷҰछ

  34. ͱΓ͋͑ͣ ίʔυϙΠϯτΛ਺͑Ε͹ ղܾʁ

  35. ͍͍͑

  36. => 1F468 + 200D + 1F469 + 200D + 1F466

  37. ࣮͸ෳ਺ͷίʔυϙΠϯτͰ ҰͭͷจࣈʹͳͬͨΓ͢Δ

  38. ਓ͕ؒೝ͍ࣝͯ͠Δ̍จࣈ͸ ॻهૉ(Grapheme cluster) ͱݺ͹Ε͍ͯΔ

  39. Ͳ͏΍Ε͹ ίʔυϙΠϯτͷྻ͔Β ॻهૉΛऔΓग़ͤΔ͔

  40. ίʔυϙΠϯτ͕ؒ ॻهૉڥքʹͳΔ͔Ͳ͏͔ͷ ݫີͳϧʔϧ͕͋Δ

  41. UAX #29 Unicode Text Segmentation

  42. None
  43. ͜ΕΛJSͰ࣮૷ͯ͠·ͨ͠ github.com/orisano/graphemesplit

  44. ৄ͘͠͸ UAX #29 Λݟͯ http://unicode.org/reports/tr29/

  45. ݟ஌Β͵ਓʹʓจࣈͱ ݴΘΕͨͱ͖ʹ͸ ͪΌΜͱ֬ೝ͠Α͏ʂ

  46. 1 byte? 1 codepoint? 1 grapheme cluster?