Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What is Unicode?

Kimtaro
December 15, 2016
Programming
0
360

What is Unicode?

A brief look at what Unicode is, what it contains and what we can do with it.

Presented at Web Platform London, December 14. https://www.meetup.com/WebPlatform-London/events/236089771/

Kimtaro

December 15, 2016
Tweet

More Decks by Kimtaro

See All by Kimtaro
Ve, a linguistic framework
kimtaro
3
4.2k

Other Decks in Programming

See All in Programming
Exploring the Implementation of “t.Run”, “t.Parallel”, and “t.Cleanup”
akarin
1
110
Let's learn code review
riofujimon
2
570
Git Rebase
bkuhlmann
11
1.6k
From Spring Boot 2 to Spring Boot 3 with Java 21 and Jakarta EE
ivargrimstad
0
550
"config" ってなんだ？ / What is "config"?
okashoi
0
320
AppRouter Panel Talk
yosuke_furukawa
PRO
1
460
Java 22 Overview
kishida
1
190
2 週間で Twitter Bot を作ってみた
contour_gara
0
770
はてなにおける CSS Modules、及び CSS Modules に足りないもの / CSS Modules in Hatena, and CSS Modules missing parts
mizdra
7
970
デフォルトにして至高、RubyMineの大好きな所
ruzia
0
770
Fast JSX: Don't clone props object #28768
yossydev
1
180
What We Can Learn From OSS
inouehi
0
440

Featured

See All Featured
Git: the NoSQL Database
bkeepers
PRO
423
63k
A Philosophy of Restraint
colly
197
16k
Statistics for Hackers
jakevdp
790
220k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
14
1.5k
The MySQL Ecosystem @ GitHub 2015
samlambert
244
12k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
188
16k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
123
39k
Building an army of robots
kneath
300
41k
Stop Working from a Prison Cell
hatefulcrawdad
267
19k
Unsuck your backbone
ammeep
663
57k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
226
51k
Typedesign – Prime Four
hannesfritz
36
2.1k

Transcript

  1. What is Unicode? Kim Ahlström – @kimtaro

  2. https://xkcd.com/927/

  3. None
  4. None
  5. None

  6. Why we need Unicode What Unicode is and how it’s

    organized Terminology Sorting Emoji Practical considerations in Javascript, MySQL, Regular Expressions, and malformed text Helper data and libraries

  7. Purpose

  8. Why do we need Unicode? Reliably write, store and send

    text Most living languages, and many historical Does away with a multitude of standards

  9. What is Unicode?

  10. What is Unicode? A numbered list of characters Ways to

    encode these numbers Information about the characters Algorithms for sorting, displaying, etc

  11. A numbered list of characters 0041;LATIN CAPITAL LETTER A;Lu;0;L;;;;;N;;;;0061; 304B;HIRAGANA

    LETTER KA;Lo;0;L;;;;;N;;;;; 1F409;DRAGON;So;0;ON;;;;;N;;;;; http://www.unicode.org/Public/UCD/latest/ucd/UnicodeData.txt

  12. 0041;LATIN CAPITAL LETTER A Code point Character name

  13. 0 - 10FFFF 1,114,112 code points 271,792 allocated

  14. Ways to encode these numbers UTF-8 UTF-16 UTF-32 A U+0041

    0x41 0x0041 0x00000041 ͔ U+304B 0xE3818B 0x304B 0x0000304B U+1F409 0xF09F9089 0xD83DDC09 0x0001F409

  15. Information about the characters 0600..0604 ; Arabic # Cf [5]

    ARABIC NUMBER SIGN..ARABIC SIGN SAMVAT 30FB;W # Po KATAKANA MIDDLE DOT 1F1F8 1F1EA ; Emoji_Flag_Sequence # 6.0 [1] (#) Flag for Sweden http://www.unicode.org/Public/UCD/latest/ucd/Scripts.txt http://www.unicode.org/Public/9.0.0/ucd/EastAsianWidth.txt http://www.unicode.org/Public/emoji/3.0//emoji-sequences.txt

  16. Algorithms for sorting, displaying, etc http://unicode.org/reports/ Bidirectional Algorithm Line Breaking

    Algorithm Text Segmentation Collation Algorithm Regular Expressions

  17. “Unicode is complicated!”

  18. Unicode is complicated because language is complicated because humans are

    complicated

  19. 1987 2016 2010 1991 Work started 1.0 7,161 characters 9.0

    128,237 characters

  20. Organization 17 planes, with 65,535 code points each Blocks of

    related characters Modern scripts in Plane 0
 BMP, 0000-FFFF Emoji in Plane 1
 SMP, 10000-1FFFF
  21. None
  22. None

  23. Terminology

  24. Glyph Á Á Grapheme Á Character Sequence A ´ Code

    Point U+0041 + U+00B4 U+1F600 or U+D83D + U+DE00 Code Unit (UTF-8) 41 C2 B4 F0 9F 98 80

  25. Bytes Code points Glyphs $ 1 1 1 ¢ 2

    1 1 € 3 1 1 4 1 1 ⛪ 3 1 1 4 1 1 1 4 2 1 ' 8 2 1 ( 11 3 1 ) 27 8 1 https://twitter.com/FakeUnicode/status/717751277342490624

  26. Encodings UCS-2 UTF-8 UTF-16 UTF-32

  27. UCS-2 First! 65,535 characters are enough! Only BMP Older programming

    languages

  28. UTF-16 BMP → One 16 bit code unit U+0041 (A)

    → 0041 Above BMP → Two 16 bit code units U+1F600 () → D83D + DE00

  29. UTF-16 Systems and programming languages Byte order matters UTF-16: FEFF

    0041 UTF-16 LE: FFFE 4100 Byte Order Mark (0xFEFF, ZERO WIDTH NO-BREAK SPACE )

  30. UTF-8 This is the encoding you are looking for One

    – four 8 bit code units Easy to parse Byte order does not matter

  31. UTF-8 U+0041 A 41 U+00D6 Ö C3 96 U+304B ͔

    E3 81 8B U+1F600 F0 9F 98 80

  32. UTF-32 32 bit code units, U+0041 (A) → 00000041 Space

    inefficient One code unit per character BORING!

  33. Han unification

  34. Han unification https://github.com/adobe-fonts/source-han-sans/raw/release/SourceHanSansReadMe.pdf zh-Hans zh-Hant ja ko

  35. Han unification Mississippi Mißißippi

  36. Indicate language <html lang="ja"> <div lang="en-US">

  37. Å Å Å U+00C5 LATIN CAPITAL LETTER A WITH RING

    ABOVE U+212B ANGSTROM SIGN U+0041 LATIN CAPITAL LETTER A U+030A COMBINING RING ABOVE

  38. Å Å Å U+00C5 LATIN CAPITAL LETTER A WITH RING

    ABOVE U+212B ANGSTROM SIGN U+0041 LATIN CAPITAL LETTER A U+030A COMBINING RING ABOVE Equivalence

  39. Å Å Å U+00C5 LATIN CAPITAL LETTER A WITH RING

    ABOVE U+212B ANGSTROM SIGN U+0041 LATIN CAPITAL LETTER A U+030A COMBINING RING ABOVE Normalization

  40. Sorting By code point - BAD! Unicode Collation Algorithm Determined

    by language Customizations

  41. Swedish
 Z sorts before Ä German
 Ä sorts before Z

  42. Emoji

  43. Emoji

  44. Emoji 2,198 emoji Most in SMP
 Two UTF-16 or four

    UTF-8 code units Separate data files

  45. Emoji

  46. Emoji

  47. ZWJ sequences , ) . / 1 2 3 4

    5 6 7 8 9 : ; < = >

  48. ZWJ sequences = WOMAN ZERO WIDTH JOINER WOMAN ZERO WIDTH

    JOINER BOY ZERO WIDTH JOINER BOY

  49. Genders A PEDESTRIAN ZERO WIDTH JOINER ♀ FEMALE SIGN VARIATION

    SELECTOR-16

  50. Fitzpatrick scale Emoji Modifier Fitzpatrick Type-1-2 Emoji Modifier Fitzpatrick Type-3

    Emoji Modifier Fitzpatrick Type-4 Emoji Modifier Fitzpatrick Type-5 Emoji Modifier Fitzpatrick Type-6 J K L M N + =

  51. Genders + skin tones O RUNNER EMOJI MODIFIER FITZPATRICK TYPE-4

    ZERO WIDTH JOINER ♀ FEMALE SIGN VARIATION SELECTOR-16
  52. None

  53. Flags # REGIONAL INDICATOR SYMBOL LETTER S REGIONAL INDICATOR SYMBOL

    LETTER E

  54. Javascript

  55. Javascript

  56. Javascript

  57. ES6

  58. ES6

  59. ES6

  60. ES6

  61. ES6

  62. ES6

  63. ES6

  64. ES6

  65. ES6

  66. MySQL “utf8” - only three bytes “utf8mb4” - four bytes

  67. Regular Expressions Unicode properties \p{} Uppercase_Letter, Math_Symbol Scripts: Latin, Arabic,

    Katakana \w and \b match beyond ASCII, like Ä

  68. Malformed text

  69. Malformed text Don’t know the character encoding Broken UTF-8, UTF-16

  70. Malformed text $ cat broken_utf8.txt ?BÄAÖC

  71. Malformed text $ iconv -f utf-8 -t utf-8 -c broken_utf8.txt

    BÄAÖC

  72. Malformed text

  73. CLDR & ICU

  74. Common Locale Data Repository Language and country data Formatting dates,

    numbers, currency Translations of scripts, languages, date units Script characters, sorting and transliteration rules

  75. International Components for Unicode C/C++ and Java libraries for Unicode

    handling Uses CLDR data Encoding conversion Sorting, formatting, normalization Time and calendar conversion Regular expressions Text segmentation

  76. – Richard Feynman “If you think you understand Unicode, you

    don't understand Unicode.”

  77. Questions? @kimtaro http://www.unicode.org http://emojipedia.org http://graphemica.com http://site.icu-project.org