U is for Unicode - solving the mystery

PyNash presentation from 2016-03-24.

Transcript

1. U is for Unicode Greg Back March 24, 2016 /gtback @gtback
2. U is for Unicode (With apologies to Sue Grafton) Undertow
3. U is for Unicode Greg Back March 24, 2016 /gtback @gtback
4. Ü is for Üñîçødé Greg Back March 24, 2016 /gtback @gtback
5. Ü is for Üñîçødé Clue #0: DO NOT use accents for stylistic (or comedic) effect. ☺
6. Four score and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal.
7. ᯜ᯹᯿᯼ ᯽ᯭ᯹᯼ᯯ ᯫ᯸ᯮ ᯽ᯯᰀᯯ᯸ ᰃᯯᯫ᯼᯽ ᯫᯱ᯹ ᯹᯿᯼ ᯰᯫ᯾᯲ᯯ᯼᯽ ᯬ᯼᯹᯿ᯱ᯲᯾ ᯰ᯹᯼᯾᯲ ᯹᯸ ᯾᯲᯳᯽ ᯭ᯹᯸᯾᯳᯸ᯯ᯸᯾, ᯫ ᯸ᯯᰁ ᯸ᯫ᯾᯳᯹᯸, ᯭ᯹᯸ᯭᯯ᯳ᰀᯯᯮ ᯳᯸ ᯡ᯳ᯬᯯ᯼᯾ᰃ, ᯫ᯸ᯮ ᯮᯯᯮ᯳ᯭᯫ᯾ᯯᯮ ᯾᯹ ᯾᯲ᯯ ᯺᯼᯹᯺᯹᯽᯳᯾᯳᯹᯸ ᯾᯲ᯫ᯾ ᯫ᯶᯶ ᯷ᯯ᯸ ᯫ᯼ᯯ ᯭ᯼ᯯᯫ᯾ᯯᯮ ᯯ᯻᯿ᯫ᯶. Four score and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal.
8. Ⓕⓞⓤⓡ ⓢⓒⓞⓡⓔ ⓐⓝⓓ ⓢⓔⓥⓔⓝ ⓨⓔⓐⓡⓢ ⓐⓖⓞ ⓞⓤⓡ ⓕⓐⓣⓗⓔⓡⓢ ⓑⓡⓞⓤⓖⓗⓣ ⓕⓞⓡⓣⓗ ⓞⓝ ⓣⓗⓘⓢ ⓒⓞⓝⓣⓘⓝⓔⓝⓣ , ⓐ ⓝⓔⓦ ⓝⓐⓣⓘⓞⓝ, ⓒⓞⓝⓒⓔⓘⓥⓔⓓ ⓘⓝ Ⓛⓘⓑⓔⓡⓣⓨ, ⓐⓝⓓ ⓓⓔⓓⓘⓒⓐⓣⓔⓓ ⓣⓞ ⓣⓗⓔ ⓟⓡⓞⓟⓞⓢⓘⓣⓘⓞⓝ ⓣⓗⓐⓣ ⓐⓛⓛ ⓜⓔⓝ ⓐⓡⓔ ⓒⓡⓔⓐⓣⓔⓓ ⓔⓠⓤⓐⓛ◎ ᯜ᯹᯿᯼ ᯽ᯭ᯹᯼ᯯ ᯫ᯸ᯮ ᯽ᯯᰀᯯ᯸ ᰃᯯᯫ᯼᯽ ᯫᯱ᯹ ᯹᯿᯼ ᯰᯫ᯾᯲ᯯ᯼᯽ ᯬ᯼᯹᯿ᯱ᯲᯾ ᯰ᯹᯼᯾᯲ ᯹᯸ ᯾᯲᯳᯽ ᯭ᯹᯸᯾᯳᯸ᯯ᯸᯾, ᯫ ᯸ᯯᰁ ᯸ᯫ᯾᯳᯹᯸, ᯭ᯹᯸ᯭᯯ᯳ᰀᯯᯮ ᯳᯸ ᯡ᯳ᯬᯯ᯼᯾ᰃ, ᯫ᯸ᯮ ᯮᯯᯮ᯳ᯭᯫ᯾ᯯᯮ ᯾᯹ ᯾᯲ᯯ ᯺᯼᯹᯺᯹᯽᯳᯾᯳᯹᯸ ᯾᯲ᯫ᯾ ᯫ᯶᯶ ᯷ᯯ᯸ ᯫ᯼ᯯ ᯭ᯼ᯯᯫ᯾ᯯᯮ ᯯ᯻᯿ᯫ᯶. Four score and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal.
9. ℉◎υя ṧ¢øяℯ @ηⅾ ṧε♥εᾔ ¥ε@ґṧ αℊø øüя ḟ@т♄εґṧ ßґ☺υℊ♄☂ ḟ☺ґ⊥н øη ⊥ℌ☤ṧ ¢øη☂☤ᾔℯηт, @ η℮ẘ ηα⊥☤øη, ¢☺ηḉℯїṽ℮ḓ їᾔ ℒ☤♭ℯя☂ƴ, αηḓ ⅾℯḓḯ¢ α⊥℮ⅾ ⊥◎ т♄℮ ρґ◎℘øṧ☤тїøᾔ тнα☂ αʟʟ μεη @яℯ ¢я℮α☂ε∂ εⓠü@ʟ▪ Ⓕⓞⓤⓡ ⓢⓒⓞⓡⓔ ⓐⓝⓓ ⓢⓔⓥⓔⓝ ⓨⓔⓐⓡⓢ ⓐⓖⓞ ⓞⓤⓡ ⓕⓐⓣⓗⓔⓡⓢ ⓑⓡⓞⓤⓖⓗⓣ ⓕⓞⓡⓣⓗ ⓞⓝ ⓣⓗⓘⓢ ⓒⓞⓝⓣⓘⓝⓔⓝⓣ , ⓐ ⓝⓔⓦ ⓝⓐⓣⓘⓞⓝ, ⓒⓞⓝⓒⓔⓘⓥⓔⓓ ⓘⓝ Ⓛⓘⓑⓔⓡⓣⓨ, ⓐⓝⓓ ⓓⓔⓓⓘⓒⓐⓣⓔⓓ ⓣⓞ ⓣⓗⓔ ⓟⓡⓞⓟⓞⓢⓘⓣⓘⓞⓝ ⓣⓗⓐⓣ ⓐⓛⓛ ⓜⓔⓝ ⓐⓡⓔ ⓒⓡⓔⓐⓣⓔⓓ ⓔⓠⓤⓐⓛ◎ ᯜ᯹᯿᯼ ᯽ᯭ᯹᯼ᯯ ᯫ᯸ᯮ ᯽ᯯᰀᯯ᯸ ᰃᯯᯫ᯼᯽ ᯫᯱ᯹ ᯹᯿᯼ ᯰᯫ᯾᯲ᯯ᯼᯽ ᯬ᯼᯹᯿ᯱ᯲᯾ ᯰ᯹᯼᯾᯲ ᯹᯸ ᯾᯲᯳᯽ ᯭ᯹᯸᯾᯳᯸ᯯ᯸᯾, ᯫ ᯸ᯯᰁ ᯸ᯫ᯾᯳᯹᯸, ᯭ᯹᯸ᯭᯯ᯳ᰀᯯᯮ ᯳᯸ ᯡ᯳ᯬᯯ᯼᯾ᰃ, ᯫ᯸ᯮ ᯮᯯᯮ᯳ᯭᯫ᯾ᯯᯮ ᯾᯹ ᯾᯲ᯯ ᯺᯼᯹᯺᯹᯽᯳᯾᯳᯹᯸ ᯾᯲ᯫ᯾ ᯫ᯶᯶ ᯷ᯯ᯸ ᯫ᯼ᯯ ᯭ᯼ᯯᯫ᯾ᯯᯮ ᯯ᯻᯿ᯫ᯶. Four score and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal.
10. ℉◎υя ṧ¢øяℯ @ηⅾ ṧε♥εᾔ ¥ε@ґṧ αℊø øüя ḟ@т♄εґṧ ßґ☺υℊ♄☂ ḟ☺ґ⊥н øη ⊥ℌ☤ṧ ¢øη☂☤ᾔℯηт, @ η℮ẘ ηα⊥☤øη, ¢☺ηḉℯїṽ℮ḓ їᾔ ℒ☤♭ℯя☂ƴ, αηḓ ⅾℯḓḯ¢ α⊥℮ⅾ ⊥◎ т♄℮ ρґ◎℘øṧ☤тїøᾔ тнα☂ αʟʟ μεη @яℯ ¢я℮α☂ε∂ εⓠü@ʟ▪ Ⓕⓞⓤⓡ ⓢⓒⓞⓡⓔ ⓐⓝⓓ ⓢⓔⓥⓔⓝ ⓨⓔⓐⓡⓢ ⓐⓖⓞ ⓞⓤⓡ ⓕⓐⓣⓗⓔⓡⓢ ⓑⓡⓞⓤⓖⓗⓣ ⓕⓞⓡⓣⓗ ⓞⓝ ⓣⓗⓘⓢ ⓒⓞⓝⓣⓘⓝⓔⓝⓣ , ⓐ ⓝⓔⓦ ⓝⓐⓣⓘⓞⓝ, ⓒⓞⓝⓒⓔⓘⓥⓔⓓ ⓘⓝ Ⓛⓘⓑⓔⓡⓣⓨ, ⓐⓝⓓ ⓓⓔⓓⓘⓒⓐⓣⓔⓓ ⓣⓞ ⓣⓗⓔ ⓟⓡⓞⓟⓞⓢⓘⓣⓘⓞⓝ ⓣⓗⓐⓣ ⓐⓛⓛ ⓜⓔⓝ ⓐⓡⓔ ⓒⓡⓔⓐⓣⓔⓓ ⓔⓠⓤⓐⓛ◎ ᯜ᯹᯿᯼ ᯽ᯭ᯹᯼ᯯ ᯫ᯸ᯮ ᯽ᯯᰀᯯ᯸ ᰃᯯᯫ᯼᯽ ᯫᯱ᯹ ᯹᯿᯼ ᯰᯫ᯾᯲ᯯ᯼᯽ ᯬ᯼᯹᯿ᯱ᯲᯾ ᯰ᯹᯼᯾᯲ ᯹᯸ ᯾᯲᯳᯽ ᯭ᯹᯸᯾᯳᯸ᯯ᯸᯾, ᯫ ᯸ᯯᰁ ᯸ᯫ᯾᯳᯹᯸, ᯭ᯹᯸ᯭᯯ᯳ᰀᯯᯮ ᯳᯸ ᯡ᯳ᯬᯯ᯼᯾ᰃ, ᯫ᯸ᯮ ᯮᯯᯮ᯳ᯭᯫ᯾ᯯᯮ ᯾᯹ ᯾᯲ᯯ ᯺᯼᯹᯺᯹᯽᯳᯾᯳᯹᯸ ᯾᯲ᯫ᯾ ᯫ᯶᯶ ᯷ᯯ᯸ ᯫ᯼ᯯ ᯭ᯼ᯯᯫ᯾ᯯᯮ ᯯ᯻᯿ᯫ᯶. Four score and seven years ago our fathers brought forth on this continent, a new nation, conceived in Liberty, and dedicated to the proposition that all men are created equal.
11. What is Unicode?
12. What is an encoding?
13. Clue #1: Humans deal with characters. Computers deal with numbers.
14. ASCII 32 40 ( 48 0 56 8 64 @ 72 H 80 P 88 X 96 ` 104 h 112 p 120 x 33 ! 41 ) 49 1 57 9 65 A 73 I 81 Q 89 Y 97 a 105 i 113 q 121 y 34 " 42 * 50 2 58 : 66 B 74 J 82 R 90 Z 98 b 106 j 114 r 122 z 35 # 43 + 51 3 59 ; 67 C 75 K 83 S 91 [ 99 c 107 k 115 s 123 { 36 $ 44 , 52 4 60 < 68 D 76 L 84 T 92 \ 100 d 108 l 116 t 124 | 37 % 45 - 53 5 61 = 69 E 77 M 85 U 93 ] 101 e 109 m 117 u 125 } 38 & 46 . 54 6 62 > 70 F 78 N 86 V 94 ^ 102 f 110 n 118 v 126 ~ 39 ' 47 / 55 7 63 ? 71 G 79 O 87 W 95 _ 103 g 111 o 119 w 127 
15. Latin-1 32 40 ( 48 0 56 8 64 @ 72 H 80 P 88 X 96 ` 104 h 112 p 120 x 33 ! 41 ) 49 1 57 9 65 A 73 I 81 Q 89 Y 97 a 105 i 113 q 121 y 34 " 42 * 50 2 58 : 66 B 74 J 82 R 90 Z 98 b 106 j 114 r 122 z 35 # 43 + 51 3 59 ; 67 C 75 K 83 S 91 [ 99 c 107 k 115 s 123 { 36 $ 44 , 52 4 60 < 68 D 76 L 84 T 92 \ 100 d 108 l 116 t 124 | 37 % 45 - 53 5 61 = 69 E 77 M 85 U 93 ] 101 e 109 m 117 u 125 } 38 & 46 . 54 6 62 > 70 F 78 N 86 V 94 ^ 102 f 110 n 118 v 126 ~ 39 ' 47 / 55 7 63 ? 71 G 79 O 87 W 95 _ 103 g 111 o 119 w 127  160 168 ¨ 176 ° 184 ¸ 192 À 200 È 208 Ð 216 Ø 224 à 232 è 240 ð 248 ø 161 ¡ 169 © 177 ± 185 ¹ 193 Á 201 É 209 Ñ 217 Ù 225 á 233 é 241 ñ 249 ù 162 ¢ 170 ª 178 ² 186 º 194 Â 202 Ê 210 Ò 218 Ú 226 â 234 ê 242 ò 250 ú 163 £ 171 « 179 ³ 187 » 195 Ã 203 Ë 211 Ó 219 Û 227 ã 235 ë 243 ó 251 û 164 ¤ 172 ¬ 180 ´ 188 ¼ 196 Ä 204 Ì 212 Ô 220 Ü 228 ä 236 ì 244 ô 252 ü 165 ¥ 173 - 181 µ 189 ½ 197 Å 205 Í 213 Õ 221 Ý 229 å 237 í 245 õ 253 ý 166 ¦ 174 ® 182 ¶ 190 ¾ 198 Æ 206 Î 214 Ö 222 Þ 230 æ 238 î 246 ö 254 þ 167 § 175 ¯ 183 · 191 ¿ 199 Ç 207 Ï 215 × 223 ß 231 ç 239 ï 247 ÷ 255 ÿ
16. Latin-2 32 40 ( 48 0 56 8 64 @ 72 H 80 P 88 X 96 ` 104 h 112 p 120 x 33 ! 41 ) 49 1 57 9 65 A 73 I 81 Q 89 Y 97 a 105 i 113 q 121 y 34 " 42 * 50 2 58 : 66 B 74 J 82 R 90 Z 98 b 106 j 114 r 122 z 35 # 43 + 51 3 59 ; 67 C 75 K 83 S 91 [ 99 c 107 k 115 s 123 { 36 $ 44 , 52 4 60 < 68 D 76 L 84 T 92 \ 100 d 108 l 116 t 124 | 37 % 45 - 53 5 61 = 69 E 77 M 85 U 93 ] 101 e 109 m 117 u 125 } 38 & 46 . 54 6 62 > 70 F 78 N 86 V 94 ^ 102 f 110 n 118 v 126 ~ 39 ' 47 / 55 7 63 ? 71 G 79 O 87 W 95 _ 103 g 111 o 119 w 127  160 168 ¨ 176 ° 184 ¸ 192 Ŕ 200 Č 208 Đ 216 Ř 224 ŕ 232 č 240 đ 248 ř 161 Ą 169 Š 177 ą 185 š 193 Á 201 É 209 Ń 217 Ů 225 á 233 é 241 ń 249 ů 162 ˘ 170 Ş 178 ˛ 186 ş 194 Â 202 Ę 210 Ň 218 Ú 226 â 234 ę 242 ň 250 ú 163 Ł 171 Ť 179 ł 187 ť 195 Ă 203 Ë 211 Ó 219 Ű 227 ă 235 ë 243 ó 251 ű 164 ¤ 172 Ź 180 ´ 188 ź 196 Ä 204 Ě 212 Ô 220 Ü 228 ä 236 ě 244 ô 252 ü 165 Ľ 173 - 181 ľ 189 ˝ 197 Ĺ 205 Í 213 Ő 221 Ý 229 ĺ 237 í 245 ő 253 ý 166 Ś 174 Ž 182 ś 190 ž 198 Ć 206 Î 214 Ö 222 Ţ 230 ć 238 î 246 ö 254 ţ 167 § 175 Ż 183 ˇ 191 ż 199 Ç 207 Ď 215 × 223 ß 231 ç 239 ď 247 ÷ 255 ˙
17. Greek (ISO/IEC 8859-7) 32 40 ( 48 0 56 8 64 @ 72 H 80 P 88 X 96 ` 104 h 112 p 120 x 33 ! 41 ) 49 1 57 9 65 A 73 I 81 Q 89 Y 97 a 105 i 113 q 121 y 34 " 42 * 50 2 58 : 66 B 74 J 82 R 90 Z 98 b 106 j 114 r 122 z 35 # 43 + 51 3 59 ; 67 C 75 K 83 S 91 [ 99 c 107 k 115 s 123 { 36 $ 44 , 52 4 60 < 68 D 76 L 84 T 92 \ 100 d 108 l 116 t 124 | 37 % 45 - 53 5 61 = 69 E 77 M 85 U 93 ] 101 e 109 m 117 u 125 } 38 & 46 . 54 6 62 > 70 F 78 N 86 V 94 ^ 102 f 110 n 118 v 126 ~ 39 ' 47 / 55 7 63 ? 71 G 79 O 87 W 95 _ 103 g 111 o 119 w 127  160 168 ¨ 176 ° 184 Έ 192 ΐ 200 Θ 208 216 Ψ 224 ΰ 232 θ 240 π 248 ψ 161 ‘ 169 © 177 ± 185 Ή 193 Α 201 209 217 Ω 225 α 233 ι 241 ρ 249 ω 162 ’ 170 ͺ 178 ² 186 Ί 194 Β 202 210 218 Ϊ 226 β 234 κ 242 ς 250 ϊ 163 £ 171 « 179 ³ 187 » 195 Γ 203 211 Σ 219 Ϋ 227 γ 235 λ 243 σ 251 ϋ 164 € 172 ¬ 180 ΄ 188 Ό 196 Δ 204 212 Τ 220 ά 228 δ 236 μ 244 τ 252 ό 165 ₯ 173 - 181 ΅ 189 ½ 197 Ε 205 213 Υ 221 έ 229 ε 237 ν 245 υ 253 ύ 166 ¦ 174 182 Ά 190 Ύ 198 Ζ 206 214 Φ 222 ή 230 ζ 238 ξ 246 φ 254 ώ 167 § 175 ― 183 · 191 Ώ 199 Η 207 215 Χ 223 ί 231 η 239 ο 247 χ 255
18. Japanese (IBM Code Page 932) 32 40 ( 48 0 56 8 64 @ 72 H 80 P 88 X 96 ` 104 h 112 p 120 x 33 ! 41 ) 49 1 57 9 65 A 73 I 81 Q 89 Y 97 a 105 i 113 q 121 y 34 " 42 * 50 2 58 : 66 B 74 J 82 R 90 Z 98 b 106 j 114 r 122 z 35 # 43 + 51 3 59 ; 67 C 75 K 83 S 91 [ 99 c 107 k 115 s 123 { 36 $ 44 , 52 4 60 < 68 D 76 L 84 T 92 \ 100 d 108 l 116 t 124 | 37 % 45 - 53 5 61 = 69 E 77 M 85 U 93 ] 101 e 109 m 117 u 125 } 38 & 46 . 54 6 62 > 70 F 78 N 86 V 94 ^ 102 f 110 n 118 v 126 ~ 39 ' 47 / 55 7 63 ? 71 G 79 O 87 W 95 _ 103 g 111 o 119 w 127  160 168 ｨ 176 ｰ 184 ｸ 192 ﾀ 200 ﾈ 208 ﾐ 216 ﾘ 161 ｡ 169 ｩ 177 ｱ 185 ｹ 193 ﾁ 201 ﾉ 209 ﾑ 217 ﾙ 162 ｢ 170 ｪ 178 ｲ 186 ｺ 194 ﾂ 202 ﾊ 210 ﾒ 218 ﾚ 163 ｣ 171 ｫ 179 ｳ 187 ｻ 195 ﾃ 203 ﾋ 211 ﾓ 219 ﾛ 164 ､ 172 ｬ 180 ｴ 188 ｼ 196 ﾄ 204 ﾌ 212 ﾔ 220 ﾜ 165 ･ 173 ｭ 181 ｵ 189 ｽ 197 ﾅ 205 ﾍ 213 ﾕ 221 ﾝ 166 ｦ 174 ｮ 182 ｶ 190 ｾ 198 ﾆ 206 ﾎ 214 ﾖ 222 ﾞ 167 ｧ 175 ｯ 183 ｷ 191 ｿ 199 ﾇ 207 ﾏ 215 ﾗ 223 ﾟ
19. Japanese (IBM Code Page 932) 32 40 ( 48 0 56 8 64 @ 72 H 80 P 88 X 96 ` 104 h 112 p 120 x 33 ! 41 ) 49 1 57 9 65 A 73 I 81 Q 89 Y 97 a 105 i 113 q 121 y 34 " 42 * 50 2 58 : 66 B 74 J 82 R 90 Z 98 b 106 j 114 r 122 z 35 # 43 + 51 3 59 ; 67 C 75 K 83 S 91 [ 99 c 107 k 115 s 123 { 36 $ 44 , 52 4 60 < 68 D 76 L 84 T 92 \ 100 d 108 l 116 t 124 | 37 % 45 - 53 5 61 = 69 E 77 M 85 U 93 ] 101 e 109 m 117 u 125 } 38 & 46 . 54 6 62 > 70 F 78 N 86 V 94 ^ 102 f 110 n 118 v 126 ~ 39 ' 47 / 55 7 63 ? 71 G 79 O 87 W 95 _ 103 g 111 o 119 w 127  160 168 ｨ 176 ｰ 184 ｸ 192 ﾀ 200 ﾈ 208 ﾐ 216 ﾘ 161 ｡ 169 ｩ 177 ｱ 185 ｹ 193 ﾁ 201 ﾉ 209 ﾑ 217 ﾙ 162 ｢ 170 ｪ 178 ｲ 186 ｺ 194 ﾂ 202 ﾊ 210 ﾒ 218 ﾚ 163 ｣ 171 ｫ 179 ｳ 187 ｻ 195 ﾃ 203 ﾋ 211 ﾓ 219 ﾛ 164 ､ 172 ｬ 180 ｴ 188 ｼ 196 ﾄ 204 ﾌ 212 ﾔ 220 ﾜ 165 ･ 173 ｭ 181 ｵ 189 ｽ 197 ﾅ 205 ﾍ 213 ﾕ 221 ﾝ 166 ｦ 174 ｮ 182 ｶ 190 ｾ 198 ﾆ 206 ﾎ 214 ﾖ 222 ﾞ 167 ｧ 175 ｯ 183 ｷ 191 ｿ 199 ﾇ 207 ﾏ 215 ﾗ 223 ﾟ Plus some 2- byte forms!!
20. What is Unicode? One ring to rule them all “encoding” https://commons.wikimedia.org/wiki/File:Unico_Anello.png
21. Unicode and Python (Demo Time!) Python 2 (“two blue”) Python 3 (“three green”)
22. Python 3 Unicode <class 'str'> Bytes <class 'bytes'> Cannot add them together Python 2 Unicode <class 'unicode'> Bytes <class 'str'> Can add them together (normally) Text types
23. Clue #2: <Unicode>.encode() → <Bytes> <Bytes>.decode() → <Unicode>
24. Clue #2: <Unicode>.encode() → <Bytes> <Bytes>.decode() → <Unicode> Clue #3: In Python, “Unicode” isn’t an “encoding”. “UTF-8”, “UTF-16”, etc. are
25. Python 3 UTF-8 Python 2 ascii? System Default Encodings sys.getdefaultencoding()
26. Clue #4: Unicode “Sandwich” 1. Convert from bytes to Unicode as early as possible 2. Convert from Unicode to bytes as late as possible 3. Deal with Unicode everywhere in the middle.
27. Detecting Encodings Q: How do I know the encoding of a stream of bytes?
28. Detecting Encodings Q: How do I know the encoding of a stream of bytes? A1: You don’t.
29. Detecting Encodings Q: How do I know the encoding of a stream of bytes? A1: You don’t. A2: Unless you’re told.
30. Detecting Encodings Q: How do I know the encoding of a stream of bytes? A1: You don’t. A2: Unless you’re told. A3: But even then you can’t be sure.
31. Detecting Encodings Q: How do I know the encoding of a stream of bytes? A1: You don’t. A2: Unless you’re told. A3: But even then you can’t be sure. A4: Fortunately, you can “guess”.
32. Clue #5: Be prepared for anything.
33. Solving the Mystery Clue #1: Humans:Characters :: Computers:Numbers Clue #2: U.encode() → B B.decode() → U Clue #3: “Unicode” isn’t an encoding. “UTF-8” is. Clue #4: Unicode Sandwich Clue #5: Be prepared for anything.
34. Resources http://www.joelonsoftware.com/articles/Unicode.html (“Spolsky Unicode”) http://nedbatchelder.com/text/unipain.html (“Unipain Python”) https://speakerdeck.com/ramalho/unicode-solutions-in-python-2-and-python-3 (“Speakerdeck Unicode Python”) https://www.safaribooksonline.com/library/view/fluent- python/9781491946237/ch04.html (“Fluent Python Unicode”) Thank you! Questions? /gtback @gtback