Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Decoding the Secrets of Binary Data (Droidcon NYC 2016)

Jesse Wilson
September 30, 2016

Decoding the Secrets of Binary Data (Droidcon NYC 2016)

Video: https://www.youtube.com/watch?v=T_p22jMZSrk
Code: https://github.com/swankjesse/encoding

Opaque blobs of data have hexed Android programmers for too long. It’s time to byte the bullet and learn how data is transmitted and persisted.

In this talk we’ll:

💾 Learn a bit about base64, little-endian, and EOF.
💾 See how inefficient encodings nibble away resources.
💾 Hash out the differences between ASCII, UTF-8, and other charsets.
💾 Zip through examples of compression, crypto, and protocol buffers.
💾 Load up on APIs and discover what Square’s Okio has in store.

This talk offers a short introduction to an array of topics. You’ll learn enough to be encode & decode whatever data you select!

Jesse Wilson

September 30, 2016
Tweet

More Decks by Jesse Wilson

Other Decks in Programming

Transcript

  1. @jessewilson
    Decoding the Secrets
    of Binary Data
    https://github.com/swankjesse/encoding

    View Slide

  2. Encoding
    @jessewilson
    https://github.com/swankjesse/encoding

    View Slide

  3. 0 1

    View Slide

  4. View Slide

  5. View Slide

  6. Goals
    • Get comfortable with bytes
    • Write simpler, more efficient programs

    View Slide

  7. Plan
    • Bottom up
    • History
    • Math
    • Code

    View Slide

  8. 1095 AD

    View Slide

  9. Date: Aprilis 11, 1095
    From: Emperor Alexios Komnenos
    To: Robert II, Count of Flanders
    Bob,
    Looks like we’re gonna war with the
    Turks. Could you see if the Pope
    could help us crusade against ’em?
    Thanks!
    Alex
    Letters!
    • Human readable
    • Signed

    View Slide

  10. Date: Aprilis 11, 1095
    From: Emperor Alexios Komnenos
    To: Robert II, Count of Flanders
    Bob,
    Looks like we’re gonna war with the
    Turks. Could you see if the Pope
    could help us crusade against ’em?
    Thanks!
    Alex
    Letters!
    • Slow, dangerous to transmit
    • Awkward to store
    • Requires Literacy!

    View Slide

  11. 1854

    View Slide

  12. Telegraphs
    NYC
    SF

    View Slide

  13. Telegraphs
    NYC
    SF

    View Slide

  14. Morse Code
    A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •

    View Slide

  15. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • •

    View Slide

  16. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • • ▬ ▬ ▬

    View Slide

  17. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • • ▬ ▬ ▬ ▬ •

    View Slide

  18. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • • ▬ ▬ ▬ ▬ • • • ▬

    View Slide

  19. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • • ▬ ▬ ▬ ▬ • • • ▬ ▬

    View Slide

  20. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • • ▬ ▬ ▬ ▬ • • • ▬ ▬

    View Slide

  21. Morse Code
    NYC
    SF

    View Slide

  22. Morse Code
    • Telegraphs!
    • 15 words per minute
    • Wireless in 1898

    View Slide

  23. Morse Code
    • Very limited characters
    • No lowercase
    • Limited punctuation*
    • Average message: 12 words
    * For example, morse code doesn’t have an asterisk symbol.

    View Slide

  24. Binary Refresher!

    View Slide

  25. 205

    View Slide

  26. 200
    205 = + 5

    View Slide

  27. 2×100
    205 = + 5×1

    View Slide

  28. 2×100
    205 = +
    + 0×10 5×1

    View Slide

  29. 2×102
    205 = +
    + 0×101 5×10

    View Slide

  30. 205

    View Slide

  31. 205

    View Slide

  32. 128 8
    +
    205 = 64 4
    +
    + + 1

    View Slide

  33. 1×128 1×8
    +
    205 = 1×64 1×4
    +
    + + 1×1

    View Slide

  34. 1×128 1×8
    +
    205 1×64 + 1×4
    +
    0×32 0×16
    + + 0×2
    + 1×1
    +
    =

    View Slide

  35. 1×27 1×23
    +
    205 1×26 + 1×22
    +
    0×25 0×24
    + + 0×21
    + 1×20
    +
    =

    View Slide

  36. 205 1×27 1×23
    1×26 1×22
    0×25 0×24 0×21 1×20
    =

    View Slide

  37. 205 1×27 1×23
    1×26 1×22
    0×25
    0×240×21
    1×20
    0b
    =

    View Slide

  38. Binary Refresher!
    • Decimal lets you represent any integer with a sequence of digits:
    0, 1, 2, 3, 4, 5, 6, 7, 8, 9
    • Binary lets you represent any integer with a sequence of bits: 0, 1

    View Slide

  39. Binary Refresher!
    • Typically prefixed with “0b”, like 0b11001101

    View Slide

  40. Binary Refresher!
    • N bits can represent 2N values
    • 8 bits: 0..255
    • 16 bits: 0..65,535
    • 32 bits: 0..4,294,967,295
    • 64 bits: 0..18,446,744,073,709,551,615

    View Slide

  41. Binary Refresher!
    NYC
    SF

    View Slide

  42. Layering
    • Given a wire that transmits a single bit, we can use binary to
    encode any integer!
    • This works because the sender and recipient agree on how to
    interpret the sequence
    • That interpretation is called an encoding

    View Slide

  43. Bytes
    • Though binary supports any number of bits, we like 8-bit integers
    • An 8-bit integer is called a byte
    • 256 values from 0 to 255

    View Slide

  44. 1967

    View Slide

  45. ASCII
    • American Standard Code for Information Interchange
    • A table of characters
    • Interpret a sequence of bytes as a string of characters!

    View Slide

  46. ASCII
    • Work started in 1960
    • Only uses 7 bits: in 1967 bits were very expensive!
    • That means there’s 27 = 128 characters

    View Slide

  47. ASCII
    0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL

    View Slide

  48. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68

    View Slide

  49. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111

    View Slide

  50. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111 110

    View Slide

  51. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111 110 117

    View Slide

  52. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111 110 117 116

    View Slide

  53. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111 110 117 116

    View Slide

  54. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111 110 117 116

    View Slide

  55. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    111 110 117 116
    0 1 0 0 0 1 0 0

    View Slide

  56. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    110 117 116
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1

    View Slide

  57. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    117 116
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0

    View Slide

  58. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    116
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1

    View Slide

  59. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 0

    View Slide

  60. 1980’s

    View Slide

  61. boss, what should
    I be working on?

    View Slide

  62. boss, what should
    I be working on?
    resume polishing

    View Slide

  63. boss, what should
    I be working on?

    View Slide

  64. boss, what should
    I be working on?
    résumé polishing

    View Slide

  65. Charset Hell
    • ASCII is English only
    • But non-English people also use computers!

    View Slide

  66. Charset Hell
    • Operating systems were installed for a specific character set and
    wouldn’t work with any others
    • Documents couldn’t mix Greek, French, and Russian characters
    • If you see ISO-8859-1, run away!

    View Slide

  67. 1991

    View Slide

  68. Unicode
    • Support all languages in a single system
    • A code point is a universal ID for a character

    View Slide

  69. UTF-16
    • 16-bit Unicode Transformation Format
    • 2 bytes per code point
    • This is Java’s char type

    View Slide

  70. ASCII
    UTF-16

    View Slide

  71. ASCII
    0 1 0 0 0 1 0 0
    0 1 0 0 0 1 0 0
    0 0 0 0 0 0 0 0
    D
    UTF-16

    View Slide

  72. ASCII
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    D
    UTF-16
    o

    View Slide

  73. ASCII
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 1 1 0 1 1 1 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    D
    UTF-16
    on

    View Slide

  74. ASCII
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0
    D
    UTF-16
    onu

    View Slide

  75. ASCII
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 0
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    D
    UTF-16
    onut

    View Slide

  76. ASCII
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 0
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    D
    UTF-16
    onut

    View Slide

  77. UTF-16 characters
    • Max code point is 65,535
    • Code point for is 127,849
    • 127,849 > 65,535

    View Slide

  78. Java’s char is broken!
    • There’s a system called “surrogate pairs” which is like multidex for
    code points
    • It splits a single code point across 2 chars
    • It’s an incredible pain

    View Slide


  79. char[] s = "Café ".toCharArray();
    C a f é
    0 1 2 3 4 5 6

    View Slide

  80. String s = “Café ";


    for (int i = 0, size = s.length(); i < size; i++) {

    char c = s.charAt(i);

    System.out.printf("The character at %d is '%c'%n", i, c);

    }

    View Slide

  81. String s = “Café ";


    for (int i = 0, size = s.length(); i < size; i++) {

    char c = s.charAt(i);

    System.out.printf("The character at %d is '%c'%n", i, c);

    }
    The character at 0 is 'C'
    The character at 1 is 'a'
    The character at 2 is 'f'
    The character at 3 is 'é'
    The character at 4 is ' '
    The character at 5 is ' '
    The character at 6 is ' '

    View Slide

  82. String s = “Café ";


    for (int i = 0, size = s.length(); i < size; ) {

    int c = s.codePointAt(i);

    System.out.printf("The code point at %d is '%c'%n", i, c);
    i += Character.charCount(c);

    }

    View Slide

  83. String s = “Café ";


    for (int i = 0, size = s.length(); i < size; ) {

    int c = s.codePointAt(i);

    System.out.printf("The code point at %d is '%c'%n", i, c);
    i += Character.charCount(c);

    }
    The code point at 0 is 'C'
    The code point at 1 is 'a'
    The code point at 2 is 'f'
    The code point at 3 is 'é'
    The code point at 4 is ' '
    The code point at 5 is ' '

    View Slide

  84. 1998

    View Slide

  85. UTF-8
    • 8-bit Unicode Transformation Format
    • Variable number of bytes per code point
    • This is how modern apps transmit & store text

    View Slide

  86. Image copyright Chris55, https://commons.wikimedia.org/wiki/File:Utf8webgrowth.svg

    View Slide

  87. UTF-8
    • Many common characters are 1-byte
    • Some are 2 and 3 bytes. ‘ ’ is 4 bytes.
    • Self-delimiting
    • Self-aligning

    View Slide

  88. 0
    1 1 0 1 0
    1 1 1 0
    1 1 1 1 0
    1 0
    1 0
    1 0
    1 0 1 0
    ≤7
    ≤11
    ≤16
    ≤21
    “How many bits do you need?”

    View Slide

  89. C
    “How many bits do you need?”
    a
    f
    é
    sp

    View Slide

  90. C
    “How many bits do you need?”
    a
    f
    é
    sp
    67

    View Slide

  91. 1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    a
    f
    é
    sp

    View Slide

  92. 1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    a
    f
    é
    sp
    97

    View Slide

  93. 1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    f
    é
    sp

    View Slide

  94. 1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    f
    é
    sp
    102

    View Slide

  95. 1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp

    View Slide

  96. 233
    1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp

    View Slide

  97. 1 1 1 0 1 0 0 1
    1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp

    View Slide

  98. 1 1 1 0 1 0 0 1
    1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp 32

    View Slide

  99. 1 0 0 0 0 0
    1 1 1 0 1 0 0 1
    1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp

    View Slide

  100. 1 0 0 0 0 0
    1 1 1 0 1 0 0 1
    1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp
    127,849

    View Slide

  101. 1 0 1 0 0 1
    0 0 1 1 0 1
    1 1 1 1 1 1
    1 0 0 0 0 0
    1 1 1 0 1 0 0 1
    1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp

    View Slide

  102. 0
    1 1 0
    1 1 1 1 0 1 0
    0
    1 0
    0
    0
    1 0
    1 0 1 0 1 0 0 1
    0 0 1 1 0 1
    0 1 1 1 1 1
    1 0 0 0 0 0
    1 0 1 0 0 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp
    1 1
    1 0 0 0 0 1 1

    View Slide

  103. 0
    1 1 0 0 0 0
    1 1 1 1 0 0 0 0 1 0
    0 0
    1 0
    0
    0
    1 0
    1 0 1 0 1 0 0 1
    0 0 1 1 0 1
    0 1 1 1 1 1
    1 0 0 0 0 0
    1 0 1 0 0 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp
    1 1
    1 0 0 0 0 1 1

    View Slide

  104. 0 1 0 0 0 0 1 1 1 1 0 0 0 0 1 1
    1 0 1 0 1 0 0 1 1 1 1 1 0 0 0 0 1 0 0 1 1 1 1 1
    1 0 0 0 1 1 0 1 1 0 1 0 1 0 0 1
    0 1 1 0 0 0 0 1 0 1 1 0 0 1 1 0
    0 0 1 0 0 0 0 0

    View Slide

  105. UTF-8
    • Basically the best thing ever
    • Superset of ASCII
    • Great for JSON and HTML because delimiter characters <, > and "
    are 1-byte

    View Slide

  106. Colors

    View Slide

  107. R G B

    View Slide

  108. View Slide

  109. View Slide

  110. View Slide

  111. R
    255
    0
    G
    255
    0
    B
    255
    0
    255
    255 255

    View Slide

  112. R
    255
    0
    G
    255
    0
    B
    255
    0
    255
    0 255

    View Slide

  113. R
    255
    0
    G
    255
    0
    B
    255
    0
    20
    206 210

    View Slide

  114. Hexadecimal Refresher!

    View Slide

  115. Hexadecimal Refresher!
    • Decimal digits: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
    • Binary bits: 0, 1
    • Hexadecimal digits: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, a, b, c, d, e, f

    View Slide

  116. 205

    View Slide

  117. 205 192
    = + 13

    View Slide

  118. 205 12×16
    = + 13×1

    View Slide

  119. 205 12×161
    = + 13×160

    View Slide

  120. 205 c×161
    = + d×160

    View Slide

  121. 205 c
    = d
    0x

    View Slide

  122. Hexadecimal Refresher!
    • Prefixed with “0x”, like 0xcd
    • Hex bytes are always two digits: 00, 01, 02 … ff
    • Sequences of bytes are okay: 0bb634

    View Slide

  123. Hexadecimal Refresher!
    • Colors: #ffffff
    • URL escaping: http://example.com/?q=hello%20world
    • Unicode code points U+2020
    • IPv6 addresses: 2001:0db8:85a3:0000:0000:8a2e:0370:7334

    View Slide

  124. R
    255
    0
    G
    255
    0
    B
    255
    0
    ced214

    View Slide

  125. Pictures

    View Slide

  126. Pictures
    • Just a 2D array of colors
    • Given 3 bytes per pixel:
    • 64 × 64 icon is 12,288 bytes
    • 1080 × 1920 picture is 5.9 MiB
    • Compression is important!

    View Slide

  127. Pictures
    • Android can use fewer bits per pixel
    • ARGB_8888: alpha, red, green, and blue get 8 bits each
    • RGB_565: red gets 5 bits, geen gets 6, blue gets 5 bits

    View Slide

  128. RGB_565
    ARGB_8888

    View Slide

  129. RGB_454
    ARGB_8888

    View Slide

  130. RGB_232
    ARGB_8888

    View Slide

  131. /** https://en.wikipedia.org/wiki/BMP_file_format */

    public void encode(BufferedSink sink) throws IOException {

    int height = pixels.length;

    int width = pixels[0].length;


    int bytesPerPixel = 3;

    int rowByteCountWithoutPadding = (bytesPerPixel * width);

    int rowByteCount = ((rowByteCountWithoutPadding + 3) / 4) * 4;

    int pixelDataSize = rowByteCount * height;

    int bmpHeaderSize = 14;

    int dibHeaderSize = 40;


    // BMP Header

    sink.writeUtf8("BM"); // ID.

    sink.writeIntLe(bmpHeaderSize + dibHeaderSize + pixelDataSize); // File size.

    sink.writeShortLe(0); // Unused.

    sink.writeShortLe(0); // Unused.

    sink.writeIntLe(bmpHeaderSize + dibHeaderSize); // Offset of pixel data.


    // DIB Header

    sink.writeIntLe(dibHeaderSize);


    View Slide

  132. int dibHeaderSize = 40;


    // BMP Header

    sink.writeUtf8("BM"); // ID.

    sink.writeIntLe(bmpHeaderSize + dibHeaderSize + pixelDataSize); // File size.

    sink.writeShortLe(0); // Unused.

    sink.writeShortLe(0); // Unused.

    sink.writeIntLe(bmpHeaderSize + dibHeaderSize); // Offset of pixel data.


    // DIB Header

    sink.writeIntLe(dibHeaderSize);

    sink.writeIntLe(width);

    sink.writeIntLe(height);

    sink.writeShortLe(1); // Color plane count.

    sink.writeShortLe(bytesPerPixel * Byte.SIZE);

    sink.writeIntLe(0); // No compression.

    sink.writeIntLe(16); // Size of bitmap data including padding.

    sink.writeIntLe(2835); // Horizontal print resolution in pixels/meter. (72 dpi).

    sink.writeIntLe(2835); // Vertical print resolution in pixels/meter. (72 dpi).

    sink.writeIntLe(0); // Palette color count.

    sink.writeIntLe(0); // 0 important colors.


    // Pixel data.


    View Slide

  133. sink.writeIntLe(0); // Palette color count.

    sink.writeIntLe(0); // 0 important colors.


    // Pixel data.

    for (int y = height - 1; y >= 0; y--) {

    int[] row = pixels[y];

    for (int x = 0; x < width; x++) {

    int pixel = row[x];

    sink.writeByte((pixel & 0x0000ff)); // Blue.

    sink.writeByte((pixel & 0x00ff00) >>> 8); // Green.

    sink.writeByte((pixel & 0xff0000) >>> 16); // Red.

    }


    // Padding for 4-byte alignment.

    for (int p = rowByteCountWithoutPadding; p < rowByteCount; p++) {

    sink.writeByte(0);

    }

    }

    }


    View Slide

  134. Pictures on Bytes
    • This bitmap writer is 50 lines of code
    • Decoders are more difficult!
    • Good specs make it easy

    View Slide

  135. sink.writeIntLe(0); // Palette color count.

    sink.writeIntLe(0); // 0 important colors.


    // Pixel data.

    for (int y = height - 1; y >= 0; y--) {

    int[] row = pixels[y];

    for (int x = 0; x < width; x++) {

    int pixel = row[x];

    sink.writeByte((pixel & 0x0000ff)); // Blue.

    sink.writeByte((pixel & 0x00ff00) >>> 8); // Green.

    sink.writeByte((pixel & 0xff0000) >>> 16); // Red.

    }


    // Padding for 4-byte alignment.

    for (int p = rowByteCountWithoutPadding; p < rowByteCount; p++) {

    sink.writeByte(0);

    }

    }

    }


    View Slide

  136. sink.writeIntLe(0); // Palette color count.

    sink.writeIntLe(0); // 0 important colors.


    // Pixel data.

    for (int y = height - 1; y >= 0; y--) {

    int[] row = pixels[y];

    for (int x = 0; x < width; x++) {

    int pixel = row[x];

    sink.writeByte((pixel & 0x0000ff)); // Blue.

    sink.writeByte((pixel & 0x00ff00) >>> 8); // Green.

    sink.writeByte((pixel & 0xff0000) >>> 16); // Red.

    }


    // Padding for 4-byte alignment.

    for (int p = rowByteCountWithoutPadding; p < rowByteCount; p++) {

    sink.writeByte(0);

    }

    }

    }


    View Slide

  137. (pixel & 0xff0000) >>> 16
    • Shifting and masking lets you access the bits within an integer
    • & and | operators treat each int like a 32-element boolean array!
    • <<, >> and >>> operators slide bits left and right

    View Slide

  138. 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0
    0000ab00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
    00ffab40 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
    00ffab40
    0000ff00 1 1 1 1 1 1 1 1
    0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0x00ffab40 & 0x0000ff00

    View Slide

  139. 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0
    0000ab00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
    00ffab40
    1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
    00ffab40
    0000ff00 1 1 1 1 1 1 1 1
    0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0x00ffab40 & 0x0000ff00

    View Slide

  140. 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0
    0000ab00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
    00ffab40
    1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
    00ffab40
    0000ff00 1 1 1 1 1 1 1 1
    0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0x00ffab40 & 0x0000ff00
    &
    =

    View Slide

  141. 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0
    00000ab00
    0x0000ab00 >> 8

    View Slide

  142. 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0
    00000ab00
    0x0000ab00 >> 8
    {
    8

    View Slide

  143. 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0
    000000ab
    0x0000ab00 >> 8

    View Slide

  144. 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0
    000000ab
    0x0000ab00 >> 8
    = 0x000000ab

    View Slide

  145. sink.writeIntLe(0); // Palette color count.

    sink.writeIntLe(0); // 0 important colors.


    // Pixel data.

    for (int y = height - 1; y >= 0; y--) {

    int[] row = pixels[y];

    for (int x = 0; x < width; x++) {

    int pixel = row[x];

    sink.writeByte((pixel & 0x0000ff)); // Blue.

    sink.writeByte((pixel & 0x00ff00) >>> 8); // Green.

    sink.writeByte((pixel & 0xff0000) >>> 16); // Red.

    }


    // Padding for 4-byte alignment.

    for (int p = rowByteCountWithoutPadding; p < rowByteCount; p++) {

    sink.writeByte(0);

    }

    }

    }


    View Slide

  146. This space
    intentionally
    left blank

    View Slide

  147. int dibHeaderSize = 40;


    // BMP Header

    sink.writeUtf8("BM"); // ID.

    sink.writeIntLe(bmpHeaderSize + dibHeaderSize + pixelDataSize); // File size.

    sink.writeShortLe(0); // Unused.

    sink.writeShortLe(0); // Unused.

    sink.writeIntLe(bmpHeaderSize + dibHeaderSize); // Offset of pixel data.


    // DIB Header

    sink.writeIntLe(dibHeaderSize);

    sink.writeIntLe(width);

    sink.writeIntLe(height);

    sink.writeShortLe(1); // Color plane count.

    sink.writeShortLe(bytesPerPixel * Byte.SIZE);

    sink.writeIntLe(0); // No compression.

    sink.writeIntLe(16); // Size of bitmap data including padding.

    sink.writeIntLe(2835); // Horizontal print resolution in pixels/meter. (72 dpi).

    sink.writeIntLe(2835); // Vertical print resolution in pixels/meter. (72 dpi).

    sink.writeIntLe(0); // Palette color count.

    sink.writeIntLe(0); // 0 important colors.


    // Pixel data.


    View Slide

  148. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    Big Endian

    View Slide

  149. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    0 0 0 0 0 0 0 0
    Big Endian

    View Slide

  150. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    1 1 1 1 1 1 1 1
    0 0 0 0 0 0 0 0
    Big Endian

    View Slide

  151. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    1 0 1 0 1 0 1 1
    1 1 1 1 1 1 1 1
    0 0 0 0 0 0 0 0
    Big Endian

    View Slide

  152. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    0 1 0 0 0 0 0 0
    1 0 1 0 1 0 1 1
    1 1 1 1 1 1 1 1
    0 0 0 0 0 0 0 0
    Big Endian

    View Slide

  153. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    0 1 0 0 0 0 0 0
    1 0 1 0 1 0 1 1
    1 1 1 1 1 1 1 1
    0 0 0 0 0 0 0 0
    Big Endian

    View Slide

  154. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    Little Endian

    View Slide

  155. 0 1 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    Little Endian

    View Slide

  156. 1 0 1 0 1 0 1 1
    0 1 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    Little Endian

    View Slide

  157. 1 0 1 0 1 0 1 1 1 1 1 1 1 1 1 1
    0 1 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    Little Endian

    View Slide

  158. 1 0 1 0 1 0 1 1 1 1 1 1 1 1 1 1
    0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    Little Endian

    View Slide

  159. 1 0 1 0 1 0 1 1 1 1 1 1 1 1 1 1
    0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    Little Endian

    View Slide

  160. Data Models

    View Slide

  161. Conference App!
    • Each talk has an ID, date, room, title,
    and speaker

    View Slide

  162. JSON is Good Stuff
    {

    "id": 72017,

    "date": "2016-09-30T18:30:00Z",

    "room": "RIGHT",

    "title": "Decoding the Secrets of Binary Data",

    "speaker": "Jesse Wilson"

    }

    View Slide

  163. JSON is Good Stuff
    {

    "id": 72017,

    "date": "2016-09-30T18:30:00Z",

    "room": "RIGHT",

    "title": "Decoding the Secrets of Binary Data",

    "speaker": "Jesse Wilson"

    }
    • A nice format that builds on UTF-8
    • Easy to read & write

    View Slide

  164. JSON is Self-Delimiting
    {

    "id": 72017,

    "date": "2016-09-30T18:30:00Z",

    "room": "RIGHT",

    "title": "Decoding the Secrets of Binary Data",

    "speaker": "Jesse Wilson"

    }
    • A JSON document has both structure and data
    • Uses escape sequences like \" to be completely unambiguous

    View Slide

  165. "2016-09-30T18:30:00Z"
    • System.currentTimeMillis() returns milliseconds since
    January 1, 1970 at 00:00:00 UTC
    • A date that’s 8 bytes in memory is 22 bytes in JSON!

    View Slide

  166. • Space: a simple message like this one is ~128 bytes
    • Time: bigger sequences take longer to decode
    JSON Space & Time

    View Slide

  167. • Google’s “small, fast, simple” structured data format
    • Upon closer inspection, it’s not that different from JSON!
    • But it has a schema
    Protocol Buffers

    View Slide

  168. {
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    "id": 72017,
    "date": "2016-09-30T18:30:00Z",
    "room": "RIGHT",
    "title": "Decoding the Secrets of Binary Data",
    "speaker": "Jesse Wilson"
    }
    "id"
    "date"
    "room"
    "title"
    "speaker"
    72017
    "2016-09-30T18:30:00Z",
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",

    View Slide

  169. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }

    View Slide

  170. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    000 enum, int32, int64...
    001 fixed64
    010 string, message
    101 fixed32
    Length Mode

    View Slide

  171. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    000 enum, int32, int64...
    001 fixed64
    010 string, message
    101 fixed32
    Length Mode
    1 0 1

    View Slide

  172. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    000 enum, int32, int64...
    001 fixed64
    010 string, message
    101 fixed32
    Length Mode
    1 0 1
    0 0 1

    View Slide

  173. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    000 enum, int32, int64...
    001 fixed64
    010 string, message
    101 fixed32
    Length Mode
    0 0 0
    1 0 1
    0 0 1

    View Slide

  174. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    000 enum, int32, int64...
    001 fixed64
    010 string, message
    101 fixed32
    Length Mode
    0 0 0
    1 0 1
    0 1 0
    0 0 1

    View Slide

  175. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    000 enum, int32, int64...
    001 fixed64
    010 string, message
    101 fixed32
    Length Mode
    0 0 0
    1 0 1
    0 1 0
    0 1 0
    0 0 1

    View Slide

  176. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    0 0 0
    1 0 1
    0 1 0
    0 1 0
    0 0 1

    View Slide

  177. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    0 0 0
    0 0 0 0 1 1 0 1
    0 1 0
    0 1 0
    0 0 1

    View Slide

  178. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    0 0 0
    0 0 0 0 1 1 0 1
    0 1 0
    0 1 0
    0 0 0 1 0 0 0 1

    View Slide

  179. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    0 0 0 1 1 0 0 0
    0 0 0 0 1 1 0 1
    0 1 0
    0 1 0
    0 0 0 1 0 0 0 1

    View Slide

  180. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    0 0 0 1 1 0 0 0
    0 0 0 0 1 1 0 1
    0 1 0
    0 0 1 0 0 0 1 0
    0 0 0 1 0 0 0 1

    View Slide

  181. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    0 0 0 1 1 0 0 0
    0 0 0 0 1 1 0 1
    0 0 1 0 1 0 1 0
    0 0 1 0 0 0 1 0
    0 0 0 1 0 0 0 1

    View Slide

  182. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    0 0 0 1 1 0 0 0
    0 0 0 0 1 1 0 1
    0 0 1 0 1 0 1 0
    0 0 1 0 0 0 1 0
    0 0 0 1 0 0 0 1

    View Slide

  183. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    0d
    11
    18
    22
    2a

    View Slide

  184. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    {
    "id": 72017,
    "date": "2016-09-30T18:30:00Z",
    "room": "RIGHT",
    "title": "Decoding the Secrets of Binary Data",
    "speaker": "Jesse Wilson"
    }
    "id"
    "date"
    "room"
    "title"
    "speaker"
    72017
    "2016-09-30T18:30:00Z",
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",

    View Slide

  185. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    72017
    "2016-09-30T18:30:00Z",
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",

    View Slide

  186. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    72017
    "2016-09-30T18:30:00Z",
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",

    View Slide

  187. 51
    19
    01
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    "2016-09-30T18:30:00Z",
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",
    00

    View Slide

  188. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    "2016-09-30T18:30:00Z",
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",
    00

    View Slide

  189. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    "2016-09-30T18:30:00Z",
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",
    00

    View Slide

  190. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",
    00
    000001577D37EE40

    View Slide

  191. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",
    00
    00
    00
    01
    57
    7D
    37
    EE
    40

    View Slide

  192. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",
    00
    00
    00
    01
    57
    7D
    37
    EE
    40

    View Slide

  193. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",
    00
    00
    00
    01
    57
    7D
    37
    EE
    40
    02

    View Slide

  194. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    "Decoding the Secrets of Binary Data"
    "Jesse Wilson",
    00
    00
    01
    57
    7D
    37
    EE
    40
    02
    00

    View Slide

  195. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    4465636f64696e67207468652053656372657473206f662042696e6172792044617461
    "Jesse Wilson",
    00
    00
    00
    01
    57
    7D
    37
    EE
    40
    02

    View Slide

  196. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    4465636f64696e67207468652053656372657473206f662042696e6172792044617461
    "Jesse Wilson",
    00
    00
    00
    01
    57
    7D
    37
    EE
    40
    02
    23

    View Slide

  197. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    4465636f64696e67207468652053656372657473206f662042696e6172792044617461
    "Jesse Wilson"
    00
    00
    00
    01
    57
    7D
    37
    EE
    40
    02
    23

    View Slide

  198. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    4465636f64696e67207468652053656372657473206f662042696e6172792044617461
    4a657373652057696c736f6e
    00
    00
    00
    01
    57
    7D
    37
    EE
    40
    02
    23

    View Slide

  199. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    4465636f64696e67207468652053656372657473206f662042696e6172792044617461
    4a657373652057696c736f6e
    00
    00
    00
    01
    57
    7D
    37
    EE
    40
    02
    23
    0c

    View Slide

  200. 511901
    0d
    11
    18
    22
    2a 4a657373652057696c736f6e
    00
    00
    00
    01
    57
    7D
    37
    EE
    40
    02
    23
    0c
    4465636f64696e67207468652053656372657473206f662042696e6172792044617461

    View Slide

  201. 511901
    0d 11 18 22
    2a
    4465636f64696e6720746865205365637265747
    4a657373652057696c736f6e
    00 00
    00
    01
    57
    7D
    37
    EE
    40 02 23
    0c
    3206f662042696e6172792044617461

    View Slide

  202. 511901
    0d 11 18 22
    2a
    4465636f64696e6720746865205365637265747
    4a657373652057696c736f6e
    00 00
    00
    01
    57
    7D
    37
    EE
    40 02 23
    0c
    3206f662042696e6172792044617461

    View Slide

  203. 511901
    0d 11 18 22
    2a
    4465636f64696e6720746865205365637265747
    4a657373652057696c736f6e
    00 00
    00
    01
    57
    7D
    37
    EE
    40 02 23
    0c
    3206f662042696e6172792044617461
    • Protocol Buffers are small and fast
    • ~1 byte for each field name
    • Compact encoding for numbers and enums
    • Strings stay the same length!
    • This message is 67 bytes in protocol buffers, vs. 128 for JSON

    View Slide

  204. Java I/O

    View Slide

  205. • Mutable!
    • equals() doesn’t work like it should: can’t be a map key!
    • Doesn’t implement Comparable
    byte[] is bad

    View Slide

  206. • Range is -128..127 but you usually want 0..255
    • But unsigned when you call InputStream.read()
    byte is signed

    View Slide

  207. • Range is -128..127 but you usually want 0..255
    • But unsigned when you call InputStream.read()
    byte is signed

    View Slide

  208. Okio

    View Slide

  209. Okio
    • Square’s library for efficient I/O
    • Used by OkHttp, Retrofit, Wire, Moshi

    View Slide

  210. • Immutable!
    • equals() works like it should
    • Implements Comparable
    ByteString is good

    View Slide

  211. ByteString makes data easy
    ByteString byteString = ByteString.decodeHex("436166c3a920f09f8da9");

    assertThat(byteString.size()).isEqualTo(10);

    assertThat(byteString.getByte(0)).isEqualTo((byte) 0x43);

    assertThat(byteString.getByte(0)).isEqualTo((byte) 67);


    String cafeDonuts = byteString.utf8();

    assertThat(cafeDonuts).isEqualTo("Café ");


    ByteString cafe = ByteString.encodeUtf8("Café");

    assertThat(byteString.startsWith(cafe)).isTrue();

    View Slide

  212. ByteString and Parcelable

    View Slide

  213. class Talk implements Parcelable {

    public static final Parcelable.Creator CREATOR = new Parcelable.Creator() {

    @Override public Talk createFromParcel(Parcel in) {

    int id = in.readInt();

    long date = in.readLong();

    Room room = Room.values()[in.readInt()];

    String title = in.readString();

    String speaker = in.readString();

    return new Talk(id, date, room, title, speaker);

    }

    };

    …


    @Override public void writeToParcel(Parcel out, int flags) {

    out.writeInt(id);

    out.writeLong(date);

    out.writeInt(room.ordinal());

    out.writeString(title);

    out.writeString(speaker);

    }

    }

    View Slide

  214. @Test public void decodeGolden() {

    Talk talk = new Talk(72017, 1475260200000L, Room.RIGHT,

    "Decoding the Secrets of Binary Data", "Jesse Wilson");

    ByteString goldenData = ByteString.decodeHex("01000000511901004034…");

    assertThat(parcelDecode(goldenData, Talk.CREATOR)).isEqualTo(talk);

    }a


    private T parcelDecode(

    ByteString byteString, Parcelable.Creator creator) {

    Parcel parcel = Parcel.obtain();

    try {

    parcel.unmarshall(byteString.toByteArray(), 0, byteString.size());

    parcel.setDataPosition(0);

    return parcel.readTypedObject(creator);

    } finally {

    parcel.recycle();

    }a

    }a

    View Slide

  215. @Test public void decodeGolden() {

    Talk talk = new Talk(72017, 1475260200000L, Room.RIGHT,

    "Decoding the Secrets of Binary Data", "Jesse Wilson");

    ByteString goldenData = ByteString.decodeHex("01000000511901004034…");

    assertThat(parcelDecode(goldenData, Talk.CREATOR)).isEqualTo(talk);

    }a


    private T parcelDecode(

    ByteString byteString, Parcelable.Creator creator) {

    Parcel parcel = Parcel.obtain();

    try {

    parcel.unmarshall(byteString.toByteArray(), 0, byteString.size());

    parcel.setDataPosition(0);

    return parcel.readTypedObject(creator);

    } finally {

    parcel.recycle();

    }a

    }a

    View Slide

  216. Next Steps

    View Slide

  217. • Everything is bytes
    • Java Strings are UTF-16. Encoded text is usually UTF-8
    • Hex is handy
    • Don’t be afraid of shifting and masking
    • Integers are big or little-endian
    • Java’s I/O APIs are trouble
    6 tips

    View Slide

  218. • You can encode almost anything
    • Okio makes it easy
    No Black Boxes

    View Slide

  219. @jessewilson
    https://github.com/swankjesse/encoding
    Thanks!

    View Slide