Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Decoding the Secrets of Binary Data (Droidcon NYC 2016)

Jesse Wilson
September 30, 2016

Decoding the Secrets of Binary Data (Droidcon NYC 2016)

Video: https://www.youtube.com/watch?v=T_p22jMZSrk
Code: https://github.com/swankjesse/encoding

Opaque blobs of data have hexed Android programmers for too long. It’s time to byte the bullet and learn how data is transmitted and persisted.

In this talk we’ll:

💾 Learn a bit about base64, little-endian, and EOF.
💾 See how inefficient encodings nibble away resources.
💾 Hash out the differences between ASCII, UTF-8, and other charsets.
💾 Zip through examples of compression, crypto, and protocol buffers.
💾 Load up on APIs and discover what Square’s Okio has in store.

This talk offers a short introduction to an array of topics. You’ll learn enough to be encode & decode whatever data you select!

Jesse Wilson

September 30, 2016
Tweet

More Decks by Jesse Wilson

Other Decks in Programming

Transcript

  1. @jessewilson
    Decoding the Secrets
    of Binary Data
    https://github.com/swankjesse/encoding

    View full-size slide

  2. Encoding
    @jessewilson
    https://github.com/swankjesse/encoding

    View full-size slide

  3. Goals
    • Get comfortable with bytes
    • Write simpler, more efficient programs

    View full-size slide

  4. Plan
    • Bottom up
    • History
    • Math
    • Code

    View full-size slide

  5. Date: Aprilis 11, 1095
    From: Emperor Alexios Komnenos
    To: Robert II, Count of Flanders
    Bob,
    Looks like we’re gonna war with the
    Turks. Could you see if the Pope
    could help us crusade against ’em?
    Thanks!
    Alex
    Letters!
    • Human readable
    • Signed

    View full-size slide

  6. Date: Aprilis 11, 1095
    From: Emperor Alexios Komnenos
    To: Robert II, Count of Flanders
    Bob,
    Looks like we’re gonna war with the
    Turks. Could you see if the Pope
    could help us crusade against ’em?
    Thanks!
    Alex
    Letters!
    • Slow, dangerous to transmit
    • Awkward to store
    • Requires Literacy!

    View full-size slide

  7. Telegraphs
    NYC
    SF

    View full-size slide

  8. Telegraphs
    NYC
    SF

    View full-size slide

  9. Morse Code
    A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •

    View full-size slide

  10. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • •

    View full-size slide

  11. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • • ▬ ▬ ▬

    View full-size slide

  12. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • • ▬ ▬ ▬ ▬ •

    View full-size slide

  13. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • • ▬ ▬ ▬ ▬ • • • ▬

    View full-size slide

  14. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • • ▬ ▬ ▬ ▬ • • • ▬ ▬

    View full-size slide

  15. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • • ▬ ▬ ▬ ▬ • • • ▬ ▬

    View full-size slide

  16. Morse Code
    NYC
    SF

    View full-size slide

  17. Morse Code
    • Telegraphs!
    • 15 words per minute
    • Wireless in 1898

    View full-size slide

  18. Morse Code
    • Very limited characters
    • No lowercase
    • Limited punctuation*
    • Average message: 12 words
    * For example, morse code doesn’t have an asterisk symbol.

    View full-size slide

  19. Binary Refresher!

    View full-size slide

  20. 200
    205 = + 5

    View full-size slide

  21. 2×100
    205 = + 5×1

    View full-size slide

  22. 2×100
    205 = +
    + 0×10 5×1

    View full-size slide

  23. 2×102
    205 = +
    + 0×101 5×10

    View full-size slide

  24. 128 8
    +
    205 = 64 4
    +
    + + 1

    View full-size slide

  25. 1×128 1×8
    +
    205 = 1×64 1×4
    +
    + + 1×1

    View full-size slide

  26. 1×128 1×8
    +
    205 1×64 + 1×4
    +
    0×32 0×16
    + + 0×2
    + 1×1
    +
    =

    View full-size slide

  27. 1×27 1×23
    +
    205 1×26 + 1×22
    +
    0×25 0×24
    + + 0×21
    + 1×20
    +
    =

    View full-size slide

  28. 205 1×27 1×23
    1×26 1×22
    0×25 0×24 0×21 1×20
    =

    View full-size slide

  29. 205 1×27 1×23
    1×26 1×22
    0×25
    0×240×21
    1×20
    0b
    =

    View full-size slide

  30. Binary Refresher!
    • Decimal lets you represent any integer with a sequence of digits:
    0, 1, 2, 3, 4, 5, 6, 7, 8, 9
    • Binary lets you represent any integer with a sequence of bits: 0, 1

    View full-size slide

  31. Binary Refresher!
    • Typically prefixed with “0b”, like 0b11001101

    View full-size slide

  32. Binary Refresher!
    • N bits can represent 2N values
    • 8 bits: 0..255
    • 16 bits: 0..65,535
    • 32 bits: 0..4,294,967,295
    • 64 bits: 0..18,446,744,073,709,551,615

    View full-size slide

  33. Binary Refresher!
    NYC
    SF

    View full-size slide

  34. Layering
    • Given a wire that transmits a single bit, we can use binary to
    encode any integer!
    • This works because the sender and recipient agree on how to
    interpret the sequence
    • That interpretation is called an encoding

    View full-size slide

  35. Bytes
    • Though binary supports any number of bits, we like 8-bit integers
    • An 8-bit integer is called a byte
    • 256 values from 0 to 255

    View full-size slide

  36. ASCII
    • American Standard Code for Information Interchange
    • A table of characters
    • Interpret a sequence of bytes as a string of characters!

    View full-size slide

  37. ASCII
    • Work started in 1960
    • Only uses 7 bits: in 1967 bits were very expensive!
    • That means there’s 27 = 128 characters

    View full-size slide

  38. ASCII
    0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL

    View full-size slide

  39. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68

    View full-size slide

  40. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111

    View full-size slide

  41. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111 110

    View full-size slide

  42. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111 110 117

    View full-size slide

  43. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111 110 117 116

    View full-size slide

  44. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111 110 117 116

    View full-size slide

  45. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111 110 117 116

    View full-size slide

  46. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    111 110 117 116
    0 1 0 0 0 1 0 0

    View full-size slide

  47. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    110 117 116
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1

    View full-size slide

  48. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    117 116
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0

    View full-size slide

  49. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    116
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1

    View full-size slide

  50. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 0

    View full-size slide

  51. boss, what should
    I be working on?

    View full-size slide

  52. boss, what should
    I be working on?
    resume polishing

    View full-size slide

  53. boss, what should
    I be working on?

    View full-size slide

  54. boss, what should
    I be working on?
    résumé polishing

    View full-size slide

  55. Charset Hell
    • ASCII is English only
    • But non-English people also use computers!

    View full-size slide

  56. Charset Hell
    • Operating systems were installed for a specific character set and
    wouldn’t work with any others
    • Documents couldn’t mix Greek, French, and Russian characters
    • If you see ISO-8859-1, run away!

    View full-size slide

  57. Unicode
    • Support all languages in a single system
    • A code point is a universal ID for a character

    View full-size slide

  58. UTF-16
    • 16-bit Unicode Transformation Format
    • 2 bytes per code point
    • This is Java’s char type

    View full-size slide

  59. ASCII
    0 1 0 0 0 1 0 0
    0 1 0 0 0 1 0 0
    0 0 0 0 0 0 0 0
    D
    UTF-16

    View full-size slide

  60. ASCII
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    D
    UTF-16
    o

    View full-size slide

  61. ASCII
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 1 1 0 1 1 1 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    D
    UTF-16
    on

    View full-size slide

  62. ASCII
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0
    D
    UTF-16
    onu

    View full-size slide

  63. ASCII
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 0
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    D
    UTF-16
    onut

    View full-size slide

  64. ASCII
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 0
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    D
    UTF-16
    onut

    View full-size slide

  65. UTF-16 characters
    • Max code point is 65,535
    • Code point for is 127,849
    • 127,849 > 65,535

    View full-size slide

  66. Java’s char is broken!
    • There’s a system called “surrogate pairs” which is like multidex for
    code points
    • It splits a single code point across 2 chars
    • It’s an incredible pain

    View full-size slide


  67. char[] s = "Café ".toCharArray();
    C a f é
    0 1 2 3 4 5 6

    View full-size slide

  68. String s = “Café ";


    for (int i = 0, size = s.length(); i < size; i++) {

    char c = s.charAt(i);

    System.out.printf("The character at %d is '%c'%n", i, c);

    }

    View full-size slide

  69. String s = “Café ";


    for (int i = 0, size = s.length(); i < size; i++) {

    char c = s.charAt(i);

    System.out.printf("The character at %d is '%c'%n", i, c);

    }
    The character at 0 is 'C'
    The character at 1 is 'a'
    The character at 2 is 'f'
    The character at 3 is 'é'
    The character at 4 is ' '
    The character at 5 is ' '
    The character at 6 is ' '

    View full-size slide

  70. String s = “Café ";


    for (int i = 0, size = s.length(); i < size; ) {

    int c = s.codePointAt(i);

    System.out.printf("The code point at %d is '%c'%n", i, c);
    i += Character.charCount(c);

    }

    View full-size slide

  71. String s = “Café ";


    for (int i = 0, size = s.length(); i < size; ) {

    int c = s.codePointAt(i);

    System.out.printf("The code point at %d is '%c'%n", i, c);
    i += Character.charCount(c);

    }
    The code point at 0 is 'C'
    The code point at 1 is 'a'
    The code point at 2 is 'f'
    The code point at 3 is 'é'
    The code point at 4 is ' '
    The code point at 5 is ' '

    View full-size slide

  72. UTF-8
    • 8-bit Unicode Transformation Format
    • Variable number of bytes per code point
    • This is how modern apps transmit & store text

    View full-size slide

  73. Image copyright Chris55, https://commons.wikimedia.org/wiki/File:Utf8webgrowth.svg

    View full-size slide

  74. UTF-8
    • Many common characters are 1-byte
    • Some are 2 and 3 bytes. ‘ ’ is 4 bytes.
    • Self-delimiting
    • Self-aligning

    View full-size slide

  75. 0
    1 1 0 1 0
    1 1 1 0
    1 1 1 1 0
    1 0
    1 0
    1 0
    1 0 1 0
    ≤7
    ≤11
    ≤16
    ≤21
    “How many bits do you need?”

    View full-size slide

  76. C
    “How many bits do you need?”
    a
    f
    é
    sp

    View full-size slide

  77. C
    “How many bits do you need?”
    a
    f
    é
    sp
    67

    View full-size slide

  78. 1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    a
    f
    é
    sp

    View full-size slide

  79. 1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    a
    f
    é
    sp
    97

    View full-size slide

  80. 1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    f
    é
    sp

    View full-size slide

  81. 1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    f
    é
    sp
    102

    View full-size slide

  82. 1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp

    View full-size slide

  83. 233
    1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp

    View full-size slide

  84. 1 1 1 0 1 0 0 1
    1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp

    View full-size slide

  85. 1 1 1 0 1 0 0 1
    1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp 32

    View full-size slide

  86. 1 0 0 0 0 0
    1 1 1 0 1 0 0 1
    1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp

    View full-size slide

  87. 1 0 0 0 0 0
    1 1 1 0 1 0 0 1
    1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp
    127,849

    View full-size slide

  88. 1 0 1 0 0 1
    0 0 1 1 0 1
    1 1 1 1 1 1
    1 0 0 0 0 0
    1 1 1 0 1 0 0 1
    1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp

    View full-size slide

  89. 0
    1 1 0
    1 1 1 1 0 1 0
    0
    1 0
    0
    0
    1 0
    1 0 1 0 1 0 0 1
    0 0 1 1 0 1
    0 1 1 1 1 1
    1 0 0 0 0 0
    1 0 1 0 0 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp
    1 1
    1 0 0 0 0 1 1

    View full-size slide

  90. 0
    1 1 0 0 0 0
    1 1 1 1 0 0 0 0 1 0
    0 0
    1 0
    0
    0
    1 0
    1 0 1 0 1 0 0 1
    0 0 1 1 0 1
    0 1 1 1 1 1
    1 0 0 0 0 0
    1 0 1 0 0 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp
    1 1
    1 0 0 0 0 1 1

    View full-size slide

  91. 0 1 0 0 0 0 1 1 1 1 0 0 0 0 1 1
    1 0 1 0 1 0 0 1 1 1 1 1 0 0 0 0 1 0 0 1 1 1 1 1
    1 0 0 0 1 1 0 1 1 0 1 0 1 0 0 1
    0 1 1 0 0 0 0 1 0 1 1 0 0 1 1 0
    0 0 1 0 0 0 0 0

    View full-size slide

  92. UTF-8
    • Basically the best thing ever
    • Superset of ASCII
    • Great for JSON and HTML because delimiter characters <, > and "
    are 1-byte

    View full-size slide

  93. R
    255
    0
    G
    255
    0
    B
    255
    0
    255
    255 255

    View full-size slide

  94. R
    255
    0
    G
    255
    0
    B
    255
    0
    255
    0 255

    View full-size slide

  95. R
    255
    0
    G
    255
    0
    B
    255
    0
    20
    206 210

    View full-size slide

  96. Hexadecimal Refresher!

    View full-size slide

  97. Hexadecimal Refresher!
    • Decimal digits: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
    • Binary bits: 0, 1
    • Hexadecimal digits: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, a, b, c, d, e, f

    View full-size slide

  98. 205 192
    = + 13

    View full-size slide

  99. 205 12×16
    = + 13×1

    View full-size slide

  100. 205 12×161
    = + 13×160

    View full-size slide

  101. 205 c×161
    = + d×160

    View full-size slide

  102. Hexadecimal Refresher!
    • Prefixed with “0x”, like 0xcd
    • Hex bytes are always two digits: 00, 01, 02 … ff
    • Sequences of bytes are okay: 0bb634

    View full-size slide

  103. Hexadecimal Refresher!
    • Colors: #ffffff
    • URL escaping: http://example.com/?q=hello%20world
    • Unicode code points U+2020
    • IPv6 addresses: 2001:0db8:85a3:0000:0000:8a2e:0370:7334

    View full-size slide

  104. R
    255
    0
    G
    255
    0
    B
    255
    0
    ced214

    View full-size slide

  105. Pictures
    • Just a 2D array of colors
    • Given 3 bytes per pixel:
    • 64 × 64 icon is 12,288 bytes
    • 1080 × 1920 picture is 5.9 MiB
    • Compression is important!

    View full-size slide

  106. Pictures
    • Android can use fewer bits per pixel
    • ARGB_8888: alpha, red, green, and blue get 8 bits each
    • RGB_565: red gets 5 bits, geen gets 6, blue gets 5 bits

    View full-size slide

  107. RGB_565
    ARGB_8888

    View full-size slide

  108. RGB_454
    ARGB_8888

    View full-size slide

  109. RGB_232
    ARGB_8888

    View full-size slide

  110. /** https://en.wikipedia.org/wiki/BMP_file_format */

    public void encode(BufferedSink sink) throws IOException {

    int height = pixels.length;

    int width = pixels[0].length;


    int bytesPerPixel = 3;

    int rowByteCountWithoutPadding = (bytesPerPixel * width);

    int rowByteCount = ((rowByteCountWithoutPadding + 3) / 4) * 4;

    int pixelDataSize = rowByteCount * height;

    int bmpHeaderSize = 14;

    int dibHeaderSize = 40;


    // BMP Header

    sink.writeUtf8("BM"); // ID.

    sink.writeIntLe(bmpHeaderSize + dibHeaderSize + pixelDataSize); // File size.

    sink.writeShortLe(0); // Unused.

    sink.writeShortLe(0); // Unused.

    sink.writeIntLe(bmpHeaderSize + dibHeaderSize); // Offset of pixel data.


    // DIB Header

    sink.writeIntLe(dibHeaderSize);


    View full-size slide

  111. int dibHeaderSize = 40;


    // BMP Header

    sink.writeUtf8("BM"); // ID.

    sink.writeIntLe(bmpHeaderSize + dibHeaderSize + pixelDataSize); // File size.

    sink.writeShortLe(0); // Unused.

    sink.writeShortLe(0); // Unused.

    sink.writeIntLe(bmpHeaderSize + dibHeaderSize); // Offset of pixel data.


    // DIB Header

    sink.writeIntLe(dibHeaderSize);

    sink.writeIntLe(width);

    sink.writeIntLe(height);

    sink.writeShortLe(1); // Color plane count.

    sink.writeShortLe(bytesPerPixel * Byte.SIZE);

    sink.writeIntLe(0); // No compression.

    sink.writeIntLe(16); // Size of bitmap data including padding.

    sink.writeIntLe(2835); // Horizontal print resolution in pixels/meter. (72 dpi).

    sink.writeIntLe(2835); // Vertical print resolution in pixels/meter. (72 dpi).

    sink.writeIntLe(0); // Palette color count.

    sink.writeIntLe(0); // 0 important colors.


    // Pixel data.


    View full-size slide

  112. sink.writeIntLe(0); // Palette color count.

    sink.writeIntLe(0); // 0 important colors.


    // Pixel data.

    for (int y = height - 1; y >= 0; y--) {

    int[] row = pixels[y];

    for (int x = 0; x < width; x++) {

    int pixel = row[x];

    sink.writeByte((pixel & 0x0000ff)); // Blue.

    sink.writeByte((pixel & 0x00ff00) >>> 8); // Green.

    sink.writeByte((pixel & 0xff0000) >>> 16); // Red.

    }


    // Padding for 4-byte alignment.

    for (int p = rowByteCountWithoutPadding; p < rowByteCount; p++) {

    sink.writeByte(0);

    }

    }

    }


    View full-size slide

  113. Pictures on Bytes
    • This bitmap writer is 50 lines of code
    • Decoders are more difficult!
    • Good specs make it easy

    View full-size slide

  114. sink.writeIntLe(0); // Palette color count.

    sink.writeIntLe(0); // 0 important colors.


    // Pixel data.

    for (int y = height - 1; y >= 0; y--) {

    int[] row = pixels[y];

    for (int x = 0; x < width; x++) {

    int pixel = row[x];

    sink.writeByte((pixel & 0x0000ff)); // Blue.

    sink.writeByte((pixel & 0x00ff00) >>> 8); // Green.

    sink.writeByte((pixel & 0xff0000) >>> 16); // Red.

    }


    // Padding for 4-byte alignment.

    for (int p = rowByteCountWithoutPadding; p < rowByteCount; p++) {

    sink.writeByte(0);

    }

    }

    }


    View full-size slide

  115. sink.writeIntLe(0); // Palette color count.

    sink.writeIntLe(0); // 0 important colors.


    // Pixel data.

    for (int y = height - 1; y >= 0; y--) {

    int[] row = pixels[y];

    for (int x = 0; x < width; x++) {

    int pixel = row[x];

    sink.writeByte((pixel & 0x0000ff)); // Blue.

    sink.writeByte((pixel & 0x00ff00) >>> 8); // Green.

    sink.writeByte((pixel & 0xff0000) >>> 16); // Red.

    }


    // Padding for 4-byte alignment.

    for (int p = rowByteCountWithoutPadding; p < rowByteCount; p++) {

    sink.writeByte(0);

    }

    }

    }


    View full-size slide

  116. (pixel & 0xff0000) >>> 16
    • Shifting and masking lets you access the bits within an integer
    • & and | operators treat each int like a 32-element boolean array!
    • <<, >> and >>> operators slide bits left and right

    View full-size slide

  117. 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0
    0000ab00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
    00ffab40 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
    00ffab40
    0000ff00 1 1 1 1 1 1 1 1
    0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0x00ffab40 & 0x0000ff00

    View full-size slide

  118. 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0
    0000ab00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
    00ffab40
    1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
    00ffab40
    0000ff00 1 1 1 1 1 1 1 1
    0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0x00ffab40 & 0x0000ff00

    View full-size slide

  119. 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0
    0000ab00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
    00ffab40
    1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
    00ffab40
    0000ff00 1 1 1 1 1 1 1 1
    0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0x00ffab40 & 0x0000ff00
    &
    =

    View full-size slide

  120. 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0
    00000ab00
    0x0000ab00 >> 8

    View full-size slide

  121. 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0
    00000ab00
    0x0000ab00 >> 8
    {
    8

    View full-size slide

  122. 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0
    000000ab
    0x0000ab00 >> 8

    View full-size slide

  123. 1 0 1 0 1 0 1 1
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0
    000000ab
    0x0000ab00 >> 8
    = 0x000000ab

    View full-size slide

  124. sink.writeIntLe(0); // Palette color count.

    sink.writeIntLe(0); // 0 important colors.


    // Pixel data.

    for (int y = height - 1; y >= 0; y--) {

    int[] row = pixels[y];

    for (int x = 0; x < width; x++) {

    int pixel = row[x];

    sink.writeByte((pixel & 0x0000ff)); // Blue.

    sink.writeByte((pixel & 0x00ff00) >>> 8); // Green.

    sink.writeByte((pixel & 0xff0000) >>> 16); // Red.

    }


    // Padding for 4-byte alignment.

    for (int p = rowByteCountWithoutPadding; p < rowByteCount; p++) {

    sink.writeByte(0);

    }

    }

    }


    View full-size slide

  125. This space
    intentionally
    left blank

    View full-size slide

  126. int dibHeaderSize = 40;


    // BMP Header

    sink.writeUtf8("BM"); // ID.

    sink.writeIntLe(bmpHeaderSize + dibHeaderSize + pixelDataSize); // File size.

    sink.writeShortLe(0); // Unused.

    sink.writeShortLe(0); // Unused.

    sink.writeIntLe(bmpHeaderSize + dibHeaderSize); // Offset of pixel data.


    // DIB Header

    sink.writeIntLe(dibHeaderSize);

    sink.writeIntLe(width);

    sink.writeIntLe(height);

    sink.writeShortLe(1); // Color plane count.

    sink.writeShortLe(bytesPerPixel * Byte.SIZE);

    sink.writeIntLe(0); // No compression.

    sink.writeIntLe(16); // Size of bitmap data including padding.

    sink.writeIntLe(2835); // Horizontal print resolution in pixels/meter. (72 dpi).

    sink.writeIntLe(2835); // Vertical print resolution in pixels/meter. (72 dpi).

    sink.writeIntLe(0); // Palette color count.

    sink.writeIntLe(0); // 0 important colors.


    // Pixel data.


    View full-size slide

  127. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    Big Endian

    View full-size slide

  128. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    0 0 0 0 0 0 0 0
    Big Endian

    View full-size slide

  129. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    1 1 1 1 1 1 1 1
    0 0 0 0 0 0 0 0
    Big Endian

    View full-size slide

  130. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    1 0 1 0 1 0 1 1
    1 1 1 1 1 1 1 1
    0 0 0 0 0 0 0 0
    Big Endian

    View full-size slide

  131. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    0 1 0 0 0 0 0 0
    1 0 1 0 1 0 1 1
    1 1 1 1 1 1 1 1
    0 0 0 0 0 0 0 0
    Big Endian

    View full-size slide

  132. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    0 1 0 0 0 0 0 0
    1 0 1 0 1 0 1 1
    1 1 1 1 1 1 1 1
    0 0 0 0 0 0 0 0
    Big Endian

    View full-size slide

  133. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    Little Endian

    View full-size slide

  134. 0 1 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    Little Endian

    View full-size slide

  135. 1 0 1 0 1 0 1 1
    0 1 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    Little Endian

    View full-size slide

  136. 1 0 1 0 1 0 1 1 1 1 1 1 1 1 1 1
    0 1 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    Little Endian

    View full-size slide

  137. 1 0 1 0 1 0 1 1 1 1 1 1 1 1 1 1
    0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    Little Endian

    View full-size slide

  138. 1 0 1 0 1 0 1 1 1 1 1 1 1 1 1 1
    0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
    Little Endian

    View full-size slide

  139. Conference App!
    • Each talk has an ID, date, room, title,
    and speaker

    View full-size slide

  140. JSON is Good Stuff
    {

    "id": 72017,

    "date": "2016-09-30T18:30:00Z",

    "room": "RIGHT",

    "title": "Decoding the Secrets of Binary Data",

    "speaker": "Jesse Wilson"

    }

    View full-size slide

  141. JSON is Good Stuff
    {

    "id": 72017,

    "date": "2016-09-30T18:30:00Z",

    "room": "RIGHT",

    "title": "Decoding the Secrets of Binary Data",

    "speaker": "Jesse Wilson"

    }
    • A nice format that builds on UTF-8
    • Easy to read & write

    View full-size slide

  142. JSON is Self-Delimiting
    {

    "id": 72017,

    "date": "2016-09-30T18:30:00Z",

    "room": "RIGHT",

    "title": "Decoding the Secrets of Binary Data",

    "speaker": "Jesse Wilson"

    }
    • A JSON document has both structure and data
    • Uses escape sequences like \" to be completely unambiguous

    View full-size slide

  143. "2016-09-30T18:30:00Z"
    • System.currentTimeMillis() returns milliseconds since
    January 1, 1970 at 00:00:00 UTC
    • A date that’s 8 bytes in memory is 22 bytes in JSON!

    View full-size slide

  144. • Space: a simple message like this one is ~128 bytes
    • Time: bigger sequences take longer to decode
    JSON Space & Time

    View full-size slide

  145. • Google’s “small, fast, simple” structured data format
    • Upon closer inspection, it’s not that different from JSON!
    • But it has a schema
    Protocol Buffers

    View full-size slide

  146. {
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    "id": 72017,
    "date": "2016-09-30T18:30:00Z",
    "room": "RIGHT",
    "title": "Decoding the Secrets of Binary Data",
    "speaker": "Jesse Wilson"
    }
    "id"
    "date"
    "room"
    "title"
    "speaker"
    72017
    "2016-09-30T18:30:00Z",
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",

    View full-size slide

  147. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }

    View full-size slide

  148. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    000 enum, int32, int64...
    001 fixed64
    010 string, message
    101 fixed32
    Length Mode

    View full-size slide

  149. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    000 enum, int32, int64...
    001 fixed64
    010 string, message
    101 fixed32
    Length Mode
    1 0 1

    View full-size slide

  150. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    000 enum, int32, int64...
    001 fixed64
    010 string, message
    101 fixed32
    Length Mode
    1 0 1
    0 0 1

    View full-size slide

  151. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    000 enum, int32, int64...
    001 fixed64
    010 string, message
    101 fixed32
    Length Mode
    0 0 0
    1 0 1
    0 0 1

    View full-size slide

  152. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    000 enum, int32, int64...
    001 fixed64
    010 string, message
    101 fixed32
    Length Mode
    0 0 0
    1 0 1
    0 1 0
    0 0 1

    View full-size slide

  153. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    000 enum, int32, int64...
    001 fixed64
    010 string, message
    101 fixed32
    Length Mode
    0 0 0
    1 0 1
    0 1 0
    0 1 0
    0 0 1

    View full-size slide

  154. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    0 0 0
    1 0 1
    0 1 0
    0 1 0
    0 0 1

    View full-size slide

  155. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    0 0 0
    0 0 0 0 1 1 0 1
    0 1 0
    0 1 0
    0 0 1

    View full-size slide

  156. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    0 0 0
    0 0 0 0 1 1 0 1
    0 1 0
    0 1 0
    0 0 0 1 0 0 0 1

    View full-size slide

  157. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    0 0 0 1 1 0 0 0
    0 0 0 0 1 1 0 1
    0 1 0
    0 1 0
    0 0 0 1 0 0 0 1

    View full-size slide

  158. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    0 0 0 1 1 0 0 0
    0 0 0 0 1 1 0 1
    0 1 0
    0 0 1 0 0 0 1 0
    0 0 0 1 0 0 0 1

    View full-size slide

  159. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    0 0 0 1 1 0 0 0
    0 0 0 0 1 1 0 1
    0 0 1 0 1 0 1 0
    0 0 1 0 0 0 1 0
    0 0 0 1 0 0 0 1

    View full-size slide

  160. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    0 0 0 1 1 0 0 0
    0 0 0 0 1 1 0 1
    0 0 1 0 1 0 1 0
    0 0 1 0 0 0 1 0
    0 0 0 1 0 0 0 1

    View full-size slide

  161. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }

    }
    0d
    11
    18
    22
    2a

    View full-size slide

  162. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    {
    "id": 72017,
    "date": "2016-09-30T18:30:00Z",
    "room": "RIGHT",
    "title": "Decoding the Secrets of Binary Data",
    "speaker": "Jesse Wilson"
    }
    "id"
    "date"
    "room"
    "title"
    "speaker"
    72017
    "2016-09-30T18:30:00Z",
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",

    View full-size slide

  163. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    72017
    "2016-09-30T18:30:00Z",
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",

    View full-size slide

  164. message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    72017
    "2016-09-30T18:30:00Z",
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",

    View full-size slide

  165. 51
    19
    01
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    "2016-09-30T18:30:00Z",
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",
    00

    View full-size slide

  166. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    "2016-09-30T18:30:00Z",
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",
    00

    View full-size slide

  167. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    "2016-09-30T18:30:00Z",
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",
    00

    View full-size slide

  168. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",
    00
    000001577D37EE40

    View full-size slide

  169. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",
    00
    00
    00
    01
    57
    7D
    37
    EE
    40

    View full-size slide

  170. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    "RIGHT",
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",
    00
    00
    00
    01
    57
    7D
    37
    EE
    40

    View full-size slide

  171. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    "Decoding the Secrets of Binary Data",
    "Jesse Wilson",
    00
    00
    00
    01
    57
    7D
    37
    EE
    40
    02

    View full-size slide

  172. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    "Decoding the Secrets of Binary Data"
    "Jesse Wilson",
    00
    00
    01
    57
    7D
    37
    EE
    40
    02
    00

    View full-size slide

  173. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    4465636f64696e67207468652053656372657473206f662042696e6172792044617461
    "Jesse Wilson",
    00
    00
    00
    01
    57
    7D
    37
    EE
    40
    02

    View full-size slide

  174. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    4465636f64696e67207468652053656372657473206f662042696e6172792044617461
    "Jesse Wilson",
    00
    00
    00
    01
    57
    7D
    37
    EE
    40
    02
    23

    View full-size slide

  175. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    4465636f64696e67207468652053656372657473206f662042696e6172792044617461
    "Jesse Wilson"
    00
    00
    00
    01
    57
    7D
    37
    EE
    40
    02
    23

    View full-size slide

  176. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    4465636f64696e67207468652053656372657473206f662042696e6172792044617461
    4a657373652057696c736f6e
    00
    00
    00
    01
    57
    7D
    37
    EE
    40
    02
    23

    View full-size slide

  177. 511901
    message Talk {

    optional fixed32 id = 1;

    optional fixed64 date = 2;

    optional Room room = 3;

    optional string title = 4;

    optional string speaker = 5;


    enum Room {

    UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

    }a

    }a
    0d
    11
    18
    22
    2a
    4465636f64696e67207468652053656372657473206f662042696e6172792044617461
    4a657373652057696c736f6e
    00
    00
    00
    01
    57
    7D
    37
    EE
    40
    02
    23
    0c

    View full-size slide

  178. 511901
    0d
    11
    18
    22
    2a 4a657373652057696c736f6e
    00
    00
    00
    01
    57
    7D
    37
    EE
    40
    02
    23
    0c
    4465636f64696e67207468652053656372657473206f662042696e6172792044617461

    View full-size slide

  179. 511901
    0d 11 18 22
    2a
    4465636f64696e6720746865205365637265747
    4a657373652057696c736f6e
    00 00
    00
    01
    57
    7D
    37
    EE
    40 02 23
    0c
    3206f662042696e6172792044617461

    View full-size slide

  180. 511901
    0d 11 18 22
    2a
    4465636f64696e6720746865205365637265747
    4a657373652057696c736f6e
    00 00
    00
    01
    57
    7D
    37
    EE
    40 02 23
    0c
    3206f662042696e6172792044617461

    View full-size slide

  181. 511901
    0d 11 18 22
    2a
    4465636f64696e6720746865205365637265747
    4a657373652057696c736f6e
    00 00
    00
    01
    57
    7D
    37
    EE
    40 02 23
    0c
    3206f662042696e6172792044617461
    • Protocol Buffers are small and fast
    • ~1 byte for each field name
    • Compact encoding for numbers and enums
    • Strings stay the same length!
    • This message is 67 bytes in protocol buffers, vs. 128 for JSON

    View full-size slide

  182. • Mutable!
    • equals() doesn’t work like it should: can’t be a map key!
    • Doesn’t implement Comparable
    byte[] is bad

    View full-size slide

  183. • Range is -128..127 but you usually want 0..255
    • But unsigned when you call InputStream.read()
    byte is signed

    View full-size slide

  184. • Range is -128..127 but you usually want 0..255
    • But unsigned when you call InputStream.read()
    byte is signed

    View full-size slide

  185. Okio
    • Square’s library for efficient I/O
    • Used by OkHttp, Retrofit, Wire, Moshi

    View full-size slide

  186. • Immutable!
    • equals() works like it should
    • Implements Comparable
    ByteString is good

    View full-size slide

  187. ByteString makes data easy
    ByteString byteString = ByteString.decodeHex("436166c3a920f09f8da9");

    assertThat(byteString.size()).isEqualTo(10);

    assertThat(byteString.getByte(0)).isEqualTo((byte) 0x43);

    assertThat(byteString.getByte(0)).isEqualTo((byte) 67);


    String cafeDonuts = byteString.utf8();

    assertThat(cafeDonuts).isEqualTo("Café ");


    ByteString cafe = ByteString.encodeUtf8("Café");

    assertThat(byteString.startsWith(cafe)).isTrue();

    View full-size slide

  188. ByteString and Parcelable

    View full-size slide

  189. class Talk implements Parcelable {

    public static final Parcelable.Creator CREATOR = new Parcelable.Creator() {

    @Override public Talk createFromParcel(Parcel in) {

    int id = in.readInt();

    long date = in.readLong();

    Room room = Room.values()[in.readInt()];

    String title = in.readString();

    String speaker = in.readString();

    return new Talk(id, date, room, title, speaker);

    }

    };

    …


    @Override public void writeToParcel(Parcel out, int flags) {

    out.writeInt(id);

    out.writeLong(date);

    out.writeInt(room.ordinal());

    out.writeString(title);

    out.writeString(speaker);

    }

    }

    View full-size slide

  190. @Test public void decodeGolden() {

    Talk talk = new Talk(72017, 1475260200000L, Room.RIGHT,

    "Decoding the Secrets of Binary Data", "Jesse Wilson");

    ByteString goldenData = ByteString.decodeHex("01000000511901004034…");

    assertThat(parcelDecode(goldenData, Talk.CREATOR)).isEqualTo(talk);

    }a


    private T parcelDecode(

    ByteString byteString, Parcelable.Creator creator) {

    Parcel parcel = Parcel.obtain();

    try {

    parcel.unmarshall(byteString.toByteArray(), 0, byteString.size());

    parcel.setDataPosition(0);

    return parcel.readTypedObject(creator);

    } finally {

    parcel.recycle();

    }a

    }a

    View full-size slide

  191. @Test public void decodeGolden() {

    Talk talk = new Talk(72017, 1475260200000L, Room.RIGHT,

    "Decoding the Secrets of Binary Data", "Jesse Wilson");

    ByteString goldenData = ByteString.decodeHex("01000000511901004034…");

    assertThat(parcelDecode(goldenData, Talk.CREATOR)).isEqualTo(talk);

    }a


    private T parcelDecode(

    ByteString byteString, Parcelable.Creator creator) {

    Parcel parcel = Parcel.obtain();

    try {

    parcel.unmarshall(byteString.toByteArray(), 0, byteString.size());

    parcel.setDataPosition(0);

    return parcel.readTypedObject(creator);

    } finally {

    parcel.recycle();

    }a

    }a

    View full-size slide

  192. • Everything is bytes
    • Java Strings are UTF-16. Encoded text is usually UTF-8
    • Hex is handy
    • Don’t be afraid of shifting and masking
    • Integers are big or little-endian
    • Java’s I/O APIs are trouble
    6 tips

    View full-size slide

  193. • You can encode almost anything
    • Okio makes it easy
    No Black Boxes

    View full-size slide

  194. @jessewilson
    https://github.com/swankjesse/encoding
    Thanks!

    View full-size slide