Decoding the Secrets of Binary Data (Droidcon NYC 2016)

Video: https://www.youtube.com/watch?v=T_p22jMZSrk
Code: https://github.com/swankjesse/encoding

Opaque blobs of data have hexed Android programmers for too long. It’s time to byte the bullet and learn how data is transmitted and persisted.

In this talk we’ll:

💾 Learn a bit about base64, little-endian, and EOF.
💾 See how inefficient encodings nibble away resources.
💾 Hash out the differences between ASCII, UTF-8, and other charsets.
💾 Zip through examples of compression, crypto, and protocol buffers.
💾 Load up on APIs and discover what Square’s Okio has in store.

This talk offers a short introduction to an array of topics. You’ll learn enough to be encode & decode whatever data you select!

Transcript

1. @jessewilson
   Decoding the Secrets
   of Binary Data
   https://github.com/swankjesse/encoding
   

   View Slide

2. Encoding
   @jessewilson
   https://github.com/swankjesse/encoding
   

   View Slide

3. 0 1
   

   View Slide

4. View Slide

5. View Slide

6. Goals
   • Get comfortable with bytes
   • Write simpler, more efficient programs
   

   View Slide

7. Plan
   • Bottom up
   • History
   • Math
   • Code
   

   View Slide

8. 1095 AD
   

   View Slide

9. Date: Aprilis 11, 1095
   From: Emperor Alexios Komnenos
   To: Robert II, Count of Flanders
   Bob,
   Looks like we’re gonna war with the
   Turks. Could you see if the Pope
   could help us crusade against ’em?
   Thanks!
   Alex
   Letters!
   • Human readable
   • Signed
   

   View Slide

10. Date: Aprilis 11, 1095
    From: Emperor Alexios Komnenos
    To: Robert II, Count of Flanders
    Bob,
    Looks like we’re gonna war with the
    Turks. Could you see if the Pope
    could help us crusade against ’em?
    Thanks!
    Alex
    Letters!
    • Slow, dangerous to transmit
    • Awkward to store
    • Requires Literacy!
    

    View Slide

11. View Slide

12. Telegraphs
    NYC
    SF
    

    View Slide

13. Telegraphs
    NYC
    SF
    

    View Slide

14. Morse Code
    A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    

    View Slide

15. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • •
    

    View Slide

16. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • • ▬ ▬ ▬
    

    View Slide

17. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • • ▬ ▬ ▬ ▬ •
    

    View Slide

18. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • • ▬ ▬ ▬ ▬ • • • ▬
    

    View Slide

19. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • • ▬ ▬ ▬ ▬ • • • ▬ ▬
    

    View Slide

20. A • ▬ G ▬ ▬ • M ▬ ▬ S • • • Y ▬ • ▬ ▬ 4 • • • • ▬
    B ▬ • • • H • • • • N ▬ • T ▬ Z ▬ ▬ • • 5 • • • • •
    C ▬ • ▬ • I • • O ▬ ▬ ▬ U • • ▬ 0 ▬ ▬ ▬ ▬ ▬ 6 ▬ • • • •
    D ▬ • • J • ▬ ▬ ▬ P • ▬ ▬ • V • • • ▬ 1 • ▬ ▬ ▬ ▬ 7 ▬ ▬ • • •
    E • K ▬ • ▬ Q ▬ ▬ • ▬ W • ▬ ▬ 2 • • ▬ ▬ ▬ 8 ▬ ▬ ▬ • •
    F • • ▬ • L • ▬ • • R • ▬ • X ▬ • • ▬ 3 • • • ▬ ▬ 9 ▬ ▬ ▬ ▬ •
    ▬ • • ▬ ▬ ▬ ▬ • • • ▬ ▬
    

    View Slide

21. Morse Code
    NYC
    SF
    

    View Slide

22. Morse Code
    • Telegraphs!
    • 15 words per minute
    • Wireless in 1898
    

    View Slide

23. Morse Code
    • Very limited characters
    • No lowercase
    • Limited punctuation*
    • Average message: 12 words
    * For example, morse code doesn’t have an asterisk symbol.
    

    View Slide

24. Binary Refresher!
    

    View Slide

25. View Slide

26. 200
    205 = + 5
    

    View Slide

27. 2×100
    205 = + 5×1
    

    View Slide

28. 2×100
    205 = +
    + 0×10 5×1
    

    View Slide

29. 2×102
    205 = +
    + 0×101 5×10
    

    View Slide

30. View Slide

31. View Slide

32. 128 8
    +
    205 = 64 4
    +
    + + 1
    

    View Slide

33. 1×128 1×8
    +
    205 = 1×64 1×4
    +
    + + 1×1
    

    View Slide

34. 1×128 1×8
    +
    205 1×64 + 1×4
    +
    0×32 0×16
    + + 0×2
    + 1×1
    +
    =
    

    View Slide

35. 1×27 1×23
    +
    205 1×26 + 1×22
    +
    0×25 0×24
    + + 0×21
    + 1×20
    +
    =
    

    View Slide

36. 205 1×27 1×23
    1×26 1×22
    0×25 0×24 0×21 1×20
    =
    

    View Slide

37. 205 1×27 1×23
    1×26 1×22
    0×25
    0×240×21
    1×20
    0b
    =
    

    View Slide

38. Binary Refresher!
    • Decimal lets you represent any integer with a sequence of digits:
    0, 1, 2, 3, 4, 5, 6, 7, 8, 9
    • Binary lets you represent any integer with a sequence of bits: 0, 1
    

    View Slide

39. Binary Refresher!
    • Typically prefixed with “0b”, like 0b11001101
    

    View Slide

40. Binary Refresher!
    • N bits can represent 2N values
    • 8 bits: 0..255
    • 16 bits: 0..65,535
    • 32 bits: 0..4,294,967,295
    • 64 bits: 0..18,446,744,073,709,551,615
    

    View Slide

41. Binary Refresher!
    NYC
    SF
    

    View Slide

42. Layering
    • Given a wire that transmits a single bit, we can use binary to
    encode any integer!
    • This works because the sender and recipient agree on how to
    interpret the sequence
    • That interpretation is called an encoding
    

    View Slide

43. Bytes
    • Though binary supports any number of bits, we like 8-bit integers
    • An 8-bit integer is called a byte
    • 256 values from 0 to 255
    

    View Slide

44. View Slide

45. ASCII
    • American Standard Code for Information Interchange
    • A table of characters
    • Interpret a sequence of bytes as a string of characters!
    

    View Slide

46. ASCII
    • Work started in 1960
    • Only uses 7 bits: in 1967 bits were very expensive!
    • That means there’s 27 = 128 characters
    

    View Slide

47. ASCII
    0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    

    View Slide

48. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68
    

    View Slide

49. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111
    

    View Slide

50. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111 110
    

    View Slide

51. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111 110 117
    

    View Slide

52. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111 110 117 116
    

    View Slide

53. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111 110 117 116
    

    View Slide

54. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    68 111 110 117 116
    

    View Slide

55. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    111 110 117 116
    0 1 0 0 0 1 0 0
    

    View Slide

56. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    110 117 116
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    

    View Slide

57. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    117 116
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0
    

    View Slide

58. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    116
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1
    

    View Slide

59. 0 NULL 16 DLE 32 SP 48 0 64 @ 80 P 96 ` 112 p
    1 SOH 17 DC1 33 ! 49 1 65 A 81 Q 97 a 113 q
    2 STX 18 DC2 34 " 50 2 66 B 82 R 98 b 114 r
    3 ETX 19 DC3 35 # 51 3 67 C 83 S 99 c 115 s
    4 EOT 20 DC4 36 $ 52 4 68 D 84 T 100 d 116 t
    5 ENQ 21 NAK 37 % 53 5 69 E 85 U 101 e 117 u
    6 ACK 22 SYN 38 & 54 6 70 F 86 V 102 f 118 v
    7 BEL 23 ETB 39 ' 55 7 71 G 87 W 103 g 119 w
    8 BS 24 CAN 40 ( 56 8 72 H 88 X 104 h 120 x
    9 HT 25 EM 41 ) 57 9 73 I 89 Y 105 i 121 y
    10 LF 26 SUB 42 * 58 : 74 J 90 Z 106 j 122 z
    11 VT 27 ESC 43 + 59 ; 75 K 91 [ 107 k 123 {
    12 FF 28 FS 44 , 60 < 76 L 92 \ 108 l 124 |
    13 CR 29 GS 45 - 61 = 77 M 93 ] 109 m 125 }
    14 SO 30 RS 46 . 62 > 78 N 94 ^ 110 n 126 ~
    15 SI 31 US 47 / 63 ? 79 O 95 _ 111 o 127 DEL
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 0
    

    View Slide

60. 1980’s
    

    View Slide

61. boss, what should
    I be working on?
    

    View Slide

62. boss, what should
    I be working on?
    resume polishing
    

    View Slide

63. boss, what should
    I be working on?
    

    View Slide

64. boss, what should
    I be working on?
    résumé polishing
    

    View Slide

65. Charset Hell
    • ASCII is English only
    • But non-English people also use computers!
    

    View Slide

66. Charset Hell
    • Operating systems were installed for a specific character set and
    wouldn’t work with any others
    • Documents couldn’t mix Greek, French, and Russian characters
    • If you see ISO-8859-1, run away!
    

    View Slide

67. View Slide

68. Unicode
    • Support all languages in a single system
    • A code point is a universal ID for a character
    

    View Slide

69. UTF-16
    • 16-bit Unicode Transformation Format
    • 2 bytes per code point
    • This is Java’s char type
    

    View Slide

70. ASCII
    UTF-16
    

    View Slide

71. ASCII
    0 1 0 0 0 1 0 0
    0 1 0 0 0 1 0 0
    0 0 0 0 0 0 0 0
    D
    UTF-16
    

    View Slide

72. ASCII
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    D
    UTF-16
    o
    

    View Slide

73. ASCII
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 1 1 0 1 1 1 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    D
    UTF-16
    on
    

    View Slide

74. ASCII
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0
    D
    UTF-16
    onu
    

    View Slide

75. ASCII
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 0
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    D
    UTF-16
    onut
    

    View Slide

76. ASCII
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1 0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 0
    0 1 0 0 0 1 0 0 0 1 1 0 1 1 1 1
    0 1 1 0 1 1 1 0 0 1 1 1 0 1 0 1 0 1 1 1 0 1 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
    D
    UTF-16
    onut
    

    View Slide

77. UTF-16 characters
    • Max code point is 65,535
    • Code point for is 127,849
    • 127,849 > 65,535
    

    View Slide

78. Java’s char is broken!
    • There’s a system called “surrogate pairs” which is like multidex for
    code points
    • It splits a single code point across 2 chars
    • It’s an incredible pain
    

    View Slide

79. 

    char[] s = "Café ".toCharArray();
    C a f é
    0 1 2 3 4 5 6
    

    View Slide

80. String s = “Café ";

    

    for (int i = 0, size = s.length(); i < size; i++) {

    char c = s.charAt(i);

    System.out.printf("The character at %d is '%c'%n", i, c);

    }
    

    View Slide

81. String s = “Café ";

    

    for (int i = 0, size = s.length(); i < size; i++) {

    char c = s.charAt(i);

    System.out.printf("The character at %d is '%c'%n", i, c);

    }
    The character at 0 is 'C'
    The character at 1 is 'a'
    The character at 2 is 'f'
    The character at 3 is 'é'
    The character at 4 is ' '
    The character at 5 is ' '
    The character at 6 is ' '
    

    View Slide

82. String s = “Café ";

    

    for (int i = 0, size = s.length(); i < size; ) {

    int c = s.codePointAt(i);

    System.out.printf("The code point at %d is '%c'%n", i, c);
    i += Character.charCount(c);

    }
    

    View Slide

83. String s = “Café ";

    

    for (int i = 0, size = s.length(); i < size; ) {

    int c = s.codePointAt(i);

    System.out.printf("The code point at %d is '%c'%n", i, c);
    i += Character.charCount(c);

    }
    The code point at 0 is 'C'
    The code point at 1 is 'a'
    The code point at 2 is 'f'
    The code point at 3 is 'é'
    The code point at 4 is ' '
    The code point at 5 is ' '
    

    View Slide

84. View Slide

85. UTF-8
    • 8-bit Unicode Transformation Format
    • Variable number of bytes per code point
    • This is how modern apps transmit & store text
    

    View Slide

86. Image copyright Chris55, https://commons.wikimedia.org/wiki/File:Utf8webgrowth.svg
    

    View Slide

87. UTF-8
    • Many common characters are 1-byte
    • Some are 2 and 3 bytes. ‘ ’ is 4 bytes.
    • Self-delimiting
    • Self-aligning
    

    View Slide

88. 0
    1 1 0 1 0
    1 1 1 0
    1 1 1 1 0
    1 0
    1 0
    1 0
    1 0 1 0
    ≤7
    ≤11
    ≤16
    ≤21
    “How many bits do you need?”
    

    View Slide

89. C
    “How many bits do you need?”
    a
    f
    é
    sp
    

    View Slide

90. C
    “How many bits do you need?”
    a
    f
    é
    sp
    67
    

    View Slide

91. 1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    a
    f
    é
    sp
    

    View Slide

92. 1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    a
    f
    é
    sp
    97
    

    View Slide

93. 1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    f
    é
    sp
    

    View Slide

94. 1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    f
    é
    sp
    102
    

    View Slide

95. 1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp
    

    View Slide

96. 233
    1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp
    

    View Slide

97. 1 1 1 0 1 0 0 1
    1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp
    

    View Slide

98. 1 1 1 0 1 0 0 1
    1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp 32
    

    View Slide

99. 1 0 0 0 0 0
    1 1 1 0 1 0 0 1
    1 0 0 0 0 1 1
    C
    “How many bits do you need?”
    1 1 0 0 0 0 1
    a
    1 1 0 0 1 1 0
    f
    é
    sp
    

    View Slide

100. 1 0 0 0 0 0
     1 1 1 0 1 0 0 1
     1 0 0 0 0 1 1
     C
     “How many bits do you need?”
     1 1 0 0 0 0 1
     a
     1 1 0 0 1 1 0
     f
     é
     sp
     127,849
     

     View Slide

101. 1 0 1 0 0 1
     0 0 1 1 0 1
     1 1 1 1 1 1
     1 0 0 0 0 0
     1 1 1 0 1 0 0 1
     1 0 0 0 0 1 1
     C
     “How many bits do you need?”
     1 1 0 0 0 0 1
     a
     1 1 0 0 1 1 0
     f
     é
     sp
     

     View Slide

102. 0
     1 1 0
     1 1 1 1 0 1 0
     0
     1 0
     0
     0
     1 0
     1 0 1 0 1 0 0 1
     0 0 1 1 0 1
     0 1 1 1 1 1
     1 0 0 0 0 0
     1 0 1 0 0 1
     C
     “How many bits do you need?”
     1 1 0 0 0 0 1
     a
     1 1 0 0 1 1 0
     f
     é
     sp
     1 1
     1 0 0 0 0 1 1
     

     View Slide

103. 0
     1 1 0 0 0 0
     1 1 1 1 0 0 0 0 1 0
     0 0
     1 0
     0
     0
     1 0
     1 0 1 0 1 0 0 1
     0 0 1 1 0 1
     0 1 1 1 1 1
     1 0 0 0 0 0
     1 0 1 0 0 1
     C
     “How many bits do you need?”
     1 1 0 0 0 0 1
     a
     1 1 0 0 1 1 0
     f
     é
     sp
     1 1
     1 0 0 0 0 1 1
     

     View Slide

104. 0 1 0 0 0 0 1 1 1 1 0 0 0 0 1 1
     1 0 1 0 1 0 0 1 1 1 1 1 0 0 0 0 1 0 0 1 1 1 1 1
     1 0 0 0 1 1 0 1 1 0 1 0 1 0 0 1
     0 1 1 0 0 0 0 1 0 1 1 0 0 1 1 0
     0 0 1 0 0 0 0 0
     

     View Slide

105. UTF-8
     • Basically the best thing ever
     • Superset of ASCII
     • Great for JSON and HTML because delimiter characters <, > and "
     are 1-byte
     

     View Slide

106. Colors
     

     View Slide

107. R G B
     

     View Slide

108. View Slide

109. View Slide

110. View Slide

111. R
     255
     0
     G
     255
     0
     B
     255
     0
     255
     255 255
     

     View Slide

112. R
     255
     0
     G
     255
     0
     B
     255
     0
     255
     0 255
     

     View Slide

113. R
     255
     0
     G
     255
     0
     B
     255
     0
     20
     206 210
     

     View Slide

114. Hexadecimal Refresher!
     

     View Slide

115. Hexadecimal Refresher!
     • Decimal digits: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
     • Binary bits: 0, 1
     • Hexadecimal digits: 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, a, b, c, d, e, f
     

     View Slide

116. View Slide

117. 205 192
     = + 13
     

     View Slide

118. 205 12×16
     = + 13×1
     

     View Slide

119. 205 12×161
     = + 13×160
     

     View Slide

120. 205 c×161
     = + d×160
     

     View Slide

121. 205 c
     = d
     0x
     

     View Slide

122. Hexadecimal Refresher!
     • Prefixed with “0x”, like 0xcd
     • Hex bytes are always two digits: 00, 01, 02 … ff
     • Sequences of bytes are okay: 0bb634
     

     View Slide

123. Hexadecimal Refresher!
     • Colors: #ffffff
     • URL escaping: http://example.com/?q=hello%20world
     • Unicode code points U+2020
     • IPv6 addresses: 2001:0db8:85a3:0000:0000:8a2e:0370:7334
     

     View Slide

124. R
     255
     0
     G
     255
     0
     B
     255
     0
     ced214
     

     View Slide

125. Pictures
     

     View Slide

126. Pictures
     • Just a 2D array of colors
     • Given 3 bytes per pixel:
     • 64 × 64 icon is 12,288 bytes
     • 1080 × 1920 picture is 5.9 MiB
     • Compression is important!
     

     View Slide

127. Pictures
     • Android can use fewer bits per pixel
     • ARGB_8888: alpha, red, green, and blue get 8 bits each
     • RGB_565: red gets 5 bits, geen gets 6, blue gets 5 bits
     

     View Slide

128. RGB_565
     ARGB_8888
     

     View Slide

129. RGB_454
     ARGB_8888
     

     View Slide

130. RGB_232
     ARGB_8888
     

     View Slide

131. /** https://en.wikipedia.org/wiki/BMP_file_format */

     public void encode(BufferedSink sink) throws IOException {

     int height = pixels.length;

     int width = pixels[0].length;

     

     int bytesPerPixel = 3;

     int rowByteCountWithoutPadding = (bytesPerPixel * width);

     int rowByteCount = ((rowByteCountWithoutPadding + 3) / 4) * 4;

     int pixelDataSize = rowByteCount * height;

     int bmpHeaderSize = 14;

     int dibHeaderSize = 40;

     

     // BMP Header

     sink.writeUtf8("BM"); // ID.

     sink.writeIntLe(bmpHeaderSize + dibHeaderSize + pixelDataSize); // File size.

     sink.writeShortLe(0); // Unused.

     sink.writeShortLe(0); // Unused.

     sink.writeIntLe(bmpHeaderSize + dibHeaderSize); // Offset of pixel data.

     

     // DIB Header

     sink.writeIntLe(dibHeaderSize);

     

     View Slide

132. int dibHeaderSize = 40;

     

     // BMP Header

     sink.writeUtf8("BM"); // ID.

     sink.writeIntLe(bmpHeaderSize + dibHeaderSize + pixelDataSize); // File size.

     sink.writeShortLe(0); // Unused.

     sink.writeShortLe(0); // Unused.

     sink.writeIntLe(bmpHeaderSize + dibHeaderSize); // Offset of pixel data.

     

     // DIB Header

     sink.writeIntLe(dibHeaderSize);

     sink.writeIntLe(width);

     sink.writeIntLe(height);

     sink.writeShortLe(1); // Color plane count.

     sink.writeShortLe(bytesPerPixel * Byte.SIZE);

     sink.writeIntLe(0); // No compression.

     sink.writeIntLe(16); // Size of bitmap data including padding.

     sink.writeIntLe(2835); // Horizontal print resolution in pixels/meter. (72 dpi).

     sink.writeIntLe(2835); // Vertical print resolution in pixels/meter. (72 dpi).

     sink.writeIntLe(0); // Palette color count.

     sink.writeIntLe(0); // 0 important colors.

     

     // Pixel data.

     

     View Slide

133. sink.writeIntLe(0); // Palette color count.

     sink.writeIntLe(0); // 0 important colors.

     

     // Pixel data.

     for (int y = height - 1; y >= 0; y--) {

     int[] row = pixels[y];

     for (int x = 0; x < width; x++) {

     int pixel = row[x];

     sink.writeByte((pixel & 0x0000ff)); // Blue.

     sink.writeByte((pixel & 0x00ff00) >>> 8); // Green.

     sink.writeByte((pixel & 0xff0000) >>> 16); // Red.

     }

     

     // Padding for 4-byte alignment.

     for (int p = rowByteCountWithoutPadding; p < rowByteCount; p++) {

     sink.writeByte(0);

     }

     }

     }

     

     View Slide

134. Pictures on Bytes
     • This bitmap writer is 50 lines of code
     • Decoders are more difficult!
     • Good specs make it easy
     

     View Slide

135. sink.writeIntLe(0); // Palette color count.

     sink.writeIntLe(0); // 0 important colors.

     

     // Pixel data.

     for (int y = height - 1; y >= 0; y--) {

     int[] row = pixels[y];

     for (int x = 0; x < width; x++) {

     int pixel = row[x];

     sink.writeByte((pixel & 0x0000ff)); // Blue.

     sink.writeByte((pixel & 0x00ff00) >>> 8); // Green.

     sink.writeByte((pixel & 0xff0000) >>> 16); // Red.

     }

     

     // Padding for 4-byte alignment.

     for (int p = rowByteCountWithoutPadding; p < rowByteCount; p++) {

     sink.writeByte(0);

     }

     }

     }

     

     View Slide

136. sink.writeIntLe(0); // Palette color count.

     sink.writeIntLe(0); // 0 important colors.

     

     // Pixel data.

     for (int y = height - 1; y >= 0; y--) {

     int[] row = pixels[y];

     for (int x = 0; x < width; x++) {

     int pixel = row[x];

     sink.writeByte((pixel & 0x0000ff)); // Blue.

     sink.writeByte((pixel & 0x00ff00) >>> 8); // Green.

     sink.writeByte((pixel & 0xff0000) >>> 16); // Red.

     }

     

     // Padding for 4-byte alignment.

     for (int p = rowByteCountWithoutPadding; p < rowByteCount; p++) {

     sink.writeByte(0);

     }

     }

     }

     

     View Slide

137. (pixel & 0xff0000) >>> 16
     • Shifting and masking lets you access the bits within an integer
     • & and | operators treat each int like a 32-element boolean array!
     • <<, >> and >>> operators slide bits left and right
     

     View Slide

138. 1 0 1 0 1 0 1 1
     0 0 0 0 0 0 0 0
     0000ab00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
     0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
     00ffab40 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
     0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
     00ffab40
     0000ff00 1 1 1 1 1 1 1 1
     0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0x00ffab40 & 0x0000ff00
     

     View Slide

139. 1 0 1 0 1 0 1 1
     0 0 0 0 0 0 0 0
     0000ab00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
     0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
     00ffab40
     1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
     0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
     00ffab40
     0000ff00 1 1 1 1 1 1 1 1
     0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0x00ffab40 & 0x0000ff00
     

     View Slide

140. 1 0 1 0 1 0 1 1
     0 0 0 0 0 0 0 0
     0000ab00 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
     0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
     00ffab40
     1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1
     0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0
     00ffab40
     0000ff00 1 1 1 1 1 1 1 1
     0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0x00ffab40 & 0x0000ff00
     &
     =
     

     View Slide

141. 1 0 1 0 1 0 1 1
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0
     00000ab00
     0x0000ab00 >> 8
     

     View Slide

142. 1 0 1 0 1 0 1 1
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0
     00000ab00
     0x0000ab00 >> 8
     {
     8
     

     View Slide

143. 1 0 1 0 1 0 1 1
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0
     000000ab
     0x0000ab00 >> 8
     

     View Slide

144. 1 0 1 0 1 0 1 1
     0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0
     000000ab
     0x0000ab00 >> 8
     = 0x000000ab
     

     View Slide

145. sink.writeIntLe(0); // Palette color count.

     sink.writeIntLe(0); // 0 important colors.

     

     // Pixel data.

     for (int y = height - 1; y >= 0; y--) {

     int[] row = pixels[y];

     for (int x = 0; x < width; x++) {

     int pixel = row[x];

     sink.writeByte((pixel & 0x0000ff)); // Blue.

     sink.writeByte((pixel & 0x00ff00) >>> 8); // Green.

     sink.writeByte((pixel & 0xff0000) >>> 16); // Red.

     }

     

     // Padding for 4-byte alignment.

     for (int p = rowByteCountWithoutPadding; p < rowByteCount; p++) {

     sink.writeByte(0);

     }

     }

     }

     

     View Slide

146. This space
     intentionally
     left blank
     

     View Slide

147. int dibHeaderSize = 40;

     

     // BMP Header

     sink.writeUtf8("BM"); // ID.

     sink.writeIntLe(bmpHeaderSize + dibHeaderSize + pixelDataSize); // File size.

     sink.writeShortLe(0); // Unused.

     sink.writeShortLe(0); // Unused.

     sink.writeIntLe(bmpHeaderSize + dibHeaderSize); // Offset of pixel data.

     

     // DIB Header

     sink.writeIntLe(dibHeaderSize);

     sink.writeIntLe(width);

     sink.writeIntLe(height);

     sink.writeShortLe(1); // Color plane count.

     sink.writeShortLe(bytesPerPixel * Byte.SIZE);

     sink.writeIntLe(0); // No compression.

     sink.writeIntLe(16); // Size of bitmap data including padding.

     sink.writeIntLe(2835); // Horizontal print resolution in pixels/meter. (72 dpi).

     sink.writeIntLe(2835); // Vertical print resolution in pixels/meter. (72 dpi).

     sink.writeIntLe(0); // Palette color count.

     sink.writeIntLe(0); // 0 important colors.

     

     // Pixel data.

     

     View Slide

148. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
     Big Endian
     

     View Slide

149. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
     0 0 0 0 0 0 0 0
     Big Endian
     

     View Slide

150. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
     1 1 1 1 1 1 1 1
     0 0 0 0 0 0 0 0
     Big Endian
     

     View Slide

151. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
     1 0 1 0 1 0 1 1
     1 1 1 1 1 1 1 1
     0 0 0 0 0 0 0 0
     Big Endian
     

     View Slide

152. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
     0 1 0 0 0 0 0 0
     1 0 1 0 1 0 1 1
     1 1 1 1 1 1 1 1
     0 0 0 0 0 0 0 0
     Big Endian
     

     View Slide

153. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
     0 1 0 0 0 0 0 0
     1 0 1 0 1 0 1 1
     1 1 1 1 1 1 1 1
     0 0 0 0 0 0 0 0
     Big Endian
     

     View Slide

154. 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
     Little Endian
     

     View Slide

155. 0 1 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
     Little Endian
     

     View Slide

156. 1 0 1 0 1 0 1 1
     0 1 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
     Little Endian
     

     View Slide

157. 1 0 1 0 1 0 1 1 1 1 1 1 1 1 1 1
     0 1 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
     Little Endian
     

     View Slide

158. 1 0 1 0 1 0 1 1 1 1 1 1 1 1 1 1
     0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
     Little Endian
     

     View Slide

159. 1 0 1 0 1 0 1 1 1 1 1 1 1 1 1 1
     0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0
     0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 0 1 0 1 0 1 1 0 1 0 0 0 0 0 0
     Little Endian
     

     View Slide

160. Data Models
     

     View Slide

161. Conference App!
     • Each talk has an ID, date, room, title,
     and speaker
     

     View Slide

162. JSON is Good Stuff
     {

     "id": 72017,

     "date": "2016-09-30T18:30:00Z",

     "room": "RIGHT",

     "title": "Decoding the Secrets of Binary Data",

     "speaker": "Jesse Wilson"

     }
     

     View Slide

163. JSON is Good Stuff
     {

     "id": 72017,

     "date": "2016-09-30T18:30:00Z",

     "room": "RIGHT",

     "title": "Decoding the Secrets of Binary Data",

     "speaker": "Jesse Wilson"

     }
     • A nice format that builds on UTF-8
     • Easy to read & write
     

     View Slide

164. JSON is Self-Delimiting
     {

     "id": 72017,

     "date": "2016-09-30T18:30:00Z",

     "room": "RIGHT",

     "title": "Decoding the Secrets of Binary Data",

     "speaker": "Jesse Wilson"

     }
     • A JSON document has both structure and data
     • Uses escape sequences like \" to be completely unambiguous
     

     View Slide

165. "2016-09-30T18:30:00Z"
     • System.currentTimeMillis() returns milliseconds since
     January 1, 1970 at 00:00:00 UTC
     • A date that’s 8 bytes in memory is 22 bytes in JSON!
     

     View Slide

166. • Space: a simple message like this one is ~128 bytes
     • Time: bigger sequences take longer to decode
     JSON Space & Time
     

     View Slide

167. • Google’s “small, fast, simple” structured data format
     • Upon closer inspection, it’s not that different from JSON!
     • But it has a schema
     Protocol Buffers
     

     View Slide

168. {
     message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }

     }
     "id": 72017,
     "date": "2016-09-30T18:30:00Z",
     "room": "RIGHT",
     "title": "Decoding the Secrets of Binary Data",
     "speaker": "Jesse Wilson"
     }
     "id"
     "date"
     "room"
     "title"
     "speaker"
     72017
     "2016-09-30T18:30:00Z",
     "RIGHT",
     "Decoding the Secrets of Binary Data",
     "Jesse Wilson",
     

     View Slide

169. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }

     }
     

     View Slide

170. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }

     }
     000 enum, int32, int64...
     001 fixed64
     010 string, message
     101 fixed32
     Length Mode
     

     View Slide

171. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }

     }
     000 enum, int32, int64...
     001 fixed64
     010 string, message
     101 fixed32
     Length Mode
     1 0 1
     

     View Slide

172. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }

     }
     000 enum, int32, int64...
     001 fixed64
     010 string, message
     101 fixed32
     Length Mode
     1 0 1
     0 0 1
     

     View Slide

173. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }

     }
     000 enum, int32, int64...
     001 fixed64
     010 string, message
     101 fixed32
     Length Mode
     0 0 0
     1 0 1
     0 0 1
     

     View Slide

174. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }

     }
     000 enum, int32, int64...
     001 fixed64
     010 string, message
     101 fixed32
     Length Mode
     0 0 0
     1 0 1
     0 1 0
     0 0 1
     

     View Slide

175. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }

     }
     000 enum, int32, int64...
     001 fixed64
     010 string, message
     101 fixed32
     Length Mode
     0 0 0
     1 0 1
     0 1 0
     0 1 0
     0 0 1
     

     View Slide

176. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }

     }
     0 0 0
     1 0 1
     0 1 0
     0 1 0
     0 0 1
     

     View Slide

177. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }

     }
     0 0 0
     0 0 0 0 1 1 0 1
     0 1 0
     0 1 0
     0 0 1
     

     View Slide

178. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }

     }
     0 0 0
     0 0 0 0 1 1 0 1
     0 1 0
     0 1 0
     0 0 0 1 0 0 0 1
     

     View Slide

179. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }

     }
     0 0 0 1 1 0 0 0
     0 0 0 0 1 1 0 1
     0 1 0
     0 1 0
     0 0 0 1 0 0 0 1
     

     View Slide

180. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }

     }
     0 0 0 1 1 0 0 0
     0 0 0 0 1 1 0 1
     0 1 0
     0 0 1 0 0 0 1 0
     0 0 0 1 0 0 0 1
     

     View Slide

181. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }

     }
     0 0 0 1 1 0 0 0
     0 0 0 0 1 1 0 1
     0 0 1 0 1 0 1 0
     0 0 1 0 0 0 1 0
     0 0 0 1 0 0 0 1
     

     View Slide

182. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }

     }
     0 0 0 1 1 0 0 0
     0 0 0 0 1 1 0 1
     0 0 1 0 1 0 1 0
     0 0 1 0 0 0 1 0
     0 0 0 1 0 0 0 1
     

     View Slide

183. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }

     }
     0d
     11
     18
     22
     2a
     

     View Slide

184. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }a

     }a
     0d
     11
     18
     22
     2a
     {
     "id": 72017,
     "date": "2016-09-30T18:30:00Z",
     "room": "RIGHT",
     "title": "Decoding the Secrets of Binary Data",
     "speaker": "Jesse Wilson"
     }
     "id"
     "date"
     "room"
     "title"
     "speaker"
     72017
     "2016-09-30T18:30:00Z",
     "RIGHT",
     "Decoding the Secrets of Binary Data",
     "Jesse Wilson",
     

     View Slide

185. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }a

     }a
     0d
     11
     18
     22
     2a
     72017
     "2016-09-30T18:30:00Z",
     "RIGHT",
     "Decoding the Secrets of Binary Data",
     "Jesse Wilson",
     

     View Slide

186. message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }a

     }a
     0d
     11
     18
     22
     2a
     72017
     "2016-09-30T18:30:00Z",
     "RIGHT",
     "Decoding the Secrets of Binary Data",
     "Jesse Wilson",
     

     View Slide

187. 51
     19
     01
     message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }a

     }a
     0d
     11
     18
     22
     2a
     "2016-09-30T18:30:00Z",
     "RIGHT",
     "Decoding the Secrets of Binary Data",
     "Jesse Wilson",
     00
     

     View Slide

188. 511901
     message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }a

     }a
     0d
     11
     18
     22
     2a
     "2016-09-30T18:30:00Z",
     "RIGHT",
     "Decoding the Secrets of Binary Data",
     "Jesse Wilson",
     00
     

     View Slide

189. 511901
     message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }a

     }a
     0d
     11
     18
     22
     2a
     "2016-09-30T18:30:00Z",
     "RIGHT",
     "Decoding the Secrets of Binary Data",
     "Jesse Wilson",
     00
     

     View Slide

190. 511901
     message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }a

     }a
     0d
     11
     18
     22
     2a
     "RIGHT",
     "Decoding the Secrets of Binary Data",
     "Jesse Wilson",
     00
     000001577D37EE40
     

     View Slide

191. 511901
     message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }a

     }a
     0d
     11
     18
     22
     2a
     "RIGHT",
     "Decoding the Secrets of Binary Data",
     "Jesse Wilson",
     00
     00
     00
     01
     57
     7D
     37
     EE
     40
     

     View Slide

192. 511901
     message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }a

     }a
     0d
     11
     18
     22
     2a
     "RIGHT",
     "Decoding the Secrets of Binary Data",
     "Jesse Wilson",
     00
     00
     00
     01
     57
     7D
     37
     EE
     40
     

     View Slide

193. 511901
     message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }a

     }a
     0d
     11
     18
     22
     2a
     "Decoding the Secrets of Binary Data",
     "Jesse Wilson",
     00
     00
     00
     01
     57
     7D
     37
     EE
     40
     02
     

     View Slide

194. 511901
     message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }a

     }a
     0d
     11
     18
     22
     2a
     "Decoding the Secrets of Binary Data"
     "Jesse Wilson",
     00
     00
     01
     57
     7D
     37
     EE
     40
     02
     00
     

     View Slide

195. 511901
     message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }a

     }a
     0d
     11
     18
     22
     2a
     4465636f64696e67207468652053656372657473206f662042696e6172792044617461
     "Jesse Wilson",
     00
     00
     00
     01
     57
     7D
     37
     EE
     40
     02
     

     View Slide

196. 511901
     message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }a

     }a
     0d
     11
     18
     22
     2a
     4465636f64696e67207468652053656372657473206f662042696e6172792044617461
     "Jesse Wilson",
     00
     00
     00
     01
     57
     7D
     37
     EE
     40
     02
     23
     

     View Slide

197. 511901
     message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }a

     }a
     0d
     11
     18
     22
     2a
     4465636f64696e67207468652053656372657473206f662042696e6172792044617461
     "Jesse Wilson"
     00
     00
     00
     01
     57
     7D
     37
     EE
     40
     02
     23
     

     View Slide

198. 511901
     message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }a

     }a
     0d
     11
     18
     22
     2a
     4465636f64696e67207468652053656372657473206f662042696e6172792044617461
     4a657373652057696c736f6e
     00
     00
     00
     01
     57
     7D
     37
     EE
     40
     02
     23
     

     View Slide

199. 511901
     message Talk {

     optional fixed32 id = 1;

     optional fixed64 date = 2;

     optional Room room = 3;

     optional string title = 4;

     optional string speaker = 5;

     

     enum Room {

     UP = 1; RIGHT = 2; DOWN = 3; LEFT = 4;

     }a

     }a
     0d
     11
     18
     22
     2a
     4465636f64696e67207468652053656372657473206f662042696e6172792044617461
     4a657373652057696c736f6e
     00
     00
     00
     01
     57
     7D
     37
     EE
     40
     02
     23
     0c
     

     View Slide

200. 511901
     0d
     11
     18
     22
     2a 4a657373652057696c736f6e
     00
     00
     00
     01
     57
     7D
     37
     EE
     40
     02
     23
     0c
     4465636f64696e67207468652053656372657473206f662042696e6172792044617461
     

     View Slide

201. 511901
     0d 11 18 22
     2a
     4465636f64696e6720746865205365637265747
     4a657373652057696c736f6e
     00 00
     00
     01
     57
     7D
     37
     EE
     40 02 23
     0c
     3206f662042696e6172792044617461
     

     View Slide

202. 511901
     0d 11 18 22
     2a
     4465636f64696e6720746865205365637265747
     4a657373652057696c736f6e
     00 00
     00
     01
     57
     7D
     37
     EE
     40 02 23
     0c
     3206f662042696e6172792044617461
     

     View Slide

203. 511901
     0d 11 18 22
     2a
     4465636f64696e6720746865205365637265747
     4a657373652057696c736f6e
     00 00
     00
     01
     57
     7D
     37
     EE
     40 02 23
     0c
     3206f662042696e6172792044617461
     • Protocol Buffers are small and fast
     • ~1 byte for each field name
     • Compact encoding for numbers and enums
     • Strings stay the same length!
     • This message is 67 bytes in protocol buffers, vs. 128 for JSON
     

     View Slide

204. Java I/O
     

     View Slide

205. • Mutable!
     • equals() doesn’t work like it should: can’t be a map key!
     • Doesn’t implement Comparable
     byte[] is bad
     

     View Slide

206. • Range is -128..127 but you usually want 0..255
     • But unsigned when you call InputStream.read()
     byte is signed
     

     View Slide

207. • Range is -128..127 but you usually want 0..255
     • But unsigned when you call InputStream.read()
     byte is signed
     

     View Slide

208. Okio
     

     View Slide

209. Okio
     • Square’s library for efficient I/O
     • Used by OkHttp, Retrofit, Wire, Moshi
     

     View Slide

210. • Immutable!
     • equals() works like it should
     • Implements Comparable
     ByteString is good
     

     View Slide

211. ByteString makes data easy
     ByteString byteString = ByteString.decodeHex("436166c3a920f09f8da9");

     assertThat(byteString.size()).isEqualTo(10);

     assertThat(byteString.getByte(0)).isEqualTo((byte) 0x43);

     assertThat(byteString.getByte(0)).isEqualTo((byte) 67);

     

     String cafeDonuts = byteString.utf8();

     assertThat(cafeDonuts).isEqualTo("Café ");

     

     ByteString cafe = ByteString.encodeUtf8("Café");

     assertThat(byteString.startsWith(cafe)).isTrue();
     

     View Slide

212. ByteString and Parcelable
     

     View Slide

213. class Talk implements Parcelable {

     public static final Parcelable.Creator CREATOR = new Parcelable.Creator() {

     @Override public Talk createFromParcel(Parcel in) {

     int id = in.readInt();

     long date = in.readLong();

     Room room = Room.values()[in.readInt()];

     String title = in.readString();

     String speaker = in.readString();

     return new Talk(id, date, room, title, speaker);

     }

     };

     …

     

     @Override public void writeToParcel(Parcel out, int flags) {

     out.writeInt(id);

     out.writeLong(date);

     out.writeInt(room.ordinal());

     out.writeString(title);

     out.writeString(speaker);

     }

     }
     

     View Slide

214. @Test public void decodeGolden() {

     Talk talk = new Talk(72017, 1475260200000L, Room.RIGHT,

     "Decoding the Secrets of Binary Data", "Jesse Wilson");

     ByteString goldenData = ByteString.decodeHex("01000000511901004034…");

     assertThat(parcelDecode(goldenData, Talk.CREATOR)).isEqualTo(talk);

     }a

     

     private T parcelDecode(

     ByteString byteString, Parcelable.Creator creator) {

     Parcel parcel = Parcel.obtain();

     try {

     parcel.unmarshall(byteString.toByteArray(), 0, byteString.size());

     parcel.setDataPosition(0);

     return parcel.readTypedObject(creator);

     } finally {

     parcel.recycle();

     }a

     }a
     

     View Slide

215. @Test public void decodeGolden() {

     Talk talk = new Talk(72017, 1475260200000L, Room.RIGHT,

     "Decoding the Secrets of Binary Data", "Jesse Wilson");

     ByteString goldenData = ByteString.decodeHex("01000000511901004034…");

     assertThat(parcelDecode(goldenData, Talk.CREATOR)).isEqualTo(talk);

     }a

     

     private T parcelDecode(

     ByteString byteString, Parcelable.Creator creator) {

     Parcel parcel = Parcel.obtain();

     try {

     parcel.unmarshall(byteString.toByteArray(), 0, byteString.size());

     parcel.setDataPosition(0);

     return parcel.readTypedObject(creator);

     } finally {

     parcel.recycle();

     }a

     }a
     

     View Slide

216. Next Steps
     

     View Slide

217. • Everything is bytes
     • Java Strings are UTF-16. Encoded text is usually UTF-8
     • Hex is handy
     • Don’t be afraid of shifting and masking
     • Integers are big or little-endian
     • Java’s I/O APIs are trouble
     6 tips
     

     View Slide

218. • You can encode almost anything
     • Okio makes it easy
     No Black Boxes
     

     View Slide

219. @jessewilson
     https://github.com/swankjesse/encoding
     Thanks!
     

     View Slide