Papers We Love Too: Shannon's Mathematical Theory of Communication

Papers We Love Too: June 2016

Transcript

1. Papers We Love Too: June 2016 A Mathematical Theory of Communication presented by Kiran Bhattaram
2. Kiran Bhattaram @kiranb
3. Agenda A Brief History The Paper Impact
4. A Brief History 1800s to 1948
5. discovering limits heat engines! speed of light! temperature! incompleteness!
6. communications telegraphy telephone wireless telegraphy AM radio FM radio television
7. “just crank up the volume’ Transatlantic Telegraph 1858
9. Transmission Speeds 0.064 bits/s 10 bit/s 10 billion bits/s
10. A Mathematical Theory of Communication Claude Shannon, Bell Labs System Journal (1948)
11. A Mathematical Theory of Communication Claude Shannon, Bell Labs System Journal (1948) The
13. Contributions
14. Contributions 1. All communication is the same.
15. Contributions 1. All communication is the same. 2. Information is measurable.
16. Contributions 1. All communication is the same. 2. Information is measurable. 3. Information can be transmitted without error over noisy channels.
17. 1. All communication is the same
19. An Overview! data estimated data
20. An Overview! data estimated data Compress Huffman, LZW, etc
21. An Overview! data estimated data Compress Huffman, LZW, etc Channel Coding Viterbi, Turbo codes, LPDC, etc.
22. An Overview! Modulation! data estimated data Compress Huffman, LZW, etc Channel Coding Viterbi, Turbo codes, LPDC, etc.
23. An Overview! NOISY CHANNELS Modulation! data estimated data Compress Huffman, LZW, etc Channel Coding Viterbi, Turbo codes, LPDC, etc.
24. An Overview! NOISY CHANNELS Modulation! Demodulation! Decoding Decompression data estimated data Compress Huffman, LZW, etc Channel Coding Viterbi, Turbo codes, LPDC, etc.
25. 2. Information is Measurable
26. A Series of Approximations to English
27. A Series of Approximations to English XFOML RXKHRJFFJUJ ZLPWCFWKCYJ FFJEYVKCQSGHYD QPAAMKBZAACIBZLHJQD. 1. Symbols independent and equiprobable.
28. A Series of Approximations to English OCRO HLI RGWR NMIELWIS EU LL NBNESEBYA TH EEI ALHENHTTPA OOBTTVA NAH BRL. 2. Symbols independent and with the frequency of English text.
29. A Series of Approximations to English ON IE ANTSOUTINYS ARE T INCTORE ST BE S DEAMY ACHIN D ILONASIVE TUCOOWE AT TEASONARE FUSO TIZIN ANDY TOBE SEACE CTISBE. 3. Digram structure as in English
30. A Series of Approximations to English Representing and speedily is an good apt or come can different natural Here he the a in came the to of to expert gray come to furnishes the line message had be these 4. Word frequency as English; independent words
31. A Series of Approximations to English The head and in frontal attack on an English writer that the character of this point is therefore another method for the letters that the time of whoever told the problem for an unexpected 5. Word digrams as in English
32. Markov Processes
33. Markov Processes
34. Markov Processes
35. Markov Processes
36. Encoding Messages Dinner Probability Encoding thai 1/2 ’00' szechuan 1/3 ’01' pizza 1/12 ’10' ice cream 1/12 ’01'
37. Huffman Codes (1951) thai szechuan pizza ice cream 0 1 0 0 1 1 Dinner Probability Encoding thai 1/2 ’0' szechuan 1/3 ’10' pizza 1/12 ’110' ice cream 1/12 ‘111'
38. Huffman Codes Dinner Probability Encoding thai 1/2 ’0' szechuan 1/3 ’10' pizza 1/12 ’110' ice cream 1/12 ‘111'
39. Huffman Codes Dinner Probability Encoding thai 1/2 ’0' szechuan 1/3 ’10' pizza 1/12 ’110' ice cream 1/12 ‘111'
40. Information content
41. - Claude Shannon Von Neumann told me, ‘You should call it entropy, for two reasons. In the second place, and more important, nobody knows what entropy really is, so in a debate you will always have the advantage.’ In the first place your uncertainty function has been used in statistical mechanics under that name, so it already has a name.
42. Source Coding Theorem setting limits on compression
43. Source Coding Theorem setting limits on compression the average codeword length < average information per symbol
44. Review! NOISY CHANNELS Modulation! Demodulation! Decompression data estimated data Compress Huffman, LZW, etc Decoding Channel Coding Viterbi, Turbo codes, LPDC, etc.
45. Review! NOISY CHANNELS Modulation! Demodulation! Decompression data estimated data Compress Huffman, LZW, etc Decoding Channel Coding Viterbi, Turbo codes, LPDC, etc.
46. Review! NOISY CHANNELS Modulation! Demodulation! Decompression data estimated data Compress Huffman, LZW, etc Decoding Channel Coding Viterbi, Turbo codes, LPDC, etc.
47. Review! NOISY CHANNELS Modulation! Demodulation! Decompression data estimated data Compress Huffman, LZW, etc Decoding Channel Coding Viterbi, Turbo codes, LPDC, etc.
48. 3. Information can be transmitted without error * up to a certain limit
49. Conditional Probabilities 0 1 0 1 Sent Received
50. Conditional Probabilities 0 1 0 1 0.9 Sent Received
51. Conditional Probabilities 0 1 0 1 0.9 0.1 Sent Received
52. Conditional Probabilities 0 1 0 1 0.9 0.9 0.1 Sent Received
53. Conditional Entropy
54. Conditional Entropy
55. Conditional Entropy
56. Conditional Entropy entropy of source
57. Conditional Entropy entropy of source uncertainty in the received signal, if the message is known
 (uncertainty added by noise)
58. Conditional Entropy entropy of source uncertainty in the received signal, if the message is known
 (uncertainty added by noise) entropy of received message
59. Conditional Entropy entropy of source uncertainty in the message,
 if the received signal is known uncertainty in the received signal, if the message is known
 (uncertainty added by noise) entropy of received message
60. Channel Capacity entropy of received message uncertainty in the message,
 if the received signal is known
61. The surprising thing about capacity a simple error-correcting code: add parity bits 0 P(error) = 1/4
62. The surprising thing about capacity a simple error-correcting code: add parity bits 0 0 P(error) = 1/16
63. The surprising thing about capacity a simple error-correcting code: add parity bits 0 0 0 P(error) = 1/64
64. The surprising thing about capacity a simple error-correcting code: add parity bits 0 0 0 P(error) = 0 1/256
65. The surprising thing about capacity a simple error-correcting code: add parity bits 0 0 0 0
66. The surprising thing about capacity P(error) —> 0 overhead —> ∞ a simple error-correcting code: add parity bits 0 0 0 0
67. Shannon-Hartley Theorem C = B log2 (1+ ) S_ N
68. Shannon-Hartley Theorem C = B log2 (1+ ) Channel capacity in bits/s S_ N
69. Shannon-Hartley Theorem C = B log2 (1+ ) Channel capacity in bits/s bandwidth of the channel S_ N
70. Shannon-Hartley Theorem C = B log2 (1+ ) Channel capacity in bits/s bandwidth of the channel signal-to-noise ratio S_ N
71. Review! NOISY CHANNELS Modulation! Demodulation! Decompression data estimated data Compress Huffman, LZW, etc Decoding Channel Coding Viterbi, Turbo codes, LPDC, etc.
72. Review! NOISY CHANNELS Modulation! Demodulation! Decompression data estimated data Compress Huffman, LZW, etc Decoding Channel Coding Viterbi, Turbo codes, LPDC, etc.
73. Review! NOISY CHANNELS Modulation! Demodulation! Decompression data estimated data Compress Huffman, LZW, etc Decoding Channel Coding Viterbi, Turbo codes, LPDC, etc.
74. Review! NOISY CHANNELS Modulation! Demodulation! Decompression data estimated data Compress Huffman, LZW, etc Decoding Channel Coding Viterbi, Turbo codes, LPDC, etc.
75. Review! NOISY CHANNELS Modulation! Demodulation! Decompression data estimated data Compress Huffman, LZW, etc Decoding Channel Coding Viterbi, Turbo codes, LPDC, etc.
76. Impact
77. Hamming Codes
78. Hamming Codes D1 P4 P3 P1 P2 D2 D3 D4
79. Hamming Codes 0 1 0 1 0 1 1 1
80. Hamming Codes 0 1 0 1 0 0 1 1
81. Convolutional Codes 011100101010010100010 + p0 p1 +
82. Convolutional Codes 011100101010010100010 + p0 p1 +
84. “Just use more power; more bandwidth”
86. Pioneer IX 1958:
87. 1962: Mariner 2
88. Mariner VI 1969:
89. Images from Mars 1964: Mariner IV 1969: Mariner VI using no encoding using Reed-Muller encoding
90. The Grand Tour Courtesy NASA/ JPL-Caltech
91. Voyager 1
93. Review! NOISY CHANNELS Modulation! Demodulation! Decompression data estimated data Compress Huffman, LZW, etc Decoding Channel Coding Viterbi, Turbo codes, LPDC, etc.
94. Review! NOISY CHANNELS Modulation! Demodulation! Decompression data estimated data Compress Huffman, LZW, etc Decoding Channel Coding Viterbi, Turbo codes, LPDC, etc.
95. Review! NOISY CHANNELS Modulation! Demodulation! Decompression data estimated data Compress Huffman, LZW, etc Decoding Channel Coding Viterbi, Turbo codes, LPDC, etc.
96. Review! NOISY CHANNELS Modulation! Demodulation! Decompression data estimated data Compress Huffman, LZW, etc Decoding Channel Coding Viterbi, Turbo codes, LPDC, etc.
97. Review! NOISY CHANNELS Modulation! Demodulation! Decompression data estimated data Compress Huffman, LZW, etc Decoding Channel Coding Viterbi, Turbo codes, LPDC, etc.
99. Weird Shit/Current Areas of Research
100. is information physical? can you measure the energy of a bit?
101. black holes and information
102. reversible logic