How the Web Works: Lecture 9

This talk was designed for a class (98-135) taught at Carnegie Mellon University in Spring 2010.

Transcript

2. Lecture 9 Distributed Computing & Scaling
4. Homeworks
5. Homeworks Overall, I failed =(
6. Homeworks Overall, I failed =( Should’ve done it in winter
7. Homeworks Overall, I failed =( Should’ve done it in winter Get 3 Points to Pass
8. Homeworks Overall, I failed =( Should’ve done it in winter Get 3 Points to Pass Hopefully, 10 by the end
9. Homeworks Overall, I failed =( Should’ve done it in winter Get 3 Points to Pass Hopefully, 10 by the end I don’t want to fail anyone
10. Zeliveau Please Start Soon
12. Rankmaniac http://scienceoftheweb.org/15-396/assignments/hw6.pdf
15. “Essentially, using nofollow causes us to drop the target links from our overall graph of the web”
18. SSL/TLS That HTTPS business...
25. Visible to Wireless Network, ISP, Server LAN
27. Let encrypt with a key!
28. Let encrypt with a key! ENC(K, “MES”) = “NFT” | DEC(K, “NFT”) = “MES” K = “Shift One Alphabet”
29. Let encrypt with a key! But how do we share the key? ENC(K, “MES”) = “NFT” | DEC(K, “NFT”) = “MES” K = “Shift One Alphabet”
30. Public Key Encryption Insanely Awesomely Brilliant
31. n = p * q
32. n = p * q Given These
33. n = p * q Given These Easy to Compute
34. n = p * q Given These Easy to Compute Given This
35. n = p * q Given These Easy to Compute Given This Possible but...
36. RSA
37. Rivest RSA
38. Rivest Shamir RSA
39. Rivest Shamir Adleman RSA
40. Public Key Encryption
41. Public Key Encryption Create an Algorithm that...
42. Public Key Encryption Create an Algorithm that... uses n to encrypt
43. Public Key Encryption Create an Algorithm that... uses n to encrypt but needs p & q to decrypt
44. Public Key Encryption Create an Algorithm that... uses n to encrypt but needs p & q to decrypt Publish n as public key
45. Public Key Encryption Create an Algorithm that... uses n to encrypt but needs p & q to decrypt Publish n as public key Keep p & q
46. Public Key Encryption Create an Algorithm that... uses n to encrypt but needs p & q to decrypt Publish n as public key Keep p & q Heard of PGP?
48. Who are you anyway?
49. Who are you anyway? Aha, an Imposter!
50. Who are you anyway? Aha, an Imposter! The Verification Problem
54. Browsers Preinstalled with some CAs
55. Browsers Preinstalled with some CAs
57. Install if you trust CMU
60. P2P ... and the indexing problem
61. Client-Server Model
62. Client-Server Model I can haz music!
64. P2P Model
65. P2P Model Who has my file?
66. More Generally Distributed Hash Table Given a Key, get the Value Stored across computers Google’s Index (GFS) So, how do you find a file?
67. Ask Everyone
68. Ask Everyone What Not to Do!
70. Computers (N)
71. Computers (N) Files (K)
72. Computers (N) Files (K) 2^m > max{N,K}
73. 8 12 16 Example
74. 1 3 4 5 7 9 12 15
75. Label Nodes between 0 and 2^m 1 3 4 5 7 9 12 15
76. 1 6 12 2 8 13 4 9 15 5 11 16
77. Label Keys between 0 and 2^m 1 6 12 2 8 13 4 9 15 5 11 16
78. Assignment Assign Key K to Node K If Node K doesn’t exist ... assign to next node 1 3 4 5 7 9 1 1 1 6 12 2 8 13 4 9 15 5 11 16
79. 1 6 12 2 8 13 4 9 15 5 11 16 1 3 4 5 7 9 12 15
80. 1 6 12 2 8 13 4 9 15 5 11 16 1 3 4 5 7 9 12 15
81. 16 1 6 12 2 8 13 4 9 15 5 11 1 3 4 5 7 9 12 15
82. Searching For key K ~ For Node K Linear Search Start at Machine 1, goto next ... so on until found! 1 3 4 5 7 9 1 1 1 6 12 2 8 13 4 9 15 5 11 16
83. But Wait They’re sorted, seems familiar?
84. Binary Search Is 8 in the list? What position is it? 1 3 5 6 8 9
85. Binary Search Is 8 in the list? What position is it? 1 3 5 6 8 9 1 3 5 6 8 9
86. Binary Search Is 8 in the list? What position is it? 1 3 5 6 8 9 1 3 5 6 8 9 6 8 9
87. Binary Search Is 8 in the list? What position is it? 1 3 5 6 8 9 1 3 5 6 8 9 6 8 9 8 9
88. 16 1 6 12 2 8 13 4 9 15 5 11 1 3 4 5 7 9 12 15
89. 16 1 6 12 2 8 13 4 9 15 5 11 1 3 4 5 7 9 12 15 Each machine stores address to some others!
90. Finger Table 1 3 4 5 7 9 1 1 1 6 12 2 8 13 4 9 15 5 11 16
91. Finger Table Total Machines (2^m) = 8 1 3 4 5 7 9 1 1 1 6 12 2 8 13 4 9 15 5 11 16
92. Finger Table Total Machines (2^m) = 8 Machine N7 stores: 1 3 4 5 7 9 1 1 1 6 12 2 8 13 4 9 15 5 11 16
93. Finger Table Total Machines (2^m) = 8 Machine N7 stores: addr(N7 + 1) addr(N7 + 2) addr(N7 + 4) = addr(N7 + 2^(m-1)) 1 3 4 5 7 9 1 1 1 6 12 2 8 13 4 9 15 5 11 16
94. Finger Table Total Machines (2^m) = 8 Machine N7 stores: addr(N7 + 1) addr(N7 + 2) addr(N7 + 4) = addr(N7 + 2^(m-1)) Can Take short-cuts! 1 3 4 5 7 9 1 1 1 6 12 2 8 13 4 9 15 5 11 16
95. 1 3 4 5 7 9 12 15
96. 1 3 4 5 7 9 12 15
97. 16 1 6 12 2 8 13 4 9 15 5 11 1 3 4 5 7 9 12 15
98. 16 1 6 12 2 8 13 4 9 15 5 11 1 3 4 5 7 9 12 15
99. 16 1 6 12 2 8 13 4 9 15 5 11 1 3 4 5 7 9 12 15 Make Biggest Jump | Too Low | Use N7’s table
100. 16 1 6 12 2 8 13 4 9 15 5 11 1 3 4 5 7 9 12 15 Make Biggest Jump | Too Low | Use N7’s table
101. 16 1 6 12 2 8 13 4 9 15 5 11 1 3 4 5 7 9 12 15
102. 16 1 6 12 2 8 13 4 9 15 5 11 1 3 4 5 7 9 12 15
103. 16 1 6 12 2 8 13 4 9 15 5 11 1 3 4 5 7 9 12 15 Halve the remaining ring
104. 16 1 6 12 2 8 13 4 9 15 5 11 1 3 4 5 7 9 12 15 Halve the remaining ring
105. 16 1 6 12 2 8 13 4 9 15 5 11 1 3 4 5 7 9 12 15
106. 16 1 6 12 2 8 13 4 9 15 5 11 1 3 4 5 7 9 12 15
107. 16 1 6 12 2 8 13 4 9 15 5 11 1 3 4 5 7 9 12 15 Done!
108. Chord Protocol 1 3 4 5 7 9 1 1 1 6 12 2 8 13 4 9 15 5 11 16 http://en.wikipedia.org/wiki/Chord_%28peer-to-peer%29
109. Chord Protocol log(N) Performance 1 3 4 5 7 9 1 1 1 6 12 2 8 13 4 9 15 5 11 16 http://en.wikipedia.org/wiki/Chord_%28peer-to-peer%29
110. Chord Protocol log(N) Performance Fully Decentralizes P2P 1 3 4 5 7 9 1 1 1 6 12 2 8 13 4 9 15 5 11 16 http://en.wikipedia.org/wiki/Chord_%28peer-to-peer%29
111. Chord Protocol log(N) Performance Fully Decentralizes P2P Napster was Centralized 1 3 4 5 7 9 1 1 1 6 12 2 8 13 4 9 15 5 11 16 http://en.wikipedia.org/wiki/Chord_%28peer-to-peer%29
112. Chord Protocol log(N) Performance Fully Decentralizes P2P Napster was Centralized ... hence closed down! 1 3 4 5 7 9 1 1 1 6 12 2 8 13 4 9 15 5 11 16 http://en.wikipedia.org/wiki/Chord_%28peer-to-peer%29
113. Chord Protocol log(N) Performance Fully Decentralizes P2P Napster was Centralized ... hence closed down! RIAA/MPAA: Oh Noes! 1 3 4 5 7 9 1 1 1 6 12 2 8 13 4 9 15 5 11 16 http://en.wikipedia.org/wiki/Chord_%28peer-to-peer%29
114. MapReduce But first, an OCD Programmer
115. alert("get the lobster"); PutInPot("lobster"); PutInPot("water"); alert("get the chicken"); BoomBoom("chicken"); BoomBoom("coconut");
116. function Cook( i1, i2, f ) { alert("get the " + i1); f(i1); f(i2); } Cook( "lobster", "water", PutInPot ); Cook( "chicken", "coconut", BoomBoom );
117. Map
118. var a = [1,2,3]; for (i=0; i<a.length; i++) { a[i] = a[i] * 2; } for (i=0; i<a.length; i++) { alert(a[i]); }
119. function map(fmap, a) { for (i = 0; i < a.length; i++) { a[i] = fmap(a[i]); } } map( function(x){return x*2;}, a ); map( alert, a );
120. Reduce
121. function sum(a) { var s = 0; for (i = 0; i < a.length; i++) s += a[i]; return s; } function join(a) { var s = ""; for (i = 0; i < a.length; i++) s += a[i]; return s; }
122. function reduce(fred, a, init) { var s = init; for (i = 0; i < a.length; i++) s = fred( s, a[i] ); return s; }
123. function sum(a) { return reduce( function(a, b){ return a + b; }, a, 0 ); } function join(a) { return reduce( function(a, b){ return a + b; }, a, "" ); }
124. Map Reduce
125. Map [1, 2, 3, 4, 5] Reduce
126. Map [1, 2, 3, 4, 5] [2, 4, 6, 8 , 10] Reduce
127. Map [1, 2, 3, 4, 5] [2, 4, 6, 8 , 10] [One, Two, Three, Four , Five] Reduce
128. Map [1, 2, 3, 4, 5] [2, 4, 6, 8 , 10] [One, Two, Three, Four , Five] Reduce [1, 1, 1, 1, 1]
129. Map [1, 2, 3, 4, 5] [2, 4, 6, 8 , 10] [One, Two, Three, Four , Five] Reduce [1, 1, 1, 1, 1] 5
130. Map [1, 2, 3, 4, 5] [2, 4, 6, 8 , 10] [One, Two, Three, Four , Five] Reduce [1, 1, 1, 1, 1] 5 “11111”
131. “Without understanding functional programming, you can't invent MapReduce. The very fact that Google invented MapReduce, and Microsoft didn't, says something about why Microsoft is still playing catch up” - Joel Spolsky
132. Pop Quiz [1, 2, 3, 4, 5] [“odd”, “even”, “odd”, “even”, “odd”] “oddevenoddevenoddeven”
133. How is that useful?
134. Word Count Given a document # occurrences of each word Let’s try the intuitive way...
136. bigFile is too big? Have two files!
139. Lets See that Again
141. BoomBoom("chicken"); BoomBoom("coconut");
142. BoomBoom("chicken"); BoomBoom("coconut"); Map
143. BoomBoom("chicken"); BoomBoom("coconut"); Map
144. BoomBoom("chicken"); BoomBoom("coconut"); Map function reduce(union, [d1,d2], [])
145. BoomBoom("chicken"); BoomBoom("coconut"); Map Reduce function reduce(union, [d1,d2], [])
146. foo foo baz bar gor baz goo bar
147. foo foo baz bar gor baz goo bar foo foo baz bar gor baz goo bar
148. foo foo baz bar gor baz goo bar foo foo baz bar gor baz goo bar Mapper Mapper
149. foo foo baz bar gor baz goo bar foo foo baz bar gor baz goo bar Mapper Mapper foo 1 foo 1 baz 1 bar 1 gor 1 baz 1 goo 1 bar 1
150. foo foo baz bar gor baz goo bar foo foo baz bar gor baz goo bar Mapper Mapper foo 1 foo 1 baz 1 bar 1 gor 1 baz 1 goo 1 bar 1 Bucket by Key
151. foo foo baz bar gor baz goo bar foo foo baz bar gor baz goo bar Mapper Mapper foo 1 foo 1 baz 1 bar 1 gor 1 baz 1 goo 1 bar 1 Bucket by Key [foo, foo] [baz, baz] [bar, bar] [goo] [gor]
152. foo foo baz bar gor baz goo bar foo foo baz bar gor baz goo bar Mapper Mapper foo 1 foo 1 baz 1 bar 1 gor 1 baz 1 goo 1 bar 1 Bucket by Key [foo, foo] [baz, baz] [bar, bar] [goo] [gor] foo reducer foo reducer foo reducer foo reducer foo reducer
153. foo foo baz bar gor baz goo bar foo foo baz bar gor baz goo bar Mapper Mapper foo 1 foo 1 baz 1 bar 1 gor 1 baz 1 goo 1 bar 1 Bucket by Key [foo, foo] [baz, baz] [bar, bar] [goo] [gor] foo reducer foo reducer foo reducer foo reducer foo reducer 2 2 2 1 1
154. Who Does What?
155. Who Does What? User: Write Mapper and Reducer
156. Who Does What? User: Write Mapper and Reducer Hadoop: Splitting, Bucketing
157. Who Does What? User: Write Mapper and Reducer Hadoop: Splitting, Bucketing Cons: Restricted Paradigm
158. Who Does What? User: Write Mapper and Reducer Hadoop: Splitting, Bucketing Cons: Restricted Paradigm Pros: Generalized, Safe
159. Who Does What? User: Write Mapper and Reducer Hadoop: Splitting, Bucketing Cons: Restricted Paradigm Pros: Generalized, Safe Implementing can be tricky!
161. Abstract, Complex? But you know what...
162. The Point People talk about scaling ... but now you know it! Distributing Files Distributing Computation
163. Higher Level Point This isn’t a CS class... ... but I’m a CS major =P Its not all HTML/CSS There’s some serious CS here!
164. http://hadoop.apache.org/ http://www.cloudera.com/resources/?type=Training
165. Poor Man’s Scaling
166. Redundancy Replicate across computers Main server balances load Other servers serve content Also useful for data backups Usually Host Managed
167. Caching PHP is dynamic ... usually unnecessarily Calculate, cache, reserve Memoization PHP/memcached http://en.wikipedia.org/wiki/Memcached
168. Bottlenecks Content Bandwidth Databases External APIs Script busy computing etc...
169. S3 EC2 http://www.youtube.com/watch?v=Iaxu-NLecm4 http://www.youtube.com/watch?v=bBajLxeKqoY
170. Homework 7 is out No Class Next Week
172. Photo Credits http://mi9.com/datawallpapers/data/12/993/1217993797/eye-with-black-background_1280x1024.jpg http://www.aemmp.org/site/wp-content/uploads/2009/10/imgname-riaa_training_video_leaked_more_stupid_than_expected-50226711-RIAA.jpg http://jasonjeffrey.files.wordpress.com/2007/09/drm.jpg http://www.sdtimes.com/blog/post/2009/image.axd?picture=2009%2F7%2Fhadoopephant.jpg