MakerSquare - Algorithms

by Elben Shira

Transcript

1. Data Structures and Algorithms
2. Elben Shira Software Engineer Mass Relevance
3. Why? At Mass Relevance we have to analyze lots of things really fast. ! Millions hits/minute.
 10,000 Tweets/second.
4. Please ask questions!
5. Arrays and Hashes Questions?
6. Problem Solving Once you understand the data structures, the code is often obvious.
7. Example: Top Words
8. Input: Book text ! Output: Ten most-mentioned words in the book.
9. King James Bible ! s ="In the beginning God created the heaven and the earth…” ! Top Words ! God 1000 Jesus 300 Good 300 Evil 300 How do we do this?! What data structures can we use? Array, Hash, Set
10. Brainstorm #1 Let's try array. Say we have an array like this: [["in", 1], ["the", 1"], ["beginning", 1"], ["god", 1], ["created", 1”]]
11. Brainstorm #1 [["in", 1], ["the", 1], ["beginning", 1], ["god", 1], ["created", 1]] What happens when I increment the word earth? [["in", 1], ["the", 1], ["beginning", 1], ["god", 1], ["created", 1], [“earth”, 1]]
12. Brainstorm #1 [["in", 1], ["the", 1], ["beginning", 1], ["god", 1], ["created", 1], [“earth”, 1]] What happens when I increment the word beginning? [["in", 1], ["the", 1], ["beginning", 2], ["god", 1], ["created", 1], [“earth”, 1]]
13. Brainstorm #2 Say we have a hash like this: {
    "in"                =>  1,
    "the"              =>  1,
    "beginning"  =>  1,
    "god"              =>  1,
    "created"      =>  1,
 } Increment the word earth.
14. Brainstorm #2 Say we have a hash like this: {
    "in"                =>  1,
    "the"              =>  1,
    "beginning"  =>  1,
    "god"              =>  1,
    "created"      =>  1,
    "earth"          =>  1,
 }
15. Brainstorm #2 Say we have a hash like this: {
    "in"                =>  1,
    "the"              =>  1,
    "beginning"  =>  1,
    "god"              =>  1,
    "created"      =>  1,
    "earth"          =>  1,
 } Increment the word beginning.
16. Brainstorm #2 Say we have a hash like this: {
    "in"                =>  1,
    "the"              =>  1,
    "beginning"  =>  2,
    "god"              =>  1,
    "created"      =>  1,
    "earth"          =>  1,
 }
17. The code s  =  File.read("king_james_bible.txt")   words  =  s.split("\s")   counts  =  Hash.new(0)   words.each  do  |w|          counts[w]  +=  1   end
18. The code sorted  =  counts.to_a.sort_by  do  |word,  count|      -­‐count   end   ! sorted.take(10)  
19. The results the 64106 and 51383 of 34708 to 13630 that 12799 in 12560 he 10264 shall 9840 unto 8987 for 8838 Not very useful!! ! Need stop words.
20. Stop words a,able,about,across,after,all,almost,also,am,amo ng,an,and,any,are,as,at,be,because,been,but,by, can,cannot,could,dear,did,do,does,either,else,ev er,every,for,from,get,got,had,has,have,he,her,her s,him,his,how,however,i,if,in,into,is,it,its,just,least ,let,like,likely,may,me,might,most,must,my,neith er,no,nor,not,of,off,often,on,only,or,other,our,ow n,rather,said,say,says,she,should,since,so,some,t han,that,the,their,them,then,there,these,they,this ,tis,to,too,twas,us,wants,was,we,were,what,when ,where,which,while,who,whom,why,will,with,wou ld,yet,you,your
21. The code stopwords  =  
    Set.new(File.read(“stop.txt”).split(“,”))   ! s  =  File.read(“king_james_bible.txt")   words  =  s.split(“\s”).map(&:downcase)   counts  =  Hash.new(0)   words.each  do  |w|      counts[w]  +=  1  if  stopwords.include?(w)   end  
22. The results unto 8987 thou 5202 lord 4739 thy 4600 ye 3847 god 2303 hath 2242 him, 2034 came 1991 man 1978 Need better stop! words!
23. Project Gutenberg
24. Project Gutenberg Input: Books, categorized into astronomy, physics, religion, philosophy and archeology. ! Output: Some method that takes in a new book we've never seen before and spits out the category it's in.
25. Project Gutenberg predict(some_physics_book)   =>  :physics   ! predict(some_philosophy_book)   =>  :archeology   ! Accuracy:  50%
26. Project Gutenberg Dataset Training set Test set ! Algorithm Base algorithm or your own
27. Getting Started Read README.md data/ gutenberg.rb lib/predictor.rb lib/simple_predictor.rb lib/complex_predictor.rb ! Implement ComplexPredictor#train! ComplexPredictor#predict
28. Tips Use arrays, hashes, and sets. Draw your data. Think: word counts, stop words. ! Have fun! ! https://bitbucket.org/elben/makersquare-project-gutenberg/
29. Questions? ! Thank You!