from mapping? A: Map operation is stateless i.e. one iteration doesn't depend on previous iteration. Q: What is the single biggest takeaway from reducing? A: Reduce represents an operation for a particular key.
you to notice, here, is that as soon as you think of map and reduce as functions that everybody can use, and they use them, you only have to get one supergenius to write the hard code to run map and reduce on a global massively parallel array of computers, and all the old code that used to work fine when you just ran a loop still works only it's a zillion times faster which means it can be used to tackle huge problems in an instant." - Joel Spolsky
2. Iterate over data chunks, convert each chunk into meaningful key, value pairs 3. Do this for all your data using massive parallelization 4. Group all the keys and their respective values 5. Take values for a key and convert into desired meaningful format 6. Step 2 is called mapper 7. Step 5 is called reducer
Put data into Hadoop 2. Define your mapper 3. Define your reducer 4. Run your jobs 5. Read processed data from Hadoop Other advantages: • Encapsulations over common problems like large files, process management, disk / node failure
Slaves slaves MapReduce Boss mapred-site.xml MapReduce Slave mapred-site.xml User's window into Hadoop, through the command hadoop Convention over Configuration? You wish. Job Task NameNode DataNode JobTracker TaskTracker Client
Shell scripts. Yuck! • Respite: ◦ Option for specifying a configuration directory ◦ Shell script configuration is mostly ENV variables • Which means: ◦ Configuration can be written in YML or JSON or Ruby and exported in XML ◦ ENV variables can be set using rake, thor or just plain Ruby • Caveats: ◦ No standard wrapper to do this (Go write one!)
are defined in Java • Other languages supported using Streaming API • Streaming API makes use of STDIN and STDOUT to read and output data and executable binaries for processing • Caveats ◦ No dependency management, we are on our own
of books in txt format 2. Find out the total usage of each character in the english alphabet. 3. Establish that e is the most used. 4. Why this example? a. Perfect use case for MapReduce. b. Algorithm is simple. c. Results are simple to analyze. d. Txt formatted books are easily available in Project Gutenberg.