Upgrade to Pro — share decks privately, control downloads, hide ads and more …

MongoDB and Hadoop

MongoDB and Hadoop

Why and How MongoDB and Hadoop are working together?

Learn this in this presentation.

This presentation was delivered during MongoDB Day Paris 2014

Aab9ac774f61c5d9bf143b5a1bfe901b?s=128

Tugdual Grall

October 28, 2014
Tweet

More Decks by Tugdual Grall

Other Decks in Technology

Transcript

  1. Paris Tugdual Grall Technical Evangelist tug@mongodb.com @tgrall

  2. MongoDB & Hadoop Tugdual Grall Technical Evangelist tug@mongodb.com @tgrall

  3. Agenda Evolving Data Landscape MongoDB & Hadoop Use Cases MongoDB

    Connector Features Demo
  4. Evolving Data Landscape

  5. • Terabyte and Petabyte datasets • Data warehousing • Advanced

    analytics Hadoop “The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models.” http://hadoop.apache.org
  6. ‹#› Enterprise IT Stack

  7. ‹#› Operational vs. Analytical: Enrichment Warehouse, Analytics Applications, Interactions

  8. Operational: MongoDB First-Level Analytics Internet of Things Social Mobile Apps

    Product/Asset Catalog Security & Fraud Single View Customer Data Management Churn Analysis Risk Modeling Sentiment Analysis Trade Surveillance Recommender Warehouse & ETL Ad Targeting Predictive Analytics
  9. Analytical: Hadoop First-Level Analytics Internet of Things Social Mobile Apps

    Product/Asset Catalog Security & Fraud Single View Customer Data Management Churn Analysis Risk Modeling Sentiment Analysis Trade Surveillance Recommender Warehouse & ETL Ad Targeting Predictive Analytics
  10. Operational & Analytical: Lifecycle First-Level Analytics Internet of Things Social

    Mobile Apps Product/Asset Catalog Security & Fraud Single View Customer Data Management Churn Analysis Risk Modeling Sentiment Analysis Trade Surveillance Recommender Warehouse & ETL Ad Targeting Predictive Analytics
  11. MongoDB & Hadoop Use Cases

  12. Commerce Applications powered by Analysis powered by Products & Inventory

    Recommended products Customer profile Session management Elastic pricing Recommendation models Predictive analytics Clickstream history MongoDB Connector
 for Hadoop
  13. Insurance Applications powered by Analysis powered by Customer profiles Insurance

    policies Session data Call center data Customer action analysis Churn analysis Churn prediction Policy rates MongoDB Connector
 for Hadoop
  14. Fraud Detection MongoDB Connector
 for Hadoop Payments Nightly Analysis  

       3rd Party
 Data Sources Results Cache Fraud   Detection Query Only Query Only
  15. MongoDB Connector for Hadoop

  16. ‹#› Connector Overview DATA • Read/Write MongoDB • Read/Write BSON


    TOOLS • MapReduce • Pig • Hive • Spark
 PLATFORMS • Apache Hadoop • Cloudera CDH • Hortonworks HDP • MapR • Amazon EMR

  17. ‹#› Connector Features and Functionality • Computes splits to read

    data • Single Node, Replica Sets, Sharded Clusters • Mappings for Pig and Hive • MongoDB as a standard data source/destination • Support for • Filtering data with MongoDB queries • Authentication • Reading from Replica Set tags • Appending to existing collections
  18. ‹#› MapReduce Configuration • MongoDB input/output mongo.job.input.format = com.mongodb.hadoop.MongoInputFormat mongo.input.uri

    = mongodb://mydb:27017/db1.collection1 mongo.job.output.format = com.mongodb.hadoop.MongoOutputFormat mongo.output.uri = mongodb://mydb:27017/db1.collection2 • BSON input/output mongo.job.input.format = com.hadoop.BSONFileInputFormat mapred.input.dir = hdfs:///tmp/database.bson mongo.job.output.format = com.hadoop.BSONFileOutputFormat mapred.output.dir = hdfs:///tmp/output.bson
  19. ‹#› Pig Mappings • Input: BSONLoader and MongoLoader data =

    LOAD ‘mongodb://mydb:27017/db.collection’ 
 using com.mongodb.hadoop.pig.MongoLoader • Output: BSONStorage and MongoInsertStorage STORE records INTO ‘hdfs:///output.bson’
 using com.mongodb.hadoop.pig.BSONStorage
  20. ‹#› Hive Support • Access collections as Hive tables •

    Use with MongoStorageHandler or BSONStorageHandler CREATE TABLE mongo_users (id int, name string, age int)
 STORED BY "com.mongodb.hadoop.hive.MongoStorageHandler"
 WITH SERDEPROPERTIES("mongo.columns.mapping” = "_id,name,age”) TBLPROPERTIES("mongo.uri" = "mongodb://host:27017/test.users”)
  21. ‹#› Spark • Use with MapReduce input/output formats • Create

    Configuration objects with input/output formats and data URI • Load/save data using SparkContext Hadoop file API
  22. ‹#› Data Movement Dynamic  queries  to  MongoDB  vs.  BSON  snapshots

     in  HDFS Dynamic queries with most recent data Puts load on operational database Snapshots move load to Hadoop Snapshots add predictable load to MongoDB
  23. Demo : Recommendation Platform

  24. ‹#› Movie Web

  25. ‹#› MovieWeb Web Application • Browse - Top movies by

    ratings count - Top genres by movie count • Log in to - See My Ratings - Rate movies • Recommendations - Movies You May Like - Recommendations
  26. ‹#› MovieWeb Components • MovieLens dataset – 10M ratings, 10K

    movies, 70K users – http://grouplens.org/datasets/movielens/ • Python web app to browse movies, recommendations – Flask, PyMongo • Spark app computes recommendations – MLLib collaborative filter • Predicted ratings are exposed in web app – New predictions collection
  27. ‹#› Spark Recommender • Apache Hadoop (2.3) - HDFS &

    YARN - Top genres by movie count • Spark (1.0) - Execute within YARN - Assign executor resources • Data - From HDFS, MongoDB - To MongoDB
  28. ‹#› MovieWeb Workflow Snapshot db as BSON Predict ratings for

    all pairings Write Prediction to MongoDB collection Store BSON 
 in HDFS Read BSON into Spark App Create user movie pairing Web Application exposes recommendations Repeat Process Train Model from existing ratings
  29. ‹#› Execution $ spark-submit --master local \
 --driver-memory 2G --executor-memory

    2G \
 --jars mongo-hadoop-core.jar,mongo-java-driver.jar \
 --class com.mongodb.workshop.SparkExercise \
 ./target/spark-1.0-SNAPSHOT.jar \
 hdfs://localhost:9000 \
 mongodb://127.0.0.1:27017/movielens \
 predictions \
  30. Should I use MongoDB or Hadoop?

  31. ‹#› Business First! First-Level Analytics Internet of Things Social Mobile

    Apps Product/ Asset Catalog Security & Fraud Single View Customer Data Manageme Churn Analysis Risk Modeling Sentiment Analysis Trade Surveillanc e Recommen der Warehouse & ETL Ad Targeting Predictive Analytics What/Why How
  32. ‹#› The good tool for the task • Dataset size

    • Data processing complexity • Continuous improvement V1.0
  33. ‹#› The good tool for the task • Dataset size

    • Data processing complexity • Continuous improvement V2.0
  34. ‹#› Resources / Questions • MongoDB Connector for Hadoop -

    http://github.com/mongodb/mongo-hadoop
 • Getting Started with MongoDB and Hadoop - http://docs.mongodb.org/ecosystem/tutorial/getting- started-with-hadoop/
 • MongoDB-Spark Demo - https://github.com/crcsmnky/mongodb-hadoop- workshop
  35. MongoDB & Hadoop Tugdual Grall Technical Evangelist tug@mongodb.com @tgrall