MongoDB Europe 2016: Using Beam and BigQuery with MongoDB

Transcript

1. Warehousing MongoDB Data Using Apache Beam and BigQuery Sandeep Parikh Head of Solutions Architecture, Americas East @crcsmnky
2. Agenda MongoDB on Google Cloud Platform What is Data Warehousing Tools & Technologies Example Use Case
3. Confidential & Proprietary Google Cloud Platform 3 MongoDB on Google Cloud Platform
4. Google Cloud Platform 4 MongoDB on Google Cloud Platform
5. Google Cloud Platform 5 Manually Deploying MongoDB
6. Google Cloud Platform 6 Google Cloud Launcher
7. Google Cloud Platform 7 MongoDB Cloud Manager
8. Google Cloud Platform 8 MongoDB Cloud Manager How do you automate this?
9. Google Cloud Platform 9 Bootstrapping MongoDB Cloud Manager Deployment Manager Template
10. Google Cloud Platform 10 Cloud Deployment Manager Provision, configure your deployment Configuration as code Declarative approach to configuration Template-driven Supports YAML, Jinja, and Python Use schemas to constrain parameters References control order and dependencies
11. Google Cloud Platform 11 Bootstrapping Cloud Manager Schema, Configuration & Template Posted on Github https://github.com/GoogleCloudPlatform/mongodb-cloud-manager Three Compute Engine instances, each with 500 GB PD-SSD MongoDB Cloud Manager automation agent pre-installed and configured $ gcloud deployment-manager deployments create mongodb-cloud-manager \ --config mongodb-cloud-manager.jinja \ --properties mmsGroupId=MMSGROUPID,mmsApiKey=MMSAPIKEY
12. Confidential & Proprietary Google Cloud Platform 12 What’s a Data Warehouse
13. Data Warehouses are central repositories of integrated data from one or more disparate sources https://en.wikipedia.org/wiki/Data_warehouse
14. Google Cloud Platform 14 Data Warehouse Money Data Data Data Insights Profit!
15. Confidential & Proprietary Google Cloud Platform 15 Tools and Technologies
16. Where: BigQuery
17. Google Cloud Platform 17 BigQuery Complex, Petabyte-scale data warehousing made simple Scales automatically; No setup or admin Foundation for analytics and machine learning
18. Google Cloud Platform 18 RUN QUERY
19. Google Cloud Platform 19
20. How: Apache Beam (incubating)
21. 21 Modern data processing Pipeline-centric approach Batch and streaming, from the same codebase Portable across runtime environments Build pipelines using Java (GA), Python (alpha) Apache Beam
22. Google Cloud Platform 22 Apache Beam Lineage MapReduce BigTable Dremel Colossus Flume Megastore Spanner PubSub Millwheel Apache Beam Google Cloud Dataflow
23. Google Cloud Platform 23 Beam, Modes of Operation 3 Streaming 4 Streaming + Accumulation 1 Classic Batch 2 Windowed Batch
24. Google Cloud Platform 24 Pipelines in Beam Pipeline p = Pipeline.create(); p.begin() .apply(TextIO.Read.from(“gs://…”)) .apply(ParDo.of(new ExtractTags()) .apply(Count.create()) .apply(ParDo.of(new ExpandPrefixes()) .apply(Top.largestPerKey(3)) .apply(TextIO.Write.to(“gs://…”)); p.run(); Pipeline p = Pipeline.create(); p.begin() .apply(TextIO.Read.from(“gs://…”)) .apply(ParDo.of(new ExtractTags()) .apply(Count.create()) .apply(ParDo.of(new ExpandPrefixes()) .apply(Top.largestPerKey(3)) .apply(TextIO.Write.to(“gs://…”)); p.run(); .apply(PubsubIO.Read.from(“input_topic”)) .apply(Window.<Integer>by(FixedWindows.of(5, MINUTES)) .apply(PubsubIO.Write.to(“output_topic”)); Batch to Streaming
25. Google Cloud Platform 25 Apache Beam Vision Beam Model: Fn Runners Apache Flink Apache Spark Beam Model: Pipeline Construction Other Languages Beam Java Beam Python Execution Execution Cloud Dataflow Execution
26. Google Cloud Platform 26 Running Apache Beam Cloud Dataflow Local Runner
27. 27 A great place for executing Beam pipelines which provides: • Fully managed, no-ops execution environment • Integration with Google Cloud Platform • Java support in GA. Python in Alpha Cloud Dataflow Service
28. Deploy Tear Down Fully Managed: Worker Lifecycle Management
29. Fully Managed: Dynamic Worker Scaling
30. 100 mins. 65 mins. vs. Fully Managed: Dynamic Work Rebalancing
31. Integrated: Monitoring UI
32. Integrated: Distributed Logging
33. Cloud Logs Google App Engine Google Analytics Premium Cloud Pub/Sub BigQuery Storage (tables) Cloud Bigtable (NoSQL) Cloud Storage (files) Cloud Dataflow BigQuery Analytics (SQL) Capture Store Analyze Batch Cloud DataStore Process Stream Cloud Monitoring Cloud Bigtable Real time analytics and Alerts Cloud Dataflow Cloud Dataproc Integrated: Google Cloud Platform Cloud Dataproc 33
34. Confidential & Proprietary Google Cloud Platform 34 Example Use Case
35. Google Cloud Platform 35 Sensor Data 1000 Devices Cloud Storage ID, Type, Name 27M Log Entries MongoDB Device ID, Value, Timestamp
36. What’s the average reading per sensor type?
37. Google Cloud Platform 37 Beam + MongoDB Export (JSON,CSV) TextIO.Read MongoClient Find
38. Google Cloud Platform 38 Beam + MongoDB (coming soon!) MongoDbIO.Read
39. Google Cloud Platform 39 Pipeline Execution
40. Google Cloud Platform 40 // Read sensor logs from MongoDB and create PCollection of Documents PCollection<Document> sensorLogs = p.apply(MongoDbIO.read() .withUri("mongodb://" + options.getMongoDBHost() + ":27017") .withDatabase(options.getMongoDBDatabase()) .withCollection(options.getMongoDBCollection())); // Extract "Device ID -> Value" PCollection PCollection<KV<String,Double>> sensorIdValue = sensorLogs .apply("ExtractValues", ParDo.of(new DoFn<Document, KV<String, Double>>() { @ProcessElement public void processElement(ProcessContext c) { String deviceId = c.element().getObjectId("_id").toString(); Double value = c.element().getDouble("v"); c.output(KV.of(deviceId, value)); } })); Using MongoDbIO.Read
41. Google Cloud Platform 41 Transforming Data Document → Device ID, Value CSV → Device ID, Type Type, Value Type, Mean Value Output to BigQuery
42. Questions? Apache Beam http://beam.incubator.apache.org Cloud Dataflow http://cloud.google.com/dataflow BigQuery http://cloud.google.com/bigquery