Big Data and Google Cloud Platform

An introduction to the services in Google Cloud Platform for use with Big Data processing including BigQuery, Google Cloud Storage and Google Compute Engine for running Hadoop.

Transcript

1. Sharif Salah Google Developer Expert 2014 Google+ profile @sharif_salah about.me/sharif.salah Big Data and Google Cloud Platform
2. Image courtesy of Google/Connie Zhou
3. Agenda Research at Google Cloud Platform Big Data overview Resources Q&A
4. Research at Google
5. GFS Map Reduce Bigtable Dremel Spanner Colossus https://research.google.com/
6. “Bigtable: A Distributed Storage System for Structured Data Fay Chang, Jeffrey Dean, Sanjay Ghemawat, Wilson C. Hsieh, Deborah A. Wallach, Mike Burrows, Tushar Chandra, Andrew Fikes, and Robert E. Gruber Abstract Bigtable is a distributed storage system for managing structured data that is designed to scale to a very large size: petabytes of data across thousands of commodity servers. Many projects at Google store data in Bigtable, including web indexing, Google Earth, and Google Finance. These applications place very different demands on Bigtable, both in terms of data size (from URLs to web pages to satellite imagery) and latency requirements (from backend bulk processing to real-time data serving). Despite these varied demands, Bigtable has successfully provided a flexible, high-performance solution for all of these Google products. In this paper we describe the simple data model provided by Bigtable, which gives clients dynamic control over data layout and format, and we describe the design and implementation of Bigtable.” To appear in: OSDI'06: Seventh Symposium on Operating System Design and Implementation, Seattle, WA, November, 2006. Source: http://research.google.com/archive/bigtable.html
7. IaaS PaaS SaaS Gmail, Google Drive and Google Calendar App Engine Compute Engine Cloud Storage
8. Cloud Platform overview
9. Compute Compute Engine App Engine
10. Compute Engine IaaS Compute, Storage & Network Sub-hour billing Hugely scalable Consistent performance Support for containers
11. Storage Cloud Datastore Cloud Storage Cloud SQL
12. Cloud Datastore NoSQL-like Fully managed Highly available Hugely scalable Flexible access Auto scaling
13. Cloud Storage IaaS Multi-petabyte Globally replicated Tightly integrated into GCP Flexible security model
14. Services Cloud Endpoints BigQuery
15. BigQuery SQL-like queries Easy to use interface Supports 3rd party tools Analyse terabytes of data Streaming data Batch processing
16. Get started with BigQuery
17. Cloud Console Cloud SDK REST APIs
18. BigQuery Demo
19. Resources
20. Google Developer Experts directory Google Cloud Platform Blog Official Google Cloud Platform Community Stack Overflow
22. Case studies Compliance information Pricing calculator Research at Google Support packages Further reading relating to Google Cloud Platform
23. Dremel Getting started with BigQuery An inside look at BigQuery Cloud Dataflow Further reading relating to BigQuery
24. Google Cloud Platform would like to offer you $500 of credit to build your applications! To get started, follow the three steps below: 1. http://g.co/cloudstarterpack 2. Click "apply now" 3. Use the code: gde-in
25. These slides: http://goo.gl/GOPeyN Credits: use code ‘gde-in’ http://g.co/cloudstarterpack Thank you! Any questions?