[Dmytro Timofeev] BigTable as a data source for massively scalable computations

Presentation from GDG DevFest Ukraine 2017 - the biggest community-driven Google tech conference in the CEE.

Learn more at: https://devfest.gdg.org.ua

Transcript

1. Proprietary + Confidential Google Cloud Bigtable: petabyte-scale NoSQL database for massively scalable computation Dima Timofeev Tools Specialist Google
2. Proprietary + Confidential Agenda Introduction Cloud Bigtable Up and running Processing data Q&A
3. Proprietary + Confidential Introduction
4. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem Motivation Variety of structured/semi-structured data • URL content and associated metadata (contents, crawl metadata, links, anchors, pagerank, etc.) • Geographic locations (physical entities: shops, restaurants, etc.; roads, satellite image data, user annotations, etc.) Scalable system for dealing with high volume and high velocity • Concurrent support for services supporting billions of users • Petabyte-scale data size
5. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem What is Bigtable? • NoSQL database • Large datasets • High throughput
6. Proprietary + Confidential Google Cloud Bigtable
7. Cloud Bigtable is the same service Google uses
8. How does Cloud Bigtable work? Clients Processing Storage Colossus file system Bigtable node Bigtable node Bigtable node
9. Cloud Bigtable learns access patterns... Clients Processing Storage Colossus file system Bigtable node Bigtable node Bigtable node A B C D E
10. … and rebalances data accordingly Clients Processing Storage Colossus file system Bigtable node Bigtable node Bigtable node A B C D E
11. Throughput can be controlled by node count Node Node Node Nodes 80,000 60,000 40,000 20,000 QPS Bigtable Nodes 6 4 2 0 0
12. Throughput can be controlled by node count 400,000 300,000 200,000 100,000 QPS Bigtable Nodes 20 10 0 0 Nodes Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node
13. Throughput can be controlled by node count 4,000,000 3,000,000 2,000,000 1,000,000 QPS Bigtable Nodes 400 300 200 100 0 0 Nodes Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node Node
14. Proprietary + Confidential Up and running
15. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem Entry point https://cloud.google.com/bigtable/
16. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem Creating a cluster
17. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem Libraries • Java client — compatible with Apache HBase API • Go client • Python client • cbt command-line tool
18. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem Python connection example from google.cloud import bigtable client = bigtable.Client(project=project_id, admin=True) instance = client.instance(instance_id) table = instance.table(table_id) table.create() column_family_id = 'cf1' cf1 = table.column_family(column_family_id) cf1.create()
19. Proprietary + Confidential Processing data
20. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem Writing / reading
21. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem Cloud Bigtable and Cloud Dataflow https://cloud.google.com/bigtable/docs/dataflow-hbase
22. Source: Lorem ipsum dolor sit amet, consectetur adipiscing elit. Duis non erat sem Cloud Bigtable and Cloud Dataproc
23. THANK YOU