Optimizing your GCP Setup for Scale

Tips on how to optimize your GCP setup for scale and keeping the spendings on a minimum.

Transcript

2. Optimizing your GCP setup for Scale Harshit Dwivedi : Co Founder, Roobits
3. What this Talk is About
4. A brief intro about me
5. Cloud is a better (and cheaper) way to run your business
6. Format of the talk 1. Product 2. Tips to effectively scale it 3. Tips for reducing the pricing incurred
7. App Engine
8. App Engine • App Engine is a fully managed serverless platform that scales with your users.
9. App Engine • App Engine is a fully managed serverless platform that scales with your users. • Supports programming in a variety of languages like Kotlin, Java, GoLang, Nodejs, Python and others.
10. Scaling App Engine 1. Reduce the frequency of health checks
11. Scaling App Engine 1. Reduce the frequency of health checks 2. Standard environment scales to 0 instances, so only use it when you have period of inactivity.
12. Scaling App Engine 1. Reduce the frequency of health checks 2. Standard environment scales to 0 instances, so only use it when you have period of inactivity. 3. Tweak the auto scaling parameters to ensure you are utilizing your existing instances efficiently.
13. Scaling App Engine 1. Reduce the frequency of health checks 2. Standard environment scales to 0 instances, so only use it when you have period of inactivity. 3. Tweak the auto scaling parameters to ensure you are utilizing your existing instances efficiently 4. Keep the AppEngine in the same region as your other GCP services .
14. Reducing the Pricing 1. Disable unwanted stackdriver logs
15. Reducing the Pricing 1. Disable unwanted stackdriver logs 2. Use multiple smaller sized instances instead of a single larger instance
16. Reducing the Pricing 1. Disable unwanted stackdriver logs 2. Use multiple smaller sized instances instead of a single larger instance 3. Network Egress in App Engine is costly, only send what you really want to!
17. Dataflow
18. Dataflow • Dataflow is a Batch and Stream event processing pipeline
19. Dataflow • Dataflow is a Batch and Stream event processing pipeline • Allows you to read data from an input source and modify that data at scale.
20. Dataflow • Dataflow is a Batch and Stream event processing pipeline • Allows you to read data from an input source and modify that data at scale. • Built on Open Sourced Apache Beam, so the SDK can be tweaked accordingly
21. Dataflow • Dataflow is a Batch and Stream event processing pipeline • Allows you to read data from an input source and modify that data at scale. • Built on Open Sourced Apache Beam, so the SDK can be tweaked accordingly • Works with Java, Python, Go and Kotlin.
22. Scaling Dataflow 1. Identify your use case and choose an appropriate machine
23. Scaling Dataflow 1. Identify your use case and use an appropriate machine 2. Use a SSD enabled pipeline if your use case involves extensive Disk I/O
24. Scaling Dataflow 1. Identify your use case and use an appropriate machine 2. Use a SSD enabled pipeline if your use case involves extensive Disk I/O 3. Add Deduplication support when using with PubSub
25. Reducing the Pricing 1. Reduce the disk size
26. Reducing the Pricing 1. Reduce the disk size 2. Specify a custom machine type if the prebuilt ones are not specific enough
27. Reducing the Pricing 1. Reduce the disk size 2. Specify a custom machine type if the prebuilt ones are not specific enough 3. Disable public IPs if you don’t want your pipeline data made available to your users
28. Reducing the Pricing 1. Reduce the disk size 2. Specify a custom machine type if the prebuilt ones are not specific enough 3. Disable public IPs if you don’t want your pipeline data made available to your users 4. Enable Dataflow Streaming engine
29. BigQuery
30. BigQuery • BigQuery is a scalable data storage warehouse
31. BigQuery • BigQuery is a scalable data storage warehouse • Backed by SQL, it allows you to perform complex manipulations to you stored data
32. BigQuery • BigQuery is a scalable data storage warehouse • Backed by SQL, it allows you to perform complex manipulations to you data • Has client libraries available in most commonly used languages to access the stored data
33. Scaling BigQuery
34. Reducing the Pricing 1. Partition your table based on Date or the number of columns
35. Reducing the Pricing 1. Partition your table based on Date or the number of columns 2. Use File Loads instead of Streaming data into your table
36. Reducing the Pricing 1. Partition your table based on Date or the number of columns 2. Use File Loads instead of Streaming data into your table 3. Use aggregation for columns if you need to access those columns frequently
37. Reducing the Pricing 1. Partition your table based on Date or the number of columns 2. Use File Loads instead of Streaming data into your table 3. Use aggregation for columns if you need to access those columns frequently 4. Be judicious with your queries
38. Reducing the Pricing 1. Partition your table based on Date or the number of columns 2. Use File Loads instead of Streaming data into your table 3. Use aggregation for columns if you need to access those columns frequently 4. Be judicious with your queries 5. In a partitioned table, set the partition name to “NULL” to query through the data from the past hour
39. GCP at Roobits
40. About
41. Some Numbers More than 500 mil daily request tracked and stored
42. Some Numbers Average latency of 1.5 seconds
43. Services Used App Engine, PubSub, Dataflow, BigQuery and GCS
44. Some Numbers Following the tips above, we cut down spendings on GCP by over 70% while maintaining the scale
45. For more info https://roobits.com
46. Thanks, Questions!? Twitter: twitter.com/harshithdwivedi Medium: medium.com/@harshithdwivedi Email: harshit@roobits.com