Google's Production Environment

Google datacenters are very different from most conventional datacenters and small-scale server farms. These differences present both extra problems and opportunities. This talk discusses the challenges and opportunities that characterize Google datacenters. It is based on the Google SRE book: https://landing.google.com/sre/sre-book/chapters/production-environment/

Transcript

1. #GoogleSandbox Google's Production Environment Florian Rathgeber, Site Reliability Engineer, Google Cloud
2. #GoogleSandbox Florian Site Reliability Engineer Google Cloud SRE for 2+ years • On the Cloud Console SRE team • Spend most of my time on SLOs Previous life • Computational Scientist @ Imperial College • Data Engineer @ ECMWF Co-founded PyData London
3. #GoogleSandbox Google's Globe Spanning Network Submarine cable investments Current fiber network https://cloud.google.com/about/locations/
4. #GoogleSandbox • B4 • Edge Network • GSLB • Jupiter Google Data Centers • GFE Current Regions & Number of Zones Future Regions & Number of Zones https://cloud.google.com/about/locations/
5. #GoogleSandbox • Campus • Data center • Cluster • Row • Rack • Machine Data Center Setup
6. #GoogleSandbox Scheduler BorgMaster Persistent store Cluster Config Files Tools Borglet Borglet Borglet BNS addresses: /bns/<cluster>/<user>/<job name>/<task number> Cluster Management
7. #GoogleSandbox Chubby Consistent data, e.g. - BNS paths->IP addresses - master election Chubby Chubby Paxos Paxos Cluster Cluster Cluster Lock Service
8. #GoogleSandbox D HDD SSD Colossus Bigtable Spanner ... ... Cluster Spanner Cluster ... ... ... Storage
9. #GoogleSandbox Server Scraping Borgmon Cluster Borgmon Cluster Scraping Borgmon Cluster Borgmon Cluster Scraping Borgmon Cluster Borgmon Global Borgmon Cluster Time Series Database Alert Manager 1 Prober Server Prober Server Prober Data Alerts Monitoring
10. #GoogleSandbox Service Client Client Stubby Server Stubby Stub Stubby Stub C++ Java Ruby protobuf request protobuf response protobuf request protobuf response Server Communication
11. #GoogleSandbox Piper Code Repository Author Changelist Reviewer Looks Good To Me Owner Approval Presubmit Checks OK! submit...done! change Code Repository
12. #GoogleSandbox MPM Piper Code Repository Blaze Continuous Testing Framework Binaries Tests PASS FAIL PASS ... Rapid Sisyphus Production Continuous Build and Deployment
13. #GoogleSandbox Tying it all together... • Develop the software: Piper, Blaze • Build the MPMs: Rapid • Run it in a cluster: Borg, which uses Chubby • Route requests/responses: GFE, GSLB, ProtoBuf, Stubby • Store and read messages: Colossus, Bigtable, Spanner • Monitor and fire alerts: Borgmon • Roll out new versions: Sisyphus
14. #GoogleSandbox • Cluster management: Kubernetes kubernetes.io • Lock service: ZooKeeper zookeeper.apache.org, etcd coreos.com/etcd • Storage: HDFS hadoop.apache.org, Cassandra cassandra.apache.org • Monitoring: Prometheus prometheus.io • RPC: gRPC grpc.io • Data serialization: Protocol Buffers developers.google.com/protocol-buffers • Google style guides github.com/google/styleguide • The Go programming language golang.org • Code repository: Git git-scm.com • Code review: Rietveld github.com/rietveld-codereview/rietveld • Building: Bazel bazel.io List of related open-source projects
15. #GoogleSandbox Cover images used with permission. These books can be found on shop.oreilly.com The full text of the Google SRE Books are available at www.google.com/sre