Swift at Scale: The IBM SoftLayer Story

Transcript

1. Swift at Scale:
 The IBM SoftLayer Story Brian Cline, Object Storage Development Lead OpenStack Summit • Ocata series 2016.10.25 Barcelona, Spain twitter/irc: @briancline
2. Our History with Public Object Storage • 2012 — First three clusters go live (DAL, AMS, SNG) • 2014 — Dedicated development team • 2014 — Launch 11 clusters in new datacenters • 2015 — Launch 5 clusters in new DCs • 2015 — Product integrations with IBM Bluemix • 2016 — Launch 3 clusters in new DCs
 (and expand an existing cluster into multiple DCs)
3. 2012: When things were [mostly] simpler… • 7-10 nodes in each cluster • Two node types • Proxy • Data - account, container, object services • Load balancer • FreeBSD with ZFS ⚠ Do not attempt. • No centralized logs • No log analysis tools
4. 2016: Adjusted for scale (blood, sweat, tears, dreams, starlight…) • Up to hundreds of nodes per cluster • Three node types • Proxy • Meta - account and container services • Data - object services • Load balancer cluster • Debian Linux • Centralized and searchable logs • Analytics via Spark and Hadoop
5. Our Scale
7. 22 Swift Clusters 24 Datacenters 16 Countries
8. Tens of Billions of Objects 7 Million Containers Hundreds of thousands of Swift Accounts
9. 90 PB of Capacity Thousands of Nodes 40,000+ Disks
10. Tens of thousands requests per second GET HEAD PUT DELETE (with notable variability between clusters)
11. Hardware
12. Hardware we like • Supermicro 36-disk chassis • 12-16 physical cores
 (24-32 HT cores) • 128GB RAM for proxies • 256GB RAM for data nodes • 10Gbps NICs (separate API vs. storage/replication networks) • 3 - 4 TB disks • Controller card • 2 disks for OS (RAID1) • 1 disk for OS hotswap • 4 disks for SSD caching • 29 disks for data storage • Usually expand by ½-row or a
 full row at a time
13. Software
14. Our Stack — Software OS Debian Base Swift (duh) — sometimes with backports Authentication Swauth — some internal patches and enhancements Keystone (APIv3) — starting with Bluemix accounts Metadata Search Elasticsearch Monitoring & Logging collectd, Nagios, Capacity Dashboard Logstash, Kibana, Graphite, Grafana slogging Automation Chef, Jenkins, Fabric
15. Our Stack — Custom Middlewares • CDN operations (purge, load, CNAMEs, TTL, compression, etc.) • CDN origin pull • Search indexer (on successful PUT/POST/DELETE) • Search query operations • Checkpoint (account enable/disable/etc. abilities for resellers) • Internal management (sysmeta read/write, proxy-level recon)
16. Lessons Learned
17. Lessons Learned: Automation • Make automation a must-have, day-one deliverable • Never launch something new without test/deploy automation • Must work across all environments (dev, QA, UAT/staging, prod) • Automation needs tests and metrics, too — it is code! • Functional testing should be an automated part of every deploy • Remember your orchestration (knowledge of Swift zones)
18. Lessons Learned: Monitoring • Scale test any monitoring/logging infrastructure you put into place • Very obvious stuff: • Space and IOPS, errors from SMART/XFS/kernel/controller, etc. • HTTP response code aggregates, latency aggregates by verb, etc. • Swift metrics: • If nothing else, async pendings • Replicator failures and partitions/sec rates • Replicator last completion timestamp vs. ring push timestamp
19. Lessons Learned: Monitoring • Any middleware you create needs to emit ops metrics • New features benefit from emitting usage metrics • Don’t forget debug-level log messages • Automatic checks for precipitating conditions that lead to failures
 (not just for the error log lines that result from them afterwards)
20. Lessons Learned: Rebalancing • Keep tabs on your rebalance times
 (and keep them small when possible) • Coordinate rebalances around node/cluster maintenance • Don’t let IOPS levels grow too high before expanding capacity • Customer IOPS vs. Replicator & Auditor IOPS — know your limits
21. Lessons Learned: Swift • Use 256 byte inode sizes (or the smallest you can get away with) • Using swauth? Use an SSD storage policy for AUTH_.auth containers • Namespace any custom API additions (and be consistent) • When possible, ask community about new middleware thoughts • Upstream is important! Stay involved and give back when possible
22. Thank you! Questions?