Polyglot Persistence at Parse

from data day 2014 in Austin, Texas

Transcript

1. Polyglot Persistence at Parse Charity Majors @mipsytipsy Sunday, January 12, 14
2. Architecture What is Parse? Databases Questions ? Sunday, January 12, 14
3. •Ops-oriented case study •Overview of architecture diagram and products •Deeper dive into each backend • some challenges and success stories • scaling tips and performance tuning details Case study on Parse Sunday, January 12, 14
4. What is Parse? Sunday, January 12, 14
5. What is Parse? Platform for mobile developers •iOS, Android, WinRT •API and native SDKs •Scales automatically to handle traffic •Analytics, cloud code, file storage, push notifications, hosting Sunday, January 12, 14
6. Sunday, January 12, 14
7. Parse is magic Sunday, January 12, 14
8. Sunday, January 12, 14
9. Parse has never touched bare metal Elasticity is key Sunday, January 12, 14
10. • Developers • Apps • API usage • Nodes and compute resources • Connected devices Parse is growing fast Sunday, January 12, 14
11. Developers 06 ‘11 07 ‘11 08 ‘11 09 ‘11 10 ‘11 11 ‘11 12 ‘11 01 ’12 02 ’12 03 ’12 04 ’12 05 ’12 06 ’12 07 ’12 08 ’12 09 ’12 10 ’12 11 ’12 12 ’12 01 ’13 02 ’13 03 ’13 04 ’13 05 ’13 06 ’13 07 ’13 08 ’13 09 ’13 10 ’13 11 ’13 12 ’13 When PARSE was acquired Sunday, January 12, 14
12. Parse node count last year Sunday, January 12, 14
13. Parse compute units last year Sunday, January 12, 14
14. Active PPNS connections last year Sunday, January 12, 14
15. Sunday, January 12, 14
16. Parse Stack Sunday, January 12, 14
17. Sunday, January 12, 14
18. Infrastructure design choices • Chef • Route53 • Use real hostnames • Distribute evenly across 3 AZs • Fail over automatically • Single source of truth • All inbound traffic through ELBs • Autoscaling groups Sunday, January 12, 14
19. Sunday, January 12, 14
20. API path • ELB • nginx • haproxy • Ruby app servers (unicorns) • Go api servers (go rewrite from the ground up) • Go logging servers to FB endpoint Sunday, January 12, 14
21. Sunday, January 12, 14
22. Hosting • ELB • Elastic IPs for apex domain redirect service • Go service that wraps cloud code and S3 Sunday, January 12, 14
23. Sunday, January 12, 14
24. Cloud code • Server-side javascript in v8 virtual machine • Third-party modules for partners (Stripe, Twilio, etc) • Restrictive security groups • Scrub IPs with squid Sunday, January 12, 14
25. Push • Resque on redis • Billions of pushes per month • 700/sec steady state • Spikes to 10k/sec (15x burst) • PPNS holds sockets open to all android devices • PDNS to serve android phone-home IPs (only place we run our own DNS) Sunday, January 12, 14
26. Databases Sunday, January 12, 14
27. Sunday, January 12, 14
28. MongoDB Sunday, January 12, 14
29. MongoDB summary • 20 replica sets, ~70 nodes, 2-6 TB per rs • Running 2.4.6 - 2.4.8 (current) • Over 1M collections • Over 200k schemas • Used for structured mobile app data, real-time query analysis, backend billing & analytics, central routing • Fully instrumented provisioning with chef Sunday, January 12, 14
30. MongoDB hardware • m2.4xlarge • 68 gigs of RAM, 26 virtual cpus • 3 nodes + 2 arbiters • one node set to priority = 0 and hidden = true for snapshots • PIOPS (striped RAID, 4000-16000 PIOPS/vol) • totally saved our bacon. EBS was a killer. • i2.4xlarge • 122 gb RAM, 53 virtual cpus • 4x800 gb SSD striped Sunday, January 12, 14
31. MongoDB tuning • blockdev --setra • warm up your working set before an election • monitor page faults • continuous compaction on snapshot nodes • perf goes off a cliff after an initial sync (for us -- not most people!) Sunday, January 12, 14
32. Latency before/after compaction Sunday, January 12, 14
33. MongoDB sharding • Implemented our own app-level sharding • Mongo sharding works ok for small # of large collections • We use a separate mongo replset for routing apps -> shards • Apps are local to a replset • lets us do things like balance performance • Transfer unused apps to graveyard Sunday, January 12, 14
34. MongoDB indexing • Every query should have an index • Auto-indexing of the keys based on entropy of values • Analyze real query traffic to generate intelligent compound indexes Sunday, January 12, 14
35. MongoDB noc • Latency, counts and errors for each API query • Aggregated by api key and shard Sunday, January 12, 14
36. MongoDB logs & analytics • Logs uploaded to S3 hourly from nodes • pulled down by ruby workers, aggregated by app id & time • events written back to mongo Sunday, January 12, 14
37. Hive analytics pipeline • parallel pipeline to replace our mongo logs • Each log event tagged with unique id • Custom service written in go • Interfaces with Facebook’s Scribe endpoint • Scribe dumps data to daily Hive table partitions • Run aggregations on hive • Load per-app aggregated data into mongo on AWS • 4k msg/sec steady, 10k msg/sec peak Sunday, January 12, 14
38. MongoDB closing thoughts • Mongo is still a young database • Flexible and powerful • Elections work • MongoDB is not magic. You still have to understand your data store! Sunday, January 12, 14
39. Sunday, January 12, 14
40. Cassandra • Powers the client side Parse analytics • Super fast writes and increments • We use lots of counters • 90k writes/sec • 25k reads/sec • Hashed by app id Sunday, January 12, 14
41. Cassandra hardware • 24 node ring • c3.4xlarges • 55 cpu units • ephemeral storage • 2x160gb ssd • never i/o bound, always cpu bound Sunday, January 12, 14
42. Cassandra + Priam • Initial token assignments • Incremental backups to S3 • Uses Auto Scaling Groups • SimpleDB for tracking tokens, instance identities • Non-trivial to set up but WORTH IT Sunday, January 12, 14
43. Cassandra perf stuff • increase sstable size -- default was 5mb! • log GC pauses • compaction strategy • restart every couple weeks to reduce heap contention • we are still on 1.1 so a lot of our issues are solved later Sunday, January 12, 14
44. Sunday, January 12, 14
45. Redis • Queueing using resque • Android outboxes • Production utilities Sunday, January 12, 14
46. Redis stats & tuning • Sets of 3 m2.4xlarge instances • ephemeral storage • Currently ~20k ops/sec, down from max ~40k ops/sec • Always disable flush to disk on primary • Use RPS to distribute eth0 software interrupts across multiple cores • watch cat /proc/interrupts, cat /proc/softirqs Sunday, January 12, 14
47. Redis • Limited applications but rock solid at what it does • Single threaded (all that shiny wasted cpu) • Problematic automatic failover schemes • but twemproxy looks rad Sunday, January 12, 14
48. Sunday, January 12, 14
49. MySQL • Trivially tiny and we would love to get rid of it • ... but rails • Considered RDS • No chained replication • Visibility is challenging • Even tiny periodic blips impact the API • ... but AZ failover would be sooo nice Sunday, January 12, 14
50. Zookeeper Sunday, January 12, 14
51. Zookeeper • Service registration and discovery • Distributed lock coordination • Single consistent source of truth about your infrastructure • (there are up and coming RAFT alternatives but none of them are production ready) Sunday, January 12, 14
52. Sunday, January 12, 14
53. Questions? Sunday, January 12, 14