connection pairings managed by ZooKeeper Puppet StatD Monit Ganglia API App (Python) Web App (Python) Task Processing (Python/Pyres) MySQL Service (Java/Finagle) Memcache Mux (Nutcracker) Feed Service (Python/Thrift) Follower Service (Python/Thrift) Images (S3 + CDN) MySQL Memcache Redis HBase Task Queue (Redis) Follower Service (Python/Thrift) Follower Service (Python/Thrift) Saturday, August 31, 13
• Does it meet your needs? • How mature is the product? • Is it commonly used? Can you hire people who have used it? • Is the community active? • How robust is it to failure? • How well does it scale? Will you be the biggest user? • Does it have a good debugging tools? Profiler? Backup software? • Is the cost justified? Saturday, August 31, 13
Variety of servers running Linux • Very good peripherals, such as load balancing, DNS, map reduce, basic firewalls, and more • Good reliability (don’t throw tomatoes at me!) • Very active dev community • Not cheap, but... • New instances ready in seconds Saturday, August 31, 13
• Well known and well liked • Solid active community • Very good libraries specifically targeted to web development • Effective rapid prototyping • Free Saturday, August 31, 13
• Extremely mature • Well known and well liked • Rarely catastrophic loss of data • Response time to request rate increases linearly • Very good software support: XtraBackup, Innotop, Maatkit • Solid active community • Free Saturday, August 31, 13
well liked • Active community • Consistently good performance • Variety of convenient and efficient data structures • 3 Flavors of Persistence: Now, Snapshot, Never • Free Production Data Saturday, August 31, 13
Efficient Storage • Handle large write throughput • Solid Hadoop interface • Maturing quickly, used by facebook • Built on HDFS • Free Production Data Saturday, August 31, 13
and Membase? Production Data • Does it meet your needs? • How mature is the product? • Is it commonly used? Can you hire people who have used it? • Is the community active? • How robust is it to failure? • How well does it scale? Will you be the biggest user? • Does it have a good debugging tools? Profiler? Backup software? • Is the cost justified? Saturday, August 31, 13
Engineers = Deploy Hell • Major bugs and performance issues stall deploys • Performance issues creep in under radar • 7+ development teams, 1 ops team • Workload changing more rapidly and less predictably • Want developers to not fear moving fast Employee Growth Saturday, August 31, 13
tests (careful! don’t erase your DB!) • Rings of deployment • Canary, employees only, 5% of user base, etc. • Continuous deployment • Production integration tests Employee Growth Saturday, August 31, 13
Push for better uptime and lower latency • Initially, most uptime and latency issues due to DB + caching • Fewer Instances => Few, but big failures • More Instances => More smaller failures + more complexity • How aggressively can you retry without hurting the system? Uptime & Latency Saturday, August 31, 13
configuration management • Used for service discovery • Notifies of service failures • WATCH and its callback are pretty reliable • Experiment framework Saturday, August 31, 13
Solution: Zookeeper • Centralized configuration management • Used for service discovery • Notifies of service failures • WATCH and its callback are pretty reliable • Experiment framework Saturday, August 31, 13
WATCH Solution: Zookeeper • Centralized configuration management • Used for service discovery • Notifies of service failures • WATCH and its callback are pretty reliable • Experiment framework Saturday, August 31, 13
Initially, entire app tier connected to all Memcache, Redis, MySQL • On Memcache... • 20k connections * 10kB / connection = 195MB / Memcache • 40 Memcaches means 7.6 GB used on connections • Connection space is not allocated from slab memory! • Can eventually cause Memcache process to leak into swap • On MySQL • At least 256 kB / connection Connections Saturday, August 31, 13
Data Services, Nutcracker • When? Once any service gets close to 10k connections • Success: Memcache • Once was >20k connections • Now 1.3k connections • But, aggressive fan-out causes... • Network contention • Incast congestion Connections Saturday, August 31, 13
• Twitter • Completely asynchronous • Previous experience with Finagle • Lots of compatible libraries • JVM • Lots of bells and whistles - Ostrich, Zipkin, lago Why Java Over Python? Saturday, August 31, 13
Local ID Type · A lookup data structure has physical server to shard ID range (cached by each app server process) · Shard ID denotes which shard · Type denotes object type (e.g., pins) · Local ID denotes position in table Saturday, August 31, 13
pin, board, user, comment) · Local ID MySQL blob (JSON / Serialized thrift) Mapping tables (e.g., user has boards, pin has likes) · Full ID Full ID (+ timestamp) · Naming schema is noun_verb_noun Queries are PK or index lookups (no joins) · Data DOES NOT MOVE All tables exist on all shards No schema changes required (index = new table) Saturday, August 31, 13
experience • Better uptime and lower latency • Help Pinners discover more of the things they love • Reduce spam and abuse • Continually collaborate and build bigger, better, faster products • 140 Pinployees and beyond • MySQL 5.6 What’s next? Saturday, August 31, 13
yashh
usanchuu
hironobuiga
ry
findy_eventslides
mocobeta
line_developers_tw
whisaiyo
chanyou0311
iwamot
rymiyamoto
iwashi86
davetheseo
axbom
kristinabergwall1
rasmusluckow
scottboms
bkeepers
erikaheidi
inesmontani
samanthasiow
addyosmani
baasie
![[email protected] pinterest.com/yashh Saturday, August 31, 13](https://files.speakerdeck.com/presentations/45f82fd0f48f0130a4081ec64b4e9d67/slide_63.jpg){kind=link}
![[email protected] pinterest.com/yashh Saturday, August 31, 13](https://files.speakerdeck.com/presentations/45f82fd0f48f0130a4081ec64b4e9d67/slide_64.jpg){kind=link}