From PHP to Golang: Migrating a real-time data replication service

Transcript

1. Migrating real-time data replication service from PHP to Golang @Maicon @eric_khun
2. Buffer
3. Buffer • 80 awesome people , 12 timezones, all remote • USD ~20 millions Annual Recurrent Revenue • 75,000+ paying customers ~7 millions users
5. Quick intro Infrastructure engineer buffer.com @eric_khun
8. Infrastructure metrics • 1 Kubernetes cluster | ~60 ec2 instances • ~800,000 “posts” sent per day to social medias • 1000 “external” requests per second • 1400 AWS SQS queue messages to process per second
9. Search - auto completion
10. Identifying 3rd party API errors (FB/Instagram/Twitter/Linkedin…)
11. A system to search users / profile / updates • MongoDB isn’t great for search / auto completion • Elasticsearch is better for that • We use SQS to say we need a message to copy data from mongoDB to Elasticsearch
12. Data pipeline Flow
13. Migrating real-time data replication worker From PHP to Golang WHY ?
14. Number of SQS message to process
16. Number of pods And resources used
17. Reasons to migrate • Too frequent paging alerts (using engineering time) • Using 37 CPU cores (8 servers! ), and growing • PHP libraries memory leaks (not PHP!) • Unstable resource usage , pods restarts (OOMKill) • Only used for customer support or troubleshooting - cost not justified
21. Benefits * No Price: Happier engineers, no more context switching
22. Questions? @eric_khun (golang) (hugo) (python) ✈