Upgrade to Pro — share decks privately, control downloads, hide ads and more …

From PHP to Golang: Migrating a real-time data replication service

3528f569ab96088a8a2b00b3c33de599?s=47 Eric Khun
September 01, 2020

From PHP to Golang: Migrating a real-time data replication service

3528f569ab96088a8a2b00b3c33de599?s=128

Eric Khun

September 01, 2020
Tweet

Transcript

  1. Migrating real-time data replication service from PHP to Golang @Maicon

    @eric_khun
  2. Buffer

  3. Buffer • 80 awesome people , 12 timezones, all remote

    • USD ~20 millions Annual Recurrent Revenue • 75,000+ paying customers ~7 millions users
  4. None
  5. Quick intro Infrastructure engineer buffer.com @eric_khun

  6. None
  7. None
  8. Infrastructure metrics • 1 Kubernetes cluster | ~60 ec2 instances

    • ~800,000 “posts” sent per day to social medias • 1000 “external” requests per second • 1400 AWS SQS queue messages to process per second
  9. Search - auto completion

  10. Identifying 3rd party API errors (FB/Instagram/Twitter/Linkedin…)

  11. A system to search users / profile / updates •

    MongoDB isn’t great for search / auto completion • Elasticsearch is better for that • We use SQS to say we need a message to copy data from mongoDB to Elasticsearch
  12. Data pipeline Flow

  13. Migrating real-time data replication worker From PHP to Golang WHY

    ?
  14. Number of SQS message to process

  15. None
  16. Number of pods And resources used

  17. Reasons to migrate • Too frequent paging alerts (using engineering

    time) • Using 37 CPU cores (8 servers! ), and growing • PHP libraries memory leaks (not PHP!) • Unstable resource usage , pods restarts (OOMKill) • Only used for customer support or troubleshooting - cost not justified
  18. None
  19. None
  20. None
  21. Benefits * No Price: Happier engineers, no more context switching

  22. Questions? @eric_khun (golang) (hugo) (python) ✈