From PHP to Golang: Migrating a real-time data replication service

Transcript

1. Migrating real-time data
   replication service
   from PHP to Golang
   @Maicon
   @eric_khun
   

   View Slide

2. Buffer
   

   View Slide

3. Buffer
   • 80 awesome people , 12 timezones, all remote
   • USD ~20 millions Annual Recurrent Revenue
   • 75,000+ paying customers ~7 millions users
   

   View Slide

4. View Slide

5. Quick intro
   Infrastructure engineer
   buffer.com
   @eric_khun
   

   View Slide

6. View Slide

7. View Slide

8. Infrastructure metrics
   • 1 Kubernetes cluster | ~60 ec2 instances
   • ~800,000 “posts” sent per day to social medias
   • 1000 “external” requests per second
   • 1400 AWS SQS queue messages to process per
   second
   

   View Slide

9. Search - auto completion
   

   View Slide

10. Identifying 3rd party API errors
    (FB/Instagram/Twitter/Linkedin…)
    

    View Slide

11. A system to search users /
    profile / updates
    • MongoDB isn’t great for search / auto completion
    • Elasticsearch is better for that
    • We use SQS to say we need a message to copy
    data from mongoDB to Elasticsearch
    

    View Slide

12. Data pipeline Flow
    

    View Slide

13. Migrating real-time data replication worker
    From PHP to Golang
    WHY ?
    

    View Slide

14. Number of SQS message to
    process
    

    View Slide

15. View Slide

16. Number of pods
    And resources used
    

    View Slide

17. Reasons to migrate
    • Too frequent paging alerts (using engineering time)
    • Using 37 CPU cores (8 servers! ), and growing
    • PHP libraries memory leaks (not PHP!)
    • Unstable resource usage , pods restarts (OOMKill)
    • Only used for customer support or troubleshooting
    - cost not justified
    

    View Slide

18. View Slide

19. View Slide

20. View Slide

21. Benefits
    * No Price: Happier engineers, no more context switching
    

    View Slide

22. Questions?
    @eric_khun
    (golang)
    (hugo)
    (python) ✈
    

    View Slide