Upgrade to Pro — share decks privately, control downloads, hide ads and more …

From PHP to Golang: Migrating a real-time data replication service

Eric Khun
September 01, 2020

From PHP to Golang: Migrating a real-time data replication service

Eric Khun

September 01, 2020
Tweet

More Decks by Eric Khun

Other Decks in Programming

Transcript

  1. Migrating real-time data
    replication service
    from PHP to Golang
    @Maicon
    @eric_khun

    View Slide

  2. Buffer

    View Slide

  3. Buffer
    • 80 awesome people , 12 timezones, all remote
    • USD ~20 millions Annual Recurrent Revenue
    • 75,000+ paying customers ~7 millions users

    View Slide

  4. View Slide

  5. Quick intro
    Infrastructure engineer
    buffer.com
    @eric_khun

    View Slide

  6. View Slide

  7. View Slide

  8. Infrastructure metrics
    • 1 Kubernetes cluster | ~60 ec2 instances
    • ~800,000 “posts” sent per day to social medias
    • 1000 “external” requests per second
    • 1400 AWS SQS queue messages to process per
    second

    View Slide

  9. Search - auto completion

    View Slide

  10. Identifying 3rd party API errors
    (FB/Instagram/Twitter/Linkedin…)

    View Slide

  11. A system to search users /
    profile / updates
    • MongoDB isn’t great for search / auto completion
    • Elasticsearch is better for that
    • We use SQS to say we need a message to copy
    data from mongoDB to Elasticsearch

    View Slide

  12. Data pipeline Flow

    View Slide

  13. Migrating real-time data replication worker
    From PHP to Golang
    WHY ?

    View Slide

  14. Number of SQS message to
    process

    View Slide

  15. View Slide

  16. Number of pods
    And resources used

    View Slide

  17. Reasons to migrate
    • Too frequent paging alerts (using engineering time)
    • Using 37 CPU cores (8 servers! ), and growing
    • PHP libraries memory leaks (not PHP!)
    • Unstable resource usage , pods restarts (OOMKill)
    • Only used for customer support or troubleshooting
    - cost not justified

    View Slide

  18. View Slide

  19. View Slide

  20. View Slide

  21. Benefits
    * No Price: Happier engineers, no more context switching

    View Slide

  22. Questions?
    @eric_khun
    (golang)
    (hugo)
    (python) ✈

    View Slide