Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Platform Engineering at Mercari

taichi nakashima
May 15, 2023
2.2k

Platform Engineering at Mercari

Talk at Platform Engineering Meetup #2 https://platformengineering.connpass.com/event/280339/

taichi nakashima

May 15, 2023
Tweet

Transcript

  1. Platform Engineering at Mercari
    Platform Engineering Meetup #2

    View Slide

  2. Taichi Nakashima
    @deeeet
    Director of Platform Engineering

    View Slide

  3. Agenda

    ● Overview of Mercari Developer Platform
    ● Lessons learned

    View Slide

  4. Overview of Mercari Developer Platform

    View Slide

  5. Mission of Platform Engineering at Mercari

    Providing core infrastructure, toolings and workflows for Mercari Group
    developers so that they can deliver new value and experience to Mercari
    customers quickly and reliably.

    View Slide

  6. Mission of Platform Engineering at Mercari

    Providing core infrastructure, toolings and workflows for Mercari Group
    developers so that they can deliver new value and experience to Mercari
    customers quickly and reliably.

    View Slide

  7. Platform

    View Slide

  8. Mission of Platform Engineering at Mercari

    Providing core infrastructure, toolings and workflows for Mercari Group
    developers so that they can deliver new value and experience to Mercari
    customers quickly and reliably.

    View Slide

  9. Build Test Deploy Operate
    Infrastructure Network Data
    Interface
    Documentation
    On-support
    Production readiness check
    Service catalog
    Template project
    Common language package/library
    Protobuf management
    CI platform
    QA environment & automation
    Cloud administration
    Kubernetes administration
    Terraform & Kubernetes monorepo
    CDN administration
    Cloud network
    Service mesh
    Data pipeline and lake
    Data governance (lineages)
    CD platform
    Experimentation platform
    Supply Chain Security protection
    Delivery configuration management
    Observability platform
    Production access managements

    View Slide

  10. Build Test Deploy Operate
    Platform DX CI/CD
    Infrastructure
    Network
    Platform Infra
    Network
    Interface
    Data
    Data Operations
    Experimentation

    View Slide

  11. Lessons Learned

    View Slide

  12. Lessons Learned

    ● Business as First Priority
    ● Platform as Product
    ● Start from “Collaboration”
    ● Keep Appropriate Distance from Stream-aligned Team
    ● Platform Engineering = Migration Engineering

    View Slide

  13. Lessons Learned

    ● Business as First Priority
    ● Platform as Product
    ● Start from “Collaboration”
    ● Keep Appropriate Distance from Stream-aligned Team
    ● Platform Engineering = Migration Engineering

    View Slide

  14. Business as First Priority


    View Slide

  15. Lessons Learned

    ● Business as First Priority
    ● Platform as Product
    ● Start from “Collaboration”
    ● Keep Appropriate Distance from Stream-aligned Team
    ● Platform Engineering = Migration Engineering

    View Slide

  16. Platform as Product

    View Slide

  17. Platform as Product

    "Platform as a Product" is a mindset where the internal developer platform is
    managed continuously like “Product”, by considering developers as
    “customer,” focusing on meeting its needs.
    Unlike a Project, which has a finite life cycle with a clear start and end based
    on achieving specific goals, this approach sees the platform as an ongoing
    concern that requires continuous refinement and improvement.

    View Slide

  18. https://miro.com/app/board/uXjVPhewWSg=/

    View Slide

  19. Discovery
    Exploring the issues
    developers are facing
    Evaluating the
    solution works or
    not.
    Deciding which
    issues to address in
    which order.
    Prioritization Validation

    View Slide

  20. Lessons Learned

    ● Business as First Priority
    ● Platform as Product
    ● Start from “Collaboration”
    ● Keep Appropriate Distance from Stream-aligned Team
    ● Platform Engineering = Migration Engineering

    View Slide

  21. Start from “Collaboration”

    View Slide

  22. https://teamtopologies.com/book

    View Slide

  23. X-as-a-Service / Self-service
    Stream-Aligned Teams
    Platform Team

    View Slide

  24. Collaboration X-as-a-Service / Self-service

    View Slide

  25. Start from “Collaboration”

    ● Easy Discovery & Prioritization: Platform team can actually facing the
    issue as Stream-aligned team and understand it clearly
    ● Easy Validation: Platform Team can get feedback directly from the team
    and iterate quickly

    View Slide

  26. Lessons Learned

    ● Business as First Priority
    ● Platform as Product
    ● Start from “Collaboration”
    ● Keep Appropriate Distance from Stream-aligned Team
    ● Platform Engineering = Migration Engineering

    View Slide

  27. Keep Appropriate Distance from
    Stream-aligned Teams

    View Slide

  28. Keep Appropriate Distance from
    Stream-aligned Teams

    View Slide

  29. Collaboration X-as-a-Service / Self-service

    View Slide

  30. Force of “Collaboration” …
    When “Collaboration”, there is a natural force
    to be the same team forever e.g., Platform
    Team is taking all operation or On-calling
    👉 Platform becomes specific to the team. It
    can not be scaled
    Collaboration
    Getting too close

    View Slide

  31. Against the force of “Collaboration”

    ● Have a deadline of the “Collaboration”
    ● Keep future “X-as-a-Service” in mind
    ● Have a strict line on what NOT to do

    View Slide

  32. Getting too further away
    Force of X-as-a-Service …
    When “X-as-a-Service”, there is a natural force
    to getting further and further away from the
    stream aligned team.
    Platform Team only cares about its internal …
    Steam-aligned Team does not know what
    Platform Team is doing …
    👉 Platform Team can not properly discover
    the issue to be solved or validate the solution …
    X-as-a-Service

    View Slide

  33. Against the force of “X-as-a-Service”

    ● Keep the visibility of Platform activity
    ● Have direct feedback channels with Stream-aligned team
    ● Use “Collaboration” for exploring and validation

    View Slide

  34. Lessons Learned

    ● Business as First Priority
    ● Platform as Product
    ● Start from “Collaboration”
    ● Keep Appropriate Distance from Stream-aligned Team
    ● Platform Engineering = Migration Engineering

    View Slide

  35. Platform Engineering = Migration Engineering

    View Slide

  36. Migration Project at Mercari

    ● Replacing legacy infrastructure/system with new platform
    ○ e.g., Moving monolith system from On-prem environment
    ○ e.g., Moving SaaS based CI platform to self-hosted
    ● Enabling new features
    ○ e.g., Migrating to GKE workload Identity
    ● Applying best practices to the service running on the platform
    ○ e.g., Improving resource utilization (FinOps)

    View Slide

  37. Time
    Number of internal tools
    With Migration
    Without Migration
    Reducing the maintenance costs and
    technical debt
    Increasing the leverage of tooling
    improvements to everyone
    Keeping the system complexity lows
    Lots of similar tool is built but not
    maintained…

    View Slide

  38. “X-as-a-Service” Makes Migration Hard

    ● X-as-a-Service/Self-service means delegating responsibility of
    workflows or configuration management to Stream-aligned Teams
    ○ = Platform Teams can not modify them
    ● Some can be only by Platform Team but the most of migration requires
    support from Stream-aligned Teams …

    View Slide

  39. Tips: Successful Migration

    ● Design with Migration in mind
    ● Start from “Collaboration”
    ● Reduce the cost of migration as much as possible
    ● Communicate, communicate, communicate

    View Slide

  40. Conclusion

    ● Business as First Priority
    ● Platform as Product
    ● Start from “Collaboration”
    ● Keep Appropriate Distance from Stream-aligned Team
    ● Platform Engineering = Migration Engineering

    View Slide