Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[DevDojo] Microservices Development 101

mercari
December 23, 2022

[DevDojo] Microservices Development 101

In this course, we will explain a basic overview of Microservices and various tools combined with GitOps used inside Mercari which powers the developer platform.

mercari

December 23, 2022
Tweet

More Decks by mercari

Other Decks in Technology

Transcript

  1. Microservices Development 101
    Fall 2022 Dev Dojo
    @arjun.hemrajani

    View full-size slide

  2. Goals of this course

    ● Learn about Mercari microservice platform
    ● Learn about development tools at Mercari that support
    microservices.
    You can deploy your microservices from day 1 at Mercari

    View full-size slide

  3. Non-Goals of this course

    ● We won’t cover what are microservices in details just the basics.
    ● We don’t cover how Mercari listing works
    ● We don’t cover how Mercari transaction works
    ● …
    We don’t cover domain-specific knowledges here

    View full-size slide

  4. Why Mercari uses microservices?
    Drawbacks of monolithic architecture and benefits of
    microservices.

    View full-size slide

  5. Listing
    Shipping
    Notification Review
    Purchase
    Login
    Timeline Search Search
    MySQL
    Mercari Monolith
    Solr

    View full-size slide

  6. Listing
    Shipping
    Notification Review
    Purchase
    Login
    Timeline Search Search
    MySQL
    Mercari Monolith
    Solr
    Scalability
    Need to scale everything for one
    part of functionalities.

    View full-size slide

  7. Listing
    Shipping
    Notification Review
    Purchase
    Login
    Timeline Search Search
    MySQL
    Mercari Monolith
    Solr
    Scalability
    Need to scale everything for one
    part of functionalities.

    View full-size slide

  8. Listing
    Shipping
    Notification Review
    Purchase
    Login
    Timeline Search Search
    MySQL
    Mercari Monolith
    Solr
    Scalability
    Need to scale everything for one
    part of functionalities.
    Availability
    If one part is dead,
    everything is dead...

    View full-size slide

  9. Listing
    Shipping
    Notification Review
    Purchase
    Login
    Timeline Search Search
    MySQL
    Mercari Monolith
    Solr
    Scalability
    Need to scale everything for one
    part of functionalities.
    If one part is dead,
    everything is dead...
    Availability

    View full-size slide

  10. Listing
    Shipping
    Notification Review
    Purchase
    Login
    Timeline Search Search
    MySQL
    Mercari Monolith
    Solr
    Scalability
    Need to scale everything for one
    part of functionalities.
    If one part is dead,
    everything is dead...
    Availability

    View full-size slide

  11. Listing
    Shipping
    Notification Review
    Purchase
    Login
    Timeline Search Search
    MySQL
    Mercari Monolith
    Solr
    Scalability
    Need to scale everything for one
    part of functionalities.
    If one part is dead,
    everything is dead...
    Availability

    View full-size slide

  12. Listing
    Shipping
    Notification Review
    Purchase
    Login
    Timeline Search Search
    MySQL
    Mercari Monolith
    Solr
    Scalability
    Need to scale everything for one
    part of functionalities.
    If one part is dead,
    everything is dead...
    Complexity
    Difficult to add new features
    Difficult to try new technologies
    Difficult to onboard
    Availability

    View full-size slide

  13. Listing
    Shipping
    Notification Review
    Purchase
    Login
    Timeline Search Search
    MySQL
    Mercari Monolith
    Solr
    Scalability
    Need to scale everything for one
    part of functionalities.
    If one part is dead,
    everything is dead...
    Complexity
    Difficult to add new features
    Difficult to try new technologies
    Difficult to onboard
    Velocity
    Too many overheads, no
    isolation in criticality
    Availability

    View full-size slide

  14. Listing
    DB
    Listing team
    User
    DB
    User team
    Item
    DB
    Item team
    Shipping
    DB
    Shipping team
    Timeline
    DB
    Timeline team

    View full-size slide

  15. Listing
    DB
    Listing team
    User
    DB
    User team
    Item
    DB
    Item team
    Shipping
    DB
    Shipping team
    Timeline
    DB
    Timeline team Resilient
    Even if one service is dead,
    others can continue to work

    View full-size slide

  16. Listing
    DB
    Listing team
    User
    DB
    User team
    Item
    DB
    Item team
    Shipping
    DB
    Shipping team
    Timeline
    DB
    Timeline team Resilient
    Even if one service is dead,
    others can continue to work

    View full-size slide

  17. Listing
    DB
    Listing team
    User
    DB
    User team
    Item
    DB
    Item team
    Shipping
    DB
    Shipping team
    Timeline
    DB
    Timeline team Resilient
    Even if one service is dead,
    others can continue to work

    View full-size slide

  18. Listing
    DB
    Listing team
    User
    DB
    User team
    Item
    DB
    Item team
    Shipping
    DB
    Shipping team
    Timeline
    DB
    Timeline team Resilient
    Even if one service is dead,
    others can continue to work

    View full-size slide

  19. Listing
    DB
    Listing team
    User
    DB
    User team
    Item
    DB
    Item team
    Shipping
    DB
    Shipping team
    Timeline
    DB
    Timeline team Resilient
    Even if one service is dead,
    others can continue to work
    Flexible scaling
    Each service can scale
    independently and on demand

    View full-size slide

  20. Listing
    DB
    Listing team
    User
    DB
    User team
    Item
    DB
    Item team
    Shipping
    DB
    Shipping team
    Timeline
    DB
    Timeline team Resilient
    Even if one service is dead,
    others can continue to work
    Flexible scaling
    Each service can scale
    independently and on demand

    View full-size slide

  21. Item
    Item
    Listing
    DB
    Listing team
    User
    DB
    User team
    Item
    DB
    Item team
    Shipping
    DB
    Shipping team
    Timeline
    DB
    Timeline team Resilient
    Even if one service is dead,
    others can continue to work
    Flexible scaling
    Each service can scale
    independently and on demand

    View full-size slide

  22. Item
    Item
    Listing
    DB
    Listing team
    User
    DB
    User team
    Item
    DB
    Item team
    Shipping
    DB
    Shipping team
    Timeline
    DB
    Timeline team Resilient
    Even if one service is dead,
    others can continue to work
    Flexible scaling
    Each service can scale
    independently and on demand
    Simplicity
    Easy to add new features
    Easy to use well suited technologies (e.g., DB)

    View full-size slide

  23. Item
    Item
    Listing
    DB
    Listing team
    User
    DB
    User team
    Item
    DB
    Item team
    Shipping
    DB
    Shipping team
    Timeline
    DB
    Timeline team Resilient
    Even if one service is dead,
    others can continue to work
    Flexible scaling
    Each service can scale
    independently and on demand
    Simplicity
    Easy to add new features
    Easy to use well suited technologies (e.g., DB)
    Agility
    Independent deployment
    Independent decisions in a small context

    View full-size slide

  24. Item
    Item
    Listing
    DB
    Listing team
    User
    DB
    User team
    Item
    DB
    Item team
    Shipping
    DB
    Shipping team
    Timeline
    DB
    Timeline team Resilient
    Even if one service is dead,
    others can continue to work
    Flexible scaling
    Each service can scale
    independently and on demand
    Simplicity
    Easy to add new features
    Easy to use well suited technologies (e.g., DB)
    Agility
    Independent deployment
    Independent decisions in a small context

    View full-size slide

  25. More about microservices

    ● Microservices by Martin Fowler
    ● Building Microservices by Sam Newman (Book)
    ● Production-Ready Microservices by Susan Fowler (Book)
    ● Adopting Microservices at Netflix
    ● Microservices at Spotify

    View full-size slide

  26. Roles of microservices developers
    What are the responsibilities of backend developers in the
    microservices world?

    View full-size slide

  27. Roles of microservices developers

    ● Develop software for business benefits
    ● Write unit tests and integration tests
    ● Response to alerts and incidents
    ● Automate operations and reduce toils
    ● Improve your service reliability based on SLI/SLO
    ● You can and should go bold
    You need to act as SWE, SET, and SRE

    View full-size slide

  28. Develop
    Service B
    Test Deploy Operate
    Service B Team
    Develop
    Service A
    Test Deploy Operate
    Service A Team
    Develop
    Service C
    Test Deploy Operate
    Service C Team

    View full-size slide

  29. https://medium.com/netflix-techblog/full-cycle-developers-at-netflix-a08c31f83249

    View full-size slide

  30. Mercari microservice technical
    stacks
    What kind of technologies do we use for microservice
    development and deployment? What are your
    responsibilities?

    View full-size slide

  31. Technical stack at Mercari

    ● Go for application language
    ● gRPC for communication protocol
    ● Kubernetes for container orchestration
    ● Terraform for cloud service provisioning
    ● Spinnaker for continuous delivery
    ● Argo CD and Tekton for continuous delivery
    ● CUE for configuration language
    ● Github Actions for CI
    ● AWS and GCP for Cloud

    View full-size slide

  32. Google Cloud Platform Services

    ● Provides Cloud Spanner and Cloud SQL for databases
    ● Provides Cloud Pub/Sub for message queuing
    ● Provides Google Cloud Storage (GCS) for blob stores
    ● Provides BigQuery for analytics data warehouses
    ● Provides … (*)
    * You can use any GCP service for your microservice component

    View full-size slide

  33. Go at Mercari

    ● Why:
    ○ Safety by statically typed & compiled
    ○ Maintainability (+readability) by simplicity & explicitness
    ○ High performance and minimum memory footprints

    View full-size slide

  34. More about Go

    ● Effective Go
    ● Go Code Review Comments
    ● Advanced Testing with Go
    ● High Performance Go Workshop
    ● So you want to expose Go on the Internet
    ● Go Walkthrough
    ● The Go Programming language (Book)

    View full-size slide

  35. Language Choices 

    ● Go is currently a main language for Mercari. But you can choose any
    programming language for your microservices under your responsibility
    ● You must know:
    ○ Lots of ecosystems and tools are designed for Go (at the moment)
    ○ It may be difficult to get supports from others if you get stuck
    ● Responsibilities:
    ○ Write everything by yourself
    ○ Hire engineers for your team

    View full-size slide

  36. gRPC at Mercari

    ● Why:
    ○ Simple service definitions by an Interface Definition
    Language (IDL)
    ○ Works across languages and platforms
    ○ Faster serialization

    View full-size slide

  37. More about gRPC

    ● What is gRPC?
    ● Go Quick Start
    ● gRPC Go: Beyond the basics
    ● Peformance

    View full-size slide

  38. Docker at Mercari

    ● Docker is mainly used for local development
    ● It is also used for building docker images

    View full-size slide

  39. More about Docker

    ● Dockerfile Best Practice

    View full-size slide

  40. Kubernetes at Mercari

    ● Kubernetes is:
    ○ Container orchestration system
    ○ Derived from Google internal system named Borg & Omega
    ● Why (*):
    ○ Extensibility (answer for why not PaaS)
    ○ Ecosystem (answer for why not ECS, Nomad, Mesos)
    * See more details about this in Microservices platform on Kubernetes at Mercari

    View full-size slide

  41. More about Kubernetes

    ● Kubernetes Basic
    ● Kubernetes Concept
    ● The Illustrated Children's Guide to Kubernetes
    ● A Crash Course on Container Orchestration
    ● kubectl Cheat Sheets / Basic Usages
    ● kubectl Cheat Sheets / Advanced Usages
    ● Kubernetes: Up and Running (Book)

    View full-size slide

  42. Terraform at Mercari

    ● We use Terraform to set up cloud services:
    ○ Set up GCP services
    ○ Set up Pagerduty, Sentry, GitHub, Slack, and more
    ○ Don’t control your infrastructure by hand (󰢃UI, CLI, or API)
    ● We expect developers to write terraform
    ○ Gives them full control over their resources.

    View full-size slide

  43. More about Terraform

    ● Terraform Cheat Sheets
    ● The Terraform book

    View full-size slide

  44. Spinnaker at Mercari

    ● We use Spinnaker for continuous delivery of microservices
    ● Developers have control over their CD strategy
    ○ They control the pipeline via GitOps
    ○ Allows rollback

    View full-size slide

  45. More about Spinnaker

    ● Continuous Delivery for Microservices with Spinnaker at Mercari

    View full-size slide

  46. Argo CD/Tekton

    ● Continuous delivery platform built by Intuit
    ● GitOps way
    ○ Everything is in code
    ○ Declarative and version controlled

    View full-size slide

  47. Argo CD/Tekton at Mercari

    ● We’re in the middle of migrating from Spinnaker to a combination of
    Argo CD and other components
    ○ Currently in Beta
    ● Abstractions are provided through our GitOps repos

    View full-size slide

  48. More about Argo CD

    ● Argo CD - Declarative GitOps CD for Kubernetes
    ● Towards a more stable and secure CD system replacement
    ● Tekton

    View full-size slide

  49. CUE

    ● Open source language used to define, generate, and validate all kinds
    of data
    ● Superset of JSON
    ● Provides sensible defaults and constraints

    View full-size slide

  50. CUE at Mercari

    ● Provide abstractions written in CUE
    ○ Kubernetes Kit for Kubernetes manifests
    ○ Observability Kit for Datadog dashboards & monitors

    View full-size slide

  51. More about CUE

    ● CUE
    ● Cuetorials
    ● Internal Doc

    View full-size slide

  52. Microservice basic access control
    and environments
    Which environments do you have? What kind of resources
    can you access?

    View full-size slide

  53. Basic Access Control

    ● Kubernetes namespaces
    ● GCP projects

    View full-size slide

  54. GCP Project: Kubernetes
    Platform Team
    Spanner
    Namespace: Service A
    Kubernetes
    GCP Project: Service A
    Namespace: Service B
    GCP Project: Service B
    RBAC
    IAM
    RBAC
    IAM
    IAM
    Container A
    Container A
    Container A
    Service B Team
    Service A Team

    View full-size slide

  55. Environments

    ● Development
    ● Production

    View full-size slide

  56. Mercari microservice
    development services and tools
    What kind of services and tools you can use? How to use it?

    View full-size slide

  57. SaaS

    ● GitHub Actions for CI
    ● Datadog for observability
    ● PagerDuty for alerting
    ● Sentry for error reporting

    View full-size slide

  58. GitHub Actions

    ● We use GitHub Actions for CI:
    ○ Run tests
    ○ Build container images
    ○ Push container images to container registries

    View full-size slide

  59. Datadog

    ● We use Datadog for monitoring microservices:
    ○ Send application metrics
    ○ Send distributed traces
    ○ Create dashboards for microservices
    ○ Integrate with Slack and Pagerduty for alerting

    View full-size slide

  60. Pagerduty

    ● We use PagerDuty for alerting:
    ○ Create on-call schedules
    ○ Receive alerts via phone calls

    View full-size slide

  61. Sentry

    ● We use Sentry for (critical) error reporting:
    ○ Send errors when panics happen
    ○ Send errors when unknown errors happen

    View full-size slide

  62. GitOps at Mercari
    The core component which makes microservices
    development secure and asynchronous

    View full-size slide

  63. Service Team PR generate
    proto
    The Go gopher was designed by Renee French.
    (http://reneefrench.blogspot.com/)

    View full-size slide

  64. Plan & Apply
    PR
    Platform Team
    Review
    terraform repo Cloud Build
    Service B Team
    Service A Team
    Service C Team
    PR & Review
    PR & Review

    View full-size slide

  65. Starter Kit

    ● Terraform module to set up tenant resources for services
    ○ GCP project & IAM
    ○ Kubernetes namespace
    ○ GitHub team
    ○ Pagerduty team and schedules
    ○ Sentry project
    ○ …

    View full-size slide

  66. Team Kit

    ● Terraform module to set up resources for teams
    ○ GCP IAM
    ○ Kubernetes RBAC
    ○ GitHub team
    ○ Pagerduty team and schedules
    ○ Slack channel and group
    ○ …

    View full-size slide

  67. Sync
    Platform Team
    Review
    k8s Repo
    PR
    Service B Team
    Service A Team
    Service C Team
    PR & Review
    PR & Review
    Sync

    View full-size slide

  68. Kubernetes Kit

    ● CUE-based Kubernetes manifest abstraction
    ● Recommended over YAML-based configurations
    ● Allows backend engineers to focus on their business
    ○ Reduces the amount of configuration and required
    knowledge on Kubernetes and associated technologies
    ○ Provides sensible defaults and constraints

    View full-size slide

  69. Production Readiness Checklist

    ● Production Readiness Level
    ● Production Readiness Checklist

    View full-size slide

  70. Key Takeaways

    ● Mercari Microservices platform has many tools and services
    ● We enable developers to take ownership and control
    ● We use GitOps to enforce policies and ease development
    https://engineering.mercari.com/

    View full-size slide