Upgrade to Pro — share decks privately, control downloads, hide ads and more …

[DevDojo] Microservices Development 101

mercari
PRO
December 23, 2022

[DevDojo] Microservices Development 101

In this course, we will explain a basic overview of Microservices and various tools combined with GitOps used inside Mercari which powers the developer platform.

mercari
PRO

December 23, 2022
Tweet

More Decks by mercari

Other Decks in Technology

Transcript

  1. Microservices Development 101 Fall 2022 Dev Dojo @arjun.hemrajani

  2. Goals of this course
 • Learn about Mercari microservice platform

    • Learn about development tools at Mercari that support microservices. You can deploy your microservices from day 1 at Mercari
  3. Non-Goals of this course
 • We won’t cover what are

    microservices in details just the basics. • We don’t cover how Mercari listing works • We don’t cover how Mercari transaction works • … We don’t cover domain-specific knowledges here
  4. Why Mercari uses microservices? Drawbacks of monolithic architecture and benefits

    of microservices.
  5. Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL

    Mercari Monolith Solr
  6. Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL

    Mercari Monolith Solr Scalability Need to scale everything for one part of functionalities.
  7. Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL

    Mercari Monolith Solr Scalability Need to scale everything for one part of functionalities.
  8. Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL

    Mercari Monolith Solr Scalability Need to scale everything for one part of functionalities. Availability If one part is dead, everything is dead...
  9. Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL

    Mercari Monolith Solr Scalability Need to scale everything for one part of functionalities. If one part is dead, everything is dead... Availability
  10. Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL

    Mercari Monolith Solr Scalability Need to scale everything for one part of functionalities. If one part is dead, everything is dead... Availability
  11. Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL

    Mercari Monolith Solr Scalability Need to scale everything for one part of functionalities. If one part is dead, everything is dead... Availability
  12. Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL

    Mercari Monolith Solr Scalability Need to scale everything for one part of functionalities. If one part is dead, everything is dead... Complexity Difficult to add new features Difficult to try new technologies Difficult to onboard Availability
  13. Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL

    Mercari Monolith Solr Scalability Need to scale everything for one part of functionalities. If one part is dead, everything is dead... Complexity Difficult to add new features Difficult to try new technologies Difficult to onboard Velocity Too many overheads, no isolation in criticality Availability
  14. Listing DB Listing team User DB User team Item DB

    Item team Shipping DB Shipping team Timeline DB Timeline team
  15. Listing DB Listing team User DB User team Item DB

    Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work
  16. Listing DB Listing team User DB User team Item DB

    Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work
  17. Listing DB Listing team User DB User team Item DB

    Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work
  18. Listing DB Listing team User DB User team Item DB

    Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work
  19. Listing DB Listing team User DB User team Item DB

    Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work Flexible scaling Each service can scale independently and on demand
  20. Listing DB Listing team User DB User team Item DB

    Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work Flexible scaling Each service can scale independently and on demand
  21. Item Item Listing DB Listing team User DB User team

    Item DB Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work Flexible scaling Each service can scale independently and on demand
  22. Item Item Listing DB Listing team User DB User team

    Item DB Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work Flexible scaling Each service can scale independently and on demand Simplicity Easy to add new features Easy to use well suited technologies (e.g., DB)
  23. Item Item Listing DB Listing team User DB User team

    Item DB Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work Flexible scaling Each service can scale independently and on demand Simplicity Easy to add new features Easy to use well suited technologies (e.g., DB) Agility Independent deployment Independent decisions in a small context
  24. Item Item Listing DB Listing team User DB User team

    Item DB Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work Flexible scaling Each service can scale independently and on demand Simplicity Easy to add new features Easy to use well suited technologies (e.g., DB) Agility Independent deployment Independent decisions in a small context
  25. More about microservices
 • Microservices by Martin Fowler • Building

    Microservices by Sam Newman (Book) • Production-Ready Microservices by Susan Fowler (Book) • Adopting Microservices at Netflix • Microservices at Spotify
  26. Roles of microservices developers What are the responsibilities of backend

    developers in the microservices world?
  27. Roles of microservices developers
 • Develop software for business benefits

    • Write unit tests and integration tests • Response to alerts and incidents • Automate operations and reduce toils • Improve your service reliability based on SLI/SLO • You can and should go bold You need to act as SWE, SET, and SRE
  28. Develop Service B Test Deploy Operate Service B Team Develop

    Service A Test Deploy Operate Service A Team Develop Service C Test Deploy Operate Service C Team
  29. https://medium.com/netflix-techblog/full-cycle-developers-at-netflix-a08c31f83249

  30. Mercari microservice technical stacks What kind of technologies do we

    use for microservice development and deployment? What are your responsibilities?
  31. Technical stack at Mercari
 • Go for application language •

    gRPC for communication protocol • Kubernetes for container orchestration • Terraform for cloud service provisioning • Spinnaker for continuous delivery • Argo CD and Tekton for continuous delivery • CUE for configuration language • Github Actions for CI • AWS and GCP for Cloud
  32. Google Cloud Platform Services
 • Provides Cloud Spanner and Cloud

    SQL for databases • Provides Cloud Pub/Sub for message queuing • Provides Google Cloud Storage (GCS) for blob stores • Provides BigQuery for analytics data warehouses • Provides … (*) * You can use any GCP service for your microservice component
  33. Go at Mercari
 • Why: ◦ Safety by statically typed

    & compiled ◦ Maintainability (+readability) by simplicity & explicitness ◦ High performance and minimum memory footprints
  34. More about Go
 • Effective Go • Go Code Review

    Comments • Advanced Testing with Go • High Performance Go Workshop • So you want to expose Go on the Internet • Go Walkthrough • The Go Programming language (Book)
  35. Language Choices 
 • Go is currently a main language

    for Mercari. But you can choose any programming language for your microservices under your responsibility • You must know: ◦ Lots of ecosystems and tools are designed for Go (at the moment) ◦ It may be difficult to get supports from others if you get stuck • Responsibilities: ◦ Write everything by yourself ◦ Hire engineers for your team
  36. gRPC at Mercari
 • Why: ◦ Simple service definitions by

    an Interface Definition Language (IDL) ◦ Works across languages and platforms ◦ Faster serialization
  37. More about gRPC
 • What is gRPC? • Go Quick

    Start • gRPC Go: Beyond the basics • Peformance
  38. Docker at Mercari
 • Docker is mainly used for local

    development • It is also used for building docker images
  39. More about Docker
 • Dockerfile Best Practice

  40. Kubernetes at Mercari
 • Kubernetes is: ◦ Container orchestration system

    ◦ Derived from Google internal system named Borg & Omega • Why (*): ◦ Extensibility (answer for why not PaaS) ◦ Ecosystem (answer for why not ECS, Nomad, Mesos) * See more details about this in Microservices platform on Kubernetes at Mercari
  41. More about Kubernetes
 • Kubernetes Basic • Kubernetes Concept •

    The Illustrated Children's Guide to Kubernetes • A Crash Course on Container Orchestration • kubectl Cheat Sheets / Basic Usages • kubectl Cheat Sheets / Advanced Usages • Kubernetes: Up and Running (Book)
  42. Terraform at Mercari
 • We use Terraform to set up

    cloud services: ◦ Set up GCP services ◦ Set up Pagerduty, Sentry, GitHub, Slack, and more ◦ Don’t control your infrastructure by hand (󰢃UI, CLI, or API) • We expect developers to write terraform ◦ Gives them full control over their resources.
  43. More about Terraform
 • Terraform Cheat Sheets • The Terraform

    book
  44. Spinnaker at Mercari
 • We use Spinnaker for continuous delivery

    of microservices • Developers have control over their CD strategy ◦ They control the pipeline via GitOps ◦ Allows rollback
  45. More about Spinnaker
 • Continuous Delivery for Microservices with Spinnaker

    at Mercari
  46. Argo CD/Tekton
 • Continuous delivery platform built by Intuit •

    GitOps way ◦ Everything is in code ◦ Declarative and version controlled
  47. Argo CD/Tekton at Mercari
 • We’re in the middle of

    migrating from Spinnaker to a combination of Argo CD and other components ◦ Currently in Beta • Abstractions are provided through our GitOps repos
  48. More about Argo CD
 • Argo CD - Declarative GitOps

    CD for Kubernetes • Towards a more stable and secure CD system replacement • Tekton
  49. CUE
 • Open source language used to define, generate, and

    validate all kinds of data • Superset of JSON • Provides sensible defaults and constraints
  50. CUE at Mercari
 • Provide abstractions written in CUE ◦

    Kubernetes Kit for Kubernetes manifests ◦ Observability Kit for Datadog dashboards & monitors
  51. More about CUE
 • CUE • Cuetorials • Internal Doc

  52. Microservice basic access control and environments Which environments do you

    have? What kind of resources can you access?
  53. Basic Access Control
 • Kubernetes namespaces • GCP projects

  54. GCP Project: Kubernetes Platform Team Spanner Namespace: Service A Kubernetes

    GCP Project: Service A Namespace: Service B GCP Project: Service B RBAC IAM RBAC IAM IAM Container A Container A Container A Service B Team Service A Team
  55. Environments
 • Development • Production

  56. Mercari microservice development services and tools What kind of services

    and tools you can use? How to use it?
  57. SaaS
 • GitHub Actions for CI • Datadog for observability

    • PagerDuty for alerting • Sentry for error reporting
  58. GitHub Actions
 • We use GitHub Actions for CI: ◦

    Run tests ◦ Build container images ◦ Push container images to container registries
  59. Datadog
 • We use Datadog for monitoring microservices: ◦ Send

    application metrics ◦ Send distributed traces ◦ Create dashboards for microservices ◦ Integrate with Slack and Pagerduty for alerting
  60. Pagerduty
 • We use PagerDuty for alerting: ◦ Create on-call

    schedules ◦ Receive alerts via phone calls
  61. Sentry
 • We use Sentry for (critical) error reporting: ◦

    Send errors when panics happen ◦ Send errors when unknown errors happen
  62. GitOps at Mercari The core component which makes microservices development

    secure and asynchronous
  63. Service Team PR generate proto The Go gopher was designed

    by Renee French. (http://reneefrench.blogspot.com/)
  64. Plan & Apply PR Platform Team Review terraform repo Cloud

    Build Service B Team Service A Team Service C Team PR & Review PR & Review
  65. Starter Kit
 • Terraform module to set up tenant resources

    for services ◦ GCP project & IAM ◦ Kubernetes namespace ◦ GitHub team ◦ Pagerduty team and schedules ◦ Sentry project ◦ …
  66. Team Kit
 • Terraform module to set up resources for

    teams ◦ GCP IAM ◦ Kubernetes RBAC ◦ GitHub team ◦ Pagerduty team and schedules ◦ Slack channel and group ◦ …
  67. Sync Platform Team Review k8s Repo PR Service B Team

    Service A Team Service C Team PR & Review PR & Review Sync
  68. Kubernetes Kit
 • CUE-based Kubernetes manifest abstraction • Recommended over

    YAML-based configurations • Allows backend engineers to focus on their business ◦ Reduces the amount of configuration and required knowledge on Kubernetes and associated technologies ◦ Provides sensible defaults and constraints
  69. Production Readiness Checklist
 • Production Readiness Level • Production Readiness

    Checklist
  70. Key Takeaways
 • Mercari Microservices platform has many tools and

    services • We enable developers to take ownership and control • We use GitOps to enforce policies and ease development https://engineering.mercari.com/