Tweet
Tweet

Slide 1

Slide 1 text

Microservices Development 101 Fall 2022 Dev Dojo @arjun.hemrajani

Slide 2

Slide 2 text

Goals of this course
 ● Learn about Mercari microservice platform ● Learn about development tools at Mercari that support microservices. You can deploy your microservices from day 1 at Mercari

Slide 3

Slide 3 text

Non-Goals of this course
 ● We won’t cover what are microservices in details just the basics. ● We don’t cover how Mercari listing works ● We don’t cover how Mercari transaction works ● … We don’t cover domain-specific knowledges here

Slide 4

Slide 4 text

Why Mercari uses microservices? Drawbacks of monolithic architecture and benefits of microservices.

Slide 5

Slide 5 text

Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL Mercari Monolith Solr

Slide 6

Slide 6 text

Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL Mercari Monolith Solr Scalability Need to scale everything for one part of functionalities.

Slide 7

Slide 7 text

Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL Mercari Monolith Solr Scalability Need to scale everything for one part of functionalities.

Slide 8

Slide 8 text

Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL Mercari Monolith Solr Scalability Need to scale everything for one part of functionalities. Availability If one part is dead, everything is dead...

Slide 9

Slide 9 text

Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL Mercari Monolith Solr Scalability Need to scale everything for one part of functionalities. If one part is dead, everything is dead... Availability

Slide 10

Slide 10 text

Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL Mercari Monolith Solr Scalability Need to scale everything for one part of functionalities. If one part is dead, everything is dead... Availability

Slide 11

Slide 11 text

Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL Mercari Monolith Solr Scalability Need to scale everything for one part of functionalities. If one part is dead, everything is dead... Availability

Slide 12

Slide 12 text

Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL Mercari Monolith Solr Scalability Need to scale everything for one part of functionalities. If one part is dead, everything is dead... Complexity Difficult to add new features Difficult to try new technologies Difficult to onboard Availability

Slide 13

Slide 13 text

Listing Shipping Notification Review Purchase Login Timeline Search Search MySQL Mercari Monolith Solr Scalability Need to scale everything for one part of functionalities. If one part is dead, everything is dead... Complexity Difficult to add new features Difficult to try new technologies Difficult to onboard Velocity Too many overheads, no isolation in criticality Availability

Slide 14

Slide 14 text

Listing DB Listing team User DB User team Item DB Item team Shipping DB Shipping team Timeline DB Timeline team

Slide 15

Slide 15 text

Listing DB Listing team User DB User team Item DB Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work

Slide 16

Slide 16 text

Listing DB Listing team User DB User team Item DB Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work

Slide 17

Slide 17 text

Listing DB Listing team User DB User team Item DB Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work

Slide 18

Slide 18 text

Listing DB Listing team User DB User team Item DB Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work

Slide 19

Slide 19 text

Listing DB Listing team User DB User team Item DB Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work Flexible scaling Each service can scale independently and on demand

Slide 20

Slide 20 text

Listing DB Listing team User DB User team Item DB Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work Flexible scaling Each service can scale independently and on demand

Slide 21

Slide 21 text

Item Item Listing DB Listing team User DB User team Item DB Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work Flexible scaling Each service can scale independently and on demand

Slide 22

Slide 22 text

Item Item Listing DB Listing team User DB User team Item DB Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work Flexible scaling Each service can scale independently and on demand Simplicity Easy to add new features Easy to use well suited technologies (e.g., DB)

Slide 23

Slide 23 text

Item Item Listing DB Listing team User DB User team Item DB Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work Flexible scaling Each service can scale independently and on demand Simplicity Easy to add new features Easy to use well suited technologies (e.g., DB) Agility Independent deployment Independent decisions in a small context

Slide 24

Slide 24 text

Item Item Listing DB Listing team User DB User team Item DB Item team Shipping DB Shipping team Timeline DB Timeline team Resilient Even if one service is dead, others can continue to work Flexible scaling Each service can scale independently and on demand Simplicity Easy to add new features Easy to use well suited technologies (e.g., DB) Agility Independent deployment Independent decisions in a small context

Slide 25

Slide 25 text

More about microservices
 ● Microservices by Martin Fowler ● Building Microservices by Sam Newman (Book) ● Production-Ready Microservices by Susan Fowler (Book) ● Adopting Microservices at Netflix ● Microservices at Spotify

Slide 26

Slide 26 text

Roles of microservices developers What are the responsibilities of backend developers in the microservices world?

Slide 27

Slide 27 text

Roles of microservices developers
 ● Develop software for business benefits ● Write unit tests and integration tests ● Response to alerts and incidents ● Automate operations and reduce toils ● Improve your service reliability based on SLI/SLO ● You can and should go bold You need to act as SWE, SET, and SRE

Slide 28

Slide 28 text

Develop Service B Test Deploy Operate Service B Team Develop Service A Test Deploy Operate Service A Team Develop Service C Test Deploy Operate Service C Team

Slide 29

Slide 29 text

https://medium.com/netflix-techblog/full-cycle-developers-at-netflix-a08c31f83249

Slide 30

Slide 30 text

Mercari microservice technical stacks What kind of technologies do we use for microservice development and deployment? What are your responsibilities?

Slide 31

Slide 31 text

Technical stack at Mercari
 ● Go for application language ● gRPC for communication protocol ● Kubernetes for container orchestration ● Terraform for cloud service provisioning ● Spinnaker for continuous delivery ● Argo CD and Tekton for continuous delivery ● CUE for configuration language ● Github Actions for CI ● AWS and GCP for Cloud

Slide 32

Slide 32 text

Google Cloud Platform Services
 ● Provides Cloud Spanner and Cloud SQL for databases ● Provides Cloud Pub/Sub for message queuing ● Provides Google Cloud Storage (GCS) for blob stores ● Provides BigQuery for analytics data warehouses ● Provides … (*) * You can use any GCP service for your microservice component

Slide 33

Slide 33 text

Go at Mercari
 ● Why: ○ Safety by statically typed & compiled ○ Maintainability (+readability) by simplicity & explicitness ○ High performance and minimum memory footprints

Slide 34

Slide 34 text

More about Go
 ● Effective Go ● Go Code Review Comments ● Advanced Testing with Go ● High Performance Go Workshop ● So you want to expose Go on the Internet ● Go Walkthrough ● The Go Programming language (Book)

Slide 35

Slide 35 text

Language Choices 
 ● Go is currently a main language for Mercari. But you can choose any programming language for your microservices under your responsibility ● You must know: ○ Lots of ecosystems and tools are designed for Go (at the moment) ○ It may be difficult to get supports from others if you get stuck ● Responsibilities: ○ Write everything by yourself ○ Hire engineers for your team

Slide 36

Slide 36 text

gRPC at Mercari
 ● Why: ○ Simple service definitions by an Interface Definition Language (IDL) ○ Works across languages and platforms ○ Faster serialization

Slide 37

Slide 37 text

More about gRPC
 ● What is gRPC? ● Go Quick Start ● gRPC Go: Beyond the basics ● Peformance

Slide 38

Slide 38 text

Docker at Mercari
 ● Docker is mainly used for local development ● It is also used for building docker images

Slide 39

Slide 39 text

More about Docker
 ● Dockerfile Best Practice

Slide 40

Slide 40 text

Kubernetes at Mercari
 ● Kubernetes is: ○ Container orchestration system ○ Derived from Google internal system named Borg & Omega ● Why (*): ○ Extensibility (answer for why not PaaS) ○ Ecosystem (answer for why not ECS, Nomad, Mesos) * See more details about this in Microservices platform on Kubernetes at Mercari

Slide 41

Slide 41 text

More about Kubernetes
 ● Kubernetes Basic ● Kubernetes Concept ● The Illustrated Children's Guide to Kubernetes ● A Crash Course on Container Orchestration ● kubectl Cheat Sheets / Basic Usages ● kubectl Cheat Sheets / Advanced Usages ● Kubernetes: Up and Running (Book)

Slide 42

Slide 42 text

Terraform at Mercari
 ● We use Terraform to set up cloud services: ○ Set up GCP services ○ Set up Pagerduty, Sentry, GitHub, Slack, and more ○ Don’t control your infrastructure by hand (󰢃UI, CLI, or API) ● We expect developers to write terraform ○ Gives them full control over their resources.

Slide 43

Slide 43 text

More about Terraform
 ● Terraform Cheat Sheets ● The Terraform book

Slide 44

Slide 44 text

Spinnaker at Mercari
 ● We use Spinnaker for continuous delivery of microservices ● Developers have control over their CD strategy ○ They control the pipeline via GitOps ○ Allows rollback

Slide 45

Slide 45 text

More about Spinnaker
 ● Continuous Delivery for Microservices with Spinnaker at Mercari

Slide 46

Slide 46 text

Argo CD/Tekton
 ● Continuous delivery platform built by Intuit ● GitOps way ○ Everything is in code ○ Declarative and version controlled

Slide 47

Slide 47 text

Argo CD/Tekton at Mercari
 ● We’re in the middle of migrating from Spinnaker to a combination of Argo CD and other components ○ Currently in Beta ● Abstractions are provided through our GitOps repos

Slide 48

Slide 48 text

More about Argo CD
 ● Argo CD - Declarative GitOps CD for Kubernetes ● Towards a more stable and secure CD system replacement ● Tekton

Slide 49

Slide 49 text

CUE
 ● Open source language used to define, generate, and validate all kinds of data ● Superset of JSON ● Provides sensible defaults and constraints

Slide 50

Slide 50 text

CUE at Mercari
 ● Provide abstractions written in CUE ○ Kubernetes Kit for Kubernetes manifests ○ Observability Kit for Datadog dashboards & monitors

Slide 51

Slide 51 text

More about CUE
 ● CUE ● Cuetorials ● Internal Doc

Slide 52

Slide 52 text

Microservice basic access control and environments Which environments do you have? What kind of resources can you access?

Slide 53

Slide 53 text

Basic Access Control
 ● Kubernetes namespaces ● GCP projects

Slide 54

Slide 54 text

GCP Project: Kubernetes Platform Team Spanner Namespace: Service A Kubernetes GCP Project: Service A Namespace: Service B GCP Project: Service B RBAC IAM RBAC IAM IAM Container A Container A Container A Service B Team Service A Team

Slide 55

Slide 55 text

Environments
 ● Development ● Production

Slide 56

Slide 56 text

Mercari microservice development services and tools What kind of services and tools you can use? How to use it?

Slide 57

Slide 57 text

SaaS
 ● GitHub Actions for CI ● Datadog for observability ● PagerDuty for alerting ● Sentry for error reporting

Slide 58

Slide 58 text

GitHub Actions
 ● We use GitHub Actions for CI: ○ Run tests ○ Build container images ○ Push container images to container registries

Slide 59

Slide 59 text

Datadog
 ● We use Datadog for monitoring microservices: ○ Send application metrics ○ Send distributed traces ○ Create dashboards for microservices ○ Integrate with Slack and Pagerduty for alerting

Slide 60

Slide 60 text

Pagerduty
 ● We use PagerDuty for alerting: ○ Create on-call schedules ○ Receive alerts via phone calls

Slide 61

Slide 61 text

Sentry
 ● We use Sentry for (critical) error reporting: ○ Send errors when panics happen ○ Send errors when unknown errors happen

Slide 62

Slide 62 text

GitOps at Mercari The core component which makes microservices development secure and asynchronous

Slide 63

Slide 63 text

Service Team PR generate proto The Go gopher was designed by Renee French. (http://reneefrench.blogspot.com/)

Slide 64

Slide 64 text

Plan & Apply PR Platform Team Review terraform repo Cloud Build Service B Team Service A Team Service C Team PR & Review PR & Review

Slide 65

Slide 65 text

Starter Kit
 ● Terraform module to set up tenant resources for services ○ GCP project & IAM ○ Kubernetes namespace ○ GitHub team ○ Pagerduty team and schedules ○ Sentry project ○ …

Slide 66

Slide 66 text

Team Kit
 ● Terraform module to set up resources for teams ○ GCP IAM ○ Kubernetes RBAC ○ GitHub team ○ Pagerduty team and schedules ○ Slack channel and group ○ …

Slide 67

Slide 67 text

Sync Platform Team Review k8s Repo PR Service B Team Service A Team Service C Team PR & Review PR & Review Sync

Slide 68

Slide 68 text

Kubernetes Kit
 ● CUE-based Kubernetes manifest abstraction ● Recommended over YAML-based configurations ● Allows backend engineers to focus on their business ○ Reduces the amount of configuration and required knowledge on Kubernetes and associated technologies ○ Provides sensible defaults and constraints

Slide 69

Slide 69 text

Production Readiness Checklist
 ● Production Readiness Level ● Production Readiness Checklist

Slide 70

Slide 70 text

Key Takeaways
 ● Mercari Microservices platform has many tools and services ● We enable developers to take ownership and control ● We use GitOps to enforce policies and ease development https://engineering.mercari.com/