OZON. Making microservices from scratch

How we build microservices in OZON.

Transcript

1. OZON Making microservices from scratch
2. Ivan Korolev Senior Go Developer @ Platform team ivan.korolev@ozon.ru
 
 github.com/mwf Slides: ozon.ru/t/2M
3. Agenda gRPC vs HTTP github.com/utrack/clay - HTTP server generator scratch - project generation tool
4. gRPC vs http • protobuf - fixed and extendable API contract • binary protocol - lower bandwidth • bidirectional data streams on a single socket • client/server code generation
5. gRPC + http? Shiny new gRPC API or legacy HTTP REST?
6. gRPC + http? Shiny new gRPC API or legacy HTTP REST? • Implement both "transports" separately 
 + native transport support
 – code duplication
7. gRPC + http? Shiny new gRPC API or legacy HTTP REST? • Implement both "transports" separately 
 + native transport support
 – code duplication • Implement gRPC + deploy grpc-gateway 
 + don't need http implementation
 – intermediate service deployment
8. gRPC + http? Shiny new gRPC API or legacy HTTP REST? • Implement both "transports" separately 
 + native transport support
 – code duplication • Implement gRPC + deploy grpc-gateway 
 + don't need http implementation
 – intermediate service deployment • HTTP server code generation
9. gRPC vs HTTP github.com/utrack/clay - HTTP server generator scratch - project generation tool
10. clay github.com/utrack/clay • protoc plugin
 .proto as single service description source
 mapping flexibility: Google API HTTP Annotations • Swagger (OpenAPI) support (grpc-gateway) • single handler implementation on top of gRPC • slim transport library: - errors handling - custom marshallers for any content type - HTTP headers - gRPC metadata translation • go-modules friendly :)
11. $ curl 
 http://localhost:8080/strings/to_upper/foo {"str":"FOO"} 
 
 $ curl -XPOST -d '{"str": "bar"}' 
 http://localhost:8080/strings/to_upper {"str":"BAR"}
12. $ curl -XPOST -d '["foo", "bar"]' http://localhost:8080/strings/to_upper ["FOO","BAR"] 
 
 $ curl http://localhost:8080/strings/ to_upper/v2?str=foo&str=bar ["FOO","BAR"]
13. clay github.com/utrack/clay/integration - examples GRPC_GATEWAY_PATH=$(GOPATH)/pkg/mod/github.com/grpc-ecosystem/ grpc-gateway@v1.7.0/third_party/googleapis
 
 protoc -I/usr/local/include:${GRPC_GATEWAY_PATH}/third_party/googleapis:. --gofast_out=plugins=grpc:. --goclay_out=impl=true,impl_path=../strings:. pb/strings.proto
15. clay + protoc-compatible HTTP client/server generator + bindings flexibility (Google API HTTP Annotations) + no business-logic code duplication – protoc configuration
16. gRPC vs HTTP github.com/utrack/clay - HTTP server generator scratch - project generation tool
17. scratch Goal: make new service ASAP • Framework - metrics - tracing - middlewares - ... • Project skeleton (project structure, deployments) • Library and project updates
18. scratch Framework: • public http & grpc servers • "debug" server: • swagger, • live/ready, • pprof, • metrics • default middlewares (panic, metrics, tracing, etc...) • realtime-config support
19. scratch scratch tool • init - creates new service from .proto (or without) • upgrade - drop-in binary upgrade • update - various project updates between version • vendor.pb - downloads .proto dependencies respecting go.mod • generate - various generators, like realtime-config helpers • install - some install helpers • check - checks system requirements (go, protoc versions, etc...) • version - prints version
20. scratch init scratch init --proto ./api/strings.proto[:<.proto>] go run cmd/<service>/main.go git push Gitlab CI Pipeline -> Production :)
21. Deployment configs service.proto Binary dependencies Main Implementation Client libs Vendored .proto files CI config golangCI-lint Modules stuff Custom make targets Generated makefile
22. k8s Helm values.yaml Implementation and methods Generated stuff
23. scratch vendor.pb • Respect go.mod • Respect well-known types • Full-Qualified Names for imports • All proto dependencies in one place protoc -I$(CURDIR)/api/:$(CURDIR)/vendor.pb
 ...
24. scratch generate config
25. scratch generate config
26. scratch upgrade
27. scratch install
28. scratch update • update project structure • remove vgo :) • clean go cache • .gitlab-ci stuff • ...
29. What do we get? Support from platform team Common project structure Easy updates Fun coding No problems with scratch. Using it from the very first version. Godlike tool, transforming service development to awesome Golang odyssey. I always upgrade to the latest scratch when creating a new service or updating an old one. Renars Freimanis, C# developer @ OZON team
30. scratch - Opensource? github.com/utrack/clay
 github.com/ozonru
31. Thank you! Slides: ozon.ru/t/2M Google API HTTP Annotations github.com/utrack/clay github.com/grpc-ecosystem/grpc-gateway
 
 Ashley McNamara: Gophers artwork habr.com/ru/company/ozontech github.com/ozonru Ivan Korolev Senior Go Developer @ Platform team ivan.korolev@ozon.ru
 github.com/mwf