- CTO, Co-founder of RisingStack
- @slashdotpeter on Twitter
- https://blog.risingstack.com
$ whoami
Slide 3
Slide 3 text
Microservices
Img: http://ryanjbaxter.com
Slide 4
Slide 4 text
Moving to microservices
Slide 5
Slide 5 text
Moving to microservices
Slide 6
Slide 6 text
- Co-located teams can work more efficiently
- different space / time zone
- Smaller teams are usually more efficient
- More focused people
- Faster onboarding for new developers
Possible business reasons to move
- Increasing architectural complexity
- Increasing operational complexity
- Monitoring and debugging are much harder
- Handling eventual Consistency
Technology drawbacks
Slide 9
Slide 9 text
microservice architecture is
not a silver bullet
Slide 10
Slide 10 text
Why we moved?
Slide 11
Slide 11 text
- Microservice monitoring tool
- Node.js focused
- Built in Node.js
- We migrated to microservices architecture
Trace by RisingStack
Slide 12
Slide 12 text
- We wanted to have more focused teams
- We have very different challenges
- Fault tolerance is a key in our business
Why we moved?
Slide 13
Slide 13 text
Before we moved
Slide 14
Slide 14 text
Today
Slide 15
Slide 15 text
How we moved?
Slide 16
Slide 16 text
Services and teams
Slide 17
Slide 17 text
- Separate DB per service
- Maximize the depth of service call chains
Service principles
Slide 18
Slide 18 text
- We create backward compatible endpoints
- We don’t version services (only endpoints)
- Use feature flippers/toggles (dark launches)
Service principles
Slide 19
Slide 19 text
- Good and available documentation
- Update docs and code together
Good to start here: https://github.com/Yelp/service-principles
Service principles
Slide 20
Slide 20 text
Automation
Slide 21
Slide 21 text
- Automation is a key in building and operating microservices
- Easy and fast to deploy
- Easy and fast to rollback
- Testing, Service bootstrap, DB migration etc.
Automation
Slide 22
Slide 22 text
Proxy / API Gateway
Slide 23
Slide 23 text
Proxy / API Gateway
Slide 24
Slide 24 text
- Client(s) specific things:
- Authentication: Cookie headers, JWT token etc.
- Protocols: http, WebSocket etc.
- Response format: JSON, XML etc.
- Combining resources: from multiple services
API Gateway
Slide 25
Slide 25 text
API Gateway
Img: http://bits.citrusbyte.com/microservices
Slide 26
Slide 26 text
Fault tolerance
Slide 27
Slide 27 text
- Services fail separately (in theory)
- Critical resources should be cached
- Messaging queues can help (HTTP request is not recoverable)
Fault tolerance
Slide 28
Slide 28 text
Fault tolerant data collection - CQRS
read
write
Slide 29
Slide 29 text
Fault tolerant data collecting
Queue size increasing during issue
Queue size decreasing after issue resolved
Slide 30
Slide 30 text
Caching
Slide 31
Slide 31 text
- On service level
- Automatically via response headers
- Via our communication package
- Multi store caching (in memory, Redis):
https://www.npmjs.com/package/cache-manager
Caching
Slide 32
Slide 32 text
Security
Slide 33
Slide 33 text
- Trusted sources (services) on public channel
- Request signing between services
- Significant CPU overhead
Request signing
Slide 34
Slide 34 text
Request signing
Slide 35
Slide 35 text
Infrastructure
Slide 36
Slide 36 text
- Started with PaaS
- We moved to Kubernetes
- Zero downtime deployment
Infrastructure
Microservices producing lot’s of data
- Logs, Errors
- Metrics per service
- Transactions
- Logical connections
- Side effects
Slide 41
Slide 41 text
microservice monitoring is
impossible for humans
Slide 42
Slide 42 text
How to find an issue?
Alerting, Dashboard
Topology
Metrics, Errors
Traces
Profiler
Investigate a service
Locate on code level
Understand connections
Identify participating services
Always know about it
Slide 43
Slide 43 text
Distributed tracing
Slide 44
Slide 44 text
- Transaction ID, Correlation ID
- Google Dapper white paper
- Trace by RisingStack
- Zipkin
- OpenTracing
Distributed tracing
Slide 45
Slide 45 text
Case study: Network delay
Slide 46
Slide 46 text
Case study: Network delay
3.8ms 564.8ms
95th response time:
Call depth: 1 service 2 services