.Experiences with
Microservices at
Aarón Fas
Andrés Viedma
Slide 2
Slide 2 text
Microservices?
I know what you’re
probably thinking...
Slide 3
Slide 3 text
No content
Slide 4
Slide 4 text
No content
Slide 5
Slide 5 text
No content
Slide 6
Slide 6 text
Who did you say these guys are?
Andrés Viedma
@andres_viedma
Aarón Fas
@aaronfc
Java
dinosaur
Useless gadgets
buyer
Slide 7
Slide 7 text
About
Slide 8
Slide 8 text
From Social Network...
Slide 9
Slide 9 text
From Social Network...
Slide 10
Slide 10 text
To Mobile Operator
(full MVNO)
Slide 11
Slide 11 text
The PHP
Monolith
One single source
repository
PHP???
Slide 12
Slide 12 text
Do you need a
release?
Take a ticket
and wait...
Slide 13
Slide 13 text
.Microservices
Slide 14
Slide 14 text
Microservices… again… (and take a shot)
❖ Distributed, independently deployable
components
❖ Well defined interfaces
❖ Simple communication interface (HTTP?)
❖ Each service has its own DB
❖ Each service has its own source repository
Slide 15
Slide 15 text
Microservices… again… (and take a shot)
❖ Distributed, independently deployable
components
❖ Well defined interfaces
❖ Simple communication interface (HTTP?)
❖ Each service has its own DB
❖ Each service has its own source repository
THAT IS
SOA !!!
Slide 16
Slide 16 text
Microservices… again… (and take a shot)
❖ Distributed, independently deployable
components
❖ Well defined interfaces
❖ Simple communication interface (HTTP?)
❖ Each service has its own DB
❖ Each service has its own source repository
Is that important enough to
deserve a new name???
Slide 17
Slide 17 text
Mixing technologies
❖ Allows using different languages
❖ Different platform versions
❖ Incremental technology changes / evolution
Slide 18
Slide 18 text
Separation of responsibilities
❖ Forces separation of responsibilities
➢ Subsystems with well defined facades
➢ Different source repositories
Slide 19
Slide 19 text
Separation of responsibilities
❖ Forces separation of responsibilities
➢ Subsystems with well defined facades
➢ Different source repositories
YOU DON’T NEED
MICROSERVICES!
USE
JARS !!!
Slide 20
Slide 20 text
Continuous deployment
«Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.»
«The best architectures, requirements, and designs
emerge from self-organizing teams.»
-- Principles of the Agile Manifesto
Slide 21
Slide 21 text
Continuous deployment
«Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.»
«The best architectures, requirements, and designs
emerge from self-organizing teams.»
-- Principles of the Agile Manifesto
1 Service => 1 Team?
Better than Continuous delivery!:
Continuous deployment
Team responsible of the
deployments?
Slide 22
Slide 22 text
Beware! High costs
❖ No transactions!
➢ Distributed tx?
❖ Requires a much more complex infrastructure
❖ Difficult integration testing
Slide 23
Slide 23 text
For us: Seemed like a good idea
❖ We have small self-organized teams =>
Continuous deployment is a reality
❖ We wanted Java, we had PHP
❖ Strong SRE / DevOps team
❖ Our software was intended mainly to access
3rd parties => transactions not possible anyway
Slide 24
Slide 24 text
.Communications
protocol
Slide 25
Slide 25 text
Existing libraries
❖ No PHP implementation
➢ Avro, Etch, Netflix stack
❖ Only serialization
➢ Protocol buffers
❖ Didn’t exist or were too new
➢ Cap’n Proto, gRPC
❖ Thrift?
➢ Good option, but a lot of PHP boilerplate
Slide 26
Slide 26 text
TService
❖ Own abstraction layer - RPC based
❖ Basic implementation: JSON-RPC
❖ Interface Definition Language (IDL)
❖ Generates Java / PHP / Erlang:
➢ Interchange objects
➢ Client
➢ Server stub
Slide 27
Slide 27 text
TService IDL
/**
* Manages the transfer of balance between subscriptions.
* @version 1
*/
interface BalanceTransferService {
/** Transfer money from one subscription to another one. */
String transfer(Donation donation) throws NoSuchSubscriptionException;
(...)
}
/** Donation between two subscriptions. */
class Donation {
/** Id of the donor */
long from;
/** Amount of money to transfer */
int amount;
(...)
}
class NoSuchSubscriptionException extends Exception {
int code = 100;
}
Java???
Slide 28
Slide 28 text
TService Versioning
Interface v1
Service
Client 1
Client 2
(compatible changes)
● New methods
● New fields in objects
● New parameters in
methods
● Delete methods /
parameters / fields
XConfig
❖ Own configuration system
❖ YAML files based
❖ Git repository
❖ Overriding system: by env, common / service
❖ Hot reloading
➢ Everything adjusts to changes: even DB pools!
➢ No restart required
Slide 34
Slide 34 text
Async jobs
TService request
processing Enqueue
job
Queued jobs
Executor thread pool
Slide 35
Slide 35 text
Async jobs
TService request
processing Enqueue
job
Queued jobs
Executor thread pool
Cron jobs
Cron jobs programming in
config
Slide 36
Slide 36 text
Feature disabling
❖ Activation / deactivation of features by config
➢ Is the new development risky?
➢ Is the rest of services / environment ready for the
change?
❖ Partial activation of a feature for a % of users
➢ Incremental activation of an optional risky change
➢ A / B tests
Slide 37
Slide 37 text
Integration tests
❖ Custom JUnit runner
➢ Bootstraps the platform
➢ Cleans / restarts the local database
➢ Allows the use of @Inject in tests
➢ Allows overriding in dependency injection => inject
mocks of the other services
❖ Uses special, “development” XConfig repo
Slide 38
Slide 38 text
.Monitoring
Slide 39
Slide 39 text
Monitoring, a priority
❖ What is happening or has happened?
➢ Logs
➢ Metrics
➢ Alarms
❖ Distributed architectures are much more
difficult to track
Slide 40
Slide 40 text
And basically because...
Slide 41
Slide 41 text
.Let’s talk about logs
Slide 42
Slide 42 text
Logging
❖ Logging library in Java?
➢ Log4j
❖ We needed full details
➢ Filters to expand/simplify information logged
➢ Multiple appenders logged into distinct storages
Slide 43
Slide 43 text
❖ Overview of appenders
Logging
log.info(...); Logger
MySQL Appender
LogStash Appender
Hadoop Appender
Slide 44
Slide 44 text
❖ Following call’s path (TService calls logging)
Logging
ServiceA ServiceB ServiceC
GlobalID = 100
RequestID = 1
GlobalID = 100
RequestID = 2
GlobalID = 100
RequestID = 3
Benefits
● Locate in/out for calls
● Get all interactions
Slide 45
Slide 45 text
Logging
❖ Kibana dashboard
What does it look like?
Slide 46
Slide 46 text
Change query
Slide 47
Slide 47 text
Customize filters
Slide 48
Slide 48 text
Log types by color
Slide 49
Slide 49 text
Full log details
Slide 50
Slide 50 text
.Let’s talk about metrics
Slide 51
Slide 51 text
Metrics
❖ We graphs
➢ As easy as possible to track new metrics
❖ Do not reinvent the wheel
➢ Already using StatsD/Graphite on PHP side
❖ What are we tracking?
➢ Basic monitoring metrics added by the platform
➢ Metrics from Tomcat JMX
➢ Metrics related to business
Alarms
❖ Graphs are ok, but we don’t have people 24x7
staring at them.
➢ We need notifications
❖ Different things to monitor
➢ SQL queries
➢ Graphite metrics
➢ HTTP requests
➢ ...
Slide 59
Slide 59 text
Alarms
❖ Created our own alarms system
➢ Multiple data sources and easily extensible
➢ Quick edition of conditions
➢ Observers for alarms
❖ We ended up using mainly
➢ MySQL and Graphite data sources
➢ Java Expression Language on config checkers
➢ Email notifications
Slide 60
Slide 60 text
… and then, we found Cabot
Separated by service
Slide 61
Slide 61 text
Cabot overview
Multiple integrations
Slide 62
Slide 62 text
Cabot overview
Service status overview
Slide 63
Slide 63 text
Cabot overview
Graphite checks
Slide 64
Slide 64 text
Cabot overview (Creating new check)
Set graphite metric
Slide 65
Slide 65 text
Cabot overview (Creating new check)
Check data
Slide 66
Slide 66 text
Cabot overview (Creating new check)
Set check type/value
Slide 67
Slide 67 text
Cabot overview (Creating new check)
Set importance
Slide 68
Slide 68 text
Cabot
❖ Benefits of using Cabot
➢ Friendlier UI than config files
➢ No dependency on the service monitored
➢ Opensource and many integrations
Slide 69
Slide 69 text
Alarms
❖ Where are we heading now?
➢ Moving now most Graphite alarms to Cabot
➢ Replacing thresholds with dynamic expectations (Holt
Winters)
❖ It is still the main alarms platform being used
Slide 70
Slide 70 text
.That’s all about monitoring
Slide 71
Slide 71 text
.Some Lessons
learned
Slide 72
Slide 72 text
No content
Slide 73
Slide 73 text
GO
ASYNC!!!
Slide 74
Slide 74 text
Don’t get blocked for too long
❖ Concurrent requests: don’t wait for free threads
➢ Own Rate limit mechanism
➢ Tune container thread pool size
➢ Tune database pool (and other possible blocking pools)
❖ Tune clients timeout
➢ It may depend on called service / operation
➢ It may depend on the caller
Slide 75
Slide 75 text
Asynchronous logging
log.info(...)
Appender
MySQL
Appender
Logstash
Appender
Hadoop
Logger
Slide 76
Slide 76 text
Asynchronous logging
log.info(...)
When the ring buffer is full…
WAIT!
Appender
MySQL
Appender
Logstash
Appender
Hadoop
Logger
Ring buffer
Async Logger
Not configurable!
Slide 77
Slide 77 text
Asynchronous logging
log.info(...)
When the ring buffer is full…
WAIT!
Appender
MySQL
Appender
Logstash
Appender
Hadoop
Logger
Ring buffer
Async Logger
Async
Appender
Async
Appender
Async
Appender
Not configurable!
When async appender full,
messages are discarded
Slide 78
Slide 78 text
Asynchronous operations
❖ Getters
➢ Make them fast (sacrifice consistency)
➢ Cache
➢ Use default values
❖ Setters
➢ No operation result
➢ Wait for a notification of operation finished
➢ Query the status of the change
Circuit breaker
❖ From the client, consider the status of the
service
➢ Previous calls
➢ Health checks
❖ If it’s degraded, don’t call it (close the circuit)
➢ Return a default response
➢ Enqueue the operation for later retry
➢ Throw an error
Slide 81
Slide 81 text
.Do It Yourself
Slide 82
Slide 82 text
Many implementations available
❖ Communication layer
➢ gRPC, Cap’n proto, Thrift…
➢ REST, JSON…
❖ Services platform
➢ Spring boot, Dropwizard, Spark, Ninja, Jodd…
❖ Netflix stack
➢ Hystrix, Ribbon…
Slide 83
Slide 83 text
Make your own combination!
(it can’t be so
difficult…)