This is the talk Aarón Fas and Andrés Viedma presented in the JBcnConf 2015 telling their experiences at Tuenti using a distributed architecture of microservices.
Microservices… again… (and take a shot) ❖ Distributed, independently deployable components ❖ Well defined interfaces ❖ Simple communication interface (HTTP?) ❖ Each service has its own DB ❖ Each service has its own source repository
Microservices… again… (and take a shot) ❖ Distributed, independently deployable components ❖ Well defined interfaces ❖ Simple communication interface (HTTP?) ❖ Each service has its own DB ❖ Each service has its own source repository THAT IS SOA !!!
Microservices… again… (and take a shot) ❖ Distributed, independently deployable components ❖ Well defined interfaces ❖ Simple communication interface (HTTP?) ❖ Each service has its own DB ❖ Each service has its own source repository Is that important enough to deserve a new name???
Separation of responsibilities ❖ Forces separation of responsibilities ➢ Subsystems with well defined facades ➢ Different source repositories YOU DON’T NEED MICROSERVICES! USE JARS !!!
Continuous deployment «Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.» «The best architectures, requirements, and designs emerge from self-organizing teams.» -- Principles of the Agile Manifesto
Continuous deployment «Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.» «The best architectures, requirements, and designs emerge from self-organizing teams.» -- Principles of the Agile Manifesto 1 Service => 1 Team? Better than Continuous delivery!: Continuous deployment Team responsible of the deployments?
For us: Seemed like a good idea ❖ We have small self-organized teams => Continuous deployment is a reality ❖ We wanted Java, we had PHP ❖ Strong SRE / DevOps team ❖ Our software was intended mainly to access 3rd parties => transactions not possible anyway
Existing libraries ❖ No PHP implementation ➢ Avro, Etch, Netflix stack ❖ Only serialization ➢ Protocol buffers ❖ Didn’t exist or were too new ➢ Cap’n Proto, gRPC ❖ Thrift? ➢ Good option, but a lot of PHP boilerplate
TService IDL /** * Manages the transfer of balance between subscriptions. * @version 1 */ interface BalanceTransferService { /** Transfer money from one subscription to another one. */ String transfer(Donation donation) throws NoSuchSubscriptionException; (...) } /** Donation between two subscriptions. */ class Donation { /** Id of the donor */ long from; /** Amount of money to transfer */ int amount; (...) } class NoSuchSubscriptionException extends Exception { int code = 100; } Java???
XConfig ❖ Own configuration system ❖ YAML files based ❖ Git repository ❖ Overriding system: by env, common / service ❖ Hot reloading ➢ Everything adjusts to changes: even DB pools! ➢ No restart required
Feature disabling ❖ Activation / deactivation of features by config ➢ Is the new development risky? ➢ Is the rest of services / environment ready for the change? ❖ Partial activation of a feature for a % of users ➢ Incremental activation of an optional risky change ➢ A / B tests
Integration tests ❖ Custom JUnit runner ➢ Bootstraps the platform ➢ Cleans / restarts the local database ➢ Allows the use of @Inject in tests ➢ Allows overriding in dependency injection => inject mocks of the other services ❖ Uses special, “development” XConfig repo
Logging ❖ Logging library in Java? ➢ Log4j ❖ We needed full details ➢ Filters to expand/simplify information logged ➢ Multiple appenders logged into distinct storages
Metrics ❖ We graphs ➢ As easy as possible to track new metrics ❖ Do not reinvent the wheel ➢ Already using StatsD/Graphite on PHP side ❖ What are we tracking? ➢ Basic monitoring metrics added by the platform ➢ Metrics from Tomcat JMX ➢ Metrics related to business
Alarms ❖ Graphs are ok, but we don’t have people 24x7 staring at them. ➢ We need notifications ❖ Different things to monitor ➢ SQL queries ➢ Graphite metrics ➢ HTTP requests ➢ ...
Alarms ❖ Created our own alarms system ➢ Multiple data sources and easily extensible ➢ Quick edition of conditions ➢ Observers for alarms ❖ We ended up using mainly ➢ MySQL and Graphite data sources ➢ Java Expression Language on config checkers ➢ Email notifications
Alarms ❖ Where are we heading now? ➢ Moving now most Graphite alarms to Cabot ➢ Replacing thresholds with dynamic expectations (Holt Winters) ❖ It is still the main alarms platform being used
Don’t get blocked for too long ❖ Concurrent requests: don’t wait for free threads ➢ Own Rate limit mechanism ➢ Tune container thread pool size ➢ Tune database pool (and other possible blocking pools) ❖ Tune clients timeout ➢ It may depend on called service / operation ➢ It may depend on the caller
Asynchronous logging log.info(...) When the ring buffer is full… WAIT! Appender MySQL Appender Logstash Appender Hadoop Logger Ring buffer Async Logger Not configurable!
Asynchronous logging log.info(...) When the ring buffer is full… WAIT! Appender MySQL Appender Logstash Appender Hadoop Logger Ring buffer Async Logger Async Appender Async Appender Async Appender Not configurable! When async appender full, messages are discarded
Asynchronous operations ❖ Getters ➢ Make them fast (sacrifice consistency) ➢ Cache ➢ Use default values ❖ Setters ➢ No operation result ➢ Wait for a notification of operation finished ➢ Query the status of the change
Circuit breaker ❖ From the client, consider the status of the service ➢ Previous calls ➢ Health checks ❖ If it’s degraded, don’t call it (close the circuit) ➢ Return a default response ➢ Enqueue the operation for later retry ➢ Throw an error
tuenti
pocke
skume
umizaru
mame
ojun9
xblood
stewemetal
utgwkk
jmatsu
ufoo68
numeroanddev
twada
vanstee
kneath
rmw
eitanlees
lara
notwaldorf
dougneiner
deanohume
ddemaree
schacon
addyosmani
pauljervisheath