From Distributed to Monolithic to Squeeze Your Resources

In this talk I'll walk through the process that took us from a Generic Distributed Platform which could work with any resource (email, contacts, calendars, etc.) to a monolithic platform for each resource.
In order to do that I'll provide an overview of the architecture of our migration product which currently is being used by customers such as Gmail, Comcast, and TimeWarner Cable.

Transcript

1. © 2016 ShuttleCloud All rights reserved. Distribute with and without permission. From Distributed to Monolithic to Squeeze Your Resources
2. About @flopezluis • Software Engineer. • Director of Engineering @ShuttleCloudEng • Author of Mastering Python Regular Expressions 2
4. 4 • 99.5% SLA • 6 Terabytes per hour • > 12.000 migrations per day 12 million Emails per day 2.5 million Contacts per day About ShuttleCloud
6. 6 About ShuttleCloud
7. What problems do we face? A diversity of sources 7 Technical point of view Apis? cardDev? Imap? pop?
8. What’s our problem? A diversity of data 8 Technical point of view vcard? json? csv?
9. What’s our problem? High Load 9 Technical point of view
10. What’s our problem? Fault Tolerant 10 Technical point of view
11. Orchestrator Spring Integration RabbitMQ Contacts Endpoint MySQL Redis Spring AMQP Orchestrator Spring Integration Spring AMQP Contacts Endpoint Selenium Apis Gmail Yahoo … Api How we solved it Architecture Email Endpoint Email Endpoint Imap Apis Mail. ru Comcast …
12. 12 Orchestrator Spring Integration RabbitMQ Get Element elementId: 112 N Contacts Endpoint ElementId: x Generic: BEGIN:VCARD…. Spring AMQP Get Element elementId: x Get Elements elementId: 112 N Get Elements elementId: x ElementId: 112 Generic: BEGIN:VCARD…. N ElementId: x Generic: BEGIN:VCARD…. ElementId: 112 Generic: BEGIN:VCARD…. N Put Element Generic: BEGIN:VCARD…. Put Element: N Generic: BEGIN:VCARD…. N N Contacts Endpoint Contacts Endpoint Contacts Endpoint How we solved it Architecture Migration 30 Put Element: 1 Generic: BEGIN:VCARD…. Migration 30 Put Element: 200 Generic: BEGIN:VCARD…. Migration 32 Put Element: 43 Generic: BEGIN:VCARD…. 100 Migration 30 Put Element: 23 Generic: BEGIN:VCARD…. elements > 100
 or
 elapsed time >timeout Contacts Endpoint elements > 100
 or
 elapsed time >timeout Contacts Endpoint 10
13. 13 • Generalization • Transport layer. • Latencies. • Security. • Synchronization - state. Distributed Platform Problems
14. 14 Distributed Platform Problems Generalization
15. 15 Transport layer Distributed Platform Problems
16. 16 Distributed Platform Problems Latencies Migration 30 Put Element: 1 Generic: BEGIN:VCARD…. 30 Migration 30 Put Element: 200 Generic: BEGIN:VCARD…. Migration 30 Put Element: 43 Generic: BEGIN:VCARD…. 20 Migration 30 Put Element: 23 Generic: BEGIN:VCARD…. Migration 30 Put Element: 1 Generic: BEGIN:VCARD…. 40 Migration 30 Put Element: 2 Generic: BEGIN:VCARD…. Migration 30 Put Element: 45 Generic: BEGIN:VCARD…. 10 Migration 30 Put Element: 312 Generic: BEGIN:VCARD…. elements > 100
 or
 elapsed time >timeout Contacts Endpoint elements > 100
 or
 elapsed time >timeout Contacts Endpoint elements > 100
 or
 elapsed time >timeout Contacts Endpoint elements > 100
 or
 elapsed time >timeout Contacts Endpoint
17. 17 Distributed Platform Problems Security
18. 18 Distributed Platform Problems Synchronization - state.
19. Iteration 2 for the win 19 Multilithic Monolithic Ticketea TM
20. 20 Architecture Api Capabilities CouchDb dns CouchDb CouchDb dns autheps dns CouchDb CouchDb status listener RabbitMQ RabbitMQ stats-service Security Locker Security Locker Email Email Contacts dns MySqlRDS Pacemaker Cluster status listener Capabilities jobs-api jobs-api CouchDb Contacts autheps
21. 21 Architecture Api CouchDb CouchDb dns autheps autheps dns HA HA jobs-api jobs-api HA HA
22. 22 Architecture Api Capabilities CouchDb dns CouchDb CouchDb dns autheps dns CouchDb CouchDb status listener RabbitMQ RabbitMQ stats-service SL SL Email Email Contacts dns MySqlRDS Pacemaker Cluster status listener dns Capabilities jobs-api jobs-api CouchDb Contacts autheps
23. Source Destination Gmail mail.ru Iteration 2 Outlook … Contacts Yahoo Selenium Hub API API RabbitMQ Start Migration
24. 24 • One platform for each type of content. • Best solution for each type of problem Generalization Iteration 2
25. 25 • Data movement happens in memory. • Transport layer is almost empty. • Easy debug Transport Layer Iteration 2
26. 26 • User data is encrypted as soon as It’s received. • Credentials are stored separately. • Only migration platforms can access them. • Data movement happens in memory. Security Iteration 2
27. 27 • Efective batches • No latencies added for other components. Latencies Iteration 2
28. 28 Synchronization - state. Iteration 2
29. 29 • VPC for each client • 9 microservices • Contacts migrations are done in 9s (avg) • 95% success in all migrations • 300 Instances (AWS and GCE) Figures Iteration 2
30. © 2016 ShuttleCloud All rights reserved. Distribute with and without permission. @flopezluis @shuttlecloud_eng