Cloud hopping with a stack of data

We’ve collected tons of events and aggregated data from our 20 million users for Wunderlist. However, after our acquisition by Microsoft we started to migrate our data and backend infrastructure from AWS to Azure. I am going to talk about our hiccups, challenges and what have we learnt from the journey from a completely Unix perspective.

It was presented at Big Data Universe conference. More information: http://bdu.hu

Transcript

1. Cloud Hopping with a Stack of Data @bfaludi @wunderlist @microsoft
2. Bence Faludi → Data & Applied Scientist at Microsoft → Lives in Berlin, Germany → Building data pipelines, data infrastructure → Open source addict → Working on Wunderlist, a multi platform to-do application
3. Topics → Scale and Complexity → Refactoring a Flying Airplane → Using the Zollstock → Bust a Move → Tighten Up
4. Wunderlist
6. → Organize and share your to-do, work, grocery, movies and household lists. → Set Due Dates and Reminders and Assign to-dos. → Share your lists and work collaboratively on projects with your colleagues, friends and family. → Available for free on every platform.
7. Disclaimer All opinions shared are my own.
8. Let's move an architecture from AWS to Azure
9. I. Scale and complexity
10. 21 million users
11. Monolith Ruby application ➡ Micro services (in 2014)
12. Micro services → Disposable software components → Many tiny databases → Many tiny services → Monitor everything → Multiple programming languages: Clojure, Scala, Go, etc.
14. mostly PostgreSQL as Storage → Hosted on AWS → ~33 databases → ~120 concurrent connections/database → Usually 2-3 tables per database → tasks table contains 1 billion records.
16. ... it was just the backend
17. Data infrastructure's size → Collect every event from tracking. 125M/day → Parse & load compressed logs' content. 375GB/day → Mirror every production database. 35GB inc./day → Load external sources. (e.g.: app store, payments). → Calculate KPIs, aggregates, business logic. 200+ queries
18. cold storage (Redshift) hot storage (Redshift) production database(s) external sources S3 S3 microservice applications Rsyslog Noxy EMR (Hadoop) logging clients (phone, tablet, etc.) email Tracking SNS SQS SQS dumper tracking Postamt Chart.io AWS + DWH riporting data-flow S3 2015-12
19. We used Redshift for analytics. 10 TB heavily compressed aggregated content
20. night-shift as ETL → Use cron for scheduling → Use make for dependencies, partial results, retries → Glue everything together with a handful of bash script → Inject variables and logic into SQL with Ruby's ERB → Flask application to monitor the data pipeline. → Testable in a local machine → Open source
21. All parts of our infrastructure are running in Unix
22. II. Refactoring a flying airplane
23. Goals → Remove the extra steps from data pipeline → Remove unnecessary complications like Hadoop → Add Azure support for the components → Refactor and make the code reusable
24. cold storage (Redshift) hot storage (Redshift) production database(s) external sources S3 S3 microservice applications Rsyslog Noxy EMR (Hadoop) logging clients (phone, tablet, etc.) email Tracking SNS SQS SQS dumper tracking Postamt Chart.io AWS + DWH riporting data-flow S3 2015-12
25. cold storage (Redshift) hot storage (Redshift) production database(s) external sources S3 S3 Rsyslog Noxy Jr. Beaver logging clients (phone, tablet, etc.) email Tracking SNS SQS SQS dumper tracking Postamt Chart.io AWS + DWH riporting data-flow S3 2016-01
26. EMR to Jr. Beaver → Detects the format of every log line → Log cruncher that standardizes microservices' logs → Classifies events' names based on API's URL → Filters the analytically interesting rows → Map/reduce functionality. → Hadoop+Scala to make+pypy
27. Jr. Beaver → Configurable with YAML files → Written in Pypy instead of Go → Using night-shift's make for parallelism → "Big RAM kills Big data" → No Hadoop+Scala headache anymore → Gives monitoring
28. vCPU count EMR (600+ in 20 computers): ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| |||||||||||||||||||||||||||||||||||||||||||| Jr. Beaver (8 in 1 computer): ||||||||
29. vCPU * working hours comparison EMR (600hrs): ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| |||||||||||||||||||||||||||||||||||||||||||| Jr. Beaver (64hrs): ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
30. cold storage (Redshift) hot storage (Redshift) production database(s) external sources S3 S3 Rsyslog Noxy Jr. Beaver logging clients (phone, tablet, etc.) email Tracking SNS SQS SQS dumper tracking Postamt Chart.io AWS + DWH riporting data-flow S3 2016-01
31. cold storage (Redshift) hot storage (Redshift) production database(s) external sources S3 S3 Rsyslog Noxy Jr. Beaver logging clients (phone, tablet, etc.) email Hamustro tracking Chart.io AWS + DWH riporting data-flow S3 2016-02
32. Homebrew Tracking to Hamustro → Track the activities of our registered users. → Receives events from client devices. → Saves events to cloud targets. → Tracks sessions and order of events. → Rewritten from NodeJS to Go. → Use S3 directly instead of SNS/SQS.
33. Hamustro → Supports Amazon SNS/SQS, Azure Queue Storage. → Supports Amazon S3, Azure Blob Storage. → Tracks up to 6M events/min on a single 4vCPU server. → Using Protobuf/JSON for events sending. → Written in Go. → Open source
34. vCPU count Homebrew tracking (12x1): |||||||||||| Hamustro (2x2): ||||
35. S3 vs. SNS in a single 4vCPU computer Hamustro's S3 dialect (~6M/min): ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||| |||||||||||||||||||||||||||||| Hamustro's SNS dialect (~60k/min): ||||||
36. Even a single RasberryPi is overkill for our 25k events/min
37. III. Using the Zollstock
39. we benchmarked everything ...
40. Azure Blob Storage Azure SQL Data Warehouse Ubuntu 14.04 Amazon S3 Amazon Redshift Ubuntu 14.04 Amazon SNS/SQS Chartio Chartio Hamustro Hamustro Power BI (under evaluation) Tracking
41. Amazon S3 = Azure Blob Storage
42. Amazon Redshift ~ Azure SQL Data Warehouse
43. IV. Bust a move
44. Clear the trash 1. Add Azure Blob Storage and Azure SQL Data Warehouse support for your services. 2. Build the missing tools for production. 3. Remove the not necessary data and backups to make your stack smaller. 4. Reconnect the new parts one by one.
45. Tools in UNIX for production → azrcmd: CLI to download and upload files to Azure Blob Storage. Provides s3cmd like functionality. → cheetah: CLI for MSSQL that works in OSX and Linux and also supports Azure SQL Data Warehouse. Similar to psql and superior to sql-cli and Microsoft's sqlcmd.
46. cold storage (Redshift) hot storage (Redshift) production database(s) external sources S3 S3 Rsyslog Noxy Jr. Beaver logging clients (phone, tablet, etc.) email Hamustro tracking Chart.io AWS + DWH riporting data-flow S3 2016-02
47. SQLDWH production database(s) external sources ABS Weasel Noxy Jr. Beaver clients (phone, tablet, etc.) Hamustro Chart.io Azure + DWH riporting data-flow logging tracking ABS 2016-04
48. Still left to transfer somehow the S3's content into BS
49. V. Tighten up
50. Improve SQLDW usage → Different loading strategies than in Redshift. → Scale up while the data pipeline is running. → Set up the right resource groups for every user. → Define distributions and use partitions. → Use full featured SQL. → Find the perfect balance between concurrency and speed.
51. everything else was fine at first
52. Thank you for your attention! @bfaludi