Challenges moving from AWS to Kubernetes

This talk outlined our process moving from unmanaged AWS EC2 instances to a managed Kubernetes Cluster

Transcript

1. Next Generation Movie Marketing for Cinema, Home Entertainment and VOD
2. Our Journey to Kubernetes Kubernetes Meetup 2017-05-18
3. Ever heard of JustWatch?
4. JustWatch is a new kind of international adtech and data company solely focused on the movie industry. JustWatch is providing data driven video advertising campaigns focused on the revenue relevant power users mostly on YouTube and Facebook. JustWatch has exclusive first party data about movie taste and purchase behavior and minimizes waste coverage worldwide on scale.
5. Why are able to reach this efficiency? JustWatch has exclusive first party data and technology B2C International Movie and TV-Show Apps B2B Next Generation Movie Marketing
6. JustWatch collects anonymized data about purchase behavior and movie taste of movie and TV show fans worldwide MovieGraph User Profile Genres Action Science Fiction Comedy Movie Taste WatchList Add Trailer Views Purchase Behavior Cinema Tickets DVD / BD Rent / Buy HD / SD VOD Provider Movie Theater
8. JustWatch is growing rapidly in currently 24 countries, adding more than 2 million new user profiles per month JustWatch Usage: Web, iOS, Android Launch February 2015 iOS & Android App Launch > 1 MM App Downloads > 6 MM New Users per Month
9. Act I: The Beginning
10. In 2015, we wanted to validate the business model as fast as possible
11. We started running Snowflakes, building and starting containers on the cloud instances via cluster ssh
12. Data was stored on several managed PostgreSQL RDS instances as well as unmanaged ElasticSearch and Aerospike nodes
13. Early 2016 we wanted to improve developer self-service in our evolving micro-service architecture
14. You said evolving micro-service architecture?
15. We build and maintain over 60 stateless 12 Factor micro-services written in Go and deploy to AWS, GCE and Kubernetes using ChatOps. We use gRPC and Prometheus.
17. You really want to move all this stuff to another cloud provider?
18. We’ve built a lot of automation ourself, but keeping this running was too time consuming. We needed to make a radical change anyway.
19. Custom Slack Bot, Consul Key-Value Store, a custom Agent and Log Shipping.
20. Looking for alternatives: Amazon ECS, Apache Mesos, Docker Swarm, Hashicorp Nomad, Kubernetes
22. How to deploy Kubernetes into production?
23. How to deploy production into Kubernetes?
24. Act II: Migrating to Kubernetes
25. Writing lots of manifests and discovering some hidden dependencies
26. Theory: Move each service one by one. Switch the DNS and you’re done.
27. Moving stateless containers is easy, multi-TB datastores is not
28. Several PostgreSQL nodes, AeroSpike and ElasticSearch
29. PostgreSQL was actually mostly pg_dump and pg_restore
30. AeroSpike offers Cross-Datacenter Replication
31. ElasticSearch has an snapshot / restore feature that works surprisingly well
32. Prometheus works great within Kubernetes. Persistent state not so much.
33. Our architecture helped a lot. Only a very small part of this actually affected our users.
34. External services run fine with managed GCLB, but internal services needed some hacks
35. Act III: Moving on
36. Continue our gRPC rollout
37. Kubernetes Federation and global ingress
38. Security Policies, Quotas, RBAC and Network Policies
39. Moving back to managed PostgreSQL
40. Questions?
41. We’re hiring. AdTech Engineers Site Reliability Engineers Fullstack Engineers Work with awesome people and state of the art technology in a self-funded, fast-moving startup.