Your Philips Hue Light Bulbs Are Turned On By Kubernetes

Philips Hue is one of the leading global Internet of Things platforms and has integrations with Amazon Echo, Google Home, IFTTT & Nest to make your lights work for you. The platform has grown considerably during the past years and to accommodate this has been migrated towards Google Container Engine (Kubernetes) at the end of 2015.

We needed to find out how to do zero downtime deployments, grasp how scaling works in Kubernetes, provide ourselves with monitoring and logging and above all find out how to do all these things and more while staying agile in the process. Grab something to drink and enjoy a deep dive into all the things we have encountered while running on Kubernetes.

Presented at Software Circus meetup https://www.meetup.com/Software-Circus/events/239478887/

Transcript

1. Your Philips Hue Light Bulbs Are Turned On By Kubernetes Mark van Straten, Q42 @markvanstraten
2. @markvanstraten
3. @markvanstraten
4. @markvanstraten
5. @markvanstraten
6. @markvanstraten
7. @markvanstraten
8. @markvanstraten GAE
9. @markvanstraten GAE GAE flex
10. @markvanstraten GAE GAE flex GKE
11. CURRENT SETUP
12. @markvanstraten Backend notifier Token validation service Bridge websocket API Bridge http API Hue Cloud API
13. @markvanstraten 1 cluster 10 services 50 nodes 200 pods
14. @markvanstraten 7 Developers
15. LESSONS LEARNED
16. @markvanstraten Circuit breakers can prevent epic failures
17. @markvanstraten Google manages our nodes BUT: how to modify settings? ( https://github.com/Q42Philips/king-kong )
18. @markvanstraten Scaling long living idle websocket connections ( https://goo.gl/uZJIjS )
20. @markvanstraten 1. Get replicas 2. Fetch metrics 3. Magic 4. Scale replicas https://github.com/crunchie84/kubernetes-example-custom-hpa
22. @markvanstraten • DevOps culture • Measure everything • Datadog alerting • Kubernetes tooling
23. @markvanstraten • Emit metrics instead • Only log what matters (warn/error) • Use structured JSON logging • Ours end up in BigQuery
24. @markvanstraten "jsonPayload": { "metadata": { "connectionid": "81fc90aa-26c2-4d95-92bb-ca6303998953", "bridgeid": "001788fffe200135", }, "message": "Updating connection info, bridge already connected to a different pod", “severity”: “info”, "timestamp": "2017-05-04T20:33:21.955Z" }
25. @markvanstraten
26. @markvanstraten • Decreasing latency • Diagnostics data // machine learning • Decreasing Ops where possible • Increase reliability • Continuous Delivery
28. @markvanstraten Kubernetes ♥ Microservices
29. @markvanstraten Extend and override the defaults
30. @markvanstraten Ops work feels like Dev
31. @markvanstraten
32. @markvanstraten Great monitoring is paramount
34. Mark van Straten @markvanstraten mark@q42.nl q42.nl (shameless plug: we’re hiring!)