PyConZA 2014: "Hand me the salt while I read my news" by Johann du Toit

Millions of people read news monthly, and news tend to happen in bursts. The ideal infrastructure is able to scale up or down as soon as needed and provide a way to let you proactively manage your servers rather than reactively, which in the media game is a deal breaker. It gets even worse when you're running most of the local publication sites, including some big name players.

Focusing on the infrastructure we built, using Salt for management; give insight into the news game and provide examples of challenges our team are facing on a daily basis while servicing users. Expanding also on the workflow put together using various solutions such for logging, notifications with Python acting as the coordinator to keep all these systems in line. Delving into various parts of the system from network monitoring to deploying code.

This talk is meant for anyone interested in getting a glimpse of the systems and workflow put in place that allow us to manage the required load and continue developing with the agility to make the needed quick decisions daily.

Transcript

1. HAND ME THE SALT WHILE I READ MY NEWS by Johann du Toit
4. this talk my vision of DevOps nature news industry our setup & workflow introducing Bob
5. Blame Culture
6. Humanise! your Infrastructure
7. proactive vs reactive
10. Styles Images Javascript Font Other Html ~3mb
11. extra extra !
14. Styles Images Javascript Font Other Html ~150kb
15. 50+
17. say hello Bob keeps the DevOps team updated allows entire team to follow DevOps gives everyone a view of the system
18. {website-url} is back up for me ! Congrats you mere mortals
19. After a 5 tries {website-url} seems down with response code 504. Go check on it mortals !
20. did you know ?! we are running X servers.
21. {server-name} is using 70% of available memory
22. {server-name}’s hard drive is currently 60% full
23. status feed from various providers
24. after-hours
26. invoicing per client
27. part of the development flow
28. the guts of Bob
30. monitors restarts notifications
32. grouped errors by app history for trends centralised
34. DOCS DOCS DOCS! make the world go around …
36. “Cluster”
37. Load-Balancer
39. uk-client-web1
40. Load Balancer DB Servers File Servers Web Server Web Server Web Server
42. deployment
43. Load Balancer DB Servers File Servers Web Server Web Server Web Server
44. burned by “server state”
45. Preview Servers Provisioned but not serving live traffic
46. local dev
49. vagrant up
50. testing server
51. Preview Servers Provisioned but not serving live traffic
52. server-wide test
53. primes the cache
55. backup’s
57. Thanks for Listening johanndutoit.net