Red Nose Data

Every year, Comic Relief runs a charity fundraising campaign called Red Nose Day that culminates in a 7 hour marathon of primetime TV on BBC 1. On March 15th this year millions of viewers will be encouraged to donate during the TV show, and the On The Night donations platform has to deal with the incredible spikes in demand that result from this unique event. This platform depends on MongoDB to process up to 2000 donations per second. This presentation describes how MongoDB and the patterns in use enable the event to be a success, despite the fact it is only in use once a year.

Transcript

1. Red Nose Data MongoDB London 2013
2. Red Nose Data MongoDB London 2013
3. About Comic Relief o Comic Relief is a major charity based in the UK which strives to create a just world free from poverty o Raises millions of pounds for the needy to tackle the root causes of poverty and social injustice o Two big fundraising campaigns - Red Nose Day and Sport Relief
4. o Every two years thousands of people do something funny for money o A year of planning o 6 week media campaign o 7 hours of TV on the 15th of March
5. New platform in 2013 o Maximise the donations received o Remove reliance on any single third party supplier o Cost-effective o Ability to make changes and understand the risks involved
6. New platform in 2013 600,000 transactions in 7 hours
7. New platform in 2013 > 10,000call centre operators
8. New platform in 2013 500donation journeys every second
9. New platform in 2013 One night per year no second chance
10. Donations Per Minute
11. Donations Per Second
12. Historical challenges Single points of failure
13. Historical challenges Deployment
14. Historical challenges Provider relationships
15. Historical challenges One year feedback cycle
16. Historical challenges Timely donation totals
17. Solution patterns Accept that failures happen
18. Solution patterns Multiple IaaS Providers
19. Solution patterns Eventual Consistency
20. Solution patterns Minimise time to Recover
21. Solution patterns Commoditise dependencies
22. Solution patterns Continuous Delivery
23. Why MongoDB?
24. Why MongoDB? Document-orientated
25. Why MongoDB? Ease of schema evolution
26. Why MongoDB? Rich query language
27. Mongoid scope :pending, not_in(:delta_state_labels => %w{payment_taken payment_failed})
28. Insight Layer What does the platform look like? Internet DNS PaaS 1 - AWS US Cloud Foundry (BOSH) PaaS 1 - AWS EU Cloud Foundry (BOSH) Service Layer Workers View API Insight Presentation Layer Service Layer Workers MGMT PaaS 1 - Cz Cloud Foundry (BOSH) Presentation Layer Service Layer Workers Shared Services Logging Metrics Alerting = + PLUS Presentation Layer
29. $$$ Continuous Delivery
30. Continuous Delivery Integrated application and infrastructure pipeline
31. Cucumber Shared understanding with our customer
32. mongoid-rspec RSpec Matchers for Mongoid
33. ZA Proxy Web-app pen testing
34. The Grinder Capture functional tests and replay at scale
35. The Grinder 60,000 concurrent users
36. 0 100 200 300 400 500 Redis Config Added DEAs Added DEAs Increased load test threads Moved load test platform to EU Added HA proxy & 3 Nginx nodes Increased load test threads 8 Nginx nodes 501 Monday 10/9 Tuesday 11/9 Transactions per second Load Testing
37. Conclusions Eventual consistency and statelessness keys for scaling.
38. Conclusions MongoDB ensured donations totals were available when it mattered.
39. Conclusions £75,107,851
40. Conclusions Red Nose Day makes people’s lives better. You can help!
41. http://www.armakuni.com/ Thank You