Rebuilding your engine at 200mph

Given at Cloud East 2013 in Cambridge on 24th May 2013.

The Guardian Content-Api serves a little over 11 million requests per day, mostly from internal systems that depend on it to serve the main guardian website, the mobile site and other digital distribution systems.

We undertook a project to completely re-architect the API from a poll based system based on Solr in our data center, to a message queue based cloud service based upon Elastic Search, all requiring no downtime, and no missing content.

In this session I'll explain the reasons we decided to undertake the project, and the approaches we took to ensure it was delivered successfully and I'll explain how we decided upon the technology, and what issues we had along the way.

Transcript

1. Michael Brunton-Spall Content API Lead Engineer @bruntonspall Rebuilding your engine at 200 mph Friday, 24 May 13
2. Michael Brunton-Spall • Who am I • Lead Engineer for Content Platforms • Developer Advocate for The Guardian • Scalability Engineer for guardian.co.uk • Technical Architect for GDS Friday, 24 May 13
3. Michael Brunton-Spall • What have I done? • Rebuilt architecture for guardian.co.uk • (more than once) • Built and scaled Content-API • (more than once) Friday, 24 May 13
4. Content API Beta Friday, 24 May 13
5. Friday, 24 May 13
6. From beta to production Friday, 24 May 13
7. Delivering Business Value Friday, 24 May 13
8. Architecture Guardian Data Centre DB Indexer Solr Solr API Replication Solr API Solr API AWS Friday, 24 May 13
9. Independent Isolated Servers Friday, 24 May 13
10. In house change control Friday, 24 May 13
11. Easy deployment Friday, 24 May 13
12. Changing Business Requirements Friday, 24 May 13
13. Changing Business Requirements Friday, 24 May 13
14. What do we care about now? http://www.flickr.com/photos/revdancatt/614777627 Friday, 24 May 13
15. Speed of publication Friday, 24 May 13
16. This architecture wont do it Friday, 24 May 13
17. We’re serving 12 million requests a day Friday, 24 May 13
18. We have large internal and external clients Friday, 24 May 13
19. What options are there? Friday, 24 May 13
20. The Mega Project http://www.flickr.com/photos/numberelevensmith/5104945904 Friday, 24 May 13
21. Piecewise Replacement http://www.flickr.com/photos/sk8geek/7358702704 Friday, 24 May 13
22. So how did we do it? Friday, 24 May 13
23. Work out what grand vision is Friday, 24 May 13
24. Guardian Data Centre DB Indexer Solr Solr API Replication Solr API Solr API AWS Friday, 24 May 13
25. Guardian DC Data Store Internal API API API Scalable Search Store Notifications AWS API CMS Friday, 24 May 13
26. Find isolation points Friday, 24 May 13
27. Guardian DC Data Store Internal API API API Scalable Search Store Notifications AWS API CMS Friday, 24 May 13
28. How do we get there? Friday, 24 May 13
29. Need to take steps one at a time Friday, 24 May 13
30. Each needs to deliver value Friday, 24 May 13
31. Find next task that has greatest value for cost Friday, 24 May 13
32. Iterate... Friday, 24 May 13
33. So what steps can we take next? Friday, 24 May 13
34. Change Solr to scalable search store Friday, 24 May 13
35. Build internal API + Data store Friday, 24 May 13
36. Replace replication with notifications Friday, 24 May 13
37. So what’s the most value for cost? Friday, 24 May 13
38. Move to Queue Technology http://www.flickr.com/photos/hktang/4243300265 Friday, 24 May 13
39. “Dropin” replacement for replication Friday, 24 May 13
40. Can be run side-by-side Friday, 24 May 13
41. Putting it into production Friday, 24 May 13
42. Early into production Friday, 24 May 13
43. Send notifications into the ether Friday, 24 May 13
44. Guardian Data Centre DB Indexer Solr Solr API Replication Solr API Solr API AWS Notifications Step 1 Friday, 24 May 13
45. Read notifications in test app Friday, 24 May 13
46. Guardian Data Centre DB Indexer Solr Solr API Replication Solr API AWS Notifications Step 2 Google App Engine Monitor Friday, 24 May 13
47. Insert notifications into solr Friday, 24 May 13
48. Guardian Data Centre DB Indexer Solr API Solr API AWS Notifications Step 3 Google App Engine Monitor Updater Updater Friday, 24 May 13
49. Iterate... Friday, 24 May 13
50. Current Latency (seconds) Friday, 24 May 13
51. This approach has a few questions to answer Friday, 24 May 13
52. How do you rebuild an index? Friday, 24 May 13
53. What if I need to change my data format Friday, 24 May 13
54. It’s amazing what you can get away with Friday, 24 May 13
55. Mistakes we made Friday, 24 May 13
56. Git branch http://www.flickr.com/photos/epsos/3432528120 Friday, 24 May 13
57. Branched for notification Friday, 24 May 13
58. Rewrote and renamed projects Friday, 24 May 13
59. Merged back after 10 weeks Friday, 24 May 13
60. Merging was a pain Friday, 24 May 13
61. We don’t have a solution, but it hurt Friday, 24 May 13
62. Business as usual http://www.flickr.com/photos/fras/8015914487/ Friday, 24 May 13
63. We added features Friday, 24 May 13
64. try not to do this! Friday, 24 May 13
65. Not well specified http://www.flickr.com/photos/psd/8591351 Friday, 24 May 13
66. I wrote 3 stories Friday, 24 May 13
67. Send messages Friday, 24 May 13
68. Receive messages Friday, 24 May 13
69. Insert into Solr Friday, 24 May 13
70. (whoops!) Friday, 24 May 13
71. But some things went well Friday, 24 May 13
72. JSON Representation http://www.flickr.com/photos/borkweb/1199 Friday, 24 May 13
73. We can change sending system Friday, 24 May 13
74. We can change receiving code Friday, 24 May 13
75. We have patterns for change now Friday, 24 May 13
76. Give team a vision not a spec http://www.flickr.com/photos/faraz/298921199/ Friday, 24 May 13
77. The team knew the metric we we measuring Friday, 24 May 13
78. They knew the bigger goal Friday, 24 May 13
79. They knew the scope of the change Friday, 24 May 13
80. Message Queues are awesome http://www.flickr.com/photos/darrentunnicliff/3717976312/ Friday, 24 May 13
81. Thanks Friday, 24 May 13
82. Any Questions? @bruntonspall michael@brunton-spall.co.uk Friday, 24 May 13