The Architecture of Stack Overflow - Info Share 2014

Transcript

1. The Architecture Of Marco Cecconi @sklivvz http://sklivvz.com
2. Sizing up the problem
5. *source: Quantcast, Alexa #55 network for traffic*
6. #55 network for traffic* …and #13 in Poland!? *source: Quantcast, Alexa much successful very traffic dziękuję
7. 520,523,008 pageviews in the last 30 days* *source: Quantcast
8. So, how big is our datacenter?
10. web servers load balancers redis search database http(s) http rest http protobuf sql sql protobuf tag engine
12. We are still scaling up!
13. Our development cycle
14. BATCAVE Code-build-test cycle running on home machine feature requests
15. BATCAVE DEV.SO Test on the real servers git push
16. BATCAVE DEV.SO META.SE 1-click deploy Users test on meta.stackexchange.com (“baking”) git push
17. BATCAVE DEV.SO META.SE 1-click deploy HALP! git revert git push
18. BATCAVE DEV.SO NETWORK META.SE 1-click deploy git push 1-click deploy It’s live! Tons of users use it…
19. BATCAVE DEV.SO NETWORK META.SE 1-click deploy git push 1-click deploy …and provide new feature requests
20. Move fast and break things* * Not the home page or question page :-)
21. Move fast and break things* * Not the home page or question page :-)
22. Caching
23. Network Level Caches (CDN, etc.) Server Level Cache (HttpRuntime.Cache) Site Level Cache (Redis) SQL Server Database Cache (384 gigs of RAM!) Solid State Disk
28. Too Many Allocations This is really the most basic thing that can go wrong. Too Many Pointers If you create a data structure that is a large mesh of pointers you'll have two problems. First, there will be a lot of object writes […] and, secondly, when it comes time to collect that data structure, you will make the garbage collector follow all those pointers and if necessary change them all as things move around. […] But if you create such a structure on a transitory basis, […], then you will pay the cost much more often. http://msdn.microsoft.com/en-us/library/ms973837.aspx#dotnetgcbasics_topic2
29. Too Many Allocations This is really the most basic thing that can go wrong. Too Many Pointers If you create a data structure that is a large mesh of pointers you'll have two problems. First, there will be a lot of object writes […] and, secondly, when it comes time to collect that data structure, you will make the garbage collector follow all those pointers and if necessary change them all as things move around. […] But if you create such a structure on a transitory basis, […], then you will pay the cost much more often. http://msdn.microsoft.com/en-us/library/ms973837.aspx#dotnetgcbasics_topic2
34. Abuse caching for GC performance
35. Too Many Allocations This is really the most basic thing that can go wrong. Too Many Pointers If you create a data structure that is a large mesh of pointers you'll have two problems. First, there will be a lot of object writes […] and, secondly, when it comes time to collect that data structure, you will make the garbage collector follow all those pointers and if necessary change them all as things move around. […] But if you create such a structure on a transitory basis, […], then you will pay the cost much more often. http://msdn.microsoft.com/en-us/library/ms973837.aspx#dotnetgcbasics_topic2
36. Too Many Allocations This is really the most basic thing that can go wrong. Too Many Pointers If you create a data structure that is a large mesh of pointers you'll have two problems. First, there will be a lot of object writes […] and, secondly, when it comes time to collect that data structure, you will make the garbage collector follow all those pointers and if necessary change them all as things move around. […] But if you create such a structure on a transitory basis, […], then you will pay the cost much more often. http://msdn.microsoft.com/en-us/library/ms973837.aspx#dotnetgcbasics_topic2
39. IRepository<Order> orderRepository = container.Resolve<IRepository<Order>>(); Order order = orderRepository.Get(35); This is what you think you are doing…
40. …but if you think about it a bit more…
41. ...this is what you are actually doing! IRepository<Order> repository = new ValidatingOrderRepository ( new SecurityRepository<Order> ( new LoggingRepository<Order> ( new CachingRepository<Order> ( new NHibernateRepository<Order> () ) ) ) ); Order order = repository.Get(35);
42. We don’t use dependency- injection or IoC containers
43. Our source code
44. Few projects :-)
45. Few projects :-) Few lines of code :-)
46. Few projects :-) Few lines of code :-) Eeek! very few tests :-S
47. Few projects :-) Few lines of code :-) Awesome community to help :-D Eeek! very few tests :-S
48. YAGNI* * You Ain’t Gonna Need It!
49. Libraries and open source
50. * Source http://bit.ly/1eSLr8Z
54. Wrap code in libraries and open source it
58. DEEP DIVES
59. HIRE “A” PLAYERS
60. TOOLS LEVEL: NINJA
61. • Performance is a feature • Always. Be. Shipping. • Use your circumstances. • Open source your libraries • 3 obscenely big monitors. KEY TAKEAWAYS
63. $_='Qgul=ar=ftq=pmk:=ftq=mooagzfmzf=puxqyym.=Guhqz=m=eqcgqzoq=ar= bduoqe,=IEzgyqdmnxq=ar=uzf=ruzp=ftq=rudef=N=ftmf=egy=fa=lqda,=itq dq=N=ue=m=bmdmy.=Ttq=xuef=yustf=zaf=nq=ruzufq=uz=eulq.=Sqzp=kagd= eaxgfuaz=fa=ewxuhhl&efmowahqdrxai.oay1';tr/=1m-za-l@&Z/ !a- zP@\n/&print; Marco Cecconi @sklivvz http://sklivvz.com