The Architecture of Stack Overflow - Dev Sum 2014

The Architecture of Stack Overflow - Dev Sum 2014

3fd9e5b2c59170ec3d74dde30d233fa4?s=128

Marco Cecconi

May 21, 2014
Tweet

Transcript

  1. The Architecture Of Marco Cecconi @sklivvz http://sklivvz.com

  2. Sizing up the problem

  3. None
  4. None
  5. *source: Quantcast, Alexa #56 network for traffic*

  6. #56 network for traffic* …and #8 in Sweden!? *source: Quantcast,

    Alexa much successful very traffic TACK
  7. 520,523,008 pageviews in the last 30 days* *source: Quantcast

  8. 520,523,008 pageviews in the last 30 days* *source: Quantcast

  9. So, how big is our datacenter?

  10. None
  11. web servers load balancers redis search database http(s) http rest

    http protobuf sql sql protobuf tag engine
  12. None
  13. We are still scaling up!

  14. Our development cycle

  15. BATCAVE Code-build-test cycle running on home machine feature requests

  16. BATCAVE DEV.SO Test on the real servers git push

  17. BATCAVE DEV.SO META.SE 1-click deploy Users test on meta.stackexchange.com (“baking”)

    git push
  18. BATCAVE DEV.SO META.SE 1-click deploy HALP! git revert git push

  19. BATCAVE DEV.SO NETWORK META.SE 1-click deploy git push 1-click deploy

    It’s live! Tons of users use it…
  20. BATCAVE DEV.SO NETWORK META.SE 1-click deploy git push 1-click deploy

    …and provide new feature requests
  21. Move fast and break things* * Not the home page

    or question page :-)
  22. Move fast and break things* * Not the home page

    or question page :-)
  23. Caching

  24. Network Level Caches (CDN, etc.) Server Level Cache (HttpRuntime.Cache) Site

    Level Cache (Redis) SQL Server Database Cache (384 gigs of RAM!) Solid State Disk
  25. None
  26. None
  27. None
  28. None
  29. Too Many Allocations This is really the most basic thing

    that can go wrong. Too Many Pointers If you create a data structure that is a large mesh of pointers you'll have two problems. First, there will be a lot of object writes […] and, secondly, when it comes time to collect that data structure, you will make the garbage collector follow all those pointers and if necessary change them all as things move around. […] But if you create such a structure on a transitory basis, […], then you will pay the cost much more often. http://msdn.microsoft.com/en-us/library/ms973837.aspx#dotnetgcbasics_topic2
  30. Too Many Allocations This is really the most basic thing

    that can go wrong. Too Many Pointers If you create a data structure that is a large mesh of pointers you'll have two problems. First, there will be a lot of object writes […] and, secondly, when it comes time to collect that data structure, you will make the garbage collector follow all those pointers and if necessary change them all as things move around. […] But if you create such a structure on a transitory basis, […], then you will pay the cost much more often. http://msdn.microsoft.com/en-us/library/ms973837.aspx#dotnetgcbasics_topic2
  31. None
  32. None
  33. None
  34. None
  35. Abuse caching for GC performance

  36. Too Many Allocations This is really the most basic thing

    that can go wrong. Too Many Pointers If you create a data structure that is a large mesh of pointers you'll have two problems. First, there will be a lot of object writes […] and, secondly, when it comes time to collect that data structure, you will make the garbage collector follow all those pointers and if necessary change them all as things move around. […] But if you create such a structure on a transitory basis, […], then you will pay the cost much more often. http://msdn.microsoft.com/en-us/library/ms973837.aspx#dotnetgcbasics_topic2
  37. Too Many Allocations This is really the most basic thing

    that can go wrong. Too Many Pointers If you create a data structure that is a large mesh of pointers you'll have two problems. First, there will be a lot of object writes […] and, secondly, when it comes time to collect that data structure, you will make the garbage collector follow all those pointers and if necessary change them all as things move around. […] But if you create such a structure on a transitory basis, […], then you will pay the cost much more often. http://msdn.microsoft.com/en-us/library/ms973837.aspx#dotnetgcbasics_topic2
  38. None
  39. None
  40. IRepository<Order> orderRepository = container.Resolve<IRepository<Order>>(); Order order = orderRepository.Get(35); This is

    what you think you are doing…
  41. …but if you think about it a bit more…

  42. ...this is what you are actually doing! IRepository<Order> repository =

    new ValidatingOrderRepository ( new SecurityRepository<Order> ( new LoggingRepository<Order> ( new CachingRepository<Order> ( new NHibernateRepository<Order> () ) ) ) ); Order order = repository.Get(35);
  43. We don’t use dependency- injection or IoC containers

  44. Our source code

  45. Few projects :-)

  46. Few projects :-) Few lines of code :-)

  47. Few projects :-) Few lines of code :-) Eeek! very

    few tests :-S
  48. Few projects :-) Few lines of code :-) Awesome community

    to help :-D Eeek! very few tests :-S
  49. YAGNI* * You Ain’t Gonna Need It!

  50. Libraries and open source

  51. * Source http://bit.ly/1eSLr8Z

  52. None
  53. None
  54. None
  55. Wrap code in libraries and open source it

  56. None
  57. None
  58. None
  59. DEEP DIVES

  60. HIRE “A” PLAYERS

  61. TOOLS LEVEL: NINJA

  62. • Performance is a feature • Always. Be. Shipping. •

    Use your circumstances. • Open source your libraries • 3 obscenely big monitors. KEY TAKEAWAYS
  63. None
  64. $_='Qgul=ar=ftq=pmk:=ftq=mooagzfmzf=puxqyym.=Guhqz=m=eqcgqzoq=ar= bduoqe,=IEzgyqdmnxq=ar=uzf=ruzp=ftq=rudef=N=ftmf=egy=fa=lqda,=itq dq=N=ue=m=bmdmy.=Ttq=xuef=yustf=zaf=nq=ruzufq=uz=eulq.=Sqzp=kagd= eaxgfuaz=fa=ewxuhhl&efmowahqdrxai.oay1';tr/=1m-za-l@&Z/ !a- zP@\n/&print; Marco Cecconi @sklivvz http://sklivvz.com