The Architecture of Stack Overflow - QCon Japan 2014

Slides for the Architecture of Stack Overflow talk at QCon Japan 2014

Transcript

1. The Architecture Of
   Marco Cecconi
   @sklivvz
   sklivvz@stackoverflow.com
   

   View Slide

2. Sizing up the problem
   

   View Slide

3. View Slide

4. View Slide

5. #48 network for traffic*
   *source: Quantcast, Alexa
   

   View Slide

6. #48 network for traffic*
   …and #17 in Japan!
   *source: Quantcast, Alexa
   much successful
   very traffic
   ありがとう
   wow
   

   View Slide

7. 522,678,720 pageviews in the last 30 days*
   (~100% growth year over year)
   *source: Quantcast
   

   View Slide

8. 522,678,720 pageviews in the last 30 days*
   (~100% growth year over year)
   *source: Quantcast
   

   View Slide

9. So, how big is our datacenter?
   

   View Slide

10. View Slide

11. web servers
    load balancers
    redis
    search
    database
    http(s)
    http
    rest
    http
    protobuf
    sql
    sql
    protobuf
    tag engine
    

    View Slide

12. View Slide

13. We are still scaling up!
    

    View Slide

14. Our development cycle
    

    View Slide

15. BATCAVE
    Code-build-test cycle
    running on home machine
    feature requests
    

    View Slide

16. BATCAVE DEV.SO
    Test on the real servers
    git push
    

    View Slide

17. BATCAVE DEV.SO
    META.SE
    1-click deploy
    Users test on
    meta.stackexchange.com
    (“baking”)
    git push
    

    View Slide

18. BATCAVE DEV.SO
    META.SE
    1-click deploy
    HALP!
    git revert
    git push
    

    View Slide

19. BATCAVE DEV.SO
    NETWORK META.SE
    1-click deploy
    git push
    1-click
    deploy
    It’s live! Tons of users use it…
    

    View Slide

20. BATCAVE DEV.SO
    NETWORK META.SE
    1-click deploy
    git push
    1-click
    deploy
    …and
    provide new
    feature requests
    

    View Slide

21. Move fast and break things*
    * Not the home page or question page :-)
    

    View Slide

22. Move fast and break things*
    * Not the home page or question page :-)
    

    View Slide

23. Caching
    

    View Slide

24. Network Level Caches (CDN, etc.)
    Server Level Cache (HttpRuntime.Cache)
    Site Level Cache (Redis)
    SQL Server Database Cache (384 gigs of RAM!)
    Solid State Disk
    

    View Slide

25. View Slide

26. View Slide

27. View Slide

28. View Slide

29. Too Many Allocations
    This is really the most basic thing that can go wrong.
    Too Many Pointers
    If you create a data structure that is a large mesh of pointers
    you'll have two problems. First, there will be a lot of object
    writes […] and, secondly, when it comes time to collect that
    data structure, you will make the garbage collector follow all
    those pointers and if necessary change them all as things
    move around. […] But if you create such a structure on a
    transitory basis, […], then you will pay the cost much more
    often.
    http://msdn.microsoft.com/en-us/library/ms973837.aspx#dotnetgcbasics_topic2
    

    View Slide

30. Too Many Allocations
    This is really the most basic thing that can go wrong.
    Too Many Pointers
    If you create a data structure that is a large mesh of pointers
    you'll have two problems. First, there will be a lot of object
    writes […] and, secondly, when it comes time to collect that
    data structure, you will make the garbage collector follow all
    those pointers and if necessary change them all as things
    move around. […] But if you create such a structure on a
    transitory basis, […], then you will pay the cost much more
    often.
    http://msdn.microsoft.com/en-us/library/ms973837.aspx#dotnetgcbasics_topic2
    

    View Slide

31. View Slide

32. View Slide

33. View Slide

34. View Slide

35. Abuse caching for GC
    performance
    

    View Slide

36. Too Many Allocations
    This is really the most basic thing that can go wrong.
    Too Many Pointers
    If you create a data structure that is a large mesh of pointers
    you'll have two problems. First, there will be a lot of object
    writes […] and, secondly, when it comes time to collect that
    data structure, you will make the garbage collector follow all
    those pointers and if necessary change them all as things
    move around. […] But if you create such a structure on a
    transitory basis, […], then you will pay the cost much more
    often.
    http://msdn.microsoft.com/en-us/library/ms973837.aspx#dotnetgcbasics_topic2
    

    View Slide

37. Too Many Allocations
    This is really the most basic thing that can go wrong.
    Too Many Pointers
    If you create a data structure that is a large mesh of pointers
    you'll have two problems. First, there will be a lot of object
    writes […] and, secondly, when it comes time to collect that
    data structure, you will make the garbage collector follow all
    those pointers and if necessary change them all as things
    move around. […] But if you create such a structure on a
    transitory basis, […], then you will pay the cost much more
    often.
    http://msdn.microsoft.com/en-us/library/ms973837.aspx#dotnetgcbasics_topic2
    

    View Slide

38. View Slide

39. View Slide

40. IRepository orderRepository =
    container.Resolve>();
    Order order = orderRepository.Get(35);
    This is what you think you are doing…
    

    View Slide

41. …but if you think about it a bit more…
    

    View Slide

42. ...this is what you are actually doing!
    IRepository repository =
    new ValidatingOrderRepository (
    new SecurityRepository (
    new LoggingRepository (
    new CachingRepository (
    new NHibernateRepository ()
    )
    )
    )
    );
    Order order = repository.Get(35);
    

    View Slide

43. We don’t use dependency-
    injection or IoC containers
    

    View Slide

44. Our source code
    

    View Slide

45. Few projects :-)
    

    View Slide

46. Few projects :-) Few lines of code :-)
    

    View Slide

47. Few projects :-) Few lines of code :-)
    Eeek! very few tests :-S
    

    View Slide

48. Few projects :-) Few lines of code :-)
    Awesome community to help :-D
    Eeek! very few tests :-S
    

    View Slide

49. YAGNI*
    * You Ain’t Gonna Need It!
    

    View Slide

50. Libraries and open source
    

    View Slide

51. * Source http://bit.ly/1eSLr8Z
    

    View Slide

52. View Slide

53. View Slide

54. View Slide

55. Wrap code in libraries
    and open source it
    

    View Slide

56. View Slide

57. View Slide

58. View Slide

59. DEEP DIVES
    

    View Slide

60. HIRE “A” PLAYERS
    

    View Slide

61. NINJA TOOLS
    

    View Slide

62. • Performance is a feature
    • Always. Be. Shipping.
    • Use your circumstances.
    • Open source your libraries
    • 3 obscenely big monitors.
    CONCLUSION
    

    View Slide

63. 日本語版のStack Overflow を
    開設します！
    コミュニティマネージャー
    になりませんか？
    ONE LAST THING…
    

    View Slide

64. $_='@mk=uf=radimdp1Z--&ewxuhhl';tr/=1m-za-l@&Z/ !a-zP@\n/&print;
    Marco Cecconi
    @sklivvz
    sklivvz@stackoverflow.com
    

    View Slide