The Architecture of StackOverflow - QCon China 2014

The Architecture of Stack OVerflow - Slides updated for QCon China 2014 including data for April 2014, new slides on garbage collection and China-specific content.

Transcript

1. The Architecture Of
   Marco Cecconi
   @sklivvz
   sklivvz@stackoverflow.com
   

   View Slide

2. View Slide

3. View Slide

4. #49 network for traffic*
   *source: Quantcast, Alexa
   

   View Slide

5. #49 network for traffic*
   …and #7 in China!
   *source: Quantcast, Alexa
   much successful
   very traffic
   谢谢
   wow
   

   View Slide

6. 561,027,840 pageviews in the last 30 days*
   (~100% growth year over year)
   *source: Quantcast
   

   View Slide

7. 561,027,840 pageviews in the last 30 days*
   (~100% growth year over year)
   *source: Quantcast
   

   View Slide

8. View Slide

9. web servers
   load balancers
   redis
   search
   database
   http(s)
   http
   rest
   http
   protobuf
   sql
   sql
   protobuf
   tag engine
   

   View Slide

10. View Slide

11. We are still scaling up, yo!
    

    View Slide

12. BATCAVE
    

    View Slide

13. BATCAVE DEV.SO
    

    View Slide

14. BATCAVE DEV.SO
    META.SO
    

    View Slide

15. BATCAVE DEV.SO
    META.SO
    

    View Slide

16. BATCAVE DEV.SO
    META.SO
    NETWORK
    

    View Slide

17. Move fast and break things
    

    View Slide

18. Move fast and break things*
    * Not the home page or question page :-)
    

    View Slide

19. Move fast and break things*
    * Not the home page or question page :-)
    

    View Slide

20. Network Level Caches (CDN, etc.)
    Server Level Cache (HttpRuntime.Cache)
    Site Level Cache (Redis)
    SQL Server Database Cache (384 gigs of RAM!)
    Solid State Disk
    

    View Slide

21. View Slide

22. View Slide

23. View Slide

24. View Slide

25. Too Many Allocations
    This is really the most basic thing that can go wrong.
    Too Many Pointers
    If you create a data structure that is a large mesh of pointers
    you'll have two problems. First, there will be a lot of object
    writes […] and, secondly, when it comes time to collect that
    data structure, you will make the garbage collector follow all
    those pointers and if necessary change them all as things
    move around. […] But if you create such a structure on a
    transitory basis, […], then you will pay the cost much more
    often.
    http://msdn.microsoft.com/en-us/library/ms973837.aspx#dotnetgcbasics_topic2
    

    View Slide

26. Too Many Allocations
    This is really the most basic thing that can go wrong.
    Too Many Pointers
    If you create a data structure that is a large mesh of pointers
    you'll have two problems. First, there will be a lot of object
    writes […] and, secondly, when it comes time to collect that
    data structure, you will make the garbage collector follow all
    those pointers and if necessary change them all as things
    move around. […] But if you create such a structure on a
    transitory basis, […], then you will pay the cost much more
    often.
    http://msdn.microsoft.com/en-us/library/ms973837.aspx#dotnetgcbasics_topic2
    

    View Slide

27. View Slide

28. View Slide

29. View Slide

30. View Slide

31. Too Many Allocations
    This is really the most basic thing that can go wrong.
    Too Many Pointers
    If you create a data structure that is a large mesh of pointers
    you'll have two problems. First, there will be a lot of object
    writes […] and, secondly, when it comes time to collect that
    data structure, you will make the garbage collector follow all
    those pointers and if necessary change them all as things
    move around. […] But if you create such a structure on a
    transitory basis, […], then you will pay the cost much more
    often.
    http://msdn.microsoft.com/en-us/library/ms973837.aspx#dotnetgcbasics_topic2
    

    View Slide

32. IRepository orderRepository =
    container.Resolve>();
    Order order = orderRepository.Get(35);
    This is what you think you are doing…
    

    View Slide

33. …but if you think about it a bit more…
    

    View Slide

34. ...this is what you are actually doing!
    IRepository repository =
    new ValidatingOrderRepository (
    new SecurityRepository (
    new LoggingRepository (
    new CachingRepository (
    new NHibernateRepository ()
    )
    )
    )
    );
    Order order = repository.Get(35);
    

    View Slide

35. View Slide

36. View Slide

37. View Slide

38. 馄饨！
    

    View Slide

39. Few projects :-)
    

    View Slide

40. Few projects :-) Few lines of code :-)
    

    View Slide

41. Few projects :-) Few lines of code :-)
    Eeek! very few tests :-S
    

    View Slide

42. Few projects :-) Few lines of code :-)
    Awesome community to help :-D
    Eeek! very few tests :-S
    

    View Slide

43. YAGNI*.
    It works.
    * You Ain’t Gonna Need It!
    

    View Slide

44. * Source http://bit.ly/1eSLr8Z
    

    View Slide

45. View Slide

46. View Slide

47. View Slide

48. Write libraries &
    open them to the world
    

    View Slide

49. View Slide

50. View Slide

51. View Slide

52. DEEP DIVES
    

    View Slide

53. OPINIONATED
    HIRING
    

    View Slide

54. REMOTE WORK
    

    View Slide

55. • Performance is a feature
    • Always. Be. Shipping.
    • Use your circumstances.
    • Open source your libraries
    • 3 obscenely big monitors.
    CONCLUSION
    

    View Slide

56. $_='@mk=uf=radimdp1Z--&ewxuhhl';tr/=1m-za-l@&Z/ !a-zP@\n/&print;
    Marco Cecconi
    @sklivvz
    sklivvz@stackoverflow.com
    

    View Slide