The Architecture of Stack Overflow - Merge Lisbon 2014

The Architecture of Stack Overflow slides from Merge Lisbon 2014

Transcript

1. The  Architecture  Of  
   Marco Cecconi
   @sklivvz
   http://sklivvz.com
   

   View Slide

2. Sizing  up  the  problem  
   

   View Slide

3. View Slide

4. View Slide

5. View Slide

6. *source: Quantcast, Alexa
   #50-­‐ish  network  for  traffic*  
   (between imdb.com and t.co)
   

   View Slide

7. #50-­‐ish  network  for  traffic*  
   …and  #17  in  Portugal!?    
   *source: Quantcast, Alexa
    
   much successful
   very traffic
   obrigado, yo!
   

   View Slide

8. 543,255,552  pageviews  in  the  last  30  days*  
   *source: Quantcast
   

   View Slide

9. Our  datacenter?  
   

   View Slide

10. View Slide

11. web servers
    load balancers
    redis
    search
    database
    http(s)
    http
    rest
    http
    protobuf
    sql
    sql
    protobuf
    tag engine
    

    View Slide

12. View Slide

13. We  are  sNll  scaling  up!  
    

    View Slide

14. Our  development  cycle  
    

    View Slide

15. BAT!CAVE
    Code-build-test cycle
    running on home machine
    feature requests
    

    View Slide

16. BAT!CAVE DEV.SO
    Test on the real servers
    git push
    

    View Slide

17. BAT!CAVE DEV.SO
    META.SE
    1-click deploy
    Users test on
    meta.stackexchange.com
    (“baking”)
    git push
    

    View Slide

18. BAT!CAVE DEV.SO
    META.SE
    1-click deploy
    HALP!
    git revert
    git push
    

    View Slide

19. BAT!CAVE DEV.SO
    NETWORK META.SE
    1-click deploy
    git push
    1-click
    deploy
    It’s live! Tons of users use it…
    

    View Slide

20. BAT!CAVE DEV.SO
    NETWORK META.SE
    1-click deploy
    git push
    1-click
    deploy
    …and
    provide new
    feature requests
    

    View Slide

21. Move  fast  and  break  things*  
    * Not the home page or question page :-)
    

    View Slide

22. Move  fast  and  break  things*  
    * Not the home page or question page :-)
    

    View Slide

23. Abusing  caching  
    

    View Slide

24. Network Level Caches (CDN, etc.)
    Server Level Cache (HttpRuntime.Cache)
    Site Level Cache (Redis)
    SQL Server Database Cache (384 gigs of RAM!)
    Solid State Disk
    

    View Slide

25. View Slide

26. View Slide

27. View Slide

28. RisoQo  &  SpagheS  
    

    View Slide

29. View Slide

30. View Slide

31. View Slide

32. View Slide

33. View Slide

34. Throwing away risotto… one grain at a time!
    

    View Slide

35. Abuse  caching  for  GC  
    performance  
    

    View Slide

36. IRepository  orderRepository  =    
           container.Resolve>();  
     
    Order  order  =  orderRepository.Get(35);
    This is what you think you are doing…
    

    View Slide

37. …but if you think about it a bit more…
    

    View Slide

38. ...this is what you are actually doing!
    IRepository  repository  =  
       new  ValidatingOrderRepository  (  
           new  SecurityRepository  (  
               new  LoggingRepository  (  
                   new  CachingRepository  (  
                       new  NHibernateRepository  ()  
                   )  
               )  
           )  
       );  
         
    Order  order  =  repository.Get(35);
    

    View Slide

39. Throwing away spaghetti… one strand at a time!
    

    View Slide

40. Dependency  InjecNon  is  our  
    version  of  spagheS  code  
    

    View Slide

41. Our  source  code  
    

    View Slide

42. Few projects :-)
    

    View Slide

43. Few projects :-) Few lines of code :-)
    

    View Slide

44. Few projects :-) Few lines of code :-)
    Eeek! very few tests :-S
    

    View Slide

45. Few projects :-) Few lines of code :-)
    Awesome community to help :-D
    Eeek! very few tests :-S
    

    View Slide

46. YAGNI*  
    * You Ain’t Gonna Need It!
    

    View Slide

47. Libraries  and  open  source  
    

    View Slide

48. View Slide

49. View Slide

50. View Slide

51. Wrap  code  in  libraries    
    and  open  source  it  
    

    View Slide

52. View Slide

53. View Slide

54. View Slide

55. “SUPERHERO” CODING
    

    View Slide

56. HIRE “NINJA” CODERS
    

    View Slide

57. TOOLS LEVEL: “NINJA”
    

    View Slide

58. •     Performance  is  a  feature  
    •     Always.  Be.  Shipping.  
    •     Use  your  circumstances.  
    •     Open  source  your  libraries  
    •     3  obscenely  big  monitors.    
    KEY TAKEAWAYS
    

    View Slide

59. View Slide

60. Marco Cecconi
    @sklivvz
    http://sklivvz.com
    we are hiring new ninjas from anywhere in the world!
    

    View Slide