Stack Exchange

Beginner-level look at how a company can reduce its engineering effort by changing a big chunk of its technology stack while keep delivering value to the general public.

Talk given @Porto Tech Hub, 2015 @rivoli

Transcript

1. STACK EXCHANGE JORGE SILVA, 2015
2. ~2 years ago we detected we had to change the way we were building our web apps
3. CODE PEOPLE INFRASTRUCTURE
4. • stateless monolithic codebases • highly coupled pieces of functionality • Wrecking ship phenomena: you change ->HERE<- and it breaks ->THERE<- • growing the app means more code hence it gets harder to maintain • Bespoke framework worked pretty good for us non-functionally. But • being bespoke meant we had to nurture it and make it accessible to developers • which we didn’t (no documentation/examples makes it harder for people to be sure they’re doing it the right way™) • So it evolved more in the product dimension than on the technological one (diverging priorities) CODE
5. • the guys would often silo themselves away from code they were not responsible for • the boundary was, invariably, the programming language • “You do X I do Y, are we understood?” • this leads to people waiting on other people’s work to be able to proceed • Pipeline • Big codebase means more complexity, tests included • The technical justification for the pipeline being broken was more often than not: “gases” a.k.a voodoo • “just re-kick the build will you?!” • Sad pandas everywhere • Customers were demanding faster state changes in our UIs PEOPLE
6. • stateless monolithic codebases • pieces of functionality highly coupled together • Let me change configuration for this component: CAN’T BE DONE. • 2 years later we find out it was a simple config no one was aware of • FEAR OF CHANGING? • Pipeline often takes more than 1 hour to complete • Oh and do you remember “gases”? They may occur at minute 59 you know… • Necessity for high computing resources due to the nature of the platform itself (and the way the bespoke framework was thought) INFRASTRUCTURE
7. CAN WE DO BETTER? FAILURE ISN’T FATAL, BUT FAILURE TO CHANGE MIGHT BE JOHN WOODEN
8. LETS UNDERSTAND OUR PROBLEMS Business Logic Views Data Web app Service Layer State transfer is an IO bound operation (or it should be) HTTP poll
9. LETS UNDERSTAND OUR PROBLEMS Business Logic Views Data Service Layer Some business logic implies collating data from several services A web app is a collection of specialised views on top of data. Data is king. One size may not fit all.
10. • We can do it in the client side • PROS: Nearer to the views makes it easier to understand how the data flows to match the business specification. • CONS: Browsers have IO limits per domain; Limited resources (remarkably bandwidth); Higher application size; Higher communication latency. • We need an aggregation layer • something product oriented but developer friendly • while respecting non-functional requirements • measure EVERYTHING! Data Collation
11. • We also want to take advantage of the latest user agent RFC’s in order to enrich the UX • We want to send state to the user in a reactive manner • This requirement fits inside our aggregation layer Stream fanout
12. We wanted our engineers to focus on building their applications and not just focus on setting up their environments.
13. LETS UNDERSTAND OUR PROBLEMS Business Logic Views Data Web app Service Layer Aggregation Layer HTTP poll WS push HTTP poll
14. SUM IT UP SUCCESS IS STUMBLING FROM FAILURE TO FAILURE WITH NO LOSS OF ENTHUSIASM WINSTON CHURCHILL
15. • We have products that work • We want to take it to the next level • better, faster products • keeping everyone engaged with the product • our engineers get to play with the latest technologies • the product gets to have the features they’ve dreamt about SUMMARY
16. SUMMARY • We need to continue having products that work • While we change technology stacks • Both for client and the server
17. GO
18. • Customers were demanding faster state changes in our UIs • speed is a feature • We had a thin superset on top of YUI3 acting as framework • a huge userland on top of that • changing it is going to be complex; reusing code ought to be hard • Lets gather a state of the art • Client Side Rendering promises a lot mainly because you turn your application as one downloadable avoiding unnecessary markup roundtrips THE CLIENT
19. (contd.) • Customers were demanding faster state changes in our UIs • speed is a feature • We evaluated what the market had to offer back then and we knew from day 1 we wanted to have • data binding • common “language” to build upon i.e. a framework • The result was favourable to AngularJS • We built iteratively • For each product lets change page by page • One product at a time
20. • Needs to be fast enough • Needs to be easy to develop for • Should not be a framework • should be a thin wrapper on top of proven technologies • should abstract the engineer from the non- functional requirements • should cover all our functional use cases THE SERVER (AGGREGATION LAYER)
21. • We have involved everyone that wanted in developing a solution to a common problem • just a spike • We evaluated all the solutions against a previously agreed weighted list of requirements • The winner was nodejs • excels in async computations • ability to reuse code • huge community • fast enough THE SERVER (AGGREGATION LAYER)
22. • We gathered a set of 6 engineers halftime for 3 months and built a bespoke aggregation tool • including learning javascript • most of the guys were server developers; statically typed languages was their forte; duck typing a sacrilege. • the process occurred while we were building our first new stack product • Exchange mobile site THE SERVER (AGGREGATION LAYER)
23. • The process was as smooth as possible • we knew what we wanted. • we gathered with the product engineers and developed for interfaces • In the end the pieces glued together almost perfectly • 3 months from zero to hero (server + client) THE SERVER (AGGREGATION LAYER)
24. It’s an exciting time @blip We are in the midst of running up against the limits of our monolithic, centralised systems and starting the transition into a portfolio of specialised distributed systems. Surely one of the best challenges in the industry. The possibilities are endless.