Building a highly scalable exchange - To react or not to react

A non-engineering and non-technical deep dive into building a highly scalable exchange. A self-learning and a retrospection into what we did and what we did wrong.

Transcript

1. Building a highly scalable exchange To react or not to react Sreekanth G S Chief Technology Officer Hatio Innovations Private Limited A BillDesk Company
2. Me Thinker Blogger Technologist Programmer Traveler Writer Crypto Evangelist Ideologist 2
3. What this presentation is not A coding exercise A programming tutorial A technology recommendation Or a demo presentation </> 3
4. Instead it is about Our self-learning And a retrospection on What we did And what we did wrong 4
5. We built a crypto currency exchange 5
6. Our initial stack Ruby on Rails PostgreSQL Dockerised Deployments Orchestration using Rancher 6
7. The best part Very easy to build Shortest time to market Handles things pretty well Until… 7
8. The woes Scaling Experience Availability 8
9. What went wrong The stack couldn’t scale Rails view rendering wasn’t efficient Nothing but jQuery in Frontend And Rails for Backend wasn’t the solution 9
10. RCA – The three letter horror Heavy long running SQL queries Lack of efficient caching 502 Bad Gateways on Upstream unavailability No way to let customers know that we are down 10
11. Data, data, data Almost a single page app. Most time spent on Trading. Data is extremely important. Realtime updates. Customers make decisions based on data. 11
12. Realtime updates WebSocket updates Charts & Graphs XHR Requests Events, alerts and notifications 12
13. Scaling – The mammoth task Scaling is a multistep solution Split into - Fixing the SQL queries - Caching of responses - Decoupling the frontend 13
14. Fixing the backend issues SQL Query Optimizations – Moving to Aurora Caching of requests & data – Memcached & Redis Archiving of stale data – Adopting the ELK Stack Taking care of complex weightlifting – C# & .NET 14
15. Initial JavaScript days For simple DOM manipulations Handling of event life cycles – WebSocket updates using jQuery bundled with Rails and Vue.js for simple data bindings 15
16. Limitations of this setup Cluttered and spaghetti JavaScript code Never ending conflicts across multiple framework Frontend was still tightly coupled with backend View rendering was a toll on App Server 16
17. Decoupling the frontend Deciding on a single frontend framework Multiple contenders – Vue, React, Backbone and Angular Roadmap for Mobile Apps – Android & iPhone 17
18. The difficult choice 18
19. Vue React Lightweight Flexible Vue Specific Components Pure JavaScript Better Documentation Larger Community HTML Templating Optional JSX Templating Very easy to Integrate Higher learning curve Limited support for Mobile Apps React Native Difficult to hire engineers Very good talent pool 19
20. The shortlist React it is. Rich Library Support Very good talent pool React Native for Apps 20
21. The learning curve Too minimalistic Concept of a state store was new One-way data binding was challenging Not so good documentation 21
22. JSX as a pain point Developers used to write HTML/Erb Difficulty and inertia learning JSX 22
23. The good part - Components Code became components - Reusable code, with Single Responsibility - Easier to manage - Without any or very low side effects 23
24. Components & Actions let handleMarketTradeHistory = activeMarket => { return dispatch => { const marketTradeHistoryChannel = pusher.subscribe( `market-${activeMarket}-trade-history` ); marketTradeHistoryChannel.bind("market_trade_history", data => { data.trade_history.map(item => dispatch(processTradeHistoryData(item))); }); }; }; 24
25. Dispatchers to do the work export function processTradeHistoryData(data) { return dispatch => { dispatch({ type: ActionType.ADD_TRADE_ORDER, payload: { amount: data.a, created_at: data.T, order_type: data.M, price: data.p, vol: data.q } }); }; } 25
26. The good part – Clean Data Data is better handled - Single storage of data and state - One-way data binding, with clean hierarchy - You cannot accidentally pollute data 26
27. The good part – Invisible DOM The declarative approach -You do not directly touch DOM -Updates are batched and automatic -Think about what and not how 27
28. 100s of lines of code to a few Componentization made code smaller Legacy JavaScript code wasn’t reusable No way to debug or test Now it’s decoupled and self handling 28
29. MVC to Pure API Functionality interwoven with View Single codebase, hard to maintain Transition to Pure API, everything returns data View can be anything, Web or Mobile! 29
30. React Native Future Apps in React Native Pure API model complements this approach Frontend developers can collaborate UX and Apps coming under an umbrella 30
31. Benchmarking Load times between page loads and data population: Category Load Time RoR with No Optimization 2.80s RoR with Backend Optimization 2.03s RoR with React Frontend 1.80s RoR with Backend Optimization & React Frontend 1.07s 31
32. Unsolved Issues Virtual DOM is at times slow Upgrading and updating is not easy as expected High memory consumption – 16.3 MB Slow startup times – 219 ms 32
33. The learnings Scale is not a frontend/backend only problem Customers tolerate informed issues, not 502s Developer happiness is extremely important Initial learning curve can be future productivity 33
34. The takeaways Rewriting entire code base isn’t easy. When you are forced to do it, decide wisely. React is not a one stop solution, it was a fit for us. At the end of the day, its about delivering. 34
35. Thank You @sreekanthgs 35