Building a highly scalable exchange - To react or not to react

Slide 1

Slide 1 text

Building a highly scalable exchange To react or not to react Sreekanth G S Chief Technology Officer Hatio Innovations Private Limited A BillDesk Company

Slide 2

Slide 2 text

Me Thinker Blogger Technologist Programmer Traveler Writer Crypto Evangelist Ideologist 2

Slide 3

Slide 3 text

What this presentation is not A coding exercise A programming tutorial A technology recommendation Or a demo presentation 3

Slide 4

Slide 4 text

Instead it is about Our self-learning And a retrospection on What we did And what we did wrong 4

Slide 5

Slide 5 text

We built a crypto currency exchange 5

Slide 6

Slide 6 text

Our initial stack Ruby on Rails PostgreSQL Dockerised Deployments Orchestration using Rancher 6

Slide 7

Slide 7 text

The best part Very easy to build Shortest time to market Handles things pretty well Until… 7

Slide 8

Slide 8 text

The woes Scaling Experience Availability 8

Slide 9

Slide 9 text

What went wrong The stack couldn’t scale Rails view rendering wasn’t efficient Nothing but jQuery in Frontend And Rails for Backend wasn’t the solution 9

Slide 10

Slide 10 text

RCA – The three letter horror Heavy long running SQL queries Lack of efficient caching 502 Bad Gateways on Upstream unavailability No way to let customers know that we are down 10

Slide 11

Slide 11 text

Data, data, data Almost a single page app. Most time spent on Trading. Data is extremely important. Realtime updates. Customers make decisions based on data. 11

Slide 12

Slide 12 text

Realtime updates WebSocket updates Charts & Graphs XHR Requests Events, alerts and notifications 12

Slide 13

Slide 13 text

Scaling – The mammoth task Scaling is a multistep solution Split into - Fixing the SQL queries - Caching of responses - Decoupling the frontend 13

Slide 14

Slide 14 text

Fixing the backend issues SQL Query Optimizations – Moving to Aurora Caching of requests & data – Memcached & Redis Archiving of stale data – Adopting the ELK Stack Taking care of complex weightlifting – C# & .NET 14

Slide 15

Slide 15 text

Initial JavaScript days For simple DOM manipulations Handling of event life cycles – WebSocket updates using jQuery bundled with Rails and Vue.js for simple data bindings 15

Slide 16

Slide 16 text

Limitations of this setup Cluttered and spaghetti JavaScript code Never ending conflicts across multiple framework Frontend was still tightly coupled with backend View rendering was a toll on App Server 16

Slide 17

Slide 17 text

Decoupling the frontend Deciding on a single frontend framework Multiple contenders – Vue, React, Backbone and Angular Roadmap for Mobile Apps – Android & iPhone 17

Slide 18

Slide 18 text

The difficult choice 18

Slide 19

Slide 19 text

Vue React Lightweight Flexible Vue Specific Components Pure JavaScript Better Documentation Larger Community HTML Templating Optional JSX Templating Very easy to Integrate Higher learning curve Limited support for Mobile Apps React Native Difficult to hire engineers Very good talent pool 19

Slide 20

Slide 20 text

The shortlist React it is. Rich Library Support Very good talent pool React Native for Apps 20

Slide 21

Slide 21 text

The learning curve Too minimalistic Concept of a state store was new One-way data binding was challenging Not so good documentation 21

Slide 22

Slide 22 text

JSX as a pain point Developers used to write HTML/Erb Difficulty and inertia learning JSX 22

Slide 23

Slide 23 text

The good part - Components Code became components - Reusable code, with Single Responsibility - Easier to manage - Without any or very low side effects 23

Slide 24

Slide 24 text

Components & Actions let handleMarketTradeHistory = activeMarket => { return dispatch => { const marketTradeHistoryChannel = pusher.subscribe( `market-${activeMarket}-trade-history` ); marketTradeHistoryChannel.bind("market_trade_history", data => { data.trade_history.map(item => dispatch(processTradeHistoryData(item))); }); }; }; 24

Slide 25

Slide 25 text

Dispatchers to do the work export function processTradeHistoryData(data) { return dispatch => { dispatch({ type: ActionType.ADD_TRADE_ORDER, payload: { amount: data.a, created_at: data.T, order_type: data.M, price: data.p, vol: data.q } }); }; } 25

Slide 26

Slide 26 text

The good part – Clean Data Data is better handled - Single storage of data and state - One-way data binding, with clean hierarchy - You cannot accidentally pollute data 26

Slide 27

Slide 27 text

The good part – Invisible DOM The declarative approach -You do not directly touch DOM -Updates are batched and automatic -Think about what and not how 27

Slide 28

Slide 28 text

100s of lines of code to a few Componentization made code smaller Legacy JavaScript code wasn’t reusable No way to debug or test Now it’s decoupled and self handling 28

Slide 29

Slide 29 text

MVC to Pure API Functionality interwoven with View Single codebase, hard to maintain Transition to Pure API, everything returns data View can be anything, Web or Mobile! 29

Slide 30

Slide 30 text

React Native Future Apps in React Native Pure API model complements this approach Frontend developers can collaborate UX and Apps coming under an umbrella 30

Slide 31

Slide 31 text

Benchmarking Load times between page loads and data population: Category Load Time RoR with No Optimization 2.80s RoR with Backend Optimization 2.03s RoR with React Frontend 1.80s RoR with Backend Optimization & React Frontend 1.07s 31

Slide 32

Slide 32 text

Unsolved Issues Virtual DOM is at times slow Upgrading and updating is not easy as expected High memory consumption – 16.3 MB Slow startup times – 219 ms 32

Slide 33

Slide 33 text

The learnings Scale is not a frontend/backend only problem Customers tolerate informed issues, not 502s Developer happiness is extremely important Initial learning curve can be future productivity 33

Slide 34

Slide 34 text

The takeaways Rewriting entire code base isn’t easy. When you are forced to do it, decide wisely. React is not a one stop solution, it was a fit for us. At the end of the day, its about delivering. 34

Slide 35

Slide 35 text

Thank You @sreekanthgs 35