$30 off During Our Annual Pro Sale. View Details »

Building a highly scalable exchange - To react or not to react

Building a highly scalable exchange - To react or not to react

A non-engineering and non-technical deep dive into building a highly scalable exchange. A self-learning and a retrospection into what we did and what we did wrong.

Sreekanth G S

March 02, 2019
Tweet

Other Decks in Technology

Transcript

  1. Building a highly scalable exchange
    To react or not to react
    Sreekanth G S
    Chief Technology Officer
    Hatio Innovations Private Limited
    A BillDesk Company

    View Slide

  2. Me
    Thinker Blogger
    Technologist
    Programmer
    Traveler
    Writer
    Crypto Evangelist
    Ideologist
    2

    View Slide

  3. What this presentation is not
    A coding exercise
    A programming tutorial
    A technology recommendation
    Or a demo presentation
    >
    3

    View Slide

  4. Instead it is about
    Our self-learning
    And a retrospection on
    What we did
    And what we did wrong
    4

    View Slide

  5. We built a crypto currency exchange
    5

    View Slide

  6. Our initial stack
    Ruby on Rails
    PostgreSQL
    Dockerised Deployments
    Orchestration using Rancher
    6

    View Slide

  7. The best part
    Very easy to build
    Shortest time to market
    Handles things pretty well
    Until…
    7

    View Slide

  8. The woes
    Scaling Experience
    Availability
    8

    View Slide

  9. What went wrong
    The stack couldn’t scale
    Rails view rendering wasn’t efficient
    Nothing but jQuery in Frontend
    And Rails for Backend wasn’t the solution
    9

    View Slide

  10. RCA – The three letter horror
    Heavy long running SQL queries
    Lack of efficient caching
    502 Bad Gateways on Upstream unavailability
    No way to let customers know that we are down
    10

    View Slide

  11. Data, data, data
    Almost a single page app.
    Most time spent on Trading.
    Data is extremely important. Realtime updates.
    Customers make decisions based on data.
    11

    View Slide

  12. Realtime updates
    WebSocket updates
    Charts & Graphs
    XHR Requests
    Events, alerts and notifications
    12

    View Slide

  13. Scaling – The mammoth task
    Scaling is a multistep solution
    Split into
    - Fixing the SQL queries
    - Caching of responses
    - Decoupling the frontend
    13

    View Slide

  14. Fixing the backend issues
    SQL Query Optimizations – Moving to Aurora
    Caching of requests & data – Memcached & Redis
    Archiving of stale data – Adopting the ELK Stack
    Taking care of complex weightlifting – C# & .NET
    14

    View Slide

  15. Initial JavaScript days
    For simple DOM manipulations
    Handling of event life cycles – WebSocket updates
    using jQuery bundled with Rails
    and Vue.js for simple data bindings
    15

    View Slide

  16. Limitations of this setup
    Cluttered and spaghetti JavaScript code
    Never ending conflicts across multiple framework
    Frontend was still tightly coupled with backend
    View rendering was a toll on App Server
    16

    View Slide

  17. Decoupling the frontend
    Deciding on a single frontend framework
    Multiple contenders –
    Vue, React, Backbone and Angular
    Roadmap for Mobile Apps – Android & iPhone
    17

    View Slide

  18. The difficult choice
    18

    View Slide

  19. Vue React
    Lightweight Flexible
    Vue Specific Components Pure JavaScript
    Better Documentation Larger Community
    HTML Templating Optional JSX Templating
    Very easy to Integrate Higher learning curve
    Limited support for Mobile Apps React Native
    Difficult to hire engineers Very good talent pool
    19

    View Slide

  20. The shortlist
    React it is.
    Rich Library Support
    Very good talent pool
    React Native for Apps
    20

    View Slide

  21. The learning curve
    Too minimalistic
    Concept of a state store was new
    One-way data binding was challenging
    Not so good documentation
    21

    View Slide

  22. JSX as a pain point
    Developers used to write HTML/Erb
    Difficulty and inertia learning JSX
    22

    View Slide

  23. The good part - Components
    Code became components
    - Reusable code, with Single Responsibility
    - Easier to manage
    - Without any or very low side effects
    23

    View Slide

  24. Components & Actions
    let handleMarketTradeHistory = activeMarket => {
    return dispatch => {
    const marketTradeHistoryChannel = pusher.subscribe(
    `market-${activeMarket}-trade-history`
    );
    marketTradeHistoryChannel.bind("market_trade_history", data => {
    data.trade_history.map(item =>
    dispatch(processTradeHistoryData(item)));
    });
    };
    };
    24

    View Slide

  25. Dispatchers to do the work
    export function processTradeHistoryData(data) {
    return dispatch => {
    dispatch({
    type: ActionType.ADD_TRADE_ORDER,
    payload: {
    amount: data.a,
    created_at: data.T,
    order_type: data.M,
    price: data.p,
    vol: data.q
    }
    });
    };
    }
    25

    View Slide

  26. The good part – Clean Data
    Data is better handled
    - Single storage of data and state
    - One-way data binding, with clean hierarchy
    - You cannot accidentally pollute data
    26

    View Slide

  27. The good part – Invisible DOM
    The declarative approach
    -You do not directly touch DOM
    -Updates are batched and automatic
    -Think about what and not how
    27

    View Slide

  28. 100s of lines of code to a few
    Componentization made code smaller
    Legacy JavaScript code wasn’t reusable
    No way to debug or test
    Now it’s decoupled and self handling
    28

    View Slide

  29. MVC to Pure API
    Functionality interwoven with View
    Single codebase, hard to maintain
    Transition to Pure API, everything returns data
    View can be anything, Web or Mobile!
    29

    View Slide

  30. React Native
    Future Apps in React Native
    Pure API model complements this approach
    Frontend developers can collaborate
    UX and Apps coming under an umbrella
    30

    View Slide

  31. Benchmarking
    Load times between page loads and data population:
    Category Load Time
    RoR with No Optimization 2.80s
    RoR with Backend Optimization 2.03s
    RoR with React Frontend 1.80s
    RoR with Backend Optimization & React Frontend 1.07s
    31

    View Slide

  32. Unsolved Issues
    Virtual DOM is at times slow
    Upgrading and updating is not easy as expected
    High memory consumption – 16.3 MB
    Slow startup times – 219 ms
    32

    View Slide

  33. The learnings
    Scale is not a frontend/backend only problem
    Customers tolerate informed issues, not 502s
    Developer happiness is extremely important
    Initial learning curve can be future productivity
    33

    View Slide

  34. The takeaways
    Rewriting entire code base isn’t easy.
    When you are forced to do it, decide wisely.
    React is not a one stop solution, it was a fit for us.
    At the end of the day, its about delivering.
    34

    View Slide

  35. Thank You
    @sreekanthgs
    35

    View Slide