Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Jordan Appleson - Designing systems that scale

Jordan Appleson - Designing systems that scale

Presented at Hey! #16 on 7th April, 2015.


Hey! Presents

April 07, 2015

More Decks by Hey! Presents

Other Decks in Technology


  1. Hello, again. Designing Systems that Scale

  2. Who am I? Software Engineer @ Branded3 Technical Lead on

    a Search Analytics Platform C# / F# / Go mainly. MongoDB / Cassandra
  3. Last time on HeyStac… There was some data, we thought

    it was big [August] …. it wasn’t [November]. [August] MongoDB. 700 million rows / objects in one collection. [November] 5 billion rows+. 2 tables.
  4. This Talk. Less about data, more about building things.

  5. I’m not …

  6. What do we mean by “Scale”? What do I think

    when I think scalability?
  7. Maintainability. How hard is it to change things, add to

    the system, extend it?
  8. Testability How easy is it to test the application or

    small parts of the application?
  9. Efficiency How does the system help you, as a developer,

    build on top of the already existing application?
  10. - Smallest Unit of Work - Strongly Typed Languages Ultimately,

    writing small chunks of the bigger picture.
  11. Smallest Unit of Work A real world example from our

    system. Our Task Management System. - Our Task Management system allows a developer design any type of task whether it be a web scraper or something against our dataset and scale it across our cluster of worker machines. - A developer only has to write two things: - A Task - A Task Processor
  12. Smallest Unit of Work Simple version (C#):

  13. Task Processor Serialization Me as the developer Other things that

    we don’t care about because someone else already wrote it ages ago :) (that person was you) Message Queue Deserialization
  14. Smallest Unit of Work It saves so much time. No

    one has to care about message queues, the machines running it, threading…
  15. Strongly Typed Languages - Enforce API’s like this at compile

    time. - You cannot compile a processor that does not take in an ITask - Less dumb-fuckery. - Even with documentation, people still get it wrong. - In a dynamic interpreted [PHP / JS] language, how feasible is this? But more importantly how many people will use it wrong?
  16. Testing the Processor with F#

  17. Using the Processor in Production

  18. TypeScript - we have the same power in browser -

    Compiles down JS. - Interop - You can use JS in TypeScript - Strongly Typed!
  19. Strongly Typed - This won’t compile. - Node.JS - In

  20. Less to think about, more to reuse. - Write types

    that are used both in the Frontend JS and the Backend - Large applications benefit from these features. Why? - Because you can write API’s that result in more robust applications.
  21. At the end of the day - Scalable systems design

    is about saving time for you and your peers in the future. - By taking advantage of technology and feature sets of tools that allow you to focus on building those small pieces.
  22. Thanks for listening to my ramblings…