Hello, again. Designing Systems that Scale

Who am I? Software Engineer @ Branded3 Technical Lead on a Search Analytics Platform C# / F# / Go mainly. MongoDB / Cassandra

Last time on HeyStac… There was some data, we thought it was big [August] …. it wasn’t [November]. [August] MongoDB. 700 million rows / objects in one collection. [November] 5 billion rows+. 2 tables.

This Talk. Less about data, more about building things.

I’m not …

What do we mean by “Scale”? What do I think when I think scalability?

Maintainability. How hard is it to change things, add to the system, extend it?

Testability How easy is it to test the application or small parts of the application?

Efficiency How does the system help you, as a developer, build on top of the already existing application?

- Smallest Unit of Work - Strongly Typed Languages Ultimately, writing small chunks of the bigger picture.

Smallest Unit of Work A real world example from our system. Our Task Management System. - Our Task Management system allows a developer design any type of task whether it be a web scraper or something against our dataset and scale it across our cluster of worker machines. - A developer only has to write two things: - A Task - A Task Processor

Smallest Unit of Work Simple version (C#):

Task Processor Serialization Me as the developer Other things that we don’t care about because someone else already wrote it ages ago :) (that person was you) Message Queue Deserialization

Smallest Unit of Work It saves so much time. No one has to care about message queues, the machines running it, threading…

Strongly Typed Languages - Enforce API’s like this at compile time. - You cannot compile a processor that does not take in an ITask - Less dumb-fuckery. - Even with documentation, people still get it wrong. - In a dynamic interpreted [PHP / JS] language, how feasible is this? But more importantly how many people will use it wrong?

Testing the Processor with F#

Using the Processor in Production

TypeScript - we have the same power in browser - Compiles down JS. - Interop - You can use JS in TypeScript - Strongly Typed!

Strongly Typed - This won’t compile. - Node.JS - In Browser

Less to think about, more to reuse. - Write types that are used both in the Frontend JS and the Backend - Large applications benefit from these features. Why? - Because you can write API’s that result in more robust applications.

At the end of the day - Scalable systems design is about saving time for you and your peers in the future. - By taking advantage of technology and feature sets of tools that allow you to focus on building those small pieces.

Thanks for listening to my ramblings…