Upgrade to Pro — share decks privately, control downloads, hide ads and more …

QConNY18: Help! I accidentally distributed my s...

QConNY18: Help! I accidentally distributed my system

The way engineers traditionally became Distributed Systems engineers was by creating infrastructure in house and building hands-on familiarity with how data moved through a system. But specialization among engineers and increasing levels of abstraction have created a situation in which almost no one has a complete view of how data moves through an entire system, end to end.
At the same time it’s become harder to understand the systems we’re building because we didn’t build them ourselves. The skills required for understanding and diagnosing problems within a distributed system are more important for everyone, front-end, back-end, and “full-stack”, because our systems are increasingly distributed.
Topics we’ll cover:
* We’ll give you the flight manual for when to use Bass, PaaS, IaaS, and the checklist of what to do if you’re already in flight with them.
* We’ll walk through the Operational consequences with war stories of surprise distributed systems problems at three different companies.
* How do you use existing tools (monitoring, tracing, alerts, logging, paging, etc) to reduce operator load, in particular for non-SRE's
* Tools for gaining visibility into the different parts of your stack (database, app, background jobs, queues, lambda and event-based compute).
* Criteria for when the cost to own a part of your stack is worth it.
* I’m a web application engineer, how did this become part of my job again and what does it all mean???

Rachel Myers

June 29, 2018
Tweet

More Decks by Rachel Myers

Other Decks in Technology

Transcript

  1. @rachelmyers, @eanakashima Designing distributed systems •What are the “nouns” in

    my system? •What are the patterns for reading data? •What are the patterns for writing data? •Do different flows have different performance, consistency, & availability requirements?
 

  2. @rachelmyers, @eanakashima ModCloth Web App MySQL Identity Service User request

    (from NGINX & loadbalancer) Mobile MySQL Voting Service MySQL
  3. @rachelmyers, @eanakashima ModCloth Web App MySQL Identity Service User request

    (from NGINX & loadbalancer) Mobile MySQL Voting Service Asset Service MySQL MySQL
  4. @rachelmyers, @eanakashima ModCloth Web App MySQL Identity Service Photo Service

    User request (from NGINX & loadbalancer) Mobile MySQL Voting Service Asset Service Comment Service MySQL MySQL MySQL MySQL
  5. @rachelmyers, @eanakashima ModCloth Web App MySQL Identity Service Photo Service

    Moderation Service User request (from NGINX & loadbalancer) Mobile MySQL Voting Service Asset Service Comment Service MySQL MySQL MySQL MySQL MySQL
  6. @rachelmyers, @eanakashima ModCloth Web App MySQL Checkout Service Identity Service

    Photo Service Moderation Service User request (from NGINX & loadbalancer) Mobile MySQL Voting Service Asset Service Comment Service MySQL MySQL MySQL MySQL MySQL MySQL
  7. @rachelmyers, @eanakashima ModCloth Web App MySQL Checkout Service Identity Service

    Photo Service Moderation Service User request (from NGINX & loadbalancer) Mobile MySQL Voting Service Asset Service Comment Service MySQL MySQL MySQL MySQL MySQL MySQL
  8. @rachelmyers, @eanakashima ModCloth Web App MySQL Checkout Service Identity Service

    Photo Service Moderation Service User request (from NGINX & loadbalancer) Mobile MySQL Voting Service Asset Service Comment Service MySQL MySQL MySQL MySQL MySQL MySQL
  9. @rachelmyers, @eanakashima Buy or Build •Will it improve availability? •Will

    it improve recovery? •Will you have visibility or points of control? •Will you have customizability? •Will you know when your changes take effect? •Can you audit changes?
  10. @rachelmyers, @eanakashima IaaS •Will it improve availability? ✅ •Will it

    improve recovery? ✅ •Will you have visibility or points of control? ✅ •Will you have customizability? ✅ •Will you know when your changes take effect? ✅ •Can you audit changes? ✅
  11. @rachelmyers, @eanakashima PaaS •Will it improve availability? ✅ •Will it

    improve recovery? •Will you have visibility or points of control? ✅ •Will you have customizability? ✅ •Will you know when your changes take effect? •Can you audit changes?
  12. @rachelmyers, @eanakashima BaaS •Will it improve availability? ✅ •Will it

    improve recovery? •Will you have visibility or points of control? •Will you have customizability? ✅ •Will you know when your changes take effect? •Can you audit changes?
  13. @rachelmyers, @eanakashima FaaS •Will it improve availability? •Will it improve

    recovery? •Will you have visibility or points of control? •Will you have customizability? ✅ •Will you know when your changes take effect? •Can you audit changes?
  14. @rachelmyers, @eanakashima “The browser is part of your distributed system,

    not just a client you support”
 -@danielespeset
  15. @rachelmyers, @eanakashima // if the page is long enough to

    scroll if (document.body.clientHeight > window.innerHeight) { // add a scroll event listener document.addEventListener('scroll', function(e) { // if within 100px of the bottom of the page if (window.innerHeight + window.scrollY + 100 > document.body.clientHeight) fetchNextPage(); }); // else fetch another page of results immediately } else { fetchNextPage(); }
  16. @rachelmyers, @eanakashima # instantiate the flag with a default value

    # falls back to default value if service is offline show_feature = ld_client.variation( "allow_traces", { :key => "[email protected]" }, false ) if show_feature # application code to show the feature else # the code to run if the feature is off
  17. @rachelmyers, @eanakashima Use tools that let you… •Ask, what is

    happening now to this user? Endpoint? Team? •Ask, how does this compare to the aggregate case? •Jump between levels of abstraction •See both breadth & depth •Understand how data flows

  18. @rachelmyers, @eanakashima “The ability to ask arbitrary questions about your

    system from the outside, and understand what is happening on the inside.”
 -@mipsytipsy