Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mastering Duct Tape (PyCon Balkan 2018)

51567a4f786cd8a2c41c513b592de9f9?s=47 David Cramer
November 18, 2018

Mastering Duct Tape (PyCon Balkan 2018)

There's a lot of hard problems in software, but how many of them are created from decisions we make day to day? Let's talk about pragmatism and a number of difficult, but common situations where we often over-engineer solutions.

51567a4f786cd8a2c41c513b592de9f9?s=128

David Cramer

November 18, 2018
Tweet

Transcript

  1. Mastering Duct Tape

  2. David Cramer Founder, CEO at Sentry
 @zeeg github.com/dcramer (i still

    write a lot of code)
  3. Why should you trust me? • Self-taught with 15 years

    of software engineering
 12 years in Python • Scaled Disqus (RIP) to 1 billion page views
 One page view is roughly one rendered embed • Over-engineered a multi-thousand node continuous integration platform at Dropbox
 It’s what you do when your test suite is Too Damn Slow™ • Scraped together Sentry on a budget
 sentry.io receives 2 billion exceptions/day as of Oct 2018 • I mash keys at 160 wpm
 An important strength of being able to duct tape, quickly
  4. What do we mean by duct tape?

  5. None
  6. What can we duct tape in software? Databases Infrastructure Monitoring

    Code Quality (spoiler: everything) “If you can’t fix it with duct tape,
 you aren’t using enough duct tape”
  7. •Act 1: Databases •Act 2: Everything Else •AMA

  8. (SQL) Databases (we all have the same problems) ACT 1

  9. Users Posts Comments Standard Application Schema (every app looks like

    a blog)
  10. Users Posts Comments Column Type id INT post_id INT display_name

    TEXT body TEXT date_posted DATETIME Comments Schema Standard Application Schema
  11. SELECT comments.id, comments.display_name, comments.body, comments.date_posted, FROM comments WHERE comments.post_id =

    ? ORDER BY comments.date_posted DESC Retrieving Comments
  12. What do we do when things are slow?

  13. Quick Diagnostics 1. Are our queries well indexed? • Indexes

    reduce disk IO, which is a common bottleneck 2. How large is the table / relation? • Row count (500 million) and size on disk (10 TB) 3. Physical resources (cpu, memory)? • Memory is the usual concern
  14. SELECT comments.id, comments.display_name, comments.body, comments.date_posted, FROM comments WHERE comments.post_id =

    ? ORDER BY comments.date_posted DESC Basic Indexes CREATE INDEX my_index_name ( comments.post_id ) ON comments
  15. Optimizing Reads SELECT comments.id, comments.display_name, comments.body, comments.date_posted, FROM comments WHERE

    comments.post_id = ? ORDER BY comments.date_posted DESC CREATE INDEX my_index_name ( comments.post_id, comments.date_posted ) ON comments Note: date_posted has high cardinality, which means this index is more expensive to maintain
  16. Real life isn’t that easy..

  17. Users Posts Comments Votes Revisions Whenever you edit a comment,

    we end up with a new revision or help rank the best comments, which is how we sort
  18. Column Type id INT author_id INT (FOREIGN KEY on users)

    post_id INT (FOREIGN KEY on posts) latest_revision_id INT (FOREIGN KEY on revisions) date_posted DATETIME Comments Schema Users Posts Comments Votes Revisions
  19. SELECT comments.id, comments.date_posted, revisions.body, users.display_name FROM comments JOIN revisions ON

    comments.latest_revision_id = revisions.id JOIN users ON comments.author_id = users.id WHERE comments.post_id = ? ORDER BY comments.date_posted DESC
  20. What if the fully indexed query is slow?

  21. Vertical Partitions Users Posts Main Database Comments Comments Database Revisions

  22. Vertical Partitions Users Posts Main Database Comments Comments Database Comments

    Schema Column Type id INT author_id INT (FOREIGN KEY on users) post_id INT (FOREIGN KEY on posts) latest_revision_id INT (FOREIGN KEY on revisions) date_posted DATETIME Revisions
  23. Vertical Partions (cont.) 1. Remove newly invalid foreign key constraints

    2. Replicate tables to new database server 3. Update application code to remove relations
 (common in frameworks like Django) 4. [some magic or downtime to cutover databases] General process to split off relations
  24. Vertical Partions (cont.) • (Likely) Find code which is still

    referencing the relation incorrectly • Write code to handle comments with missing posts • Write code to delete comments when posts go away 1. Remove newly invalid foreign key constraints 2. Replicate tables to new database server 3. Update application code to remove relations
 (common in frameworks like Django) 4. [some magic or downtime to cutover databases] General process to split off relations Some other things we almost certainly didn’t think about
  25. But what if we hit the same problem on the

    new database?
  26. Horizontal Partitions Users Posts Main Database Comments Comments 2018 Revisions

    Comments Comments 2020 Revisions Comments Comments 2019 Revisions Comments Comments 2021 Revisions
  27. Horizontal Partitions (cont.) Users Posts Main Database Comments Comments 2018-12

    Revisions Comments Comments 2019-03 Revisions Comments Comments 2019-01 Revisions Comments Comments 2019-04 Revisions Comments Comments 2019-02 Revisions Comments Comments 2019-05 Revisions
  28. Horizontal Partitions (cont.) Users Main Database Comments Comments 2018-12 Revisions

    Comments Comments 2019-01 Revisions Posts Posts 2018-12 Posts 2019-01 Posts
  29. Maybe we should just use MongoDB? Clickhouse? Cassandra? Elastic Search?

    HBase? Our own database? Redis?
  30. No!

  31. Just buy a bigger server! (or use your favorite cloud

    provider to provision one)
  32. Services and Things ACT 2

  33. User Interface API Database Monolith

  34. Monoliths aren’t cool anymore Monoliths slow us down

  35. User Interface API Database Monolith User Interface Database Service Oriented

    (SOA) Service Service Database
  36. SOA isn’t cool anymore SOA is an outdated idea

  37. User Interface API Database Service Service Service Service Service Database

    Database Database User Interface Monolith Microservices (a trendy way to describe a SOA)
  38. How do we get there?

  39. REPO 1 REPO 1 Database REPO 2 REPO 4 REPO

    3 REPO 5 REPO 6 Database Database Database REPO 1 Monolith Microservices
  40. Moving to a SOA 1. Build a framework
 You might

    as well use a new language while you’re at it 2. Rewrite the previous developers code
 It was old and nobody liked that developer anyways 3. Profit!
  41. Moving to a SOA (cont.) 1. Build a framework 2.

    Setup an event stream (Kafka) 3. Break apart your monolithic MySQL database 4. Write a service which owns one set of your problems 5. Attempt [and fail] to setup automated testing 6. Create a new way to deploy code 7. … 8. Profit!
  42. Let’s step back and look at our goals

  43. Goal Setting • Speed up the build/test/release process • Better

    ownership and autonomy • Improve reliability
 (through stronger API contracts, reduced complexity) • Transition away from legacy hard-to-support systems
  44. Do we really need a SOA?

  45. User Interface Database Duct Tape Oriented Architecture because we just

    need to get things done User Interface User Interface Service THE MICROLIFT Service Service Service Service
  46. Test Review Release Build Code largely automated social processes Change

    Control your most important lever
  47. Remove Humans Remove humans anywhere you can — rely on

    robots!
 Let everyone ship their own code ^ an office manager can deploy code at Sentry!
  48. Optimize Your Tests You must spend time on build/test suite

    performance
 People often overlook how slow database access is and it’s easy to fix! • Profile your test suite! • You don’t have to run every test, every time • Use transactions to create quick database tests
 https://github.com/getsentry/zeus/blob/5004a6b7c538fada3e98c8943ea5385234a8220b/zeus/testutils/pytest.py#L89 • Replace production services with no-ops where possible
 https://docs.djangoproject.com/en/2.1/topics/cache/#dummy-caching-for-development • Mock third party network calls
 https://github.com/getsentry/responses
  49. Test in Production Let production be production
 Don’t try to

    mock your production environment — it wont work v2 v1 LIVE TRAFFIC 99% 1% utilize a tool like LaunchDarkly to scope features
  50. Define Ownership Ownership doesn’t mean you need a separate codebase


    You can deploy service-isolated copies of your monolith REPO 1 sentry.io REPO 1 api.sentry.io REPO 2 docs.sentry.io REPO 1 ingest.sentry.io https://help.github.com/articles/about-codeowners/ lead Docs Team (and put teams on-call for their services)
  51. What about legacy systems? Let them be legacy. You have

    more important things to do.
  52. In Closing FINALE

  53. If nothing else.. • Remove human roadblocks
 If it annoys

    you, its probably wasting time (and money) • Enable people to do their best work
 Treat people as adults and give them the tools they need to succeed • Take off your engineering hat
 Focus on the business goals - less on academics • “Time to Ship” is your metric
 The faster you can change and react to your customers, the more fun and success your going to enjoy
  54. AMA (ask me anything)