Uncoupling Systems

We tackle unique business challenges but are solving similar problems, and all too rarely will we talk through our designs and mistakes. Integrations, external APIs, data-stores - these might not be pure code but we can learn a lot from good object design when scaling our systems.

In this talk we’ll discuss how parts of a distributed systems rely on each other, how data flow can couple and propagate failure, and how common emergent patterns can encapsulate services to be resilient.

This is a journey of close calls, errors, and design smells researched into how a service gets simplified.

Transcript

1. Uncoupling Systems Jeremy Hanna
5. class Cataloger URL = "https://www.goodreads.com/" attr_accessor :username, :password def initialize(username, password) ... end def import results = get_books if results.code == 200 begin parsed_results = JSON.parse(results.body).dig("data", "results") parsed_results.each do |result| if result["author"] && result["title"] Library::Book.create( ... ) else raise Cataloger::ParseError, "Response missing author or title" end end rescue JSON::ParserError => e raise Cataloger::ParseError, "Unparsable results format" end elsif results.code == 403 raise Cataloger::AuthenticationError else raise StandardError, "Unhandled response from client" end end def get_books RestClient::Request.execute( method: :get, url: URL + "book/list.json", {:Authorization => "Basic #{username}:#{password}" } ) end end
6. class Cataloger URL = "https://www.goodreads.com/" attr_accessor :username, :password def initialize(username, password) ... end def import results = get_books if results.code == 200 begin parsed_results = JSON.parse(results.body).dig("data", "results") parsed_results.each do |result| if result["author"] && result["title"] Library::Book.create( ... ) else raise Cataloger::ParseError, "Response missing author or title" end end rescue JSON::ParserError => e raise Cataloger::ParseError, "Unparsable results format" end elsif results.code == 403 raise Cataloger::AuthenticationError else raise StandardError, "Unhandled response from client" end end def get_books RestClient::Request.execute( method: :get, url: URL + "book/list.json", {:Authorization => "Basic #{username}:#{password}" } ) end end
7. class Cataloger URL = "https://www.goodreads.com/" attr_accessor :username, :password def initialize(username, password) ... end def import results = get_books if results.code == 200 begin parsed_results = JSON.parse(results.body).dig("data", "results") parsed_results.each do |result| if result["author"] && result["title"] Library::Book.create( ... ) else raise Cataloger::ParseError, "Response missing author or title" end end rescue JSON::ParserError => e raise Cataloger::ParseError, "Unparsable results format" end elsif results.code == 403 raise Cataloger::AuthenticationError else raise StandardError, "Unhandled response from client" end end def get_books RestClient::Request.execute( method: :get, url: URL + "book/list.json", {:Authorization => "Basic #{username}:#{password}" } ) end end shape
8. class Cataloger URL = "https://www.goodreads.com/" attr_accessor :username, :password def initialize(username, password) ... end def import results = get_books if results.code == 200 begin parsed_results = JSON.parse(results.body).dig("data", "results") parsed_results.each do |result| if result["author"] && result["title"] Library::Book.create( ... ) else raise Cataloger::ParseError, "Response missing author or title" end end rescue JSON::ParserError => e raise Cataloger::ParseError, "Unparsable results format" end elsif results.code == 403 raise Cataloger::AuthenticationError else raise StandardError, "Unhandled response from client" end end def get_books RestClient::Request.execute( method: :get, url: URL + "book/list.json", {:Authorization => "Basic #{username}:#{password}" } ) end end color
9. class Cataloger URL = "https://www.goodreads.com/" attr_accessor :username, :password def initialize(username, password) ... end def import results = get_books if results.code == 200 begin parsed_results = JSON.parse(results.body).dig("data", "results") parsed_results.each do |result| if result["author"] && result["title"] Library::Book.create( ... ) else raise Cataloger::ParseError, "Response missing author or title" end end rescue JSON::ParserError => e raise Cataloger::ParseError, "Unparsable results format" end elsif results.code == 403 raise Cataloger::AuthenticationError else raise StandardError, "Unhandled response from client" end end def get_books RestClient::Request.execute( method: :get, url: URL + "book/list.json", {:Authorization => "Basic #{username}:#{password}" } ) end end
10. class CatalogWorker URL = "https://www.goodreads.com/" @queue = :system def self.perform(options = {}) books = RestClient::Request.execute( method: :get, url: URL + "book/show.json?id=#{id}", { :Authorization => "Basic #{ENV["CAT_USERNAME"]}:#{ENV["CAT_PASSWORD"]}" }) parsed_results = JSON.parse(results.body).dig("data", "results") parsed_results.each do |result| if result["author"] && result["title"] Library::Book.create( author: response["author"], title: response["title"] ) else raise Cataloger::ParseError, "Response missing author or title" end end end end
15. managing dependencies DESIGN IS ABOUT
17. not talking about design ANTI-PATTERN 1: (and document it)
18. let so and so design it ANTI-PATTERN 2:
19. not discussing trade-offs ANTI-PATTERN 3: - The network is reliable - The network is secure - The network is homogeneous - The network topology does not change - Latency is zero - Bandwidth is infinite - Transport cost is zero
20. RestClient::Request.execute( method: :get, url: URL + "book/list.json", {:Authorization => "Basic #{username}:#{password}" } ) - Availability - Latency - Topology - Bandwidth - Consistency
22. - Asynchronous calls - Load balance the work - Replicate data
23. response = -> () { RestClient::Request.execute( ... ) } 1..10.each { |_| foo(response.call) } @response = RestClient::Request.execute( ... ).body Library::Book.find_or_create_from( @response ) Redis.cache.fetch('books', :expires_in => 1.days) do RestClient::Request.execute( ... ).body end
24. response = -> () { RestClient::Request.execute( ... ) } 1..10.each { |_| foo(response.call) } @response = RestClient::Request.execute( ... ).body Library::Book.find_or_create_from( @response ) Redis.cache.fetch('books', :expires_in => 1.days) do RestClient::Request.execute( ... ).body end
25. - Encoding - Partitioning - Specialized data-stores - Batch and stream processing
26. events != streams
27. events != streams
28. - Systems of record and derived data - Encoding - Partitioning - Specialized data-stores - Batch and stream processing
29. derived data
30. derived data
31. derived data
33. your data DESIGN FOR - Communicate through queues - Retryable, idempotent workers - Pre-compute work - Identify derived data - Determine consistency-costs
34. building blocks workers materialized views queues
35. behavior 1. External API 2. Relational Model 3. Business rules 4. Computations 5. High read, low write
36. An External API SCENARIO 1: problem: display external reviews
37. problem: display external reviews • “hard consistency” • relies on network • a request every render
38. problem: display external reviews • one request • real-time on render
39. problem: display external reviews • asynchronous • retriable • pre-compute information • hard to determine changes • too much data
40. problem: + post external review • done for client • batch process • complex • config errors • “lie” about consistency
41. problem: + post external review • asynchronous • multiple sessions • config errors • lots of book keeping
42. Relational Models SCENARIO 2: add a book and author, problem: then display that to a user
43. add a book and author, problem: then display that to a user • straight forward • consistency guarantees • simple locks • joins • limited volume • hotspot queries
44. add a book and author, problem: then display that to a user • keep a lot of the good • Increased • can’t table scan • hotspots • database connections
45. add a book and author, problem: then display that to a user • simple • no joins • no transactions • uptime
46. add a book and author, problem: then display that to a user • transactions • coupled state • no joins
47. add a book and author, problem: then display that to a user • joins • batch • circuit break • consistency cost • complexity ➔ embed stable elements
49. Business rules SCENARIO 4: source title from social problem: media platforms
50. source title from social problem: media platforms • transactions • lock on record • race-condition • lose information
51. ➔ CRDTs are cool source title from social problem: media platforms
52. • debugging • source consistency • easy to skew • different abstractions source title from social problem: media platforms
53. source title from social problem: media platforms • recreatable • versionable • broadcastable • complexity
54. Computations SCENARIO 3: predict a release from problem: author publishings
55. ➔ publish an artifact predict a release from problem: author publishings • detect anomalies • compare versions
56. • easy • memory efficient • I/O bound • hotspots • querying DB state predict a release from problem: author publishings
57. • Less I/O bound • hotspots predict a release from problem: author publishings
58. • separate stateful from non-stateful functions predict a release from problem: author publishings ➔ side-join or broadcast
59. • reuse • In-sync with display • recreatable but slow • no subset queries predict a release from problem: author publishings
60. • dynamic • query optimized • hard to debug • monitoring predict a release from problem: author publishings
61. High read, low write SCENARIO 5:
63. simplicity.
64. For more - Chad Fowler's "From Homogeneous Monolith to Microservices Architecture" - David Schmidt’s “10 Tips for Failing Badly at Microservices - Kevlin Henney’s “Old is the New New”
65. Contact me https://jeremy-hanna.github.io/ https://gitter.im/jeremy-hanna https://www.linkedin.com/in/jeremy-hanna