Building Durable, Performant Systems Integrations at Scale

Systems integrations kinda… suck. On one hand, they're among the most important parts of complex ecommerce builds, but on the other hand, they're arguably the most frequently botched. Poor design can result in integrations that test well but bog down quickly at scale, or encounter frequent, annoying errors that require a lot of "encouragement" from a human operator.

This talk explores how a decoupled approach can solve many of the problems that plague systems integrations. Architectural patterns like microservices and message buses enable more durable, performant systems integrations that can more easily recover from outages and handle load spikes. While these architectural patterns may be trendy, there's no need to abandon your whole tech stack: we can apply these principles to anything from a SQL table to an FTP bucket.

Transcript

1. Building Durable, Performant Systems Integrations at Scale Thayer Luscian @ Above The Fray ×
2. Head of Engineering abovethefray.io • thayer.dev Based in Lancaster, PA Has a cat (pictured) As of July, has a wife (not pictured) ThayerLuscian.phtml
3. Platform-agnostic (but I 🧡 Magento) Architecture-agnostic (mostly) In context of two systems Assumes a “blind” integrator Ground rules 1 2 3 4
4. Separation of concerns
5. Direct connection (Magento does the heavy lifting) pub sub pub sub
6. What happens when Magento goes down? pub sub pub sub ?
7. What happens when D365 goes down? pub sub pub sub ?
8. Stability A system’s ability to perform by consistently maintaining stasis Keeping the lights on Durability A system’s ability to perform consistently despite loss of stasis Stasis A state of equilibrium, wherein “it’s all chill bruv” (remember, from biology class?)
9. Beep beep! Here comes the message bus! pub sub pub sub A B
10. Almost anything can be a message bus, if you only believe… pub sub pub sub A B
11. … but sometimes, it does take a little extra work. pub sub pub sub A B SFTP
12. Neither snow nor rain nor heat nor gloom of night— pub sub pub sub A B ?
13. Double-decker message bus! pub sub pub sub A sub pub B C D
14. Publish, enqueue, consume
15. Entities & events • Integrations synchronize entities (nouns) • Integrations are driven by events (verbs) product updated product sales order placed sales order Transactional Non-transactional entity event
16. Event listener (observer) • Make sure to pick the right event! • A (welcome) exception to the rule • Save publication status in DB! Cron job • Use these sparingly and efficiently • Can bring other performance issues • Make sure you know what you’re doing Webhooks • Should be paired with a “full load” • Availability is a toss-up, since you’re already dealing with a closed system Polling • Extra care needed to ensure efficiency • Requires extra work for change tracking • Should be paired with a “full load” How do we know when events happen? Active Passive Closed system Open system
17. We always know what Magento’s up to pub B app/code sales order placed Mind palace Great unknown
18. APIs: a window into another world pub B app/code sales order placed Mind palace Great unknown API
19. Most systems are closed (and sometimes you should treat them that way anyway).
20. Magento isn’t really special pub B X++ product updated Someone else’s mind palace “The shopping cart” API
21. We’ll take what we can get pub B app/code sales order placed Mind palace Great unknown something something ibm? DB2
22. Use the Force, Luke pub app/code sales order placed Mind palace Not-so-great unknown API pub microservice API I/F A B web hook
23. Some entities must be polled because they change too often.
24. Consuming is a little easier! sub app/code product update Mind palace Not-so-great unknown API sub microservice API I/F A B
25. Be careful where you step! INSERT Create a new entity, even if an identical one already exists (not idempotent) REPLACE Apply all values as provided (including blank) to all fields for an existing entity (idempotent) MERGE Apply certain provided values to specified fields for an existing entity (idempotent) Idempotent Has the same effect, no matter how many times you do it POST PATCH PUT (technically)
26. Building it right
27. Rule #1 Solve “data” problems with logic solutions. State Region Equivalent entities, common key ISO 3166-2
28. Rule #2 Keep all business logic separate from the integration layer. Warehouse A = 15 Warehouse B = 12 qty = whA + whB qty: 27 Quantity = 27 Warehouse A = 15 Warehouse B = 12 whA: 15 whB: 12 Warehouse A = 15 Warehouse B = 12 qty = whA + whB Quantity = 27 Correct Incorrect ERP Integration Magento
29. Rule #3 Log errors in all three layers, and keep at least one copy of payloads. → ~/var/log tail -f integration.log Jun 16 14:12:01admin user bob@acme.co not found with role "Sales Rep" thayer - ssh - 1.ent-a1b2cd34-production@ssh.us-2.magento.cloud…
30. Rule #4 Optimize publishers and consumers to use the fewest possible resources. GET https://erp.acme.co/api/orders?status=open GET https://erp.acme.co/api/orders?status=open &updated_start=2023-06-16T23:49:00Z ↑ 5,000 orders loaded to memory & filtered ↓ 35 orders loaded to memory, no filtering needed &updated_end=2023-06-16T23:50:00Z
31. abovethefray.io • thayer.dev Questions?