Building Durable, Performant Systems Integrations at Scale Thayer Luscian @ Above The Fray × 

Head of Engineering abovethefray.io • thayer.dev Based in Lancaster, PA Has a cat (pictured) As of July, has a wife (not pictured) ThayerLuscian.phtml 

Platform-agnostic (but I 🧡 Magento) Architecture-agnostic (mostly) In context of two systems Assumes a “blind” integrator Ground rules 1 2 3 4 

Separation of concerns 

Direct connection (Magento does the heavy lifting) pub sub pub sub 

What happens when Magento goes down? pub sub pub sub ? 

What happens when D365 goes down? pub sub pub sub ? 

Stability A system’s ability to perform by consistently maintaining stasis Keeping the lights on Durability A system’s ability to perform consistently despite loss of stasis Stasis A state of equilibrium, wherein “it’s all chill bruv” (remember, from biology class?) 

Beep beep! Here comes the message bus! pub sub pub sub A B 

Almost anything can be a message bus, if you only believe… pub sub pub sub A B 

… but sometimes, it does take a little extra work. pub sub pub sub A B SFTP 

Neither snow nor rain nor heat nor gloom of night— pub sub pub sub A B ? 

Double-decker message bus! pub sub pub sub A sub pub B C D 

Publish, enqueue, consume 

Entities & events ● Integrations synchronize entities (nouns) ● Integrations are driven by events (verbs) product updated product sales order placed sales order Transactional Non-transactional entity event 

Event listener (observer) ● Make sure to pick the right event! ● A (welcome) exception to the rule ● Save publication status in DB! Cron job ● Use these sparingly and efficiently ● Can bring other performance issues ● Make sure you know what you’re doing Webhooks ● Should be paired with a “full load” ● Availability is a toss-up, since you’re already dealing with a closed system Polling ● Extra care needed to ensure efficiency ● Requires extra work for change tracking ● Should be paired with a “full load” How do we know when events happen? Active Passive Closed system Open system 

We always know what Magento’s up to pub B app/code sales order placed Mind palace Great unknown 

APIs: a window into another world pub B app/code sales order placed Mind palace Great unknown API 

Most systems are closed (and sometimes you should treat them that way anyway). 

Magento isn’t really special pub B X++ product updated Someone else’s mind palace “The shopping cart” API 

We’ll take what we can get pub B app/code sales order placed Mind palace Great unknown something something ibm? DB2 

Use the Force, Luke pub app/code sales order placed Mind palace Not-so-great unknown API pub microservice API I/F A B web hook 

Some entities must be polled because they change too often. 

Consuming is a little easier! sub app/code product update Mind palace Not-so-great unknown API sub microservice API I/F A B 

Be careful where you step! INSERT Create a new entity, even if an identical one already exists (not idempotent) REPLACE Apply all values as provided (including blank) to all fields for an existing entity (idempotent) MERGE Apply certain provided values to specified fields for an existing entity (idempotent) Idempotent Has the same effect, no matter how many times you do it POST PATCH PUT (technically) 

Building it right 

Rule #1 Solve “data” problems with logic solutions. State Region Equivalent entities, common key ISO 3166-2 

Rule #2 Keep all business logic separate from the integration layer. Warehouse A = 15 Warehouse B = 12 qty = whA + whB qty: 27 Quantity = 27 Warehouse A = 15 Warehouse B = 12 whA: 15 whB: 12 Warehouse A = 15 Warehouse B = 12 qty = whA + whB Quantity = 27 Correct Incorrect ERP Integration Magento 

Rule #3 Log errors in all three layers, and keep at least one copy of payloads. → ~/var/log tail -f integration.log Jun 16 14:12:01admin user [email protected] not found with role "Sales Rep" thayer - ssh - [email protected]… 

Rule #4 Optimize publishers and consumers to use the fewest possible resources. GET https://erp.acme.co/api/orders?status=open GET https://erp.acme.co/api/orders?status=open &updated_start=2023-06-16T23:49:00Z ↑ 5,000 orders loaded to memory & filtered ↓ 35 orders loaded to memory, no filtering needed &updated_end=2023-06-16T23:50:00Z 

abovethefray.io • thayer.dev Questions?