Event-driven: Don’t Fear the Async

In the beginning, programs were simple. They ran in one place, step by step. Sure, we had bugs. But, through engineering, we found ways to tame them. Then we started running things in parallel. The programs went fast, but they went wrong fast. We discovered new kinds of parallel bugs that tortured our human brains.

Salesforce does a great job of letting us pretend to live in a single-threaded world. But when should we build event-based asynchronous systems? And how can we get them right?

Slides from London's Calling on 9 June 2023

Transcript

1. #LDNsCall #LC23 Event-driven: Don’t Fear the Async Aidan Harding Technology Director - Nebula Consulting https://trailblazer.me/id/aidan-harding https://www.linkedin.com/in/aidan-harding/ https://twitter.com/AidanHarding https://fosstodon.org/@aidanharding
2. Asynchronous events inside a Salesforce system • What is asynchronous processing? • How does Salesforce make synchronous triggers so simple? • Why is async so hard? • Why should we give up that simplicity? • Mind-bending algorithm interlude • Examples of async issues • Some tips for handling async processes • Wrap up Overview
3. • Processes that overlap their execution time • It could be that only one process is running at once, sharing a single processor • It could be that you have multiple cores/computers so processes run in true parallel • Events can help to coordinate async processes • Zombie process: in Unix has an entry in the process table, but is not running • Race condition: two async processes competing to use the same resource What even is async?
4. • Apex Batch, Scheduled, and Queueable • Flow Asynchronous Paths • LWC interactions with the server • Platform Events • Multiple Users • External Systems How/where does async play out on Salesforce?
5. Either of these placeholders can hold text, table, charts, smart art, or media. • Atomic: transactions either fully succeed, or fully fail • Consistent: data satisfies all validation rules and trigger-enforced invariants • Isolated: transactions that happen in parallel have the same result as if they were in sequence • Durable: committed transactions are stored permanently ACID properties of a database
6. Either of these placeholders can hold text, table, charts, smart art, or media. • Atomic: ✅ • Consistent: ❌ Triggers and validation can be skipped e.g. cascading deletes, lead conversion • Isolated: ❌ Default behaviour does not use locking, so race conditions are likely • Durable: ✅ Does Salesforce satisfy ACID properties?
7. • Do. Or do not. There is no try ◦ In Yoda’s world, a business operation is always within one transaction i.e. • By breaking business operations into multiple transactions, we invert Yoda ◦ Try. Maybe do. Maybe do not • The database is working atomically and consistently, but the business operations are not • At a business operation level, we’ve given up all of ACID! Async and the inverse-Yoda
8. 1. Set the Account Number field to a UUID 2. Create a log record When an Account is inserted An example scenario • We could viably implement this as: ◦ Synchronous Apex ◦ Synchronous Flow ◦ Asynchronous Flow/Apex (scheduled) ◦ Asynchronous Flow/Apex (near real-time)
9. • It’s much easier to reason about a linear chain of events than a branching one • Keeping business operations within a transactions provides a linear chain Reasoning about code Linear Time Insert Log Set UUID Insert Account ✅ Account ✅ UUID ✅ Log ❌ Account ❌ UUID ❌ Log
10. Branching time 🤯 Reasoning about code Insert Log Set UUID Insert Account Insert Log Insert Log Set UUID Set UUID ✅ Account ✅ UUID ✅ Log ✅ Account ✅ UUID ❌ Log ✅ Account ❌ UUID ✅ Log ✅ Account ❌ UUID ❌ Log ✅ Account ✅ Log ✅ UUID ✅ Account ✅ Log ❌ UUID ✅ Account ❌ Log ✅ UUID ✅ Account ❌ Log ❌ UUID
11. • It’s the reason we moved from global variables to functions, objects, functional programming • Psychologists suggest that we can hold 4 “chunks” in working memory1 Reasoning about code Minimise state! 1. “The Magical Mystery Four: How is Working Memory Capacity Limited, and Why?” Nelson Cowan https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2864034/
12. Algorithm Interlude!
13. • For each value: sleep for time proportional to the value, then append that value to a new list • e.g. with a delay of 200ms and a list containing [3, 1, 5, 9] ◦ Sleep for 600 ms, append 3 to the result Sleep for 200 ms, append 1 to the result Sleep for 1000 ms, append 5 to the result Sleep for 1800 ms, append 9 to the result • Result: [1, 3, 5, 9] An algorithm to sort a list of numbers Sleep Sort
14. • Sleep for the value of each element, then append that value to a new list • e.g. set the delay at 200 ms for the list [3, 1, 5, 9] ◦ Sleep for 600 ms, append 3 to the result Sleep for 200 ms, append 1 to the result Sleep for 1000 ms, append 5 to the result Sleep for 1800 ms, append 9 to the result • Result: [1, 3, 5, 9] Sleep Sort
15. • Sleep for the value of each element, then append that value to a new list • e.g. set the delay at 200 ms for the list [3, 1, 5, 9] ◦ Sleep for 600 ms, append 3 to the result Sleep for 200 ms, append 1 to the result Sleep for 1000 ms, append 5 to the result Sleep for 1800 ms, append 9 to the result • Result: [1, 3, 5, 9] • Quicksort et al. are O(n*log(n)) where n is the list-size • Sleep sort is O(n) Sleep Sort
16. const input = Array.from(generateRandoms(NUMBER_OF_ITEMS, MAX_VALUE)); const promises = []; const result = []; console.log(`Input:\n${input.join(' ')}`); for(const thisValue of input) { promises.push(sleepAndAddToArray(thisValue, result)); } await Promise.all(promises); console.log(`Done!\n${result.join(' ')}`); Sleep Sort Code
17. async function sleepAndAddToArray(thisValue, result) { return new Promise(resolve => setTimeout( function() { result.push(thisValue); console.error(...result); resolve(); }, thisValue * SLEEP_MULTIPLE )); } Sleep Sort Code
18. • In the code, there is no manipulation of variables to do the sorting • Time itself is the variable! • In general, this is a reason for asynchronous code being hard to think about: ◦ Time becomes part of the state and more state makes programs harder to reason about ◦ Branching possibilities are harder to reason about ◦ Humans can’t multitask What did Sleep Sort teach us?
19. • Speed - if you have multiple processors available, elapsed time to get a result can be shortened • Responsiveness - let users carry on while heavy processing happens in the background • Salesforce limits - don’t paper over bad design, but some operations are essentially expensive e.g. callouts, large data volumes • Decoupling - prevent errors in one process from affecting another 1. Well Architected: Automation https://architect.salesforce.com/well-architected/easy/automated 2. Event-Driven Architecture https://architect.salesforce.com/decision-guides/event-driven Why even bother with async?
20. • Before starting an async operation, store that attempt somewhere • When it comes to debugging, you need information to be stored reliably • Have a consistent approach to logging across the entire system • Consider using off-platform storage of events Show your working! Tip One: Make the implicit explicit • You won’t want to be adding instrumentation to a production system that’s flapping in the wind during an incident
21. • An operation is idempotent when executing any number of times has the same effect as executing the first time What’s done is done Tip Two: Make operations idempotent • Some operations lend themselves to idempotency e.g. upsert • Some APIs can offer idempotency keys as an attribute on the request • Great for at-least-once semantics • Great for retrying on transient errors
22. • This stuff is super-hard to test, so test the hard parts once and make it commodity throughout your project • An ideal use-case for Unlocked Packages • Or document the what and why with standards of how your project will handle async Tip Three: Write the hard parts once Make a pot of coffee
23. Tip Four: Belt and Braces • Whenever you fire something off asynchronously, you need a second method to check the results • For Apex, use Transaction Finalizers or Batch Apex Error Event • Also consider a scheduled process to make sure that you’re even wearing trousers in the first place ◦ Near real-time performance most of the time ◦ Nightly checks to sync up anything that got missed • If you made your operation idempotent, you can automatically retry failed processes ◦ Make sure you have a delay!
24. Tip Five: Treat it like science • The behaviour of the system is now something that you must design experiments for and observe • How will I know if the system is working? What measures will convince me ◦ Make falsifiable hypotheses • If it’s not working, hypothesise why not. Write tests to confirm your hypothesis and to check your fix
25. global without sharing class AccountSetApexAsyncPending implements nebc.BeforeInsert { public void handleBeforeInsert(List<Account> newList) { for(Account thisAccount : newList) { if(thisAccount.Automation_Type__c == 'Asynchronous Apex') { thisAccount.Apex_Record_Creation_History_Status__c = 'Pending'; thisAccount.Apex_UUID_Status__c = 'Pending'; } } } } Set status fields to pending Before Insert
26. global without sharing class AccountStartQueueables implements nebc.AfterInsert { global void handleAfterInsert(List<Account> newList) { Set<Id> recordCreationAccounts = new Set<Id>(); Set<Id> uuidAccounts = new Set<Id>(); for(Account thisAccount : newList) { if(thisAccount.Apex_Record_Creation_History_Status__c == 'Pending') { recordCreationAccounts.add(thisAccount.Id); } if(thisAccount.Apex_UUID_Status__c == 'Pending') { uuidAccounts.add(thisAccount.Id); } } if(!recordCreationAccounts.isEmpty()) { System.enqueueJob(new AccountRecordCreationHistoryQueueable(recordCreationAccounts), 0); } if(!uuidAccounts.isEmpty()) { System.enqueueJob(new AccountUUIDQueueable(uuidAccounts), 0); } } } Start the Queueables After Insert
27. public without sharing class AccountUUIDQueueable extends AccountQueueableHandler { public AccountUUIDQueueable(Set<Id> accountIds) { super(accountIds); } public void execute(Account thisAccount) { thisAccount.AccountNumber = new UUIDGenerator().next(); thisAccount.Apex_UUID_Status__c = 'Complete'; update thisAccount; } } Set the UUID UUID Queueable
28. public without sharing class AccountRecordCreationHistoryQueueable extends AccountQueueableHandler { public AccountRecordCreationHistoryQueueable(Set<Id> accountIds) { super(accountIds); } public void execute(Account thisAccount) { insert new Record_Creation_History__c( Record_Id__c = thisAccount.Id, Account__c = thisAccount.Id, Record_Created_By__c = thisAccount.CreatedById, Record_Created_Date__c = thisAccount.CreatedDate ); thisAccount.Apex_Record_Creation_History_Status__c = 'Complete'; update thisAccount; } } Insert the log Record Creation History Queueable
29. public abstract inherited sharing class AccountQueueableHandler implements Queueable { private List<Id> accountsToProcess; public AccountQueueableHandler(Set<Id> accountsToProcess) { this.accountsToProcess = new List<Id>(accountsToProcess); } public void execute(QueueableContext qc) { Account thisAccount = [ SELECT Id, AccountNumber, CreatedById, CreatedDate, Apex_UUID_Status__c, Apex_Record_Creation_History_Status__c FROM Account WHERE Id = :accountsToProcess.remove(0) ]; execute(thisAccount.clone(true, true, true)); if(!accountsToProcess.isEmpty()) { System.enqueueJob(this, 0); } } abstract void execute(Account theAccount); } Manage querying and iteration The Queueable superclass
30. @IsTest static void sixRecords() { Test.startTest(); List<Account> testAccounts = testRecordSource.getRecord(Account.SObjectType) .put(Account.Automation_Type__c, 'Asynchronous Apex') .withInsert(6); Test.stopTest(); testAccounts = [SELECT AccountNumber, Apex_UUID_Status__c FROM Account WHERE Id IN :testAccounts]; for(Account testAccount : testAccounts) { Assert.isTrue(Uuid.isValid(testAccount.AccountNumber)); Assert.areEqual('Complete', testAccount.Apex_UUID_Status__c); } } UUID Tests
31. But what about running it? Account Number Log Status UUID Status Created Date Last Modified Date Log Id 4cf40477-0073-44 d4-87c7-20f0e3d1 3d28 Complete Pending 19:12:43 19:12:43 a007g000008XJySAAW a764007c-376e-4d 8d-a5e5-51b632da c211 Complete Pending 19:12:43 19:12:43 a007g000008XJyXAAW dbcc4ac2-2dd9-45f 5-be7e-d94285b48 d7f Complete Complete 19:12:43 19:12:43 a007g000008XJycAAG 8d0d1031-9834-4d dd-bebe-08c3b20f e0d3 Complete Complete 19:12:43 19:12:44 a007g000008XJyhAAG Pending Pending 19:12:43 19:12:43 Pending Pending 19:12:43 19:12:43 Account Number Log Status UUID Status Created Date Last Modified Date Log Id 4cf40477-0073-44 d4-87c7-20f0e3d1 3d28 Complete Pending 19:12:43 19:12:43 a007g000008XJySAAW a764007c-376e-4d 8d-a5e5-51b632da c211 Complete Pending 19:12:43 19:12:43 a007g000008XJyXAAW dbcc4ac2-2dd9-45f 5-be7e-d94285b48 d7f Complete Complete 19:12:43 19:12:43 a007g000008XJycAAG 8d0d1031-9834-4d dd-bebe-08c3b20f e0d3 Complete Complete 19:12:43 19:12:44 a007g000008XJyhAAG Pending Pending 19:12:43 19:12:43 Pending Pending 19:12:43 19:12:43 Account Number Log Status UUID Status Created Date Last Modified Date Log Id 4cf40477-0073-44 d4-87c7-20f0e3d1 3d28 Complete Pending 19:12:43 19:12:43 a007g000008XJySAAW a764007c-376e-4d 8d-a5e5-51b632da c211 Complete Pending 19:12:43 19:12:43 a007g000008XJyXAAW dbcc4ac2-2dd9-45f 5-be7e-d94285b48 d7f Complete Complete 19:12:43 19:12:43 a007g000008XJycAAG 8d0d1031-9834-4d dd-bebe-08c3b20f e0d3 Complete Complete 19:12:43 19:12:44 a007g000008XJyhAAG Pending Pending 19:12:43 19:12:43 Pending Pending 19:12:43 19:12:43
32. Tip Six: Serialise! • A common solution to the problems of running processes in parallel is stop running them in parallel • This may be done using record locks, mutexes, semaphores, tokens • Salesforce currently offers ◦ Database write locking via FOR UPDATE ◦ A form of mutex with Platform Events ◦ Possible future Queueable functionality with a token
33. public abstract inherited sharing class AccountQueueableHandler implements Queueable { private List<Id> accountsToProcess; public AccountQueueableHandler(Set<Id> accountsToProcess) { this.accountsToProcess = new List<Id>(accountsToProcess); } public void execute(QueueableContext qc) { Account thisAccount = [ SELECT Id, AccountNumber, CreatedById, CreatedDate, Apex_UUID_Status__c, Apex_Record_Creation_History_Status__c FROM Account WHERE Id = :accountsToProcess.remove(0) FOR UPDATE ]; execute(thisAccount.clone(true, true, true)); if(!accountsToProcess.isEmpty()) { System.enqueueJob(this, 0); } } abstract void execute(Account theAccount); } AccountQueueableHandler The Queueable superclass (with FOR UPDATE)
34. But what about running it? Account Number Log Status UUID Status Created Date Last Modified Date Log Id f6d5cb96-1d31-4b 3b-a626-912a3bd8 bcbd Complete Complete 19:33:55 19:33:55 a007g000008XJzLAAW 487d7996-7d12-4a 2a-b5f5-8c67fbf8e 1a6 Complete Complete 19:33:55 19:34:04 a007g000008XJzQAAW 81b6db53-1ff9-4b1 b-a0e0-c8ef87196 277 Complete Complete 19:33:55 19:34:06 a007g000008XJzVAAW 5bdb3b2f-22d1-40 60-9ede-9265d0e6 dc33 Complete Complete 19:33:55 19:34:06 a007g000008XJzaAAG Pending Pending 19:33:55 19:33:55 Pending Pending 19:33:55 19:33:55
35. Hypothesis
36. Collorary • Choosing async process may change your deployment pipeline • If you can’t rely on Apex Unit Tests, then you may need organise your own tests that run for-real in a sandbox ◦ If you’re doing async to cope with Large Data Volumes, you might already be in this position
37. • You can add asynchronous paths to Flow • The same considerations apply, but fewer tools at the moment • No finalizer to handle errors • Can debug them • Cannot test in Apex • Cannot test in Flow What about Flow?
38. • Making things asynchronous is harder than you think • Sometimes it’s essential! • In the right place, it’s useful! • It can change your ability to quickly and confidently deploy changes • The challenges are not unique to Salesforce, so reading around is helpful Subtitle placeholder Some final thoughts on async Those tips again: 1. Show your working 2. Idempotency 3. Do the hard bits once 4. Belt and braces 5. Treat it like science 6. Serialise 7. (Don’t use Flow)
39. Links Sleep Sort https://github.com/aidan-h arding/js-sleep-sort Async Fun https://github.com/aidan-h arding/async-fun Nebula Core https://github.com/aidan-h arding/nebula-core Apex UUID https://github.com/jongpie/ ApexUUID
40. #LDNsCall #LC23 Q&A Aidan Harding Technology Director - Nebula Consulting https://trailblazer.me/id/aidan-harding https://www.linkedin.com/in/aidan-harding/ https://twitter.com/AidanHarding https://fosstodon.org/@aidanharding
41. #LDNsCall #LC23 Thank You Aidan Harding Technology Director - Nebula Consulting https://trailblazer.me/id/aidan-harding https://www.linkedin.com/in/aidan-harding/ https://twitter.com/AidanHarding https://fosstodon.org/@aidanharding