Real data in dev environment: From myth to reality By Nathan Abondance

•Salesforce (admin|developer|team leader) •Started 4 years ago at MeilleursAgents •Work at Aviv Group •My passion is to empower dev teams with magic tools mainly based on sfdx /nabondance Nathan Abondance

Agenda 1. Why Aviv Group needs Salesforce ? 2. Context 3. Metadata vs Data 4. The goal 5. The journey 6. More tips 7. Example of script step by step 8. The demo 9. Ideas of improvement 10. Questions & Answers

Aviv aims to unlock everyone’s perfect place in europe. Use of Salesforce: Handling and managing the entire B2B process: • Account management • Opportunities • Product catalogue • Pricing • Invoices • Everything related to payments • Everything related to contract from quote to termination • Automatic call campaign Salesforce’s products used: Why Aviv Group needs Salesforce ?

Context 8 years ago: 1st Salesforce user created 🎉 5 years ago: 1st Salesforce developer 󰞵 3 years ago: 1st deploy with a CI + git + sfdx 🤖 Now: Handle metadata well, lots of dev tools and processes: • CI • Github repository • Packages • Pool of scratch orgs with metadata, managed packages, custom packages, configuration → Next challenge is data in dev environments 💪

Metadata vs Data

Metadata • Metadata is data that describes other data. Example: Fields, Custom Object, Layout, Label, EmailTemplate, Flow, Profile and looooooot more, including apex classes and triggers. • Easy to handle, lot of official (sfdx) and community (texei-sfdx-plugin, sfpowerkit, sfdx-git-delta, etc.) made tools. • Main developers work is about managing metadata. → Easy cute tamed cat

Data • Data is money but right now we don’t care. • Data is what represent something from real world into Salesforce (via a data model). Example: Records of Account represent rental agency. • Data is why we develop things. • Salesforce is useless if there’s no data. → Data is like an untamed monster

The Goal Why importing data to dev environment ? • To test business use cases for new development directly in scratch org • To reproduce errors in dev environment • To populate a sandbox with lots of real data to train sales teams Doing it manually is very slow, barely works, not always possible → so it’s never done The solution: Importing data magically with a script

The Journey

The journey - Main concepts Disclaimer: The language doesn’t matter. Two types of data: Business data: • Imported on the fly when needed • It’s data specific to a use case, linked together • It refers to referential data • Example: Account, Contact, Opportunity, Contract, Subscription, etc. → That’s what we’ll focus on, it’s the hardest Referential data: • Imported at the creation of the scratch org or when refreshing a sandbox • It’s data that doesn’t change much, important for the entire org regardless the account • Example: Products, CPQ rules, geographic referential, other config • Not very challenging

Import/export is based on Texeï’s plugin (augmented with our use-cases) Things that works thanks to the plugin: • Links between records of a same export • Importing in a specific order • Excluding fields to export/import • Specifying the dynamic id per object In the data-plan The journey - Technical considerations Source org: where the data is (prod, staging, etc.) Target org: where we want to import the data (SO, Sandbox)

The Journey - Data model

The Journey - Define the entry point What is the entry point ? → The object at the center of the data model, the one specified when calling the script Example: Account Utility: Import everything related to the entry point record • Using parent to child relationship • Using lookup relationship Import the entire hierarchy of the entry point A cluster: every records we want to import

The Journey - A cluster

The Journey - Define dynamic id What is a dynamic id ? → Concatenation of fields of a record that make it unique in the org Examples: Product: Name + ProductCode Geographic referential: External Id Why a dynamic id ? → To be able to refer to already imported data

The Journey - The hard part Cyclic relationship and links between objects: → Import in multiple steps First import without links Second import with links Examples: Account: first import is only top hierarchy accounts Contact: first import without the lookup fields to contact

The Journey - The hard part with Revenue Cloud Cyclic relationship and links between objects in Revenue Cloud Same idea as before, but next level because of Revenue Cloud Example of Revenue Cloud hell: 1. Quote without link to contract or fields referring to it 2. Order with Draft status without link to contract or fields referring to it 3. Contract without status 4. Contract with status 5. Quote with link to contract and fields referring to it 6. QuoteLine without link to QuoteLine 7. Subscription without link to subscription 8. Subscription with link to subscription 9. QuoteLine with link to QuoteLine 10. OrderItem with all links (almost too easy) 11. Order with real status with links to Contract and fields referring to it → 6 Objects: 11 export/import

The Journey - The GDPR part Two easy ways: • Import with real data, then run a batch to anonymise it in the target org • Do the anonymization on local json files The way to make GDPR happy: • Do everything in the cloud so the dev can never see the real data

More tips Restricted picklist: Every org is evolving: picklist values changes, can become restricted. → Unrestrict all restricted picklist to import the data, then restrict it again. → texei:picklist:unrestrict and sfdx texei:picklist:restrict Value not allowed at creation: Impossible to create Order with status ‘Activated’ → Export twice the Orders → Edit the json files of exported data to change values to an accepted one Audit Fields (CreatedDate, CreatedBy, etc.): You can import specific createdDate but only at first insert. → Exclude the fields if you have to export/import multiple times the same record

Example of script step by step

Example of script step by step - Config in data-plan Excluded fields: Dynamic Ids: sObject config:

Example of script step by step - Preparation sfdx force:org:display --targetusername=${sourceOrg} --json npm run so:adddata -- 001b000003Ztnr3AAB sfdx force:org:display --targetusername=${targetOrg} --json sfdx force:data:record:update --sobjecttype=User --where="Name=\'User User\'" --values="BypassVR__c=true" --targetusername=${targetOrg} sfdx force:user:permset:assign --permsetname "DEV, bpAll" --targetusername=${targetOrg} delete [SELECT Id FROM blng__BillingConfig__c]; sfdx texei:data:import --inputdir blng_disabled_folder sfdx texei:picklist:unrestrict --json > ./picklist.json 1. Check connection to source org 0. Call the script with the account id as parameter 2. Check connection to target org 3. On the target org, give bypasses to the user 4. On the target org, disable billing triggers 5. On the target org, unrestrict all picklists

Example of script step by step - Export sfdx force:data:soql:query --query="SELECT Id FROM Account WHERE Id = \'${entryId}\' OR ParentId = \'${entryId}\'" --targetusername=${sourceOrg} --json var accountIdListJoined = ‘id1, id2, etc.’ shell.sed('-i', /{{AccountId}}/g, accountIdListJoined, dataPlan); sfdx texei:data:export --dataplan ${dataPlan} --outputdir ${dataPath} --targetusername=${sourceOrg} shell.sed('-i', /Activated/g, 'Draft', `${dataPath}/20-Order.json`); 1. Get id of all accounts in the hierarchy of the entry point account id 2. Join the account Ids 3. Put the list of account ids in the data-plan where needed 4. Export the data 5. Do some needed update on local json (i.e. change order status)

Example of script step by step - Import sfdx texei:data:import --inputdir ${dataPath} --dataplan ${dataPlan} --ignoreerrors --ignoreunavailablefields --targetusername=${targetOrg} sfdx force:apex:execute --apexcodefile ${anonymisationBatchApexPath} sfdx force:data:record:update --sobjecttype=User --where="Name=\'User User\'" --values="BypassVR__c=false" --targetusername=${targetOrg} sfdx force:data:record:delete --sobjecttype PermissionSetAssignment --where "Assignee.name=\'User User\' PermissionSet.name=\'bpAll\'" --targetusername=${targetOrg} delete [SELECT Id FROM blng__BillingConfig__c]; sfdx texei:data:import --inputdir blng_enabled_folder sfdx texei:picklist:restrict --inputdir ./picklist.json 1. Import the data on the target org 2. Launch the GDPR batch 3. On the target org, remove user bypasses 4. On the target org, set back the billing triggers 5. On the target org, restrict all required picklist again

The demo

• Clean specific data on the org • Update already imported records • Add more possibilities of import • Make it a plugin anyone can use Ideas of improvement → There’s no limit

Q&A

/nabondance