[FrenchTouchDreamin] Real data in dev environment: From myth to reality

Nowadays, managing metadata is almost easy. But having real data and business use-cases in a development environment isn’t that simple….yet. We’ll go on a quick journey through requirements, logic needed, feedback and tips to make you able to recreate real use-cases in dev environments on the fly.

Spoiler: This talk will include a demo and maybe some GDPR considerations.

Transcript

1. Real data in dev environment: From myth to reality By Nathan Abondance
2. •Salesforce (admin|developer|team leader) •Started 4 years ago at MeilleursAgents •Work at Aviv Group •My passion is to empower dev teams with magic tools mainly based on sfdx /nabondance Nathan Abondance
3. Agenda 1. Why Aviv Group needs Salesforce ? 2. Context 3. Metadata vs Data 4. The goal 5. The journey 6. More tips 7. Example of script step by step 8. The demo 9. Ideas of improvement 10. Questions & Answers
4. Aviv aims to unlock everyone’s perfect place in europe. Use of Salesforce: Handling and managing the entire B2B process: • Account management • Opportunities • Product catalogue • Pricing • Invoices • Everything related to payments • Everything related to contract from quote to termination • Automatic call campaign Salesforce’s products used: Why Aviv Group needs Salesforce ?
5. Context 8 years ago: 1st Salesforce user created 🎉 5 years ago: 1st Salesforce developer 󰞵 3 years ago: 1st deploy with a CI + git + sfdx 🤖 Now: Handle metadata well, lots of dev tools and processes: • CI • Github repository • Packages • Pool of scratch orgs with metadata, managed packages, custom packages, configuration → Next challenge is data in dev environments 💪
6. Metadata vs Data
7. Metadata • Metadata is data that describes other data. Example: Fields, Custom Object, Layout, Label, EmailTemplate, Flow, Profile and looooooot more, including apex classes and triggers. • Easy to handle, lot of official (sfdx) and community (texei-sfdx-plugin, sfpowerkit, sfdx-git-delta, etc.) made tools. • Main developers work is about managing metadata. → Easy cute tamed cat
8. Data • Data is money but right now we don’t care. • Data is what represent something from real world into Salesforce (via a data model). Example: Records of Account represent rental agency. • Data is why we develop things. • Salesforce is useless if there’s no data. → Data is like an untamed monster
9. The Goal Why importing data to dev environment ? • To test business use cases for new development directly in scratch org • To reproduce errors in dev environment • To populate a sandbox with lots of real data to train sales teams Doing it manually is very slow, barely works, not always possible → so it’s never done The solution: Importing data magically with a script
10. The Journey
11. The journey - Main concepts Disclaimer: The language doesn’t matter. Two types of data: Business data: • Imported on the fly when needed • It’s data specific to a use case, linked together • It refers to referential data • Example: Account, Contact, Opportunity, Contract, Subscription, etc. → That’s what we’ll focus on, it’s the hardest Referential data: • Imported at the creation of the scratch org or when refreshing a sandbox • It’s data that doesn’t change much, important for the entire org regardless the account • Example: Products, CPQ rules, geographic referential, other config • Not very challenging
12. Import/export is based on Texeï’s plugin (augmented with our use-cases) Things that works thanks to the plugin: • Links between records of a same export • Importing in a specific order • Excluding fields to export/import • Specifying the dynamic id per object In the data-plan The journey - Technical considerations Source org: where the data is (prod, staging, etc.) Target org: where we want to import the data (SO, Sandbox)
13. The Journey - Data model
14. The Journey - Define the entry point What is the entry point ? → The object at the center of the data model, the one specified when calling the script Example: Account Utility: Import everything related to the entry point record • Using parent to child relationship • Using lookup relationship Import the entire hierarchy of the entry point A cluster: every records we want to import
15. The Journey - A cluster
16. The Journey - Define dynamic id What is a dynamic id ? → Concatenation of fields of a record that make it unique in the org Examples: Product: Name + ProductCode Geographic referential: External Id Why a dynamic id ? → To be able to refer to already imported data
17. The Journey - The hard part Cyclic relationship and links between objects: → Import in multiple steps First import without links Second import with links Examples: Account: first import is only top hierarchy accounts Contact: first import without the lookup fields to contact
18. The Journey - The hard part with Revenue Cloud Cyclic relationship and links between objects in Revenue Cloud Same idea as before, but next level because of Revenue Cloud Example of Revenue Cloud hell: 1. Quote without link to contract or fields referring to it 2. Order with Draft status without link to contract or fields referring to it 3. Contract without status 4. Contract with status 5. Quote with link to contract and fields referring to it 6. QuoteLine without link to QuoteLine 7. Subscription without link to subscription 8. Subscription with link to subscription 9. QuoteLine with link to QuoteLine 10. OrderItem with all links (almost too easy) 11. Order with real status with links to Contract and fields referring to it → 6 Objects: 11 export/import
19. The Journey - The GDPR part Two easy ways: • Import with real data, then run a batch to anonymise it in the target org • Do the anonymization on local json files The way to make GDPR happy: • Do everything in the cloud so the dev can never see the real data
20. More tips Restricted picklist: Every org is evolving: picklist values changes, can become restricted. → Unrestrict all restricted picklist to import the data, then restrict it again. → texei:picklist:unrestrict and sfdx texei:picklist:restrict Value not allowed at creation: Impossible to create Order with status ‘Activated’ → Export twice the Orders → Edit the json files of exported data to change values to an accepted one Audit Fields (CreatedDate, CreatedBy, etc.): You can import specific createdDate but only at first insert. → Exclude the fields if you have to export/import multiple times the same record
21. Example of script step by step
22. Example of script step by step - Config in data-plan Excluded fields: Dynamic Ids: sObject config:
23. Example of script step by step - Preparation sfdx force:org:display --targetusername=${sourceOrg} --json npm run so:adddata -- 001b000003Ztnr3AAB sfdx force:org:display --targetusername=${targetOrg} --json sfdx force:data:record:update --sobjecttype=User --where="Name=\'User User\'" --values="BypassVR__c=true" --targetusername=${targetOrg} sfdx force:user:permset:assign --permsetname "DEV, bpAll" --targetusername=${targetOrg} delete [SELECT Id FROM blng__BillingConfig__c]; sfdx texei:data:import --inputdir blng_disabled_folder sfdx texei:picklist:unrestrict --json > ./picklist.json 1. Check connection to source org 0. Call the script with the account id as parameter 2. Check connection to target org 3. On the target org, give bypasses to the user 4. On the target org, disable billing triggers 5. On the target org, unrestrict all picklists
24. Example of script step by step - Export sfdx force:data:soql:query --query="SELECT Id FROM Account WHERE Id = \'${entryId}\' OR ParentId = \'${entryId}\'" --targetusername=${sourceOrg} --json var accountIdListJoined = ‘id1, id2, etc.’ shell.sed('-i', /{{AccountId}}/g, accountIdListJoined, dataPlan); sfdx texei:data:export --dataplan ${dataPlan} --outputdir ${dataPath} --targetusername=${sourceOrg} shell.sed('-i', /Activated/g, 'Draft', `${dataPath}/20-Order.json`); 1. Get id of all accounts in the hierarchy of the entry point account id 2. Join the account Ids 3. Put the list of account ids in the data-plan where needed 4. Export the data 5. Do some needed update on local json (i.e. change order status)
25. Example of script step by step - Import sfdx texei:data:import --inputdir ${dataPath} --dataplan ${dataPlan} --ignoreerrors --ignoreunavailablefields --targetusername=${targetOrg} sfdx force:apex:execute --apexcodefile ${anonymisationBatchApexPath} sfdx force:data:record:update --sobjecttype=User --where="Name=\'User User\'" --values="BypassVR__c=false" --targetusername=${targetOrg} sfdx force:data:record:delete --sobjecttype PermissionSetAssignment --where "Assignee.name=\'User User\' PermissionSet.name=\'bpAll\'" --targetusername=${targetOrg} delete [SELECT Id FROM blng__BillingConfig__c]; sfdx texei:data:import --inputdir blng_enabled_folder sfdx texei:picklist:restrict --inputdir ./picklist.json 1. Import the data on the target org 2. Launch the GDPR batch 3. On the target org, remove user bypasses 4. On the target org, set back the billing triggers 5. On the target org, restrict all required picklist again
26. The demo
27. • Clean specific data on the org • Update already imported records • Add more possibilities of import • Make it a plugin anyone can use Ideas of improvement → There’s no limit
28. Q&A
29. /nabondance