Data as Code

Transcript

1. Data as Code: Exploring Data and Database Portability Latin America Oracle User Group Community Tour '23 July 29 - August 16, 2023 @ViscosityNA www.viscosityna.com Sean Scott Oracle ACE Director DoK Community Ambassador Managing Principal Consultant @oraclesean linkedin.com/in/soscott/ sean.scott@viscosityna.com
2. The Intersectionality of Cloud Native and "Groundhog Day" @ViscosityNA www.viscosityna.com Latin America Community Tour '23 - Uruguay August 14, 2023 Sean Scott Oracle ACE Director DoK Community Ambassador Managing Principal Consultant @oraclesean linkedin.com/in/soscott/ sean.scott@viscosityna.com
3. DATABASE RELIABILITY ENGINEERING ⁘ DEVOPS & AUTOMATION HIGH AVAILABILITY ⁘ BUSINESS CONTINUITY ⁘ DISASTER RECOVERY MODERNIZATION ⁘ OBSERVABILITY ⁘ ENGINEERED SYSTEMS AutoUpgrade ⁘ Zero Downtime Migrations ⁘ Patching Real Application Clusters ⁘ Data Guard ⁘ Sharding Docker/Containers ⁘ Terraform ⁘ Ansible Exadata ⁘ Oracle Database Appliance AHF ⁘ TFA ⁘ GIMR ⁘ CHA @ViscosityNA www.viscosityna.com Sean Scott Oracle ACE Director DoK Community Ambassador Managing Principal Consultant @oraclesean linkedin.com/in/soscott/ sean.scott@viscosityna.com
5. An Open Community for Data on Kubernetes https://dok.community
6. @ViscosityNA www.viscosityna.com Oracle on Docker Running Oracle Databases in Linux Containers Download a free sample chapter: https:/ /oraclesean.com 20% Discount Code: OracleDocker https:/ /link.springer.com
7. @ViscosityNA www.viscosityna.com "Databases are Special™"
8. @ViscosityNA www.viscosityna.com "Databases are Special™" DBA DEV
9. @ViscosityNA www.viscosityna.com Databases are Special™
10. @ViscosityNA www.viscosityna.com As creators, buyers, and influencers, developers have become instrumental in deciding what products pass muster. Ultimately, the C-suite may call the shot, but the developer is often guiding its aim. Boston Consulting Group "The Growing Influence of Developers in Enterprise Tech Sales" https://www.bcg.com/publications/2022/developers-influence-in-enterprise-tech-sales
11. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM
12. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM
13. @ViscosityNA www.viscosityna.com Feature Request Customers are complaining that the Phil microservice is narcissistic and misogynistic. Redesign Phil to be a decent human being.
14. @ViscosityNA www.viscosityna.com
15. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main └─ day-2 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM
16. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main └─ day-2 └─ bank-robber 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM
17. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main └─ day-2 └─ bank-robber └─ piano-student 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM
18. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main └─ day-2 └─ bank-robber └─ piano-student └─ redemption 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM
19. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main └─ day-2 └─ bank-robber └─ piano-student └─ redemption 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM Punxsutawney_TEST Feb 2 FUNNY
20. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM Punxsutawney_TEST Feb 3 NOT FUNNY Phil Connors: main └─ day-2 └─ bank-robber └─ piano-student └─ redemption
21. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main └─ day-2 └─ bank-robber └─ piano-student └─ redemption 6:00 AM Alarm doesn't go off! N ed is PO TU S! 7:00 AM M ultiple groundhogs? 7:20 AM M iam i Beach? 9:00 AM Punxsutawney_TEST Polluted Feb 2 BIZARRE
22. @ViscosityNA www.viscosityna.com Problems • Every test further pollutes data. • Restoring databases is operationally difficult, requiring: • Downtime • Specialized skills/DBA intervention • Every database is a production database • Making databases idempotent is technically difficult
23. DBA DEVELOPER
24. @ViscosityNA www.viscosityna.com Common Solutions • Build exceptions/elasticity into CI/CD pipelines • Technically difficult • Requires upkeep • Not absolute • Assume certain errors can be ignored • Technology solution ($$$$$)
25. @ViscosityNA www.viscosityna.com But... Data is Code
27. @ViscosityNA www.viscosityna.com Data is Code Every record in every database is created in one of three ways: • INSERT • UPDATE • DELETE
28. @ViscosityNA www.viscosityna.com Heterogenous replication = synchronous SQL Oracle Tx Log SQL MySQL
29. @ViscosityNA www.viscosityna.com Heterogenous replication = synchronous SQL Oracle Tx Log SQL MySQL Optional Transformation
30. @ViscosityNA www.viscosityna.com Databases are Special™
31. @ViscosityNA www.viscosityna.com Databases are Special™
32. @ViscosityNA www.viscosityna.com Databases are Special™
33. @ViscosityNA www.viscosityna.com "[F]or a long time, DBAs were in the business of crafting silos and snowflakes. Their tools were different, their hardware was different, and their languages were different." Laine Campbell, Charity Majors Database Reliability Engineering O'Reilly
34. @ViscosityNA www.viscosityna.com "The days in which this model can prove itself to be effective and sustainable are numbered." Laine Campbell, Charity Majors Database Reliability Engineering O'Reilly
35. @ViscosityNA www.viscosityna.com "Our systems are no more or less important than any other components serving the needs of the business. [T]he components of database clusters are not sacred." Laine Campbell, Charity Majors Database Reliability Engineering O'Reilly
36. @ViscosityNA www.viscosityna.com Databases are Special™ Software Configuration Data Not a Database Database
37. @ViscosityNA www.viscosityna.com Databases are Special™ Software Configuration Data
39. @ViscosityNA www.viscosityna.com What is Cloud Native?
40. Containers are like lightweight Virtual Machines, right?
42. @ViscosityNA viscosityna.com
45. @ViscosityNA www.viscosityna.com An Image is the game board and rules; Containers host and persist games; The superimposed Layer saves moves; Deleting a Container removes its Layer.
46. @ViscosityNA www.viscosityna.com
47. @ViscosityNA www.viscosityna.com Stalemate!
48. @ViscosityNA www.viscosityna.com
49. @ViscosityNA www.viscosityna.com
50. @ViscosityNA www.viscosityna.com Database Containers DB Software Configuration Data Image Container Layer
51. @ViscosityNA www.viscosityna.com Database Containers DB Software Configuration Data Container Local Disk Container Layer
52. @ViscosityNA www.viscosityna.com CONTAINER_NAME=punxsutawney_test mkdir -p /data/${CONTAINER_NAME}/data docker volume create \ --opt type=none \ --opt o=bind \ --opt device=/data/${CONTAINER_NAME}/data \ ${CONTAINER_NAME}_data
53. @ViscosityNA www.viscosityna.com An Image is the game board and rules. Containers host and persist games.
54. @ViscosityNA www.viscosityna.com Database Image Rule #1: If there is a database, start it. If not, create a new database.
55. @ViscosityNA www.viscosityna.com Database Container Startup docker run -d \ --name ${CONTAINER_NAME} \ -e ORACLE_SID=ORCL \ -v /data/${CONTAINER_NAME}:/opt/oracle/oradata \ oracle/database:19.3.0-ee
56. @ViscosityNA www.viscosityna.com Clone the Data Directory cp -rp /data/punxsutawney/ \ /data/${CONTAINER_NAME}
57. @ViscosityNA www.viscosityna.com Clone the Data Directory cp -rp /data/punxsutawney/ \ /data/${CONTAINER_NAME} cp -rp /NFS/gold_data/punxsutawney/ \ /data/${CONTAINER_NAME}
58. @ViscosityNA www.viscosityna.com Clone the Data Directory cp -rp /data/punxsutawney/ \ /data/${CONTAINER_NAME} cp -rp /NFS/gold_data/punxsutawney/ \ /data/${CONTAINER_NAME} cp -rp /NFS/gold_data/punxsutawney/2023-02-02.0600/ \ /data/${CONTAINER_NAME}
59. @ViscosityNA www.viscosityna.com If data can be codified, data is versionable.
60. @ViscosityNA www.viscosityna.com Versioned Data 2023-02-02 Wake up campers! Phil? Phil Connors? Six more weeks of winter? Blizzard is coming!
61. @ViscosityNA www.viscosityna.com Versioned Data 2023-02-02 ARMORED_TRUCKS: NULL LAKES_AND_RIVERS: NULL IMPRESSIVE_PIANO: NULL ICE_SCULPTING: NULL select * from knowledge where fname = 'Phil' and lname = 'Connors';
62. @ViscosityNA www.viscosityna.com Versioned Data 2023-02-02 + phil: main 2023-02-02 + phil: day-2 2023-02-02 + phil: redemption
63. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA
64. @ViscosityNA www.viscosityna.com Apply lean principles to versioned data.
65. @ViscosityNA www.viscosityna.com Versioned Data 2023-02-02 day-2.sql day-x.sql + +
66. @ViscosityNA www.viscosityna.com Versioned Data 2023-02-02 day-2.sql day-x.sql + +
67. @ViscosityNA www.viscosityna.com Entrypoints • Generally adapted but not an OCI standard • Implementation varies by image, vendor/author • Setup only • Setup & startup
68. @ViscosityNA www.viscosityna.com Versioned Data 2023-02-02 day-2.sql day-x.sql + + day-2.sql day-x.sql Phil Microservice Rita Microservice + +
69. @ViscosityNA www.viscosityna.com Save ancestor data as a dataset.
70. @ViscosityNA www.viscosityna.com Apply versioned SQL to create descendants.
71. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main └─ day-2 └─ bank-robber └─ piano-student └─ redemption 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tape segm ent 7:20 AM Blizzard! 9:00 AM Punxsutawney_TEST Feb 2 FUNNY
72. @ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main 6:00 AM Phil w akes up Punxsutawney_TEST Feb 3 PHIL & RITA LIVE HAPPILY EVER AFTER
74. @ViscosityNA www.viscosityna.com Data as Code: Standardizes data Decouples DBAs from the development lifecycle Accelerates & streamlines development Assures test reliability and quality
75. @ViscosityNA www.viscosityna.com Contact Me sean.scott@viscosityna.com https://linktr.ee/oraclesean