Data as Code

Data as Code: Exploring Data and Database Portability Latin America
Oracle User Group Community Tour '23 July 29 - August 16, 2023 @ViscosityNA www.viscosityna.com Sean Scott Oracle ACE Director DoK Community Ambassador Managing Principal Consultant @oraclesean linkedin.com/in/soscott/ [email protected]

The Intersectionality of Cloud Native and "Groundhog Day" @ViscosityNA www.viscosityna.com
Latin America Community Tour '23 - Uruguay August 14, 2023 Sean Scott Oracle ACE Director DoK Community Ambassador Managing Principal Consultant @oraclesean linkedin.com/in/soscott/ [email protected]

DATABASE RELIABILITY ENGINEERING ⁘ DEVOPS & AUTOMATION HIGH AVAILABILITY ⁘
BUSINESS CONTINUITY ⁘ DISASTER RECOVERY MODERNIZATION ⁘ OBSERVABILITY ⁘ ENGINEERED SYSTEMS AutoUpgrade ⁘ Zero Downtime Migrations ⁘ Patching Real Application Clusters ⁘ Data Guard ⁘ Sharding Docker/Containers ⁘ Terraform ⁘ Ansible Exadata ⁘ Oracle Database Appliance AHF ⁘ TFA ⁘ GIMR ⁘ CHA @ViscosityNA www.viscosityna.com Sean Scott Oracle ACE Director DoK Community Ambassador Managing Principal Consultant @oraclesean linkedin.com/in/soscott/ [email protected]

An Open Community for Data on Kubernetes https://dok.community

@ViscosityNA www.viscosityna.com Oracle on Docker Running Oracle Databases in Linux
Containers Download a free sample chapter: https:/ /oraclesean.com 20% Discount Code: OracleDocker https:/ /link.springer.com

@ViscosityNA www.viscosityna.com "Databases are Special™"

@ViscosityNA www.viscosityna.com "Databases are Special™" DBA DEV

@ViscosityNA www.viscosityna.com Databases are Special™

@ViscosityNA www.viscosityna.com As creators, buyers, and influencers, developers have become
instrumental in deciding what products pass muster. Ultimately, the C-suite may call the shot, but the developer is often guiding its aim. Boston Consulting Group "The Growing Influence of Developers in Enterprise Tech Sales" https://www.bcg.com/publications/2022/developers-influence-in-enterprise-tech-sales

@ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA 6:00 AM Phil w akes up
Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM

@ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main 6:00 AM Phil
w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM

@ViscosityNA www.viscosityna.com Feature Request Customers are complaining that the Phil
microservice is narcissistic and misogynistic. Redesign Phil to be a decent human being.

@ViscosityNA www.viscosityna.com

@ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main └─ day-2 6:00
AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM

@ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main └─ day-2 └─
bank-robber 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM

bank-robber └─ piano-student 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM

bank-robber └─ piano-student └─ redemption 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM

bank-robber └─ piano-student └─ redemption 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM Punxsutawney_TEST Feb 2 FUNNY

@ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA 6:00 AM Phil w akes up
Encounters N ed 7:00 AM Tapes segm ent 7:20 AM Blizzard! 9:00 AM Punxsutawney_TEST Feb 3 NOT FUNNY Phil Connors: main └─ day-2 └─ bank-robber └─ piano-student └─ redemption

bank-robber └─ piano-student └─ redemption 6:00 AM Alarm doesn't go off! N ed is PO TU S! 7:00 AM M ultiple groundhogs? 7:20 AM M iam i Beach? 9:00 AM Punxsutawney_TEST Polluted Feb 2 BIZARRE

@ViscosityNA www.viscosityna.com Problems • Every test further pollutes data. •
Restoring databases is operationally difficult, requiring: • Downtime • Specialized skills/DBA intervention • Every database is a production database • Making databases idempotent is technically difficult

DBA DEVELOPER

@ViscosityNA www.viscosityna.com Common Solutions • Build exceptions/elasticity into CI/CD pipelines
• Technically difficult • Requires upkeep • Not absolute • Assume certain errors can be ignored • Technology solution ($$$$$)

@ViscosityNA www.viscosityna.com But... Data is Code

@ViscosityNA www.viscosityna.com Data is Code Every record in every database
is created in one of three ways: • INSERT • UPDATE • DELETE

@ViscosityNA www.viscosityna.com Heterogenous replication = synchronous SQL Oracle Tx Log
SQL MySQL

@ViscosityNA www.viscosityna.com Heterogenous replication = synchronous SQL Oracle Tx Log
SQL MySQL Optional Transformation

@ViscosityNA www.viscosityna.com Databases are Special™

@ViscosityNA www.viscosityna.com "[F]or a long time, DBAs were in the
business of crafting silos and snowflakes. Their tools were different, their hardware was different, and their languages were different." Laine Campbell, Charity Majors Database Reliability Engineering O'Reilly

@ViscosityNA www.viscosityna.com "The days in which this model can prove
itself to be effective and sustainable are numbered." Laine Campbell, Charity Majors Database Reliability Engineering O'Reilly

@ViscosityNA www.viscosityna.com "Our systems are no more or less important
than any other components serving the needs of the business. [T]he components of database clusters are not sacred." Laine Campbell, Charity Majors Database Reliability Engineering O'Reilly

@ViscosityNA www.viscosityna.com Databases are Special™ Software Configuration Data Not a
Database Database

@ViscosityNA www.viscosityna.com Databases are Special™ Software Configuration Data

@ViscosityNA www.viscosityna.com What is Cloud Native?

Containers are like lightweight Virtual Machines, right?

@ViscosityNA viscosityna.com

@ViscosityNA www.viscosityna.com An Image is the game board and rules;
Containers host and persist games; The superimposed Layer saves moves; Deleting a Container removes its Layer.

@ViscosityNA www.viscosityna.com Stalemate!

@ViscosityNA www.viscosityna.com Database Containers DB Software Configuration Data Image Container
Layer

@ViscosityNA www.viscosityna.com Database Containers DB Software Configuration Data Container Local
Disk Container Layer

@ViscosityNA www.viscosityna.com CONTAINER_NAME=punxsutawney_test mkdir -p /data/${CONTAINER_NAME}/data docker volume create \
--opt type=none \ --opt o=bind \ --opt device=/data/${CONTAINER_NAME}/data \ ${CONTAINER_NAME}_data

@ViscosityNA www.viscosityna.com An Image is the game board and rules.
Containers host and persist games.

@ViscosityNA www.viscosityna.com Database Image Rule #1: If there is a
database, start it. If not, create a new database.

@ViscosityNA www.viscosityna.com Database Container Startup docker run -d \ --name
${CONTAINER_NAME} \ -e ORACLE_SID=ORCL \ -v /data/${CONTAINER_NAME}:/opt/oracle/oradata \ oracle/database:19.3.0-ee

@ViscosityNA www.viscosityna.com Clone the Data Directory cp -rp /data/punxsutawney/ \
/data/${CONTAINER_NAME}

/data/${CONTAINER_NAME} cp -rp /NFS/gold_data/punxsutawney/ \ /data/${CONTAINER_NAME}

/data/${CONTAINER_NAME} cp -rp /NFS/gold_data/punxsutawney/ \ /data/${CONTAINER_NAME} cp -rp /NFS/gold_data/punxsutawney/2023-02-02.0600/ \ /data/${CONTAINER_NAME}

@ViscosityNA www.viscosityna.com If data can be codified, data is versionable.

@ViscosityNA www.viscosityna.com Versioned Data 2023-02-02 Wake up campers! Phil? Phil
Connors? Six more weeks of winter? Blizzard is coming!

@ViscosityNA www.viscosityna.com Versioned Data 2023-02-02 ARMORED_TRUCKS: NULL LAKES_AND_RIVERS: NULL IMPRESSIVE_PIANO:
NULL ICE_SCULPTING: NULL select * from knowledge where fname = 'Phil' and lname = 'Connors';

@ViscosityNA www.viscosityna.com Versioned Data 2023-02-02 + phil: main 2023-02-02 +
phil: day-2 2023-02-02 + phil: redemption

@ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA

@ViscosityNA www.viscosityna.com Apply lean principles to versioned data.

@ViscosityNA www.viscosityna.com Versioned Data 2023-02-02 day-2.sql day-x.sql + +

@ViscosityNA www.viscosityna.com Entrypoints • Generally adapted but not an OCI
standard • Implementation varies by image, vendor/author • Setup only • Setup & startup

@ViscosityNA www.viscosityna.com Versioned Data 2023-02-02 day-2.sql day-x.sql + + day-2.sql
day-x.sql Phil Microservice Rita Microservice + +

@ViscosityNA www.viscosityna.com Save ancestor data as a dataset.

@ViscosityNA www.viscosityna.com Apply versioned SQL to create descendants.

bank-robber └─ piano-student └─ redemption 6:00 AM Phil w akes up Encounters N ed 7:00 AM Tape segm ent 7:20 AM Blizzard! 9:00 AM Punxsutawney_TEST Feb 2 FUNNY

@ViscosityNA www.viscosityna.com www.viscosityna.com @ViscosityNA Phil Connors: main 6:00 AM Phil
w akes up Punxsutawney_TEST Feb 3 PHIL & RITA LIVE HAPPILY EVER AFTER

@ViscosityNA www.viscosityna.com Data as Code: Standardizes data Decouples DBAs from
the development lifecycle Accelerates & streamlines development Assures test reliability and quality

@ViscosityNA www.viscosityna.com Contact Me [email protected] https://linktr.ee/oraclesean

Data as Code

Data as Code

More Decks by Sean Scott

Other Decks in Technology

Featured

Transcript