its-time-for-relational-databases-to-focus-on-developers_shlomi-noach.pdf

The relational model is one of the oldest surviving models in computer science. And while it is still popular, it is gradually losing the developers’ good opinion. While relational databases have invested efforts in optimizing speed, latency and overall read and write performance, they have not adapted to the evolving change in developer’s needs.

The relational model comes with an operational complexity, and schema changes remain an operational blocker, outside the domain and ownership of the very developers who plan those changes. As a result, developers perceive relational databases unlike any other production system and seek ways to avoid schema changes.

Exactly what went wrong? And what would it take for relational databases to close the gap and regain the developer’s confidence? This session highlights the problems around making and deploying schema changes. We discuss scheduling, operational conflicts, production impact and blast radius, blue/green deployments, idempotency and more. We provide a paradigm that relational databases can follow, illustrated by Vitess, an OSS CNCF project which implements that paradigm.

Transcript

1. It’s time for relational databases to focus on developers Shlomi Noach Reversim Summit 2022
2. The relational model
3. ALTER TABLE node_health ADD COLUMN extra_info varchar(128) NOT NULL, DROP KEY instance_tag_idx DROP TABLE instance_tags Schema changes (DDLs)
4. About me Engineer at PlanetScale; previously at GitHub, Booking.com, Outbrain Author of open source projects orchestrator, gh-ost, freno and others Maintainer for Vitess Blog at http://openark.org github.com/shlomi-noach @ShlomiNoach
5. Vitess A database clustering system for horizontal scaling of MySQL • CNCF graduated project • Open source, Apache 2.0 licence • Contributors from around the community • Originally developed at YouTube to mass scale their load
6. MySQL-compatible serverless database platform, built for developers Main sponsors for Vitess
7. ALTER TABLE node_health ADD COLUMN extra_info varchar(128) NOT NULL, DROP KEY instance_tag_idx DROP TABLE instance_tags Avoiding schema changes
8. Why?
9. Interacting with the relational database
10. Schema change as an operation The relational database does not have a concept of developers. It only has the concept of operations people.
11. Schema change as an operation
12. Schema change as a deployment We should apply our deployment terminology and practices to schema changes
13. Operational time ALTER TABLE node_health ADD COLUMN extra_info varchar(128) NOT NULL; [Three days later…] Query OK, 0 rows affected (70h 12m 56s)
14. Operational overhead
15. External online schema change tools • pt-online-schema-change • gh-ost • pg-osc
16. app
17. app
18. app Row copy
19. app Row copy
20. app Row copy Apply changes
21. app
22. app
23. app
24. Operational conflicts
26. Blast radius
27. user
28. user 10% deployment
29. user 20% deployment
30. user 50% deployment
31. user 100% deployment
32. app
33. app
34. Blue-green deployments
35. user
36. user
37. user
38. user
39. user ? ? ? ?
40. Undeployment (revert)
41. app
42. app
43. app
44. Apply changes app
45. Apply changes app sync
46. Apply changes sync app
47. Apply changes app
48. Apply changes app
49. Undeploy via blue-green deployments
50. Undeploy as first class citizen
51. Redeployment
52. CREATE TABLE IF NOT EXISTS node_health ( hostname varchar(128) CHARACTER SET ascii NOT NULL, token varchar(128) NOT NULL, last_seen_active timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (hostname, token) ) ENGINE=InnoDB DEFAULT CHARSET=ascii “Base schema”
53. ALTER TABLE node_health DROP PRIMARY KEY, ADD PRIMARY KEY (hostname, token) ALTER TABLE node_health ADD COLUMN extra_info varchar(128) CHARACTER SET utf8 NOT NULL ALTER TABLE node_health MODIFY last_seen_active timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP “patches”
54. if flags2&BinlogThroughGTID != 0 { dataSize, pos, ok := readUint32(data, pos) if !ok { return logFile, logPos, position, readPacketErr } if gtid := string(data[pos : pos+int(dataSize)]); gtid != "" { position, err = DecodePosition(gtid) if err != nil { return logFile, logPos, position, err } } } return logFile, logPos, position, nil Source code
56. ALTER TABLE node_health DROP PRIMARY KEY, ADD PRIMARY KEY (hostname, token) ALTER TABLE node_health ADD COLUMN extra_info varchar(128) CHARACTER SET utf8 NOT NULL ALTER TABLE node_health MODIFY last_seen_active timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP Human calculated diff
57. CREATE TABLE IF NOT EXISTS node_health ( hostname varchar(128) CHARACTER SET ascii NOT NULL, token varchar(128) NOT NULL, last_seen_active timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, PRIMARY KEY (hostname, token) ) ENGINE=InnoDB DEFAULT CHARSET=ascii Declarative schema
58. CREATE TABLE IF NOT EXISTS node_health ( hostname varchar(128) CHARACTER SET ascii NOT NULL, token varchar(128) NOT NULL, last_seen_active timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP, extra_info varchar(128) CHARACTER SET utf8 NOT NULL PRIMARY KEY (hostname, token) ) ENGINE=InnoDB DEFAULT CHARSET=ascii Declarative schema
59. ALTER TABLE node_health ADD COLUMN extra_info varchar(128) CHARACTER SET utf8 NOT NULL Imperative deployments via DDL statements
60. Idempotent deployments
61. More to discuss Asynchronous and scheduled deployments, job management, load shedding and throttling, discoverability, multi change deployments, multi target deployments, artifacts and garbage collection
62. How did we let this happen?
63. What’s next?
64. Thank you! @ShlomiNoach github.com/shlomi-noach Images source: https://www.pexels.com/