FloorPlan: Visualize databases' evolution

As software matures, more functionality is added, code gets moved, re-written or removed. In the same rhythm, team members change and with them historical knowledge might get lost. For that reason, it is important to have a well-organized project, that is inviting for new contributors while still digestible for one-off readers.

Understanding how data is structured can be very helpful as an initial step for diving into the abstract modeling of the business domain. Database schemas, then, provide that entry window, but building the mental model out of a machine-readable format can be a daunting task even for seasoned engineers, especially when the project grows large and changes rapidly.

In this talk, we will learn how this challenge drove the creation of FloorPlan, an open source Kotlin library to translate database schemas into ER diagrams, discuss its development and application for the team of 20+ Android engineers at SoundCloud.

Transcript

1. FloorPlan: Visualize databases’ evolution www.juliozynger.com
2. SoundCloud www.juliozynger.com • 200+ million tracks → world’s largest open audio platform • Over 25 million creators • Available in 190 countries • 90%+ of usage comes from mobile devices
3. SoundCloud www.juliozynger.com • Started as Rails application • Microservice extraction • Dozens of engineers, hundreds of services • Polyglot development and tooling
4. SoundCloud www.juliozynger.com • ~20 engineers • ~60 merged PRs per week • ~250 modules, ~8000 tests, ~15 databases
5. SoundCloud Bare SQL www.juliozynger.com
6. SoundCloud www.juliozynger.com Bare SQL Propeller Lean API Type Safety Observability, etc.
7. SoundCloud www.juliozynger.com SQLDelight Code generation SQL / Java,Kt separation Multiplatform Bare SQL Propeller Lean API Type Safety Observability, etc.
8. SoundCloud www.juliozynger.com Room Google’s solution Onboardable Well-documented SQLDelight Code generation SQL / Java,Kt separation Multiplatform Bare SQL Propeller Lean API Type Safety Observability, etc.
9. www.juliozynger.com
10. www.juliozynger.com Documentation
11. www.juliozynger.com
12. www.juliozynger.com
13. www.juliozynger.com
14. www.juliozynger.com
15. A document shouldn’t try to do what the code already does well. www.juliozynger.com “ ― Eric Evans, Domain-Driven Design: Tackling Complexity in the Heart of Software
16. www.juliozynger.com ?
17. www.juliozynger.com FloorPlan
18. Architecture www.juliozynger.com Consumer Schema DBML
19. Architecture www.juliozynger.com Consumer Schema DBML impl
20. www.juliozynger.com
21. www.juliozynger.com
22. Architecture www.juliozynger.com Consumer Schema DBML Room Consumer
23. Architecture www.juliozynger.com Consumer Schema DBML SQLite Consumer
24. DBML (dbml.org) www.juliozynger.com • Open-source DSL language • Define and document DB schemas • Designed to be simple, consistent and highly-readable
25. DBML (dbml.org) www.juliozynger.com Table users { id integer [pk] username varchar role varchar } Table posts { id integer [pk] title varchar body text [note: 'Content of the post'] user_id integer } Ref: posts.user_id > users.id
26. Rendering: GraphViz www.juliozynger.com • Open-source graph visualization software • Java bindings through graphviz-java val graph: Graph = mutGraph() graph.addNodes(tables.map { it.toNode() }) graph.addLinks(references.map { it.toLink() }) g.render(Format.SVG)
27. www.juliozynger.com Interactivity through metadata
28. Workflow www.juliozynger.com
29. www.juliozynger.com CLI
30. www.juliozynger.com Gradle Plugin floorPlan { schemaLocation = "$projectDir/schemas" outputLocation = "$projectDir/floorplan-output" outputFormat { svg { enabled = true } } }
31. www.juliozynger.com Gradle Plugin apply plugin: 'kotlin-kapt' apply plugin: 'com.juliozynger.floorplan' kapt { arg("room.schemaLocation", "$projectDir/schemas") ... } floorPlan { schemaLocation = "$projectDir/schemas" ... } dependencies { implementation Libs.room_runtime ... } room-support.gradle apply from: "$rootProject.projectDir/android-library.gradle" apply from: "$rootProject.projectDir/room-support.gradle" dependencies { implementation Libs.dagger implementation Libs.appcompat testImplementation Libs.junit ... } some-module/build.gradle
32. www.juliozynger.com IDE Plugin
33. www.juliozynger.com IDE Plugin
34. www.juliozynger.com SoundCloud-Android ├─ app ├─ docs ├─ modules │ ├─ discovery │ │ ├─ src │ │ ├─ schemas │ │ └─ floorplan-output │ ├─ player │ │ ├─ src │ │ ├─ schemas │ │ └─ floorplan-output │ ├─ upload │ │ ├─ src │ │ ├─ schemas │ │ └─ floorplan-output └─ ... Serving SoundCloud-Android ├─ app ├─ docs │ └─ databases │ ├─ discovery │ ├─ player │ └─ upload │ ├─ 1.svg │ ├─ 2.svg │ └─ ... ├─ modules │ └─ ... └─ mkdocs.yml </>
35. www.juliozynger.com
36. www.juliozynger.com Code Review
37. www.juliozynger.com Code Review
38. Wrap-up www.juliozynger.com
39. www.juliozynger.com Wrap-up • Source code is source for docs
40. www.juliozynger.com Wrap-up • Source code is source for docs • No disruption through full automation
41. www.juliozynger.com Wrap-up • Source code is source for docs • No disruption through full automation • Value generation in each step of the pipeline
42. www.juliozynger.com Wrap-up • Source code is source for docs • No disruption through full automation • Value generation in each step of the pipeline • Workflow augmentation
43. www.juliozynger.com Wrap-up • Source code is source for docs • No disruption through full automation • Value generation in each step of the pipeline • Workflow augmentation • Enable potential extension
44. www.juliozynger.com Wrap-up • Source code is source for docs • No disruption through full automation • Value generation in each step of the pipeline • Workflow augmentation • Enable potential extension • Accessibility through environment agnostic tooling
45. www.juliozynger.com Wrap-up • Source code is source for docs • No disruption through full automation • Value generation in each step of the pipeline • Workflow augmentation • Enable potential extension • Accessibility through environment agnostic tooling • Inter business department involvement
46. www.juliozynger.com Wrap-up julioz.github.io/FloorPlan
47. Thanks! @juliozynger soundcloud.com/jobs