Pro Yearly is on sale from $80 to $50! »

FloorPlan: Visualize databases' evolution

FloorPlan: Visualize databases' evolution

As software matures, more functionality is added, code gets moved, re-written or removed. In the same rhythm, team members change and with them historical knowledge might get lost. For that reason, it is important to have a well-organized project, that is inviting for new contributors while still digestible for one-off readers.

Understanding how data is structured can be very helpful as an initial step for diving into the abstract modeling of the business domain. Database schemas, then, provide that entry window, but building the mental model out of a machine-readable format can be a daunting task even for seasoned engineers, especially when the project grows large and changes rapidly.

In this talk, we will learn how this challenge drove the creation of FloorPlan, an open source Kotlin library to translate database schemas into ER diagrams, discuss its development and application for the team of 20+ Android engineers at SoundCloud.

2bf14ad96e09ea7eda4973795ce70fa6?s=128

Júlio Zynger

October 22, 2020
Tweet

Transcript

  1. FloorPlan: Visualize databases’ evolution www.juliozynger.com

  2. SoundCloud www.juliozynger.com • 200+ million tracks → world’s largest open

    audio platform • Over 25 million creators • Available in 190 countries • 90%+ of usage comes from mobile devices
  3. SoundCloud www.juliozynger.com • Started as Rails application • Microservice extraction

    • Dozens of engineers, hundreds of services • Polyglot development and tooling
  4. SoundCloud www.juliozynger.com • ~20 engineers • ~60 merged PRs per

    week • ~250 modules, ~8000 tests, ~15 databases
  5. SoundCloud Bare SQL www.juliozynger.com

  6. SoundCloud www.juliozynger.com Bare SQL Propeller Lean API Type Safety Observability,

    etc.
  7. SoundCloud www.juliozynger.com SQLDelight Code generation SQL / Java,Kt separation Multiplatform

    Bare SQL Propeller Lean API Type Safety Observability, etc.
  8. SoundCloud www.juliozynger.com Room Google’s solution Onboardable Well-documented SQLDelight Code generation

    SQL / Java,Kt separation Multiplatform Bare SQL Propeller Lean API Type Safety Observability, etc.
  9. www.juliozynger.com

  10. www.juliozynger.com Documentation

  11. www.juliozynger.com

  12. www.juliozynger.com

  13. www.juliozynger.com

  14. www.juliozynger.com

  15. A document shouldn’t try to do what the code already

    does well. www.juliozynger.com “ ― Eric Evans, Domain-Driven Design: Tackling Complexity in the Heart of Software
  16. www.juliozynger.com ?

  17. www.juliozynger.com FloorPlan

  18. Architecture www.juliozynger.com Consumer Schema DBML

  19. Architecture www.juliozynger.com Consumer Schema DBML impl

  20. www.juliozynger.com

  21. www.juliozynger.com

  22. Architecture www.juliozynger.com Consumer Schema DBML Room Consumer

  23. Architecture www.juliozynger.com Consumer Schema DBML SQLite Consumer

  24. DBML (dbml.org) www.juliozynger.com • Open-source DSL language • Define and

    document DB schemas • Designed to be simple, consistent and highly-readable
  25. DBML (dbml.org) www.juliozynger.com Table users { id integer [pk] username

    varchar role varchar } Table posts { id integer [pk] title varchar body text [note: 'Content of the post'] user_id integer } Ref: posts.user_id > users.id
  26. Rendering: GraphViz www.juliozynger.com • Open-source graph visualization software • Java

    bindings through graphviz-java val graph: Graph = mutGraph() graph.addNodes(tables.map { it.toNode() }) graph.addLinks(references.map { it.toLink() }) g.render(Format.SVG)
  27. www.juliozynger.com Interactivity through metadata

  28. Workflow www.juliozynger.com

  29. www.juliozynger.com CLI

  30. www.juliozynger.com Gradle Plugin floorPlan { schemaLocation = "$projectDir/schemas" outputLocation =

    "$projectDir/floorplan-output" outputFormat { svg { enabled = true } } }
  31. www.juliozynger.com Gradle Plugin apply plugin: 'kotlin-kapt' apply plugin: 'com.juliozynger.floorplan' kapt

    { arg("room.schemaLocation", "$projectDir/schemas") ... } floorPlan { schemaLocation = "$projectDir/schemas" ... } dependencies { implementation Libs.room_runtime ... } room-support.gradle apply from: "$rootProject.projectDir/android-library.gradle" apply from: "$rootProject.projectDir/room-support.gradle" dependencies { implementation Libs.dagger implementation Libs.appcompat testImplementation Libs.junit ... } some-module/build.gradle
  32. www.juliozynger.com IDE Plugin

  33. www.juliozynger.com IDE Plugin

  34. www.juliozynger.com SoundCloud-Android ├─ app ├─ docs ├─ modules │ ├─

    discovery │ │ ├─ src │ │ ├─ schemas │ │ └─ floorplan-output │ ├─ player │ │ ├─ src │ │ ├─ schemas │ │ └─ floorplan-output │ ├─ upload │ │ ├─ src │ │ ├─ schemas │ │ └─ floorplan-output └─ ... Serving SoundCloud-Android ├─ app ├─ docs │ └─ databases │ ├─ discovery │ ├─ player │ └─ upload │ ├─ 1.svg │ ├─ 2.svg │ └─ ... ├─ modules │ └─ ... └─ mkdocs.yml </>
  35. www.juliozynger.com

  36. www.juliozynger.com Code Review

  37. www.juliozynger.com Code Review

  38. Wrap-up www.juliozynger.com

  39. www.juliozynger.com Wrap-up • Source code is source for docs

  40. www.juliozynger.com Wrap-up • Source code is source for docs •

    No disruption through full automation
  41. www.juliozynger.com Wrap-up • Source code is source for docs •

    No disruption through full automation • Value generation in each step of the pipeline
  42. www.juliozynger.com Wrap-up • Source code is source for docs •

    No disruption through full automation • Value generation in each step of the pipeline • Workflow augmentation
  43. www.juliozynger.com Wrap-up • Source code is source for docs •

    No disruption through full automation • Value generation in each step of the pipeline • Workflow augmentation • Enable potential extension
  44. www.juliozynger.com Wrap-up • Source code is source for docs •

    No disruption through full automation • Value generation in each step of the pipeline • Workflow augmentation • Enable potential extension • Accessibility through environment agnostic tooling
  45. www.juliozynger.com Wrap-up • Source code is source for docs •

    No disruption through full automation • Value generation in each step of the pipeline • Workflow augmentation • Enable potential extension • Accessibility through environment agnostic tooling • Inter business department involvement
  46. www.juliozynger.com Wrap-up julioz.github.io/FloorPlan

  47. Thanks! @juliozynger soundcloud.com/jobs