Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Taming a huge, sprawling system with Clojure

Avatar for rsslldnphy rsslldnphy
December 05, 2017

Taming a huge, sprawling system with Clojure

#clojurex 2017 conference talk by Mahsa Mohammadkhani and Russell Dunphy from HealthUnlocked

Avatar for rsslldnphy

rsslldnphy

December 05, 2017

Other Decks in Technology

Transcript

  1. What this talk is about healthunlocked.com z A warts and

    all look at HealthUnlocked’s adopAon of Clojure, 4 years on. A discussion of some of the problems we encountered. A grab bag of pracAcal techniques we found for miAgaAng them.
  2. z healthunlocked.com Search Search for arthritis, running, methotrexate, COPD… Top

    3 health website in the UK Top 17 health website in the world A social network for health
  3. had never met someone else with the same health condiAon

    as themselves before coming to HealthUnlocked A plaForm for peer to peer support z healthunlocked.com 63%
  4. 54% 96% feel more confident managing their health visit their

    GP/doctor less frequently had be3er interacAons with their GP/doctor would recommend HealthUnlocked We posiDvely affect people’s health z healthunlocked.com 27% 77%
  5. A technical history healthunlocked.com z The first version of HealthUnlocked,

    an ASP.NET Web Form applicaAon, was released. 2010 2012 The first big rewrite, into a .NET API and Javascript frontend. 2013 An a3empt to introduce NodeJS backfired, and in the rush to deal with the fallout, our first Clojure app was born. We decided we quite liked Clojure, actually, and built preIy much everything in it from then on
  6. Why is this bad? healthunlocked.com z We have a lot

    of (nearly) duplicated code Each app has its own code for cross-cuYng concerns such as logging, configuraAon, error reporAng, connecAon management…. Our understanding of how to do those things has evolved over the years So we have many variaAons on similar themes sca3ered across our codebases
  7. Why is this bad? healthunlocked.com z Knowledge is pre3y isolated

    And they’re probably busy because they’re also the only person who knows how apps Y and Z work… Usually there’s only one person who knows how app X works and/or how to deploy it
  8. Other difficulDes we faced healthunlocked.com z Several of our apps

    combined data from mulAple other apps or data sources. To understand what data you had, it was someAmes necessary to follow the thread through mulAple different applicaAons. Dependencies between apps or of apps on services made it difficult to test all features locally.
  9. Unhelpful unit tests healthunlocked.com z Most of these apps had

    decent (Midje) unit test coverage, but: • they seemed to break whenever we made any changes • rarely caught any of the bugs we introduced • mocked out all external services, so didn’t test these interacAons

  10. Conway’s Law healthunlocked.com z “organizaAons … are constrained to produce

    designs which are copies of [their] communicaAon structures” “The 'Inverse Conway Maneuver' recommends evolving your team and organizaAonal structure to promote your desired architecture” Your code, organisaAon, and teams are interrelated parts of the same ecosystem, and should be designed together
  11. What did we plan? healthunlocked.com z So we decided to:

    • Consolidate our code into fewer, bigger applicaAons • Create mulA-app repos to group together related apps We wanted to: • Promote collaboraAon and pairing • Create good examples of best pracAces • Have these examples followed consistently
  12. MulD-app repositories (not quite a monorepo) healthunlocked.com z ApplicaAons grouped

    by funcAonal area Database access isolated to apps in the repository Everything needed to run the set of applicaAons collected in one place Easy start-up of dev- and test-Ame services using docker-compose
  13. healthunlocked.com z PRESENTING automatically map ports assigned by docker-compose into

    your environ environment https://github.com/HealthUnlocked/lein-docker-compose lein-docker-compose
  14. Moving from horizontal to verDcal slices healthunlocked.com z Inspired by

    the re-frame template A simple change but one we found to have a profound effect Code is grouped by funcAonal area Shared code dealing with cross-cuYng concerns at the top-level - this is scary to change Code sharing between subfolders is not allowed, so the effect of changes here is localised
  15. A change in tesDng strategy healthunlocked.com z We decided to:

    • Focus on integraAon tests over unit tests • Move from Midje to clojure.test • Write tests that hit the DB rather than mocked it out • InvesAgate generaAve tesAng
  16. Reaping the benefits of integraDon tests healthunlocked.com z eHNA is

    an applicaAon we built in partnership with Macmillan Cancer Support We used an append-only data model to make the data more easily auditable. Halfway into the project we ran into some performance issues with our table design Thanks to our integraAon tests we were able to completely replace every database table and all our data access code with a new, performant design with no bugs introduced.
  17. A useful paiern healthunlocked.com z • DeclaraAve at the top

    level • Schemas to validate config, inputs and outputs • Isolated imperaAve code for each individual feature • The tricky to get right stuff pushed into shared code
  18. Aside: How cool is Clojure? healthunlocked.com z Look up a

    job definiAon Do a depth-first traversal of its dependency graph Reverse it to find a safe order of execuAon In 6 lines of code…
  19. Summary healthunlocked.com z Design your code with your team in

    mind, and vice versa An integraAon test is worth a thousand unit tests VerAcal slices allow you to group code by funcAonal area LimiAng the potenAal scope of changes makes them much less scary Schemas at applicaAon boundaries are invaluable for validaAon and documentaAon