How we migrated a complex JavaScript application to ClojureScript step-by-step

As software evolves along with the languages and frameworks it is built upon, it’s easy to create a monster. This is the tale of tackling complexity at the heart of Exoscale, and how a data-driven approach with ClojureScript enabled us to seamlessly replace a JavaScript app without user outage.

Transcript

1. Migrating a complex JS app to ClojureScript Alex King
2. Alexander James King • github.com/alexanderjamesking • Clojure dev at Exoscale • Clojure(script) / Scala / Java / JS • Previous: JUXT / BBC / HMRC
4. • European cloud provider • Data centres in Switzerland, Germany and Austria • CPU, GPU, Object Storage, Elastic IPs, DNS, Run Status • Data Security, GDPR Compliance, Data Privacy
7. Product Problems Going fast Implementation Lessons learned Conclusion
11. Browser Python API Architecture JS Platform API Gateway
12. Product Problems Going fast Implementation Lessons learned Conclusion
13. #1 Feature Slowdown Features Time
14. #2 Tests UI Integration Unit Manual
15. #3 Tooling
21. Problems • Velocity • Lack of tests • Tooling • Build tools • Dependencies • Speed of change
22. Product Problems Going fast Implementation Lessons learned Conclusion
23. –Robert Cecil Martin “The higher the quality, the faster you go. The only way to go fast is to go well.”
24. Improving the Quality • Tests at all levels of the pyramid • Fewer dependencies • Small, focused libraries • Better tooling
25. ClojureScript • Functional programming • Immutability • Data Driven • Small, focused libraries • Browser REPL
26. ClojureScript Reitit Integrant re-frame More Dependencies! Reagent
27. “Sometimes, things have to get worse before they can get better”
28. Product Problems Going fast Implementation Lessons learned Conclusion
29. The grand redesign in the sky This time we will get it right!
33. Greenfield vs Brownfield Build a Replacement Not very agile ∴ High risk Moving target ∴ High risk Migrate the existing app Agile ∴ Low risk Complex ∴ Medium risk
34. Brownfield: Migrate the existing app
35. How can we migrate? URL: Server Page reloads (e.g nginx) URL: Browser HTML5 History API Component Lots of JS Interop 1 2 3
36. Clojure Python :option-1 URL: Server JS CLJS
37. Clojure Python :option-1 URL: Server JS CLJS
38. Clojure Python :option-1 URL: Server JS CLJS + Clean codebase - Page reloads required - Multiple apps to manage
39. Python :option-2 URL: Browser JS CLJS A2 Router A1 Router History API
40. Python :option-2 URL: Browser JS CLJS A2 Router Router A1 History API
41. Python :option-2 URL: Browser JS CLJS A2 Router A1 +Single app +No page reloads - Challenging Interop? History API
42. :option-3 Component ClojureScript JS
43. :option-3 Component ClojureScript +Single app +No page reloads +Agile - Challenging Interop - High complexity
44. (and :option-2 :option-3) ClojureScript JS
45. <div class="ng-scope“ ng-controller=“NavController"> <div id=“alerts-wrapper”>…</div> <div class="col-nav primary”>…</div> <div class="internal-wrapper"> <div class="col-nav secondary ng-scope”>…</div> <div class="col-content"> <div class="top-bar"></div> <div id="cljs-app"></div> <ng-view class=“ng-scope"></ng-view> </div> </div> </div>
46. One URL at a time • Bootstrap both JS and CLJS apps • Routing: History API • Minimal JS <-> CLJS interop • Write UI tests against the JS app
47. Browser Python API Architecture JS CLJS Platform API Gateway
48. UI Tests Browser Python API JS CLJS Mock API Scope
49. UI Tests • Selenium and Chrome Headless • Clojure mock of the API • Independent of implementation • Write tests against JS • Swap JS with CLJS
51. Python API CLJS Architecture Controller Router Data Instance IP IP Addresses View Model
52. Python API Initial Release Controller Router Data Instance IP IP Addresses View Model JS API CLJS API Poller
53. Python API Additional Modules Controller Router Instances Data Instance Firewall IP View Model IP Addresses View Model JS API CLJS API Poller
54. Libraries & Frameworks
55. reagent-project / reagent • Define React components with data (Hiccup-like syntax) • reagent.core/atom for component state • reagent.core/render
56. ;; basic component (defn title-bar [] [:div.title-bar [:h1.page-title.ellipsis "New Instance”]]) ;; with local state (defn confirm-delete [callback] (let [confirming? (reagent/atom false) toggle #(swap! confirming? not)] (fn [delete-fn] (if @confirming? [:li [:button {:on-click callback} "Delete"] [:button {:on-click toggle} "Cancel"]] [:li [:button {:on-click toggle} "Delete"]]))))
57. Day8 / re-frame • State management (e.g. Flux/Redux) • Unidirectional data flow DB Subscriptions View Events
58. (ns portal.compute.ssh-keys.model) (rf/reg-event-db ::quick-filter (fn [db [_ value]] (assoc db ::quick-filter value))) (rf/reg-sub ::quick-filter (fn [db _] (not-empty (::quick-filter db)))) (ns portal.compute.ssh-keys.view) (defn quick-filter-names [] [:input {:on-change #(rf/dispatch [::model/quick-filter (.. % -target -value)]) :value @(rf/subscribe [::model/quick-filter]) :type "text"}])
59. weavejester / integrant • Application structure (e.g. Component/Mount) • System as data • Why? • Dependency Injection • Multiple configurations
60. (ns portal.system) {::router/init {:routes routes/app-routes :route-data route-data} ::notification/init {:callback js/window.jsAPI.displayNotification} ::http/init {:api-base-url api-base-url :csrf-token csrf-token} ::controller/init {:target-id target-id}} (ns portal.notification) (defmethod ig/init-key ::init [_ {:keys [callback]}] (rf/reg-fx ::info (partial display-notification callback “info")))
61. Primary Content Only ClojureScript JS
62. Primary Content + Top Bar ClojureScript JS
63. Everything in CLJS :) ClojureScript
64. Mock JS Interop ClojureScript Mock
65. Subset of JS Interop ClojureScript Mock
66. metosin/reitit • Bi-directional • Data-driven • Attach data to routes ! • Define in CLJC, add data in CLJ or CLJS
67. (ns portal.routes) ;; CLJC (def app-routes ["" ["/u/:workspace" ["/compute" ["/ip" ["" :compute-ip] ["/:zone" :compute-ip-zone]] ["/instances" ["" :compute-instances] ["/add" :compute-instance-add] ["/:id" ["" :compute-instance] ["/move" :compute-instance-move] ["/scale" :compute-instance-scale]]]]]])
68. ;; reitit route data (ns portal.system) ;; CLJS {:compute-instance {:view instance-view/view :mount ::instance-model/mount :unmount ::instance-model/unmount :query->ttl {::instance/list 10 ::volume/list 10 ::private-network/list 10 ::snapshot/list 10}}} ;; route-data at construction (reitit/router routes {:conflicts nil :expand (expand route-data)})
69. Product Problems Going fast Implementation Lessons learned Conclusion
70. – Alan J. Perlis “It is better to have 100 functions operate on one data structure than 10 functions on 10 data structures.”
71. Small functions • Performance • Re-frame subscriptions • Reagent components • React Developer Tools
72. ::namespaced/keywords • Namespace owns the data • Avoid deeply nested DB • Avoid circular dependencies • (::ip/list not :portal.data.ip/list) • Separate transient and persistent state
73. ::namespaced/keywords in re-frame db cljs.user> (-> @re-frame.db/app-db keys sort) (:portal.compute.instance.model/editing-displayname? :portal.compute.instance.model/input-displayname :portal.data/last-fetch :portal.data/subscriptions :portal.data.account/account :portal.data.firewall/list :portal.data.instance/list :portal.data.private-network/list :portal.data.service-offering/list :portal.data.ssh-key/list :portal.data.volume/list :portal.data.zone/list :portal.router/route :portal.router/workspace)
74. ::namespaced/keywords in re-frame event handler (ns portal.data.ip) (rf/reg-event-fx ::allocate-success (fn [_ _] {::notification/info "Successfully allocated IP” ::js-api/fetch [::list] ::data/fetch-n [[::list] [::instance/list]]}))
75. Encapsulation
76. ;; Java 101 public class Person { private String name; public String getName() { return name; } public void setName(String name) { this.name = name; } }
77. (ns portal.router) (defn route-handler [db] ;; encapsulation (get-in db [::route :data :name])) (rf/reg-sub ::route-handler route-handler) (ns portal.compute.firewalling.model) (when (= :compute-firewalling (router/route-handler db)) ;; ...)
78. Embrace JS Interop • Interop by URL alone is not enough • Outgoing: JS API • Incoming: CLJS API
79. window.jsAPI = { fetchAccount: function() { reduxDispatch(fetch_account(false)); }, fetchInstances: function() { reduxDispatch(fetch_instances()); }, displayNotification: function(msg, level) { getModule('notify')({ message: msg, classes: `alert-${level}`, }); }, ... };
80. (ns portal.cljs-api) (defn ^:export match-route [route] (rf/dispatch-sync [::router/match-route route])) (defn ^:export fetch-ips [] (rf/dispatch [::data/fetch [::ip/list]])) (defn ^:export fetch-instances [] (rf/dispatch [::data/fetch [::instance/list]])) ...
81. ;; Calling JS from CLJS (js/window.jsAPI.fetchInstances) (js/window.jsAPI.displayNotification "info" "VM Created") ;; Calling CLJS from JS portal.cljs_api.fetch_instances(); portal.cljs_api.match_route(action.payload); ;; Converting data structures cljs.user> (clj->js {:hello "Dutch Clojure Days"}) #js {:hello "Dutch Clojure Days"} cljs.user> (js->clj #js {:hello "Dutch Clojure Days"} :keywordize-keys true)
82. Model data as maps • Even lists! • Easier data access • Optimistic updates • get-in, assoc-in, update-in • keys, vals
83. ;; vector cljs.user> (->> [{:id "a1" :ipaddress "1.2.3.4"} {:id "a2" :ipaddress "1.2.3.3"}] (filter #(= "a2" (:id %))) first :ipaddress) “1.2.3.3” ;; map cljs.user> (-> {"a1" {:id "a1" :ipaddress "1.2.3.4"} "a2" {:id "a2" :ipaddress "1.2.3.3"}} (get-in ["a2" :ipaddress])) “1.2.3.3"
84. Don’t re-frame all the things!
85. Product Problems Going fast Implementation Lessons learned Conclusion
86. What went well? • UI Tests • One URL at a time • Data layer • JS Interop
87. Things we can improve • More JS Interop • Smaller initial scope • Integration tests • Reusable components
88. If we could change one thing…
89. Python API Initial Release Controller Router Data Instance IP IP Addresses View Model JS API CLJS API Poller
90. Share the data layer Controller Router Data IP Addresses View Model JS API CLJS API
91. Thank you for listening! www.exoscale.com alexanderjamesking.com github.com/alexanderjamesking