Testing Business Logic Using Domain Specific Languages (DSLs) in Clojure

Talk presented at FuConf'15
Video - https://www.youtube.com/watch?v=YOsfPrgNY_M

Transcript

1. Testing Business Logic Using Domain Specific Languages (DSLs) in Clojure Mayank Jain Test Engineer
2. WHAT IS THE PROBLEM? Testing real world stateful business logic is hard
3. Microwave Oven State Machine Business logic is Stateful
4. Difficult to enumerate all possible cases Microwave Oven State Machine
5. Tests become unreadable Microwave Oven State Machine
6. A Domain Specific Language (DSL) is a programming language designed specifically to express solutions to problems in a specific domain. DSL
7. HELPSHIFT • Embeddable support desk for native apps • Main Features: • Frequently Asked Questions which customers can search • File issues/tickets from within the app
8. Helpshift Supercell Clash of Clans Gaana General Billing Domain App Section FAQ 1 FAQ 2 FAQs Boom Beach Gaana App Translations English Content Hindi Content …. …. …. …. …. Top Down View of FAQs
9. Example of Gaana FAQ Page Customers can search FAQs Available FAQs FAQ Sections Domain App
10. FAQ Title FAQ Body FAQ is Visible?
11. PROBLEM You cannot share FAQs across apps. Big Customers have multiple apps which have same FAQ translations content example “Privacy Policy”
12. FEATURE: LINKED FAQS
13. {:published? false :id “faq-id-1” :app_id “app-1” :section_id “section-1” :translations {:en {:published? false :stags [] :body “Privacy Body” :title “Privacy Title”}} :linked_faq_ids [“faq-id-2”] :publish_id “1”} App-1 {:published? false :id “faq-id-2” :app_id “app-2” :section_id “section-2” :translations {:en {:published? false :stags [] :body “Privacy Body" :title “Privacy Title"}} :linked_faq_ids [“faq-id-1”] :publish_id “2”} App-2 faq 1 (Sync faq-1 to App-2) General faq 2 General
14. {:published? false :id “faq-id-1” :app_id “app-1” :section_id “section-1” :translations {:en {:published? false :stags [] :body “Update Body” :title “Privacy Title”}} :linked_faq_ids [“faq-id-2”] :publish_id “1”} {:published? false :id “faq-id-2” :app_id “app-2” :section_id “section-2” :translations {:en {:published? false :stags [] :body “Privacy Body" :title “Privacy Title"}} :linked_faq_ids [“faq-id-1”] :publish_id “2”} faq 1 Sync faq 2 Update FAQ 1’s Body
15. {:published? false :id “faq-id-1” :app_id “app-1” :section_id “section-1” :translations {:en {:published? false :stags [] :body “Update Body” :title “Privacy Title”}} :linked_faq_ids [“faq-id-2”] :publish_id “1”} {:published? false :id “faq-id-2” :app_id “app-2” :section_id “section-2” :translations {:en {:published? false :stags [] :body “Update Body" :title “Privacy Title"}} :linked_faq_ids [“faq-id-1”] :publish_id “2”} faq 1 Sync faq 2 Update FAQ 1’s Body
16. EXAMPLE TEST CASE
17. 1. Add 1st App with only English languages. 2. Add 2nd App with only English languages 3. Add 1st FAQ under 1st App 4. Link 1st FAQ to 2nd App to create 2nd FAQ 5. Check -> translations of 2nd FAQ == 1st FAQ 6. Update English title of FAQ-1 7. Check -> translations of 2nd FAQ == 1st FAQ 8. Delete FAQ-1 9. Check if 1st FAQ is deleted in DB 10. Assert 2nd FAQ should remain as it is in database. Simulation Verification
18. PARTS OF EACH ACTION “Add APP-1 with only English language” 1. Type of Action Add an App 2. Names App-1 3. Arguments English Language 4. Expected Result As per Spec/Modal of the system 5. Actual Result Database
19. EXPRESS ACTIONS AS DATA USING DEFRECORD “Add app APP-1 with English language” (->AddApp :app-1 {:langs-config [:en]})
20. SERIES OF ACTIONS [(->AddApp :app-1 {:langs-config nil}) (->AddFaq :faq-1 :app-1 <faq-config> <faq-translations>) (->AddApp :app-2 {:langs-config nil}) (->LinkFaq :faq-1 :faq-2 :app-2 {}) (->UpdateFaq :faq-2 :app-2 {} <new-translations>}) (->DeleteFaq :faq-1 {})]
21. DSL CONTEXT {:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}}} :faqs {:faq-1 {:faq-id “112”}}}} :latest-simulated-state {….} :log [{:action-taken :add-faq :simulated-state {….} :actual-mongo {….} :actual-es {….}} {…more}]} Environment (:env) - Contains bindings of vars
22. DSL CONTEXT :latest-simulated-state - Current Generated Expected State {:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}}} :faqs {:faq-1 {:faq-id “112”}}}} :latest-simulated-state {….} :log [{:action-taken :add-faq :simulated-state {….} :actual-mongo {….} :actual-es {….}} {…more}]}
23. DSL CONTEXT Result (:log) - Contains Actual And Expected Result Immutable snapshot of action in time {:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}}} :faqs {:faq-1 {:faq-id “112”}}}} :latest-simulated-state {….} :log [{:action-taken :add-faq :simulated-state {….} :actual-mongo {….} :actual-es {….}} {…more}]}
24. HIGH LEVEL OVERVIEW OF DSL run-dsl Initial Context First Action Context After First Action run-dsl Second Action Context After Second Action And so on… Context After First Action
25. RUN-DSL run-dsl {:env {} :latest-simulated-state {} :log []} (->AddApp :app-1 {:langs-config nil}) {:env {:apps {:app-1 {…}}} :log [{log-1}] :latest-simulated-state {}} …..} run-dsl (->AddApp :app-2 {:langs-config nil}) {:env {:apps {:app-1 {…} :app-2 {…}}} :log [{log-1} {log-2}] :latest-simulated-state {}} …..} And so on… {:env {:apps {:app-1 {…}}} :log [{log-1}] :latest-simulated-state {}}
26. What happens inside the run-dsl function?
27. Dispatch On Action Update The Variables in Action to its bindings Call args with relevant function Update the result to the given var in env Update latest simulate state Log Actual DB State and Updated Simulate State resolve-action-vars run-action! update-env update-latest-simulate-state update-log
28. Each Defrecord Action implements the following DSL Protocol functions 1. resolve-action-vars 2. run-action! 3. update-env 4. update-latest-simulate-state 5. update-log
29. Dispatch On Action resolve-action-vars run-action! update-env update-latest-simulate- state update-log (->AddApp :app-1 {:langs-config nil}) (->AddApp :app-1 {:langs-config nil}) (add-app {:langs-config nil}) {:env {:apps {:app-1 {:app-id “970"}}} :latest-simulated-state {} :log []} {:env {:apps {:app-1 {:app-id “970"}}} :latest-simulated-state {} :log []} {:env {:apps {:app-1 {:app-id “970"}}} :latest-simulated-state {} :log []}
30. Dispatch On Action resolve-action-vars run-action! update-env update-latest-simulate- state update-log (->AddApp :app-2 {:langs-config nil}) (->AddApp :app-2 {:langs-config nil}) (add-app {:langs-config nil}) {:env {:apps {:app-1 {:app-id “970”} :app-2 {:app-id “142”}} :latest-simulated-state {} :log []} {:env {:apps {:app-1 {:app-id “970”} :app-2 {:app-id “142”}} :latest-simulated-state {} :log []} {:env {:apps {:app-1 {:app-id “970”} :app-2 {:app-id “142”}} :latest-simulated-state {} :log []}
31. Dispatch On Action resolve-action-vars (->AddFaq :faq-1 :app-1 {…} {:en {…}})) {:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}}}
32. Dispatch On Action resolve-action-vars (->AddFaq :faq-1 :app-1 {…} {:en {…}})) (->AddFaq :faq-1 :app-1 {:app-id “970”} {:en {}}]) {:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}}}
33. Dispatch On Action resolve-action-vars run-action! update-env (->AddFaq :faq-1 :app-1 {…} {:en {…}})) (->AddFaq :faq-1 :app-1 {:app-id “970”} {:en {}}]) (add-faq {:app-id “970”} {:en {…}})
34. Dispatch On Action resolve-action-vars run-action! update-env (->AddFaq :faq-1 :app-1 {…} {:en {…}})) (->AddFaq :faq-1 {:app-id “970” …} {:en {…}}]) (add-faq {:app-id “970”} {:en {…}}) {:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}} :faqs {:faq-1 {:faq-id “112”}}}}
35. Dispatch On Action resolve-action-vars run-action! update-env update-latest-simulate- state (->AddFaq :faq-1 :app-1 {…} {:en {…}})) (->AddFaq :faq-1 {:app-id “970” …} {:en {…}}]) (add-faq {:app-id “970”} {:en {…}}) {:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}} :faqs {:faq-1 {:faq-id “112”}}}} {:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}}} :faqs {:faq-1 {:faq-id “112”}}} :latest-simulated-state {<Expected>}}
36. Dispatch On Action resolve-action-vars run-action! update-env update-latest-simulate- state update-log {:env {:apps {:app-1 {:app-id “970"} :app-2 {:app-id “142”}}} :faqs {:faq-1 {:faq-id “112”}}} :latest-simulated-state {<Expected>} :log [{:action-taken :add-faq :actual-es <Actual> :actual-mongo <Actual> :simulated-state <Expected>}]
37. VERIFICATION: COMPARE EXPECTED VS ACTUAL FOR EACH STEP
38. COMPARE RESULT FOR FIRST ACTION {"112" {:translations {:en {:published? false, :stags [], :body "Temp body", :title “Title 1"}}, :linked-faq-ids #{}}} Actual Data in DB Expected Data from my Generated result {"112" {:translations {:en {:published? false, :stags [], :body "Temp body", :title “Title 1"}}, :linked-faq-ids #{}}} == PASS
39. {"112" {:translations {:en {:published? false, :stags [], :body "Temp body", :title "Title 1"}}, :linked-faq-ids #{}}} Actual Data in DB Expected Data from my Generated result {"112" {:translations {:en {:published? false, :stags [], :body "Temp body", :title "Title 2"}}, :linked-faq-ids #{}}} == FAIL COMPARE RESULT FOR SECOND ACTION
40. DEMO TIME
41. APPROACHES TO VERIFY Actions : Hardcoded Expected Output : Hardcoded Actions : Simulated Expected Output : Hardcoded Actions : Simulated Expected Output : Generated Actions : Generated Expected Output : Generated
42. Hardcoded Actions, Hardcoded Expected Output ADVANTAGES DISADVANTAGES Requires no extra knowledge to understand Very cumbersome to enumerate Anyone can add/edit tests Modification is hard False negatives are not possible
43. Simulate list of Actions, Hardcoded Final Expected Output ADVANTAGES DISADVANTAGES Easy to enumerate Requires knowledge of the DSL Very readable Maintenance overhead of DSL Shareable with dev to simulate bugs Does not check intermediate state False negatives are not possible Expected output may have data which is available only at runtime like faq-ids
44. Simulate list of actions, Generate Expected Output ADVANTAGES DISADVANTAGES Easy to enumerate Requires knowledge of the DSL Very readable Maintenance overhead of DSL Shareable with dev to simulate bugs Maintenance overhead of Expected Modal Checks intermediate state False negatives are possible Runtime data is available like faq-ids
45. Generate Actions, Generate Expected Output ADVANTAGES DISADVANTAGES Possible to generate large number of tests Requires knowledge of the DSL Possible to Shrink failed test case using test.check library Maintenance overhead of DSL Shareable with dev to simulate bugs Maintenance overhead of Expected Modal Checks intermediate state False negatives are possible Maintenance overhead of generative code for list of actions.
46. Generate Actions, Generate Expected Output Feature - Issue Audit Trail
47. Issue Audit Trail • Maintains a log of • Who Took an action • What Action they took
48. Who - Types of Users App User Support User Agents Admins Helpshift
49. What - Types of Actions App User • Create Issue • Reply Issue • etc… Admin User • Create Issue • Reply Issue • Resolve Issue • Edit Tags • etc… Agent User • Create Issue • Reply Issue • Resolve Issue • Edit Tags • etc…
50. App User Admin “Mayank” Agent “Agent-2” Example Issue
51. Example Issue Issue Audit Logs
52. DEMO - ISSUE AUDIT TRAIL Generate Actions, Generate Expected Output
53. (let [r (simulate-actions [[:user :create-issue {:domain “mayanklocal", :message “0” :hs-tags [“ux" “bug ""game"] :platform-type :ios}] [:agent :change-status {:status :resolved}] [:admin :edit-tags {:tags ["paid" "crash" "bug"]}] [:user :reply {:message-type :confirmation-rejected}] [:user :reply {:message “1”}] [:agent :change-status {:status :resolved}] [:admin :edit-tags {:tags ["paid" "bug" "free"]}] [:user :reply {:message “2”}] [:user :reply {:message “3”}] [:agent :edit-tags {:tags ["paid" "crash"]}]])] (run-trails-tests* r) => true (run-messages-tests domain r) => true) Original Input Data for Edge Case
54. (fact "Simpler Edge case failure" (let [response (simulate-actions [[:user :create-issue {:domain "mayanklocal" :platform-type :ios :message "0"}] [:agent :change-status {:status :resolved}] [:user :reply {:message "1"}] [:user :reply {:message "2"}] [:user :reply {:message "3"}] [:user :reply {:message "4"}] [:user :reply {:message "5"}] [:user :reply {:message "6"}] [:user :reply {:message "7"}] [:user :reply {:message "8"}] [:user :reply {:message "9"}] [:user :reply {:message "10"}] [:user :reply {:message "11"}] [:user :reply {:message "12"}]])] Shrink Input Data for Edge Case
55. Bug • Pagination for older messages happens after 15 messages • This bug happens when two conditions are met • More than 15 messages to load • Certain audit trail appears at the border of pagination.
56. ADVANTAGES OF WRITING DSL Discovering very hard to find bugs, for example we found: Duplicate issue messages being rendered only if the number of messages were "just right". Finding ordering bugs. Increase in developer/tester productivity
57. DISADVANTAGES OF WRITING DSL Cost of maintaining DSLs is high. If your DSL is just data being evaluated at run time, changes in feature code will not throw any compile time errors like function parameters being changed in DSL code. You have to educate your team members to learn your specific DSL for that feature to be able to understand the tests.
58. CONCLUSION DSL can be used as a workflow for writing tests. Separating simulation vs verification of tests. A step towards ability to generate tests instead of writing them.
59. Further Resources • Presentation on Domain Specific Languages by Martin Fowler • DSLs in Action by Debasish Ghosh • Growing a DSL with Clojure • Jeanine Adkisson - Design and Prototype a Language In Clojure • Clojure Tutorials on DSLs with Tim Baldridge (Paid) • John Hughes - Testing the Hard Stuff and Staying Sane • Reid Draper - Powerful Testing with test.check • Clojure Made Simple - Rich Hickey
60. Any Questions? @firesofmay facebook.com/firesofmay/ firesofmay@gmail.com