Historical Details: a case study

Think about your phone number while growing up. You can still remember it even though it isn't your phone number anymore. Historical Details retrieves information about a domain object as it was at a given time. Historical Details was built to remember information like a phone number and then recall it by time like "while growing up".

The presentation is divided into three sections:

A high level discussion about arriving to Historical Details as a solution
A code-level walkthrough of the Historical Details
A reflective evaluation of challenges and possibilities of Historial Details

Transcript

1. Historical Details a case study on tracking how information changes over time Nate Davis Olds Bene ts Data Trust, http://www.bdtrust.org https://github.com/natedavisolds/historical-details-talk
2. INTRO TALKING POINTS A high level discussion about arriving to Historical Details as a solution
3. INTRO TALKING POINTS A code-level walkthrough of the Historical Details
4. INTRO TALKING POINTS A re ective evaluation of challenges and possibilities of Historical Details
5. DISCOVERY BRIEF OVERVIEW OF BDT BDT nds and assists people in applying for government bene ts
6. DISCOVERY BRIEF OVERVIEW OF PRISM -Guides phone conversations to collect relevant information -Determines potential eligibility -Calculates estimation amount -Tracks progress of applications from outreach to submission
7. DISCOVERY THE PROBLEM What was the information of the household at the time the application was submitted?
8. DISCOVERY THE PROBLEM Multiple applications for different bene ts (SNAP, PACE, LIHEAP)
9. DISCOVERY THE PROBLEM Multiple applications for the same bene ts spread over time (Someone applies for SNAP each for the last 3 years)
10. DISCOVERY THE PROBLEM Information can change at any time Reported gas bill amount might change when we get the actual bill
11. DISCOVERY FIND A SOLUTION Keep all the information for every bene t application
12. DISCOVERY FINDING A SOLUTION SNAP 2012 (215) 444-4444 SNAP 2013 (215) 444-4445 phone_number PACE 2012 (215) 444-4444 LIHEAP 2012 (215) 444-4444
13. SECTION FINDING A SOLUTION Information in one place and reference when needed
14. DISCOVERY FINDING A SOLUTION SNAP 2012 (215) 444-4445 SNAP 2013 phone_number PACE 2012 LIHEAP 2012 Household
15. DISCOVERY FINDING A SOLUTION SNAP 2012 SNAP 2013 phone_number PACE 2012 LIHEAP 2012 By Period of Time (215) 444-4444 2012 Household (215) 444-4445 2013
16. DISCOVERY FINDING A SOLUTION Store everything; construct a household from the information a given time
17. DISCOVERY FINDING A SOLUTION SNAP 2012 (215) 444-4447 1-14-2012 1:00 SNAP 2013 phone_number PACE 2012 LIHEAP 2012 Household (215) 444-4445 1-4-2013 10:23 (215) 444-4444 1-1-2012 3:45 (215) 444-4444 1-1-2012 1:01 1-1-2012 For 1-5-2013 1-15-2012
18. DISCOVERY FINDING A SOLUTION by Rich Hickey “Database as a value” http://www.youtube.com/watch?v=EKdV1IgAaFc
19. DISCOVERY FINDING A SOLUTION SNAP 2012 Interview Store Estimator LetterBuilder Household (215) 444-4445 1-4-2013 10:23 (215) 444-4444 1-1-2012 3:45 $40 PDF HTML PAGE
20. THE PLAN WHY THIS SOLUTION FEELS RIGHT? - ts my vision of passing data in isolated components -no information is lost -no duplicate information -transactional storage leaves the most room reporting
21. DISCOVERY THE DOMAIN MODEL HOUSEHOLDS PEOPLE INCOMES FINANCIALS EXPENSES VEHICLES EMPLOYERS ADDITIONAL_ OWNERS APPLICANTS PORTFOLIOS BENEFIT_ APPLICATIONS PRESENTER
22. THE PLAN GENERALIZING THE DOMAIN HOUSEHOLDS PEOPLE INCOMES FINANCIALS EXPENSES VEHICLES EMPLOYERS ADDITIONAL_ OWNERS APPLICANTS PORTFOLIOS BENEFIT_ APPLICATIONS PRESENTERS Facts
23. THE PLAN FACT -it holds value -it identi es as a key (i.e. fact_key) -it knows at what time the change happened_at -it converts the value to the proper type: values and collections -conceptual “thing” being passed around
24. THE PLAN GENERALIZING THE DOMAIN HOUSEHOLDS PEOPLE INCOMES FINANCIALS EXPENSES VEHICLES EMPLOYERS ADDITIONAL_ OWNERS APPLICANTS PORTFOLIOS BENEFIT_ APPLICATIONS PRESENTERS Entities
25. THE PLAN HistoricalEntity -it has an id -it contains facts -it reports the most recent value for a fact -it reports an array of all the unique values for collections -it is included in a domain class
26. THE PLAN GENERALIZING THE DOMAIN HOUSEHOLDS PEOPLE INCOMES FINANCIALS EXPENSES VEHICLES EMPLOYERS ADDITIONAL_ OWNERS APPLICANTS PORTFOLIOS BENEFIT_ APPLICATIONS PRESENTERS DataObject
27. THE PLAN HistoricalDataObject -it has an id -it has many entities -it has a class constructor to retrieve facts -it is included in a domain class
28. THE PLAN GENERALIZING THE DOMAIN HOUSEHOLDS PEOPLE INCOMES FINANCIALS EXPENSES VEHICLES ENTITY DATAOBJECT ENTITY ENTITY ENTITY ENTITY ENTITY
29. DEFINING HistoricalDetail - it persists the “Fact” concept (ActiveRecord) - it retrieve facts before a time - it has a fact_key column - it has a value column - it has a happened_at column - it has an entity column - it has an entity_id column - it has a household_id column
30. DEFINING Transactor -it writes to HistoricalDetails (inserts only) -it handles the logic of remembering a fact -it handles the logic of forgetting a fact -it knows which entity it is writing for
31. THE PLAN INFORMATION FLOW Request for information retrieve historical facts from store starting from time group facts by entity group facts by entity_id sort facts by fact_key and happened_at convert value to proper type DATAOBJECT HISTORICAL DETAILS ENTITY FACTS id, time FACTS FACTS VALUE
32. THE PLAN INFORMATION FLOW: Example Household 2 Person 4’s phone number as of January 4, 2012 DATAOBJECT nd 2, at: DateTime.new(2012,1,4) Request for information fact_key value happend_at entity entity_id household_id phone_numbe r 2155678543 2013-2-18 Person 4 2 rst-name Fred 2011-12-2 Person 4 2 phone_numbe r 2151234567 2011-12-2 Person 4 2 phone_numbe r 2151234564 2011-12-2 Person 5 2 zip_code 19119 2011-12-2 Household 2 2 phone_numbe r 2151234566 2011-11-2 Person 4 2 phone_numbe r 2151234569 2011-11-2 Person 2 3
33. THE PLAN INFORMATION FLOW: Example All facts before January 4, 2012 for household_id 2 HISTORICAL DETAILS retrieve historical facts from store starting from a time fact_key value happend_at entity entity_id household_id rst-name Fred 2011-12-2 Person 4 2 phone_numbe r 2151234567 2011-12-2 Person 4 2 phone_numbe r 2151234564 2011-12-2 Person 5 2 zip_code 19119 2011-12-2 Household 2 2 phone_numbe r 2151234566 2011-11-2 Person 4 2
34. THE PLAN INFORMATION FLOW: Example group facts by entity Group facts by entity DATAOBJECT Build each entity with grouped facts fact_key value happend_at entity entity_id household_id rst-name Fred 2011-12-2 Person 4 2 phone_numbe r 2151234567 2011-12-2 Person 4 2 phone_numbe r 2151234564 2011-12-2 Person 5 2 phone_numbe r 2151234566 2011-11-2 Person 4 2 fact_key value happend_at entity entity_id household_id zip_code 19119 2011-12-2 Household 2 2
35. THE PLAN INFORMATION FLOW: Example build an array of entities from entity name group facts by entity_id fact_key value happend_at entity entity_id household_id rst-name Fred 2011-12-2 Person 4 2 phone_numbe r 2151234567 2011-12-2 Person 4 2 phone_numbe r 2151234566 2011-11-2 Person 4 2 fact_key value happend_at entity entity_id household_id phone_numbe r 2151234564 2011-12-2 Person 5 2 ENTITY: Person 5 ENTITY: Person 4 DATAOBJECT
36. THE PLAN INFORMATION FLOW: Example Find person 4 DATAOBJECT nd entity fact_key value happend_at entity entity_id household_id rst-name Fred 2011-12-2 Person 4 2 phone_numbe r 2151234567 2011-12-2 Person 4 2 phone_numbe r 2151234566 2011-11-2 Person 4 2 ENTITY: Person 4
37. THE PLAN INFORMATION FLOW: Example fact_key value happend_at entity entity_id household_id rst-name Fred 2011-12-2 Person 4 2 ENTITY: Person 4 sort facts by fact_key and happened_at fact_key value happend_at entity entity_id household_id phone_numbe r 2151234567 2011-12-2 Person 4 2 phone_numbe r 2151234566 2011-11-2 Person 4 2
38. THE PLAN INFORMATION FLOW: Example convert rst value to proper type fact_key value happend_at entity entity_id household_id phone_numbe r 2151234567 2011-12-2 Person 4 2 phone_numbe r 2151234566 2011-11-2 Person 4 2 Get phone_number facts Grab the rst one Return value
39. THE PLAN INFORMATION FLOW: Example convert rst value to proper type Get all facts Group by value Return array of the most recent value fact_key value happend_at entity entity_id household_id household_age_ranges 20-30 2011-12-2 Person 4 2 household_age_ranges 65 or older 2011-11-2 Person 4 2 [“20-30”, “65 or older”]
40. THE PLAN FORGETTING fact_key value happend_at entity entity_id household_id forget household_age_range s nil 2011-12-2 Person 4 2 TRUE household_age_range s 65 or older 2011-12-1 Person 4 2 FALSE household_age_range s 20-30 2011-12-1 Person 4 2 FALSE Forgetting all collection values Get all facts Group by value ignoring everything after forget is TRUE and value.nil? []
41. THE PLAN INFORMATION FLOW: Example Forgetting entities Group facts by entity DATAOBJECT Build each entity with grouped facts fact_key value happend_at entity entity_id household_id Forget first-name Fred 2011-12-2 Person 4 2 FALSE phone_numbe r 2151234567 2011-12-2 Person 4 2 FALSE phone_numbe r 2151234564 2011-12-2 Person 5 2 FALSE phone_numbe r 2151234566 2011-11-2 Person 4 2 FALSE 2011-12-3 Person 4 2 TRUE Filter out entities where forget is TRUE and fact_key and value are empty
42. TOUR
43. REFLECTION DIFFICULTIES -Immutability of data is different than how ActiveRecord handles data changes -Search is hard -Reporting directly from database requires logic -Large data table
44. REFLECTION STRENGTHS -Adding facts are easy -Nothing lost -Fast Tests -Separating concerns of storage and domain logic
45. REFLECTION FUTURE -Develop a class named Facts which would hold the functionality -Refactor Entity and DataObject to have only the single responsibility of de ning methods -Add “Who” to the fact so we can know a user that made the change -Archiving facts
46. REFLECTION FUTURE -I don’t like the name HistoricalDetails. Factus? -Namespace everything -Turn it into a gem -Documentation
47. Thank you.
48. Nate Davis Olds Bene ts Data Trust, http://www.bdtrust.org https://github.com/natedavisolds/historical-details-talk Questions? @natedavisolds nate@davisolds.com TDD Example: http://www.youtube.com/watch?v=Nu_wtmXVBHo