Aggregating data accountably and consistently

Overview of the Event Sourcing pattern and a story of how we used it on a project to aggregate data from bunch of various social networks. All for the sake of preventing conflicts in data and being able to pin down transient bugs by tracking changes in the data.

Transcript

1. Aggrega&ng  data   accountably  and  consistently neektza@gmail.com @neektza #wczg  2014  
2. Accountably?   ! ! Consistently?
3. 3 –  Someone  on  Hacker  News "Every  large  enough  soBware  system   has  an  badly  implemented  subset  of  git   contained  within  itself."
12. Context  of  the  problem 5 Submissions Replies
13. Problems  of  this  system 6 System  state  can  change  seemingly  at  random Source  (Reddit) Our  system
14. Problems  of  this  system 6 System  state  can  change  seemingly  at  random Source  (Reddit) Our  system
15. Problems  of  this  system 6 System  state  can  change  seemingly  at  random Source  (Reddit) Our  system
16. Problems  of  this  system 6 System  state  can  change  seemingly  at  random Source  (Reddit) Our  system
17. Problems  of  this  system 6 System  state  can  change  seemingly  at  random Source  (Reddit) Our  system
18. Problems  of  this  system 6 System  state  can  change  seemingly  at  random Source  (Reddit) Our  system
19. Problems  of  this  system 7 Hard  to  filter  out  repeaOng  (noisy)  data
20. Problems  of  this  system 7 Hard  to  filter  out  repeaOng  (noisy)  data
21. Problems  of  this  system 7 Hard  to  filter  out  repeaOng  (noisy)  data
22. Problems  of  this  system 7 Hard  to  filter  out  repeaOng  (noisy)  data
23. Problems  of  this  system 8 Hard  to  filter  out  repeaOng  (noisy)  data
24. Problems  of  this  system 8 Hard  to  filter  out  repeaOng  (noisy)  data [{…},  {…},  {…},  {…},  {…},  {…},  {…},  {…}]
25. 9 –  Mar1n  Fowler “The  fundamental  idea  of  Event  Sourcing  is  that   of  ensuring  every  change  to  the  state  of  an   applicaOon  is  captured  in  an  event  object,  and   that  these  event  objects  are  themselves  stored   in  the  sequence  they  were  applied  for  the  same   lifeOme  as  the  applicaOon  state  itself.”
28. CharacterisOcs  of  an  Event 12
29. CharacterisOcs  of  an  Event value  objects 12
30. CharacterisOcs  of  an  Event value  objects immutable  /  not  updateable 12
31. CharacterisOcs  of  an  Event value  objects immutable  /  not  updateable bring  new  informaOon 12
32. Sum  of  all  Events  =  ApplicaOon  state
33. CharacterisOcs  of  ApplicaOon  State 14
34. CharacterisOcs  of  ApplicaOon  State also  called  EnOOes 14
35. CharacterisOcs  of  ApplicaOon  State also  called  EnOOes have  relaOonships  to  other  things  in  the  system 14
36. CharacterisOcs  of  ApplicaOon  State also  called  EnOOes have  relaOonships  to  other  things  in  the  system we’re  able  to  update  them 14
37. Events  vs.  EnOOes 15
38. Events  vs.  EnOOes  immutable  vs.  mutable 15
39. Events  vs.  EnOOes  immutable  vs.  mutable  value  vs.  idenOty 15
40. Events  vs.  EnOOes  immutable  vs.  mutable  value  vs.  idenOty  ∆  of  informaOon  vs.  ∫  of  informaOon 15
41. 16 { "number": 1347, "state": "open", "title": "Found a bug", "body": "I'm having a problem …”, "url": “https://api.github…”, "labels": [{ /* label */ }], "user": { /* user object */ }, "assignee": { /* user object */ }, "closed_by": { /* user object */ } } Github  Issues  API
42. 16 { "number": 1347, "state": "open", "title": "Found a bug", "body": "I'm having a problem …”, "url": “https://api.github…”, "labels": [{ /* label */ }], "user": { /* user object */ }, "assignee": { /* user object */ }, "closed_by": { /* user object */ } } ! { "id":"2299367813", "user":"neektza", "repo_id":3228363, "issue_id": :43320689, "action":"opened", } Github  Issues  API
43. 16 { "number": 1347, "state": "open", "title": "Found a bug", "body": "I'm having a problem …”, "url": “https://api.github…”, "labels": [{ /* label */ }], "user": { /* user object */ }, "assignee": { /* user object */ }, "closed_by": { /* user object */ } } ! { "id":"2299367813", "user":"neektza", "repo_id":3228363, "issue_id": :43320689, "action":"opened", } { "id":"2299367814", "user":"neektza", "repo_id":3228363, "issue_id":43320689, "action":"updated", } Github  Issues  API
44. 16 { "id":"2299367815", "user":"neektza", "repo_id":3228363, "issue_id":43320689, "action":"closed", } ! { "id":"2299367813", "user":"neektza", "repo_id":3228363, "issue_id": :43320689, "action":"opened", } { "id":"2299367814", "user":"neektza", "repo_id":3228363, "issue_id":43320689, "action":"updated", } { "number": 1347, "state": "closed", "title": "Found a bug", "body": "I'm having a problem …”, "url": “https://api.github…”, "labels": [{ /* label */ }], "user": { /* user object */ }, "assignee": { /* user object */ }, "closed_by": { /* user object */ } } Github  Issues  API
45. 17 Taming  the  monster
46. 17 Taming  the  monster
47. 17 Taming  the  monster
48. 18 The  structure  of  data
49. 18 Events ID ENTITY_ID MD5 JSON … … … … The  structure  of  data
50. 18 Events ID ENTITY_ID MD5 JSON … … … … Entities ID TYPE TITLE BODY … … … … The  structure  of  data
51. The  fetch 19
52. The  event 20
53. State 21
54. State 21 Events ID ENTITY_ID MD5 JSON 123 ? 845b7853f39adb1… {  ..  }
55. The  enOty 22 22
56. The  state 23 Events ID ENTITY_ID MD5 JSON 123 987 845b7853f39adb1… {  ..  } Entities ID TYPE TITLE BODY 987 Post Cocurency  primiOves  and  abstracOons  in  R …
57. “Cocurrency?” 24
58. The  fetch 25
59. The  event 26
60. The  state 27
61. The  state 27 Events ID ENTITY_ID MD5 JSON 123 987 845b7853f39adb1… {  ...  } 124 ? c9155434978967… {  …  }
62. The  enOty 28
63. Final  state 29 Events ID ENTITY_ID MD5 JSON 123 987 845b7853f39adb1… {  …  } 124 987 c9155434978967… {  …  } Entities ID TYPE TITLE BODY 987 Post Concurency  primiOves  and  abstracOons  in   …
64. Problems?  Solved! 30
65. Problems?  Solved! 1st  problem  solved!  —>  tracking  changes 30
66. Problems?  Solved! 1st  problem  solved!  —>  tracking  changes 2nd  problem  solved!    —>  able  to  filter  out  noise 30
67. Problems?  Solved! 1st  problem  solved!  —>  tracking  changes 2nd  problem  solved!    —>  able  to  filter  out  noise a  natural  soluOon 30
68. Much  power 31
69. Much  power complete  rebuild  of  state 31
70. Much  power complete  rebuild  of  state temporal  query  of  past  states 31
71. Much  power complete  rebuild  of  state temporal  query  of  past  states easy  to  capture  business  intent 31
72. Much  power complete  rebuild  of  state temporal  query  of  past  states easy  to  capture  business  intent goes  nicely  with  FP  philosophy 31
73. There’s  no  silver  bullet 32
74. There’s  no  silver  bullet more  complex  architecture   32
75. There’s  no  silver  bullet more  complex  architecture   very  hard  to  introduce  retroacOvely 32
76. There’s  no  silver  bullet more  complex  architecture   very  hard  to  introduce  retroacOvely applicaOon  state  is  eventually  consistent 32
77. QuesOons? 33 neektza@gmail.com @neektza Further  reading:   1.  MicrosoB  Developer  Network  (MSDN)   2.  LosTechies  community  blog