Unified Log Architecture

An introduction to using unified logs for systems architecture and real-time processing.

Transcript

1. Unified log architecture
3. #chinchillafacts
4. –Alan Perlis, “Epigrams on Programming” “It is better to have 100 functions operate on one data abstraction than 10 functions on 10 data structures.”
5. Hello Food Bar Fetch Cap Fool Arch Fooled
6. Hello Food Bar Fetch Cap Fool Arch Fooled *
7. Hello Food Bar Fetch Cap Fool Arch Fooled * *
8. Hello Food Bar Fetch Cap Fool Arch Fooled * * *
9. F O E O D L E D T C H * * * *
10. “F”: [“Food”, “Fetch”, “Fool”, “Fooled”] “FO”: [“Food”, “Fool”, “Fooled”] “FOO”: [“Food”, “Fool”, “Fooled”] “FOOD”: [“Food”] “FOOL”: [“Fool”, “Fooled”] “FOOLE”: [“Fooled”] “FOOLED”: [“Fooled”] “FE”: [“Fetch”] “FET”: [“Fetch”] “FETC”: [“Fetch”] “FETCH”: [“Fetch”]
11. cat input.txt | sort | uniq | wc -l
12. (the other D&D)
13. Distributed systems n systems linked by a network Communicate through message passing Concurrency of components Independent component failure
14. –Jay Z (father of concurrent computing) “I’ve got problems 99 a system but ain’t concurrent one.”
15. Consensus Who did what when?
17. #chinchillafacts
18. A relational sidebar
19. 0 100 200 * * 0 50 100 * * 100 150 200 * *
20. Write-ahead log (WAL) 01 02 03 04 05 06 07 08 09 10
21. WAL Leader Follower
22. –Alan Perlis, “Structure and Interpretation of Computer Programs” “It is better to have 100 functions operate on one data abstraction than 10 functions on 10 data structures.”
24. The log Append-only event sequence Ordered by time
25. Advantages Resiliency Messaging guarantees Unified Single source of truth
27. amazon kinesis
28. Terminology Event Stream/topic Partition/shard
29. Up and running
30. Architecture with a unified log
31. LMS
32. LMS Monitoring service
33. LMS Monitoring service Messaging Service
34. LMS Reporting Service Monitoring service Messaging Service
35. LMS Reporting Service Attachments Service Monitoring service Messaging Service
37. Log-based architecture Log Producers Consumers
38. Unified Log
39. Unified Log LMS Permissions Service Reporting Service Email Service Attachments Service
40. Example application
41. Real-time processing
43. #chinchillafacts
44. Consumer patterns Single event Multiple event
46. Storm topology Spout
47. Storm topology Spout Bolt
48. Storm topology Spout Bolt Bolt Bolt
49. Storm topology Spout Bolt Bolt Bolt Bolt
50. Advantages Supports Kafka, Kinesis as spouts Distributed, high performance system Single abstraction for all workers Multi-language support
51. Example application
52. Example application Trending courses or topics Real-time alerting and notifications Fraud detection
53. Word counter
54. [“Be not afeard; the isle is full of noises, sounds and sweet airs, that give delight and hurt not.”] Spout
55. [“Be”, “not”, “afeard”, “the”, “isle”, “is”, “full”, “of”, “noises”, “sounds”, “and”, “sweet”, “airs”, “that”, “give”, “delight”, “and”, “hurt”, “not”] Spout Bolt
56. [“afeard”, “isle”, “full”, “noises”, “sounds”, “sweet”, “airs”, “give”, “delight”, “hurt”] Spout Bolt Bolt
57. {“afeard”: 1, “isle”: 1, “full”: 1, “noises”: 1, “sounds”: 1, “sweet”: 1, “airs”: 1, “give”: 1, “delight”: 1, “hurt”: 1} Spout Bolt Bolt Bolt
58. {“afeard”: 13, “isle”: 2, “full”: 100, “noises”: 25, “sounds”: 13, “sweet”: 17, “airs”: 31, “give”: 310, “delight”: 72, “hurt”: 5} Spout Bolt Bolt Bolt Database Tick Tuple
59. {“afeard”: 13, “isle”: 2, “full”: 100, “noises”: 25, “sounds”: 13, “sweet”: 17, “airs”: 31, “give”: 310, “delight”: 72, “hurt”: 5} Spout Bolt Bolt Bolt Database Tick Tuple Bolt Bolt Bolt Bolt
60. Questions?