.NET Conf 2024 Taiwan 初探 Drasi：微軟開源的 Data Change Processing platform

在現代化的系統架構中，微服務化（Microservices）已成為趨勢。
然而，隨之而來的一個常見問題就是：資料共享該如何處理？
過去，當所有系統都集中在同一個地方時，直接存取資料是理所當然的選擇。
但在微服務架構中，我們是否還應該這樣做？如果不應該，那有哪些建議的做法？

Drasi 的誕生正是為了解決這些挑戰。
讓我們一起探索 Drasi，了解何為一個資料變更處理平台，以及帶來什麼好處。

Transcript

1. 初探 Drasi 微軟開源的 Data Change Processing Platform
2. Alan Tsai 蔡孟玹 • 後端工程師 - .NET 技術為主 • 喜歡學習不同東西 • Azure • Data Science、Chatbot • Container、DevOps • 加强開發的Tools、架構 • 翻譯文章/軟體 • 兩本翻譯書 • 三門線上課程 • 看小說、玩手游
3. 教就是最好的學習方式 - 喜歡考證驗證所學
4. 喜歡技術分享 2024 11 月 GDG 小聚 探索 GitHub Copilot 的新功能讓我們的應用程式開發更快一步 2024 08 月 TSMC IT Community Meetup 聊聊生成式 AI 衝擊下工程師可以做好什麼準備 2024 06 月 Global DevOps Experience Day 2023 .NET Conf Taiwan 探索 API 開發的挑戰與解決之道 從 GitHub Copilot 到 Enterprise Copilot：打造符合企業需求的智能開發助手之路 2023 10 月 GDG 小聚 開始邁上你的 Azure 成本管控大師之路 2022 .NET Conf Taiwan 談 Event Driven Architecture 之前是不是該把 Event 規格搞定？ CloudEvents 是什麼？ | 邁上 Cloud Native App 之路
5. 喜歡技術分享 Tibame AZ-900、AZ-104、AZ-204 GitHub Copilot 企業導入，在製造業、金融業、服務業，已培訓近 2000+ 人次 Trainocate AZ-104、AZ-204、AZ-400、DP-200、DP-201、PL-900、DA-100 台灣智慧自動化與機器人協會 運用Python進行大數據分析、機器學習基礎理論課程及人工智慧 – ML.NET 中華電信學院 DevOps與CI/CD實務研習班 Windows Container 技術實務班 使用Azure AI打造有人工智能的Line聊天機器人
6. 教就是最好的學習方式 - 喜歡考證驗證所學 FB 粉絲頁 http://fb.alantsai.net http://yt.alantsai.ne t http://bili.alantsai.net contact@alantsai.net contact@alantsai.net Alan Tsai 的學習筆記 https://blog.alantsai.net 影片
7. Agenda What is the problem What is Drasi How to use Drasi 02 01 03
8. What is the problem
9. What is a Program
10. Data + Logic
11. Death Star Architecture
12. It appears powerful but is fundamentally vulnerable.
13. What is Death Star Architecture and why you should care 1.Take the cheaper faster option. 2.Take the cheaper faster option. 3.Goto 1.
14. 改 A 壞 B => 有程式碼品質問題 Gate Keepers 變成英雄 為了加速開發，我們導入 “敏捷” 專注在 Process, Documentation, Planning 整個糾結在一起，value stream 散落在各個解決方案 開始有很多分支 (但是 DB 是同一座…) 商業需求不會因為開發變慢而停下 各個部門解決自己的問題，而沒有思考整體公司。沒 有一致步調前進
15. We need to share Data not Data Source
16. Three way of communication Logic and Data Synchronous API Asynchronous Event Data CDC 02 01 03
17. Organizations often need to take some action when data stored in a database changes, or, conversely, when it doesn’t change as expected
18. https://blog.bytebytego.com/p/data-sharing-between-microservices
19. What is the challenge? Change Detection Challenges Real-Time Response Issues Developer and Business Impact
20. instantaneous query
21. What is the current solution Complex Logic
22. What is Propose to CNCF Sandbox Open Source Project
24. •Azure Cosmos DB Gremlin API •PostgreSQL •Event Hubs •Microsoft SQL Server •Microsoft Dataverse •Kubernetes Source Connection of data
25. Continuous Queries Cypher query •Real-Time Data Updates: •Continuously incorporate changes from the source database. •Maintain accurate, up-to-date query results. •Flexible Query Capabilities: •Use Cypher Query Language for detailed change detection. •Rich query logic for properties and relationships. •Multi-Source Integration: •Queries span multiple data sources without complex joins. •Integrates relational and graph data seamlessly.
26. Cypher query: Node and Relation Incident Alert Detection
27. Incident Alert Detection MATCH (e:Employee)-[:ASSIGNED_TO]->(t:Team), (m:Employee)-[:MANAGES]->(t:Team), (e:Employee)-[:LOCATED_IN]->(:Building)-[:LOCATED_IN]->(r:Region), (i:Incident {type:'environmental'})-[:OCCURS_IN]->(r:Region) WHERE elementId(e) <> elementId(m) AND i.severity IN [‘critical’, ‘extreme’] AND i.endTimeMs IS NULL RETURN m.name AS ManagerName, m.email AS ManagerEmail, e.name AS EmployeeName, e.email AS EmployeeEmail, r.name AS RegionName, elementId(i) AS IncidentId, i.severity AS IncavidentSeverity, i.description AS IncidentDescription Employees located in Buildings within Regions where there are active Incidents of type ’environmental’ that have a severity level of ‘critical’ or ‘extreme’ The name and email address of the at risk employee and their manage
28. Incident Alert Detection MATCH (e:Employee)-[:ASSIGNED_TO]->(t:Team), (m:Employee)-[:MANAGES]->(t:Team), (e:Employee)-[:LOCATED_IN]->(:Building)-[:LOCATED_IN]->(r:Region), (i:Incident {type:'environmental'})-[:OCCURS_IN]->(r:Region) WHERE elementId(e) <> elementId(m) AND i.severity IN [‘critical’, ‘extreme’] AND i.endTimeMs IS NULL RETURN m.name AS ManagerName, m.email AS ManagerEmail, e.name AS EmployeeName, e.email AS EmployeeEmail, r.name AS RegionName, elementId(i) AS IncidentId, i.severity AS IncavidentSeverity, i.description AS IncidentDescription { “added”: [ { “ManagerName”: “Allen”, “ManagerEmail”: “allen@contos “Southern California”, “IncidentId”: “in1000”, “IncidentSeverit { “ManagerName”: “Allen”, “ManagerEmail”: “allen@contos “Southern California”, “IncidentId”: “in1000”, “IncidentSeverit ], “updated”: [], “deleted”: [] }
29. Incident Alert Detection { “added”: [], “updated”: [], “deleted”: [ { “ManagerName”: “Allen”, “ManagerEmail”: “allen@contoso.com”, “EmployeeName”: “Bob…” } ] } If Bob Change Location If severity change { “added”: [], “updated”: [ { “before”: { “Manager “after”: { “ManagerName” “deleted”: [] }
30. Continuous Query Abstract
31. Reaction
32. How to use Propose to CNCF Sandbox Open Source Project
33. Dependency 1. Kubernetes 2. Dapr Monitor 1. OpenTelemetry 2. Prometheus drasi cli
34. Install on AKS 1. Connect to AKS 2. Download drasi cli 3. Call drasi init az aks get-credentials --resource-group <group> --name <name> curl -fsSL https://raw.githubusercontent.com/drasi- project/drasi- platform/main/cli/installers/install-drasi- cli.sh | /bin/bash https://drasi.io/how-to-guides/installation/install-on-
35. Solution Architecture 1.Which people have sent the message “Hello World”? 2.How many times has the same message been sent? 3.Which people haven’t sent a message in the last 20 seconds?
36. Prepare the data - PostgreSQL
37. Deploy PostrgreSQL kubectl apply -f drasi-postgres.yaml kubectl port-forward svc/postgres 5432:5432​
38. Test PostgreSQL psql -h localhost -U test -d hello-world
39. Source
40. Query
41. Reaction
42. Debug kubectl port-forward services/hello- world-debug-gateway 8080:8080 -n drasi-system
43. Insert Hello World INSERT INTO public."Message" VALUES (5, 'Allen', 'Hello World’); INSERT INTO public."Message" VALUES (5, 'Allen', 'Hello World'); Which people have sent the message “Hello World”? How many times has the same message been sent?
44. 20 seconds inactive
45. Conclustion
46. connected fleet
47. Observing Changes, Entity CRUD Observing Conditions, an Incident becomes critical. an Order becomes ready for pickup. a Room’s temperature exceeds 80 degrees. the occupancy of a Store exceeds 100 people. Observing Collections There are new customer orders. These need to be picked from stock. Once picked, orders need to be packed and prepared for dispatch. Once prepared, the orders need to be dispatched through various delivery options. Once dispatched, the orders need to be tracked until delivery. Once the Order is delivered, it is complete, unless the customer has an issue, in which case a customer support process is initiated… Solution Design How to Design Solutions with Drasi Solution for Down Stream Application
48. What is it not good for has a mature change notification capability Continuous Query includes data types for which there is a lot of data stream analytics or streaming data transformation over high volume data streams 02 01 03
49. 1. Drasi: Microsoft’s newest open-source project simplifies change detection and reaction in complex systems 2. Detect and react intelligently to changes in data with Drasi 3. Drasi.io 4. Learning Reference
51. 特別感謝 以及各位參與活動的各位