EDA for the Win

Slides from my conference talk: Event-Driven Architecture for the win - A successful strategy for a journey to the cloud!

Transcript

1. Event-Driven Architecture for the win! Mirna Alaisami, Novatec Consulting GmbH 1 A successful strategy for a journey to the cloud
2. Whoami @alaisamiM ▪ Senior Technology Consultant @Novatec Consulting GmbH ▪ People Lead - Cloud Platforms & Infrastructure Automation ▪ Trainer | Speaker | Blogger Mirna Alaisami
3. About my employer
4. Modernize the Architecture The story behind the journey
5. Customer Story - Customer is one of the best machine manufacturer worldwide! - Common use case causing problems! - IT architecture is outdated..
6. Common Use Case - Huge amount of machine data are maintained in various systems - Data are filled using a user interface - Data are updated via VPN connection by service technicians in a very time consuming task!
7. Use Case in Detail - - -
8. Propose a Solution A strategic decision
9. Let’s move to cloud using EDA!
10. Why Cloud? Why EDA? - Customer recognized the need to modernize the architecture! - Let’s search for a solution… - “Agile Integration”? Wait that seems to be cool. We need to get support here!
11. “Agile integration is an approach to compartmentalizing connectivity logic to bring your entire infrastructure together through APIs & events.” RedHAt
12. Agile Integration Aspects - Distributed ownership - Fine-grained architecture - Cloud-native infrastructure
13. Agile Integration Benefits - Agility - Scalability - Resilience
14. Choose the Tools An essential discussion
15. Cloud Provider? - Azure - AWS - Google Cloud Platform - …
16. Cloud Provider - Customer already has an Azure Enterprise Agreement - Customer has Windows native systems => Makes sense to use Azure!
17. EDA Platform? - Apache Kafka - Confluent Cloud - Azure Event Hub - …
18. EDA Platform - Apache Kafka (Great features but need to install & operate) - Confluent Cloud (Fully managed but need Enterprise Agreement) - Azure Event Hub (Fully managed & existing Enterprise Agreement) => Tendency towards Azure Event Hub!
19. Event Hubs with Kafka - Start with fewer infrastructure components + - Avoid vendor lock-in when it comes to event handling => Azure Event Hub with Kafka clients
20. Data Storage - Azure Blob Storage - Amazon S3 - Google Cloud Storage - …
21. Data Storage - We need a storage to upload/download the data to/from - Already decided to use Azure - Azure has Storage Accounts with a Blob Storage solution that fits very well with the use case => Let’s use Azure Blob Storage to manage machines data
22. Implement the Solution A detailed explanation
23. Business Logic machine “geo- location(s)” upload download local DC “central location” download local DC “geo-location(s)” delete
24. Azure Event Hub Event Hubs Namespace Event Hub Event Hub Partition 1 Partition 2 Partition n Event Hub Event Hub Partition Key X Partition Key Y f(x)
25. Azure Event Hub Event Hub Partition 1 Partition 2 Partition n Consumer Group 1 Consumer 1 Consumer 2 Producer Key X Key Y
26. Azure Schema Registry Event Hubs Namespace Consumer Producer Event Hub Event Hub Schema Registry serialize deserialize
27. Azure Blob Storage Account Container Blob customer container 1 container 2 file 1 file 2 file 3
28. At a Glance
29. At a Glance On-Premise Azure Event Hub Namespace datadwonloaded datauploaded Blob Storage “central location” “location!= central location” Cleaner Downloader Downloader Uploader Blob+Metadata delete [if all locs downloaded] [if machine loc=my loc] publish receive receive download download publish publish receive
30. EDA on Cloud
31. On-Premise On-Premise Security Components - Azure Key Vault Downloader Uploader Azure Blob Storage Key Vault encrypt decrypt getKey getKey
32. Security Components - Azure Active Directory Azure AD Tenant App (Uploader/Downloader /Cleaner) register-App Application Service Principal Subscription Access Policy & Permissions
33. Security - Authentication Service Principal ClientId TenantId ClientSecret On-Premise CronJob exchange DB “ClientSecret ” set App (Downloader/Cleaner/ Uploader) use
34. Security - Authentication (C#) export AZURE_TENANT_ID export AZURE_CLIENT_ID export AZURE_CLIENT_SECRET DefaultAzureCredential credentialToken = new DefaultAzureCredential();
35. Security - Authentication (Java) export AZURE_TENANT_ID export AZURE_CLIENT_ID export AZURE_CLIENT_SECRET ClientSecretCredential clientSecretCredential = new ClientSecretCredentialBuilder() .clientId(AZURE_CLIENT_ID) .clientSecret(AZURE_CLIENT_SECRET) .tenantId(AZURE_TENANT_ID) .build();
36. Security - Authentication App (Uploader) sign-in Azure AD Tenant Application delegates sign-in
37. Security - Authentication (C#) services.AddAuthentication (OpenIdConnectDefaults .AuthenticationScheme ) .AddMicrosoftIdentityWebApp (MicrosoftIdentityOptions => { MicrosoftIdentityOptions .Instance = "https://login.microsoftonline.com/ "; MicrosoftIdentityOptions .Domain = System.Environment.GetEnvironmentVariable ("DOMAIN"); MicrosoftIdentityOptions .TenantId = System.Environment.GetEnvironmentVariable ("AZURE_TENANT_ID" ); MicrosoftIdentityOptions .ClientId = System.Environment.GetEnvironmentVariable ("AZURE_CLIENT_ID" ); MicrosoftIdentityOptions .CallbackPath = new PathString("/signin-oidc" ); });
38. Security - User Authorization App (Uploader) sign-in Azure AD Tenant Application App Roles delegates sign-in
39. Security - User Authorization (C#) [Authorize(Policy = AuthorizationPolicies .AssignmentToAzureUploadRoleRequired )] public async Task<IActionResult> Blob(IFormFile postedFile) { … } services.AddAuthorization (options => { options.AddPolicy(AuthorizationPolicies .AssignmentToAzureUploadRoleRequired , policy => policy.RequireRole(AppRole.AzureUpload)); });
40. Security - User Authorization (Java) @PreAuthorize("hasAuthority('APPROLE_AzureUpload')") public String uploadData(@RequestParam("location") String location) { … }
41. Security - App Authorization Options OAuth2 SAS Azure AD (RBAC) Storage Account Access Key Azure AD (RBAC) Key Vault Access Policy
42. Security - Kafka App Config (C#) public void InitConsumer(...) { … BootstrapServers = System.Environment.GetEnvironmentVariable("EVENTHUB_NAMESPACE") + ".servicebus.windows.net:9093", SaslMechanism = SaslMechanism.OAuthBearer, SecurityProtocol = SecurityProtocol.SaslSsl, AutoOffsetReset = AutoOffsetReset .Earliest, … }
43. Security - Kafka App Config (Java) bootstrap.servers=${EVENTHUB_NAMESPACE}.servicebus.windows.net:9093 security.protocol=SASL_SSL sasl.mechanism=OAUTHBEARER sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required; sasl.login.callback.handler.class=de.nt.publisher.CustomAuthenticateCallbackHandler
44. Demo
46. Future
47. Next Steps - Automating the setup and management of the infrastructure - Structuring the Blob Storage hierarchy & adjusting the Access Policy accordingly - Implementing event capturing - Utilization monitoring of the various cloud components
48. Thanks for Listening! mirna.alaisami@novatec-gmbh.de @alaisamiM Mirna Alaisami Scan me for code examples!
49. Novatec Consulting GmbH Bertha-Benz-Platz 1 D-70771 Leinfelden-Echterdingen T. +49 711 22040-700 info@novatec-gmbh.de www.novatec-gmbh.de 49 Senior Consultant Mirna Alaisami mirna.alaisami@novatec-gmbh.de