Microsoft Azure <3 Open Source Tech

> Slides of my session at the Global Azure Bootcamp 2019 in Linz, Austria

https://coding-club-linz.github.io/global-azure-bootcamp-2019/

Over the last few years, Microsoft has time and again proven the high priority given to open source technologies, especially in different Azure Cloud Computing Services.

A brief introduction will show the key aspects of Azure Event Hubs and Azure Cosmos DB. Apart from the standard operation, alternative APIs are the focus of this session, which should allow for these two Azure services a seamless integration or a "drop-in" replacement to very popular open source projects.

While Azure Event Hubs offers an Apache Kafka-compliant API, Azure CosmosDB can also be used in a MongoDB-compatible variant. Based on simple use cases and practical live demos, these possibilities are discussed.

> Folien meiner Session am Global Azure Bootcamp 2019 in Linz, Österreich

https://coding-club-linz.github.io/global-azure-bootcamp-2019/

Über die letzten Jahre hat Microsoft stets aufs Neue bewiesen, welchen hohen Stellenwert Open Source Technologien insbesondere auch in unterschiedlichen Azure Cloud Computing Services eingeräumt bekommen.

Im Rahmen einer kurzen Einführung werden zunächst die wichtigsten Aspekte von Azure Event Hubs sowie Azure CosmosDB vorgestellt. Abseits vom Standard Betrieb liegen alternative APIs im Fokus dieser Session, welche für diese beiden Azure Services eine nahtlose Integration bzw. einen “Drop-In” Ersatz zu sehr populären Open Source Projekten ermöglichen sollen.

Während Azure Event Hubs eine zu Apache Kafka kompatible API offeriert, kann Azure CosmosDB auch in einer MongoDB-kompatiblen Variante genutzt werden. Anhand einfacher Use-Cases und praktischer Live-Demos werden diese Möglichkeiten diskutiert.

Transcript

1. Microsoft Azure ❤ Open Source Tech
2. Microsoft + Open Source ? @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 2
3. GitHub Octoverse 2018 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 3
4. GitHub Octoverse 2018 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 4
5. GitHub Octoverse 2018 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 5
6. GitHub Octoverse 2018 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 6
7. HDInsight Services @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 7
8. HDInsight Services @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 8
9. $ whoami • Hans-Peter Grahsl • living & working in Graz • technical trainer at • independent engineer & consultant • associate lecturer • " irregular conference speaker @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 9
10. Application Needs ? • "It depends" ! • plenty of pieces & components • irrespective of concrete use cases • two architectural pieces... @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 10
11. operational data store @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 11
12. Cosmos DB @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 12
13. messaging platform @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 13
14. Event Hubs @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 14
15. @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 15
16. Cosmos DB • global distribution • high availability & elastic scaling • multi-model & native NoSQL APIs • consistency choices • leading security, compliance & SLAs @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 16
17. Multi-Model • keys + values • documents • column families • graphs @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 17
18. Native API Support • SQL • Table Storage • MongoDB • Cassandra • Gremlin @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 18
19. Throughput Provisioning • request units (RUs) • abstract performance metric • combination of CPU + Memory + IOPS • backed by SLAs @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 19
20. exciting ...in theory
21. Azure Docs on RUs "The cost of all database operations is normalized by Azure Cosmos DB and is expressed in terms of Request Units (RUs). The cost to read a 1-KB item is 1 Request Unit (1 RU) and minimum RUs required to consume 1 GB of storage is 40." @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 21
22. @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 22
23. daunting ...in practice
24. Measure via API • write operation consumed ??? RUs • given 1000 provisioned RUs at most ☛ 1000 ÷ 20.19 ≅ 49 docs/sec • no parallel activity considered @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 24
25. Measure via API • read operation consumed ??? RUs • given 1000 provisioned RUs at most ☛ 1000 ÷ 3.11 ≅ 321 reads/sec • no parallel activity considered @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 25
26. Influences on RUs • indexing • property count • consistency level • query patterns • scripts ...and any other concurrent operation @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 26
27. Cosmos DB for MongoDB API • native implementation • wire protocol compatibility • transparent re-use (code + tools) • migration benefits: in both ways @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 27
28. Showtime! @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 28
29. Important Note • feature support is limited of course • carefully check what does (NOT) work • wire protocol versions: 3.2 GA, 3.4 preview @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 29
31. 3.6 ?? 4.0 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 31
33. Currently Missing Out On... • powerful operators from 3.2 + 3.4 - $graphLookup, $facet, $bucket(Auto) - $reduce, $zip, $switch, $replaceRoot, ... • great features & major advances from 3.6 + 4.0 - views, change streams, - type conversions, schema validation, - multi-element array updates, array filters, - sessions & transactions, ... @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 33
34. @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 34
35. Event Hubs • distributed event ingestion platform • decouple producers ⬌ consumers • source for stream processing • high availability & scalability • fully-managed @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 35
36. Concepts: Event Hub @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 36
37. Concepts: Partition @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 37
38. Concepts: Partition Keys @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 38
39. Concepts: Partition Offsets @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 39
40. Concepts: Big Picture @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 40
41. Event Hubs for Apache Kafka • overlay on top of Event Hubs • binary compatible with Kafka 1.0+ • transparent re-use (code + tools) • migration benefits: in both ways @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 41
42. The Promise... "You update the connection string in configurations to point to the Kafka endpoint exposed by your event hub instead of pointing to your Kafka cluster. Then, you can start streaming events from your applications that use the Kafka protocol into Event Hubs." @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 42
43. Showtime! @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 43
44. Unsupported Kafka Features • idempotent producers & transactions • message compression • size-based retention or log compaction • parition resize for existing topics • HTTP Kafka API support • Kafka Streams & KSQL @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 44
45. The devil is in the detail
46. THANK YOU Q & A ? https://bit.ly/2W7KGn1 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 46