Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Microsoft Azure <3 Open Source Tech

Microsoft Azure <3 Open Source Tech

> Slides of my session at the Global Azure Bootcamp 2019 in Linz, Austria

https://coding-club-linz.github.io/global-azure-bootcamp-2019/

Over the last few years, Microsoft has time and again proven the high priority given to open source technologies, especially in different Azure Cloud Computing Services.

A brief introduction will show the key aspects of Azure Event Hubs and Azure Cosmos DB. Apart from the standard operation, alternative APIs are the focus of this session, which should allow for these two Azure services a seamless integration or a "drop-in" replacement to very popular open source projects.

While Azure Event Hubs offers an Apache Kafka-compliant API, Azure CosmosDB can also be used in a MongoDB-compatible variant. Based on simple use cases and practical live demos, these possibilities are discussed.

> Folien meiner Session am Global Azure Bootcamp 2019 in Linz, Österreich

https://coding-club-linz.github.io/global-azure-bootcamp-2019/

Über die letzten Jahre hat Microsoft stets aufs Neue bewiesen, welchen hohen Stellenwert Open Source Technologien insbesondere auch in unterschiedlichen Azure Cloud Computing Services eingeräumt bekommen.

Im Rahmen einer kurzen Einführung werden zunächst die wichtigsten Aspekte von Azure Event Hubs sowie Azure CosmosDB vorgestellt. Abseits vom Standard Betrieb liegen alternative APIs im Fokus dieser Session, welche für diese beiden Azure Services eine nahtlose Integration bzw. einen “Drop-In” Ersatz zu sehr populären Open Source Projekten ermöglichen sollen.

Während Azure Event Hubs eine zu Apache Kafka kompatible API offeriert, kann Azure CosmosDB auch in einer MongoDB-kompatiblen Variante genutzt werden. Anhand einfacher Use-Cases und praktischer Live-Demos werden diese Möglichkeiten diskutiert.

744f1c2c6cbea2ff5104b0ac512936bd?s=128

Hans-Peter Grahsl

April 27, 2019
Tweet

Transcript

  1. Microsoft Azure ❤ Open Source Tech

  2. Microsoft + Open Source ? @hpgrahsl | #Azure @GlobalAzure, 27th

    April 2019, Linz - Austria 2
  3. GitHub Octoverse 2018 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019,

    Linz - Austria 3
  4. GitHub Octoverse 2018 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019,

    Linz - Austria 4
  5. GitHub Octoverse 2018 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019,

    Linz - Austria 5
  6. GitHub Octoverse 2018 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019,

    Linz - Austria 6
  7. HDInsight Services @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz

    - Austria 7
  8. HDInsight Services @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz

    - Austria 8
  9. $ whoami • Hans-Peter Grahsl • living & working in

    Graz • technical trainer at • independent engineer & consultant • associate lecturer • " irregular conference speaker @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 9
  10. Application Needs ? • "It depends" ! • plenty of

    pieces & components • irrespective of concrete use cases • two architectural pieces... @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 10
  11. operational data store @hpgrahsl | #Azure @GlobalAzure, 27th April 2019,

    Linz - Austria 11
  12. Cosmos DB @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz

    - Austria 12
  13. messaging platform @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz

    - Austria 13
  14. Event Hubs @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz

    - Austria 14
  15. @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

    15
  16. Cosmos DB • global distribution • high availability & elastic

    scaling • multi-model & native NoSQL APIs • consistency choices • leading security, compliance & SLAs @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 16
  17. Multi-Model • keys + values • documents • column families

    • graphs @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 17
  18. Native API Support • SQL • Table Storage • MongoDB

    • Cassandra • Gremlin @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 18
  19. Throughput Provisioning • request units (RUs) • abstract performance metric

    • combination of CPU + Memory + IOPS • backed by SLAs @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 19
  20. exciting ...in theory

  21. Azure Docs on RUs "The cost of all database operations

    is normalized by Azure Cosmos DB and is expressed in terms of Request Units (RUs). The cost to read a 1-KB item is 1 Request Unit (1 RU) and minimum RUs required to consume 1 GB of storage is 40." @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 21
  22. @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

    22
  23. daunting ...in practice

  24. Measure via API • write operation consumed ??? RUs •

    given 1000 provisioned RUs at most ☛ 1000 ÷ 20.19 ≅ 49 docs/sec • no parallel activity considered @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 24
  25. Measure via API • read operation consumed ??? RUs •

    given 1000 provisioned RUs at most ☛ 1000 ÷ 3.11 ≅ 321 reads/sec • no parallel activity considered @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 25
  26. Influences on RUs • indexing • property count • consistency

    level • query patterns • scripts ...and any other concurrent operation @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 26
  27. Cosmos DB for MongoDB API • native implementation • wire

    protocol compatibility • transparent re-use (code + tools) • migration benefits: in both ways @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 27
  28. Showtime! @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz -

    Austria 28
  29. Important Note • feature support is limited of course •

    carefully check what does (NOT) work • wire protocol versions: 3.2 GA, 3.4 preview @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 29
  30. None
  31. 3.6 ?? 4.0 @hpgrahsl | #Azure @GlobalAzure, 27th April 2019,

    Linz - Austria 31
  32. None
  33. Currently Missing Out On... • powerful operators from 3.2 +

    3.4 - $graphLookup, $facet, $bucket(Auto) - $reduce, $zip, $switch, $replaceRoot, ... • great features & major advances from 3.6 + 4.0 - views, change streams, - type conversions, schema validation, - multi-element array updates, array filters, - sessions & transactions, ... @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 33
  34. @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria

    34
  35. Event Hubs • distributed event ingestion platform • decouple producers

    ⬌ consumers • source for stream processing • high availability & scalability • fully-managed @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 35
  36. Concepts: Event Hub @hpgrahsl | #Azure @GlobalAzure, 27th April 2019,

    Linz - Austria 36
  37. Concepts: Partition @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz

    - Austria 37
  38. Concepts: Partition Keys @hpgrahsl | #Azure @GlobalAzure, 27th April 2019,

    Linz - Austria 38
  39. Concepts: Partition Offsets @hpgrahsl | #Azure @GlobalAzure, 27th April 2019,

    Linz - Austria 39
  40. Concepts: Big Picture @hpgrahsl | #Azure @GlobalAzure, 27th April 2019,

    Linz - Austria 40
  41. Event Hubs for Apache Kafka • overlay on top of

    Event Hubs • binary compatible with Kafka 1.0+ • transparent re-use (code + tools) • migration benefits: in both ways @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 41
  42. The Promise... "You update the connection string in configurations to

    point to the Kafka endpoint exposed by your event hub instead of pointing to your Kafka cluster. Then, you can start streaming events from your applications that use the Kafka protocol into Event Hubs." @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 42
  43. Showtime! @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz -

    Austria 43
  44. Unsupported Kafka Features • idempotent producers & transactions • message

    compression • size-based retention or log compaction • parition resize for existing topics • HTTP Kafka API support • Kafka Streams & KSQL @hpgrahsl | #Azure @GlobalAzure, 27th April 2019, Linz - Austria 44
  45. The devil is in the detail

  46. THANK YOU Q & A ? https://bit.ly/2W7KGn1 @hpgrahsl | #Azure

    @GlobalAzure, 27th April 2019, Linz - Austria 46
  47. None