Cosmos DB, Graph and Azure Search, building a compelling cloud solution

CosmosDB is a new PaaS service available in Azure providing support for various models, global scale and availability. The supported Graph model unlocked a unique potential to explicitly relate content and combined with Azure Search you can build a compelling solution. During this session, a real-world scenario, pitfalls, experiences and best practices will be discussed.

Transcript

1. 1 Azure Saturday 2018 Cosmos DB, Graph and Azure Search, building a compelling cloud solution Steef-Jan Wiggers| Codit
2. 2 Azure Saturday 2018 Thank you, sponsors!
3. steefjan.wiggers@codit.eu +31 653 12 29 57 @SteefJan nl.linkedin.com/in/steefjan
4. Not my first visit to Munich!
6. The next race …
7. Who we are Customers Entities 2000 Belgium 2004 France 2013 Portugal 2016 Switzerland 2016 UK 2016 The Netherlands 2017 Malta 180 worldwide
8. Codit & Microsoft
9. DATA & AI What the future will bring
10. What can you expected in this session • Scenario • Cosmos DB • Graph model • Azure Search • Demo
11. Scenario
12. Scenario • Old knowledge base implementation with SOLR struggled with related content • Dependency on hosting - and service partner • High cohesion and tight coupling Knowledge Platform Tax Returns
13. Business: • Increase quality of content • Better user experience • New business models & revenue streams • Independent Technology: • Completely PaaS • Azure (Pay as you go) • No IT management support only DevOps • Independent A new future proof knowledge base
14. High Level Architecture Editing CMS Content Creation Knowledge Platform Integrate Cleanse Match & Merge Connectors Ingestion Standardize Validation Meta data Enrichment Value decay XRef Data quality, workflows & monitoring Content Constitution Content collection Relation Store Index Search Models Content API Knowledge base
15. Solution Building Blocks Content collection Azure Search Document DB Graph Integrate Match & Merge Content API Importer/.NET Web App/.Net Search Index Relations Store
16. Solution Architecture CMS Output FRONT-END Content Related Content
17. • What are the costs (ROI, TCO) • Will it work for content and related content (Quality) • Meet business requirements • New revenue streams, business models Business challenges
18. • Architectural Fit – POA • Microsoft Support • Training • Compare with other search solutions, and graph solutions • Performance • Scale • Complexity Technology challenges
19. Cosmos DB
20. Cosmos DB Column-family Document Graph Turnkey global distribution Elastic scale out of storage & throughput Guaranteed low latency at the 99th percentile Comprehensive SLAs Five well-defined consistency models Table API Key-value MongoDB API A globally distributed, massively scalable, multi-model database service Cassandra API
21. • Turnkey global distribution • Multiple datacenters • Auto replication • 99,99% Availability • All resources are horizontally partitioned and vertically distributed • Replication topology is dynamic based on consistency level and network conditions Global Distribution
22. Multi-model + multi-API • Different models: • Graph • Key-Value • Document DB • No schema or index management • Automatic indexing • API support: • SQL • JavaScript • Gremlin • MongoDB • Azure Table Storage • Cassandra
23. Scale • Pay as go for storage and throughput • Elastic scale across regions • Partitions
24. • Five levels of consistency • Programmatically change at anytime • Can be overridden on a per-request basis • Writing correct distribution applications is hard • Global distribution forces CAP theorem • Intuitive and practical with clear PACELC tradeoffs Consistency
25. • Simultaneously read/write Latency
26. SLA • Fully managed service • 99,99% SLA for latency • Guaranteed throughput, consistency and high availability
27. Request Units • Request Units (RU) is a rate-based currency • Abstracts physical resources for performing requests • 1 RU = 1 read of 1 Kb document • Each request consumes fixed Rus • Provisioned in terms of RU/sec and RU/min • Rate limiting based on provisioned throughput • Can be in- and decreased instantly • Metered hourly
28. Capacity Planner
29. • Azure Search • Apache Spark Connector • Azure Functions Integrations
30. Graph Model
31. • TinkerPop is a developer group creating an open-source stack for graphs (http://tinkerpop.apache.org/) • Graph database and analytics systems Graph
32. Data is your model, model is your data
33. • LinkedIn (Business) • Facebook (Social Media) • Walmart (Recommendation) • Google (Search) • Airbnb (Search) • Cisco (Master Management) Graph implementation examples
34. • Gremlin is a graph traversal language • Vertices, Edges, and Properties Graph API
35. Add Vertices (V)
36. Add Edge (E)
37. Execute Gremlin
38. Demo Graph Explorer
39. Azure Search
40. Azure Search
41. “Search service” ▪ Scope for capacity ▪ Bound to a region ▪ Has keys, indexes, indexers, data sources Provisioning ▪ Azure Portal ▪ Azure resource management API Elastic scale ▪ Capacity can be changed dynamically ▪ Replicas ~ more QPS, HA ▪ Partitions ~ more documents, write throughput Provisioning
42. “Index” ▪ Container for data, think “table” ▪ Has schema, CORS options, search options ▪ Create in portal or during app initialization Typical schema ▪ Fields definition: name, type, key Search specifics ▪ Field attributes – searchable, facetable, etc. ▪ Linguistics and analysis ▪ Suggesters for auto-complete ▪ Scoring profiles for ranking tuning Index
43. Push - using indexing API ▪ POST to /indexes/<name>/docs/index ▪ Up to 1000 actions per batch ▪ Actions can be upload, merge, delete, etc. ▪ WebJobs are great for regular execution Pull - using indexers ▪ Azure SQL DB and Document DB ▪ Change detection, deletion markers ▪ Point it at the data source, define policy, done Index data
44. Search + typical data operations ▪ Simple search options, + - * () “” ▪ Filter, sort, project, page over results ▪ Options work with search and suggest Search from client or server ▪ Use query keys when searching from clients ▪ CORS allows direct calls from browsers Render from search results ▪ Include necessary non-searchable data ▪ E.g. URLs for pictures, keys to main content Search
45. Scoring profiles ▪ Field weights ▪ Scoring functions ▪ magnitude, freshness, distance, tags 3 main patterns ▪ Known data directly available in the index ▪ Personalization using tag boosting ▪ Analytics, compute externally and push to the index Customization
46. Integrate with Azure Search
47. Integrate with Azure Search
48. Demo Search
49. Scenario Revisted
50. Development Content collection Azure Search Document DB Graph Integrate Match & Merge Content API Importer/.NET Web App/.Net Search Index Relations Store
51. Solution Architecture CMS Output FRONT-END Content Related Content
52. Search Content FRONT-END MENU BAR Content Dividend tax SEARCH Related Content Related Content Content B A A
53. Demo API
54. • Cosmos DB Graph fit for related content purpose • Cosmos DB Document fit for content • Cosmos DB + Search good combination • Meets business requirements • Complete Architecture on PaaS • Cool eh! Summary
55. • CosmosDB • CosmosDB Graph Explorer • Azure Search • Azure Search Resources Resources
56. Azure Saturday 2018 We appreciate your feedback! https://form.responster.com/mWJ1VI
57. Azure Saturday 2018 Thank you!