SharePoint with (artificial) intelligence: Using Azure Cognitive Services for automated text recognition and metadata extraction

Even with all the changes and transformations that SharePoint is going through, proper metadata and tagging remains a cornerstone of each and every SharePoint implementation, regardless if in cloud or on premises. But even there, we’ve got some help!

In this session we will see how we can utilise Azure Cognitive Services to recognise texts and pictures in SharePoint, and then how to use that information for the automatic metadata population. We will speak about concepts, prerequisites, and implementation possibilities. We will see how we can implement this using both development techniques and using no-code/low-code platforms such as PowerApps and Flow.

Transcript

2. Using Azure Cognitive Services for Metadata Automation in SharePoint Online ADIS JUGO, CSO SysKit Microsoft Regional Director, Microsoft MVP
3. Adis Jugo Microsoft MVP Office Apps and Services Microsoft MVP Azure Microsoft Regional Director In IT for way too long (first money earned with development back in 1991) Chief Strategy Officer at SysKit Ltd Mastermind behind European Collaboration Summit (www.collabsummit.eu) and European Cloud Summit (www.cloudsummit.eu) Born in Sarajevo, Bosnia, living in Bingen, Germany Blogger, speaker, author. adisjugo.com
5. https://tinyurl.com/syskit
7. AGENDA • Short intro into Azure Cognitive Services • Short intro into SharePoint • Azure Cognitive Services API Explorer • Using Cognitive Services with SharePoint • Microsoft PowerApps • Microsoft Flow • SharePoint Framework (SPFx)
8. AGENDA • Short intro into Azure Cognitive Services • Short intro into SharePoint • Azure Cognitive Services API Explorer • Using Cognitive Services with SharePoint • Microsoft PowerApps • Microsoft Flow • SharePoint Framework (SPFx)
9. WHAT ARE WE TALKING ABOUT HERE? The memex is the name of the hypothetical proto-hypertext system that Vannevar Bush described in his 1945 The Atlantic Monthly article "As We May Think". Bush envisioned the memex as a device in which individuals would compress and store all of their books, records, and communications, "mechanized so that it may be consulted with exceeding speed and flexibility". The memex would provide an "enlarged intimate supplement to one's memory". The concept of the memex influenced the development of early hypertext systems and personal knowledge base software.
10. 2.5 quintillion bytes per day
11. 80% of business relevant information is unstructured
13. TIFF HTML … JPG
14. And it all ends up in SharePoint (hopefully!)
16. Azure Cognitive Services: The AI for poor people ☺
18. Cognitive Services capabilities Infuse your apps, websites, and bots with human-like intelligence
19. Built-in skills Azure Databricks Machine Learning VMs Azure Machine Learning Custom skills Your custom skill goes here!
20. So what do we potentially want to achieve? • Language Detection • Text context understanding • Sentiment detection (ratings) • Keyword Extraction (metadata) • Objects and people extraction (metadata) • Image understanding • Image keyword extraction (metadata) • OCR (search indexing)
21. Object, scene, and activity detection Face recognition and identification Celebrity and landmark recognition Explicit or offensive content moderation Custom image recognition Language detection Text sentiment analysis Key phrase extraction Entity recognition Text translation Contextual language understanding Cognitive Services capabilities Infuse your apps, websites, and bots with human-like intelligence
22. OK, so how do we do it? SharePoint image analysis Limited capability, JSON outputs Power Apps Directly in Modern SPForms Power Automate In processes #SPFx Highest Flexibility #SPFx
23. Capabilities built into SP Online • Any image file is automatically scanned for object recognition • identifying geolocation data • Identifying extractable text images. • tagged in metadata columns on each document library https://www.cloudappie.nl/image-analysis-sharepoint-online/
24. SharePoint image analysis Limited capability, JSON outputs Power Apps Directly in Modern SPForms Flow In processes #SPFx Highest Flexibility #SPFx Azure Cognitive Services
25. DEMO 1 • Cognitive Services registration • Keys • Cognitive Services REST API Explorer
26. DEMO 2 • In-Form operations • Using PowerApps to extract and update metadata
27. DEMO 3 • BING Translation
28. DEMO 4 • Background operations • Using Flow update metadata
29. RECAP • Short intro into Azure Cognitive Services • Short intro into SharePoint • Azure Cognitive Services API Explorer • Using Cognitive Services with SharePoint • Microsoft PowerApps • Microsoft Flow • SharePoint Framework (SPFx) QUESTIONS? Twitter: @adisjugo