Unleash Some AI into the Wild door Henk Boelman

devNetNoord #13

Transcript

1. @hboelman | www.henkboelman.com
3. How can AI improve the efficiency of medical field workers in remote Ugandan villages?
4. I'm I'm Henk Boelman Henk Boelman
5. @hboelman https://nl.linkedin.com/in/henkboelman https://github.com/hnky https://www.henkboelman.com
6. Like to make software Like to make software Like to make software Like to make software Like to make software Like to make software
7. Volunteer Volunteer for Suubi Reachout
9. TendoCare.AI TendoCare.AI TendoCare.ai is a platform that enables medical workers to be more effective in the field. TendoCare.ai is a platform that enables medical workers to be more effective in the field. TendoCare.AI TendoCare.AI
10. AI first AI first How can AI make this app easier to use
11. Develop on the Mac Develop on the Mac Just something fun...
12. Xamarin Visual Studio App Center .NET core Backend Visual Studio Team Services GitHub
14. Put intelligence APIs to work Put intelligence APIs to work Cognitive Services Cognitive Services Microsoft Azure
15. Speech Knowledge Language Search Labs Vision
16. Patient identification Patient identification Search by typing a name Search by typing a name Kiyemba Buyondo Ngenyi Nyanjura
17. Take a picture Tendocare API Patient File
18. Take a picture Tendocare API Patient File
19. Face Api Face Api faceAttributes": { "smile": 0.003, "gender": "male", "age": 56.3, "glasses": "NoGlasses", } Face detection & description Person identification
20. Demo Demo What we get back from the API
21. Register a face Register a face Create a PersonGroup Create a Person Add a Face to the Train the persongroup PersongroupID PersonName PersongroupName PersonID Image (stream / url) PersongroupName PersongroupName
22. Identify a face Identify a face Detect Identify PersongroupID FaceID Image (stream / url)
23. Take a picture api.tendocare.ai Patient File
24. Detect Face /LoadPatientFileByFace 1. Load PatientFile by personId Identify the face Return PatientFile 1. Register with FaceAPI 2. Create a new PatientFile
25. Demo Demo Tendocare FaceId demo
26. PatientFile by object PatientFile by object
27. An easy-to-use tool for creating your own custom image classifier No background in computer vision or deep learning required. Custom Vision Service Custom Vision Service
28. Model "Sticker A" "Sticker B" Custom Vision Service Creating a Creating a custom image classifier custom image classifier
29. Model "Sticker B" Creating a Creating a custom image classifier custom image classifier
30. Creating a Creating a custom image classifier custom image classifier Export to: API
31. Demo Demo Creating a custom image classifier
32. PatientFile by object PatientFile by object Object detection
33. Computer Vision Api Computer Vision Api "a man flying through the air while riding a skateboard"
34. { "name": "water", "confidence": 0.997288346 }, { "name": "outdoor", "confidence": 0.9940638 }, { "name": "sky", "confidence": 0.9912407 }, { "name": "fishing", "confidence": 0.970712841 }, "a man water skiing behind a boat"
35. { "name": "JOHN TRAVOLTA", "faceRectangle": { "left": 516, "top": 152, "width": 206, "height": 206 }, "confidence": 0.9996151 } { "name": "SAMUEL L. JACKSON", "faceRectangle": { "left": 1107, "top": 172, "width": 203, "height": 203 }, "confidence": 0.988874555 }
36. Computer Vision Api Computer Vision Api Read text in images Read text in images
37. Computer Vision Api Computer Vision Api Read handwritten text Read handwritten text in images in images
38. Demo Demo Detect handwriting in an image
39. PatientFile by object PatientFile by object Object detection Read handwriting
40. Custom Vision + Computer Vision Custom Vision + Computer Vision 1. Train a model with Custom Vision (CoreML) 2. Send the detected image to the Computer Vision API Scan object 90% confidence Read handwriting frame Classify Send to Vision API
41. Demo Demo Xamarin IOS vision prototype
42. Making notes Making notes
43. Bing Speech API Bing Speech API Convert speech to text and back again to understand user intent
44. Speech to text Speech to text Convert speech to text. REST API or WebSocket 3 Modes (Interactive, Conversation, Dictation) 10 Languages
45. Text to speech Text to speech Convert text to speech. REST API call OutputFormat Gender Name Language
46. Record the audio file. Post to api Put it on blob storage Azure function (blob trigger) Call Bing Speech to text API Update the patientfile
47. Demo Demo Using an Azure Function for Speech to text
48. Security Security
50. Speaker Recognition API Speaker Recognition API Use speech to identify and authenticate individual speakers
51. Create a speaker profile Create a speaker profile Load phrases Create verification profile Enrole phrase 3x created Profile "my voice is my passport verify me"
52. Verify speaker Verify speaker Container WAV Encoding PCM Rate 16K Sample Format 16 bit Channels Mono Record the phrase Call the API (VerificationProfileId & Enrollment sentence as bytes) Handle response (result: Accept | confidence: High)
53. Demo Demo Xamarin voice prototype
54. Summary Summary Most of the AI Cognitive services work in happy flow.
55. Future Steps Future Steps Project Prague Images for training model Hololens
56. @hboelman https://nl.linkedin.com/in/henkboelman https://github.com/hnky https://www.henkboelman.com Thank you Thank you tendocare.ai tendocare.ai more information on: