React and MediaPipe: Powering web application with AI and Computer Vision

Talk that I gave at React Global Online Summit 2021 - Junior Track.

Live image recognition is one of the most researched topics in artificial intelligence these days. Google's MediaPipe allows to easily integrate cutting edge machine learning models for live inference on video and live media. Good news is: we can use this power in React too!
By integrating MediaPipe frameworks, it's possible to extend the power of our React applications, bringing groundbreaking possibilities, such as: gesture based interactions, selfie filters, live video effects, face detection, object tracking and human pose estimation. Get to know in this talk, how you can bring your's web application experience to the next level, by using the power of artificial intelligence and computer vision with React!

Transcript

1. React and MediaPipe Powering Web Applications with AI and Computer Vision This Photo by Unknown author is licensed under CC BY-SA-NC.
2. whoami Mateus Bueno https://www.linkedin.com/in/mateusmb/ https://twitter.com/themagusdev https://github.com/mateusmb
4. Overview AI and Computer Vision Machine Learning and React Google's MediaPipe Using MediaPipe Frameworks in React Use casesand some nice ideas
5. AI and Computer Vision
8. A bunch of data Training
9. It's a box New data
10. Inference
11. It's a dog New data
12. No, but try again Testing
13. Computer Vision
15. Machine Learning and React
18. • View training process live on application • Use pre-trained models straight at the client • Make inferences with little latency
19. • View training process live on application • Use pre-trained models straight at the client • Make inferences with little latency
20. Google's MediaPipe
21. ML solutions for live and streaming media
22. Hands Pose Face
23. Supported Platforms This Photo by Unknown author is licensed under CC BY-SA. This Photo by Unknown author is licensed under CC BY-NC.
24. mediapipe.dev
26. Using MediaPipe Frameworks with React • Install required npm packages • Import them on your app/component • Get a reference to the webcam and also a canvas • Setup the framework on app/component mount • Setup a callback function for detection results
29. Get the Webcam and Canvas refs
30. Setup the framework on startup
31. Setup the framework on startup
32. Setup the framework on startup
33. Setup the framework on startup
34. https://google.github.io/mediapipe/solutions/hands.html
35. Use cases and some nice ideas
38. Thank you