Bust the burglars: Machine Learning with TensorFlow and Apache Kafka

The integration of event streams is becoming increasingly complex in the world of sensors, media streams and the Internet of Things. In this session I will discuss how `Apache Kafka` and `TensorFlow` can be used to process real time video events and send alerts when a potential burglary is detected.

Specifically, I'll discuss how `Kafka Connect` and `Kafka Streams` can be used to read images from a media stream and how this can be integrated with TensorFlow to analyse the resulting event streams.

Although the use case presented is based on home security, the same technology used to analyse event streams can be applied to other use cases, as I will demonstrate during this session.

Transcript

1. @fbascheper #Devoxx #Busted Bust the burglars - ML with TensorFlow and Apache Kafka Erik-Berndt Scheper
2. @fbascheper #Devoxx #Busted Let’s meet • I’m delighted to be here in Ukraine! • Java Architect @ Ordina JTech • Contracted by the Dutch Tax administration • Passion for ‣ Big Data technologies ‣ Machine learning 2
3. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video Kafka Connectors Demo Wrap-up 3 https://pixabay.com/p-683065/
4. @fbascheper #Devoxx #Busted https://nl.wikipedia.org/wiki/Bartje#/media/File:Standbeeld-bartje.jpg Introduction 4
5. @fbascheper #Devoxx #Busted Statistics https://reolink.com/home-burglary-crime-statistics/ 5 Lack of evidence! Why?
6. @fbascheper #Devoxx #Busted https://upload.wikimedia.org/wikipedia/commons/5/56/Haren_Rijksstraatweg_377.JPG Entry points 6 8% 4% 9% 22% 23% 34%
7. @fbascheper #Devoxx #Busted https://www.flickr.com/photos/takomabibelot/2134214940 7
8. @fbascheper #Devoxx #Busted Components 8
9. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video Kafka Connectors Demo Wrap-up 9 https://pixabay.com/p-683065/
10. @fbascheper #Devoxx #Busted Event streams • Input - Images from IP camera(s) - Lock status changes • Output - Burglar Alerts 10
11. @fbascheper #Devoxx #Busted Event processing 11 camera image analyse with TensorFlow burglary? alerts enabled? STATELESS …
12. @fbascheper #Devoxx #Busted Event processing 12 poll lock REST API alerts enabled! all locks alerts disabled (flat)map response to individual locks
13. @fbascheper #Devoxx #Busted Event processing 13 poll lock REST API alerts enabled! all locks alerts disabled (flat)map response to individual locks STATE ?
14. @fbascheper #Devoxx #Busted val builder = new StreamsBuilder(); Kafka Streams application 14
15. @fbascheper #Devoxx #Busted val builder = new StreamsBuilder(); val cameraSourceStream = builder.stream( SOURCE_TOPIC_CAMERA_IMAGES, Consumed.with(Serdes.String(), Serdes.ByteArray())); Kafka Streams application 15 NOW… DISCARD IMAGES WHEN ALERTS ARE DISABLED
16. @fbascheper #Devoxx #Busted << Attribution >> Intermezzo 16 K-Table
17. @fbascheper #Devoxx #Busted STREAM (changelog) (“alert-enabled”, 0) ⊗ (“alert-enabled”, 0) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, 0) ⊗ KStream - KTable duality 17 (no change) (no change) KTABLE | alert-enabled | 0 | ⊗ ⊗ ⊗ | alert-enabled | -1 | ⊗ ⊗ ⊗ | alert-enabled | 0 |
18. @fbascheper #Devoxx #Busted STREAM (changelog) (“alert-enabled”, 0) ⊗ (“alert-enabled”, 0) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, 0) ⊗ KStream - KTable duality 18 (no change) (no change) KTABLE | alert-enabled | 0 | ⊗ ⊗ ⊗ | alert-enabled | -1 | ⊗ ⊗ ⊗ | alert-enabled | 0 | STATE !
19. @fbascheper #Devoxx #Busted val nukiRestApiStream = builder .stream(SOURCE_TOPIC_SMARTLOCK_POLL, Consumed.with(Serdes.String(), Serdes.String())) .mapValues(SmartLockMapper::toNukiResponse) .map(SmartLockMapper::alertingEnabledMapper) .to(BURGLAR_ALERTING_STATE_TOPIC, Produced.with(Serdes.String(), Serdes.Integer())); Adding state - part 1 19 RESULT: A CHANGELOG STREAM WITH (“alert-enabled”, 0) (“alert-enabled”, 0) (“alert-enabled”, -1) public static KeyValue<~> alertingEnabledMapper(String key, NukiResponse rp) { boolean disabled = rp.getLocks().stream() .anyMatch(lock -> LockStates.UNLOCKED.equals(lock.getState())); return KeyValue.pair(ALERT_STATE, disabled ? 0 : -1); }
20. @fbascheper #Devoxx #Busted /* Build global KTable from changelog stream BURGLAR_ALERTING_STATE_TOPIC */ val matd = Materialized.<String, Integer, KeyValueStore<Bytes, byte[]>> as(BURGLAR_ALERTING_STATE_STORE) .withKeySerde(Serdes.String()) .withValueSerde(Serdes.Integer()); val alertingEnabledGlobalKTable = builder .globalTable(BURGLAR_ALERTING_STATE_TOPIC, matd); Adding state - part 2 20 RESULT: A TABLE WITH | alert-enabled | 0 or -1 |
21. @fbascheper #Devoxx #Busted << Attribution >> Let’s continue … 21 K-Table
22. @fbascheper #Devoxx #Busted val builder = new StreamsBuilder(); val cameraSourceStream = builder.stream( SOURCE_TOPIC_CAMERA_IMAGES, Consumed.with(Serdes.String(), Serdes.ByteArray())); val filteredSourceStream = cameraSourceStream .leftJoin( alertingEnabledGlobalKTable, (filename, image) -> ALERT_STATE, (image, enabled) -> enabled.equals(-1) ? image :new byte[]{}) .filter((key, value) -> value.length > 0); Kafka Streams application 22
23. @fbascheper #Devoxx #Busted val imageStream = filteredSourceStream .mapValues((readOnlyKey, value) -> new SerializableImage(readOnlyKey, ByteBuffer.wrap(value))); Kafka Streams application 23
24. @fbascheper #Devoxx #Busted val imageStream = filteredSourceStream .mapValues((readOnlyKey, value) -> new SerializableImage(readOnlyKey, ByteBuffer.wrap(value))); val telegramPhotoMessage = imageStream .mapValues((readOnlyKey, value) -> { val caption = TensorFlowMatcher.matchImage(labels, graphDef, image); return TelegramMessageMapper.photoMessage(image, caption); }); Kafka Streams application 24
25. @fbascheper #Devoxx #Busted val imageStream = filteredSourceStream .mapValues((readOnlyKey, value) -> new SerializableImage(readOnlyKey, ByteBuffer.wrap(value))); val telegramPhotoMessage = imageStream .mapValues((readOnlyKey, value) -> { val caption = TensorFlowMatcher.matchImage(labels, graphDef, image); return TelegramMessageMapper.photoMessage(image, caption); }); telegramPhotoMessage.to(SINK_TOPIC_BURGLAR_ALERTS, Produced.with(Serdes.String(), telegramMessageSerde)); Kafka Streams application 25
26. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video Kafka Connectors Demo Wrap-up 26 https://pixabay.com/p-683065/
27. @fbascheper #Devoxx #Busted TensorFlow - ‘ML for everyone’ Familiar with TensorFlow? Used TensorFlow before? Which API’s? Python Java Keras Other 27 https://commons.wikimedia.org/wiki/File:Polling_Station_sign_2017.jpg
28. @fbascheper #Devoxx #Busted https://medium.com/marketing-and-entrepreneurship/10-companies-using-machine-learning-in-cool-ways-887c25f913c3 28
29. @fbascheper #Devoxx #Busted https://medium.com/@tifa2up/image-classification-using-deep-neural-networks-a-beginner-friendly-approach-using-tensorflow-94b0a090ccd4 Image classification 29
30. @fbascheper #Devoxx #Busted Binary image classification 30 NOT A BURGLAR ! BURGLAR ⚠
31. @fbascheper #Devoxx #Busted https://www.flickr.com/photos/111692634@N04/11406964665 Oops … 31 SOLUTION: USE A (PRE-TRAINED) MULTI-CLASS MODEL
32. @fbascheper #Devoxx #Busted public static String matchImage(List<String> tfLabels, byte[] tfGraphDef, SerializableImage image) { byte[] imageBytes = image.getImageData().array(); TensorFlow interface 32
33. @fbascheper #Devoxx #Busted public static String matchImage(List<String> tfLabels, byte[] tfGraphDef, SerializableImage image) { byte[] imageBytes = image.getImageData().array(); try (Tensor tensor = constructAndExecuteGraph(imageBytes)) { float[] labelProbability = executeInceptionGraph(tfGraphDef, tensor); int bestLabelIdx = maxIndex(labelProbabilities); TensorFlow interface 33
34. @fbascheper #Devoxx #Busted public static String matchImage(List<String> tfLabels, byte[] tfGraphDef, SerializableImage image) { byte[] imageBytes = image.getImageData().array(); try (Tensor tensor = constructAndExecuteGraph(imageBytes)) { float[] labelProbability = executeInceptionGraph(tfGraphDef, tensor); int bestLabelIdx = maxIndex(labelProbabilities); String imageClassification = tfLabels.get(bestLabelIdx); float imageProbability = labelProbabilities[bestLabelIdx] * 100f; return String.format("BEST MATCH for image %s was %s (%.2f%% likely)”, image.getName(), imageClassification, imageProbability); } } TensorFlow interface 34
35. @fbascheper #Devoxx #Busted https://www.flickr.com/photos/59195512@N00/88683459 Forgot something? 35
36. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video Kafka Connectors Demo Wrap-up 36 https://pixabay.com/p-683065/
37. @fbascheper #Devoxx #Busted https://www.flickr.com/photos/16210667@N02/25300692725 Streaming video ? 37
38. @fbascheper #Devoxx #Busted String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter = new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); Acquiring images - take 1… 38
39. @fbascheper #Devoxx #Busted String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter = new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); frameGrabber.setFrameRate(5); frameGrabber.setImageWidth(640); frameGrabber.start(); Acquiring images - take 1… 39
40. @fbascheper #Devoxx #Busted String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter = new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); frameGrabber.setFrameRate(5); frameGrabber.setImageWidth(640); frameGrabber.start(); Frame frame = frameGrabber.grabImage(); BufferedImage image = frameConverter.convert(frame); ImageIO.write(image, "png", new File("./video-frame.png")); Acquiring images - take 1… 40
41. @fbascheper #Devoxx #Busted Acquiring images - take 1… 41 WE’LL NEED SOMETHING BETTER THAN THIS ⚠
42. @fbascheper #Devoxx #Busted Acquiring images - take 2 • KISS • Use out-of-the box support - Motion detection - Movie snapshots • Auto upload using FTP(S) 42 NO CODING REQUIRED
43. @fbascheper #Devoxx #Busted https://cdn.pixabay.com/photo/2016/11/22/16/04/dive-1849534_640.jpg One more thing … 43 FEEDING KAFKA
44. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video Kafka Connectors Demo Wrap-up 44 https://pixabay.com/p-683065/
45. @fbascheper #Devoxx #Busted Kafka Connectors 1. FTP source (FOSS) ‣ Loads images into Kafka 2. REST source (FOSS) ‣ Loads lock status changes into Kafka 3. Telegram sink ‣ Newly developed for this project ‣ Sends burglar alerts 45
46. @fbascheper #Devoxx #Busted https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.….connect.ftp.source.FtpSourceConnector", Configuring Kafka Connect 46
47. @fbascheper #Devoxx #Busted https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.….connect.ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": "com. … .connect.converters.ByteArrayConverter" Configuring Kafka Connect 47
48. @fbascheper #Devoxx #Busted https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.….connect.ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": "com. … .connect.converters.ByteArrayConverter" , "connect.ftp.refresh": "PT1M" , "connect.ftp.file.maxage": "P14D" , "connect.ftp.fileconverter": "com. … .connect.ftp.source.SimpleFileConverter" , "connect.ftp.monitor.update": "/foscam-download/:burglar-alerts-camera-ftp-topic" , "connect.ftp.protocol": "sftp" } } Configuring Kafka Connect 48
49. @fbascheper #Devoxx #Busted Applying custom value-converters 49 INFRARED DAYLIGHT HOW TO HANDLE THIS ? ⚠
50. @fbascheper #Devoxx #Busted public class ImageGrayScaleConverter implements Converter, HeaderConverter { @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); Image grayscale converter 50
51. @fbascheper #Devoxx #Busted public class ImageGrayScaleConverter implements Converter, HeaderConverter { @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); val resized = ImageSizeTransformer.INSTANCE.transform(convertedImg); val grayscale = GrayScaleTransformer.INSTANCE.transform(resized); Image grayscale converter 51
52. @fbascheper #Devoxx #Busted public class ImageGrayScaleConverter implements Converter, HeaderConverter { @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); val resized = ImageSizeTransformer.INSTANCE.transform(convertedImg); val grayscale = GrayScaleTransformer.INSTANCE.transform(resized); val bos = new ByteArrayOutputStream(); ImageIO.write(grayscale, "jpg", bos); return bos.toByteArray(); }} Image grayscale converter 52
53. @fbascheper #Devoxx #Busted Kafka Telegram connector • Newly developed for this project (FOSS @ GitHub) • Can be used to send multiple message types ‣ Text ‣ Photo ‣ Videos • Dependencies ‣ Apache Avro ‣ Telegram Bot API 53
54. @fbascheper #Devoxx #Busted << Attribution >> Kafka Telegram connector 54
55. @fbascheper #Devoxx #Busted Kafka Telegram - photo message 55
56. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video Kafka Connectors Demo Wrap-up 56 https://pixabay.com/p-683065/
57. Demo @fbascheper #Devoxx #Busted 57
58. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video Kafka Connectors Demo Wrap-up 58 https://pixabay.com/p-683065/
59. @fbascheper #Devoxx #Busted https://www.flickr.com/photos/30478819@N08/42695313601 Wrap-up 59
60. @fbascheper #Devoxx #Busted https://core.telegram.org/file/811140184/1/5YJxx-rostA/ad3f74094485fb97bd and https://pixabay.com/en/waterfall-water-source-stream-1746227/ Kafka Telegram - planned features 60
61. @fbascheper #Devoxx #Busted Useful links • Source code ‣ https://github.com/fbascheper/kafka-devoxx-2018 • Kafka Connect Telegram plugin ‣ https://github.com/fbascheper/kafka-connect-telegram ‣ https://github.com/fbascheper/kafka-connect-telegram-avro-model ‣ https://github.com/fbascheper/kafka-connect-storage-converters • Kafka connectors overview ‣ https://www.confluent.io/product/connectors-repository 61
62. @fbascheper #Devoxx #Busted http://www.thebluediamondgallery.com/handwriting/q/questions.html 62
63. @fbascheper #Devoxx #Busted 63
64. @fbascheper #Devoxx #Busted https://commons.wikimedia.org/wiki/File:Thank-you-word-cloud.jpg 64