Bust the burglars with Apache Kafka and TensorFlow

In this session I have shown how Apache Kafka and TensorFlow can be used to process real time video events and send alerts when a potential burglary is detected.

Specifically, I've discussed how Kafka Connect and Kafka Streams can be used to read images from a media stream and how this can be integrated with TensorFlow to analyse the resulting event streams.

Transcript

1. Bust the burglars ML with TensorFlow and Apache Kafka Erik-Berndt Scheper
2. Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 2 https://pixabay.com/p-683065/
3. Introduction - Let’s meet • It’s great to be in Barcelona! • Java Architect @ Ordina JTech • Contracted by Ministry of Justice and Security • Passion for ‣ Big Data technologies ‣ Machine learning 3
4. https://nl.wikipedia.org/wiki/Bartje#/media/File:Standbeeld-bartje.jpg Introduction - My home town 4 https://www.ttcircuit.com/wp-content/uploads/2018/11/0P9A1643AA-1024x683.jpg A REPLICA !?
5. https://upload.wikimedia.org/wikipedia/commons/5/56/Haren_Rijksstraatweg_377.JPG Burglary-crime statistics 5 8% 4% 9% 22% 23% 34%
6. Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 6 https://pixabay.com/p-683065/
7. Components 7
8. Application event flow 8 camera image analyse with TensorFlow burglary? alerts enabled?
9. Event streams • Input - Images from IP camera(s) - Lock status changes • Output - Burglar Alerts 9 Integrated using Kafka Connect }
10. Kafka Connectors 1. FTP source ‣ Loads images into Kafka 2. REST source ‣ Loads lock status changes into Kafka 3. Telegram sink ‣ Developed specifically for this project ‣ Sends burglar alerts 10
11. Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 11 https://pixabay.com/p-683065/
12. Application event flow 12 camera image analyse with TensorFlow burglary? alerts enabled?
13. https://www.flickr.com/photos/16210667@N02/25300692725 Streaming video ? 13
14. String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter = new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); Acquiring images - take 1… 14
15. String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter = new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); frameGrabber.setFrameRate(5); frameGrabber.setImageWidth(640); frameGrabber.start(); Acquiring images - take 1… 15
16. String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter = new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); frameGrabber.setFrameRate(5); frameGrabber.setImageWidth(640); frameGrabber.start(); Frame frame = frameGrabber.grabImage(); BufferedImage image = frameConverter.convert(frame); ImageIO.write(image, "png", new File("./video-frame.png")); Acquiring images - take 1… 16
17. Acquiring images - take 1… 17 WE’LL NEED SOMETHING BETTER THAN THIS ⚠
18. Acquiring images - take 2 • KISS • Use out-of-the box support - Motion detection - Movie snapshots • Auto upload using FTP(S) 18 NO CODING REQUIRED
19. https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.datamountaineer.….ftp.source.FtpSourceConnector", Configuring Kafka Connect 19
20. https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.datamountaineer.….ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": "com.github.fbascheper.kafka.connect.….ByteArrayConverter" Configuring Kafka Connect 20
21. https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.datamountaineer.….ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": "com.github.fbascheper.kafka.connect.….ByteArrayConverter" , "connect.ftp.refresh": "PT1M" , "connect.ftp.file.maxage": "P14D" , "connect.ftp.fileconverter": "com.datamountaineer.….SimpleFileConverter" , "connect.ftp.monitor.update": "/cam-download/:burglar-alerts-camera-ftp-topic" , "connect.ftp.protocol": "sftp" } } Configuring Kafka Connect 21
22. Applying custom value-converters 22 INFRARED DAYLIGHT HOW TO HANDLE THIS ? ⚠
23. public class ImageGrayScaleConverter implements Converter, HeaderConverter { @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); Image grayscale converter 23
24. public class ImageGrayScaleConverter implements Converter, HeaderConverter { @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); val resized = ImageSizeTransformer.INSTANCE.transform(convertedImg); val grayscale = GrayScaleTransformer.INSTANCE.transform(resized); Image grayscale converter 24
25. public class ImageGrayScaleConverter implements Converter, HeaderConverter { @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); val resized = ImageSizeTransformer.INSTANCE.transform(convertedImg); val grayscale = GrayScaleTransformer.INSTANCE.transform(resized); val bos = new ByteArrayOutputStream(); ImageIO.write(grayscale, "jpg", bos); return bos.toByteArray(); } } Image grayscale converter 25
26. { "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.datamountaineer.….ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": "com.github.fbascheper.kafka.connect.….ByteArrayConverter" , "connect.ftp.refresh": "PT1M" , "connect.ftp.file.maxage": "P14D" , "connect.ftp.fileconverter": "com.datamountaineer.….SimpleFileConverter" , "connect.ftp.monitor.update": "/cam-download/:burglar-alerts-camera-ftp-topic" , "connect.ftp.protocol": "sftp" } } Configuring the image converter 26 { "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.datamountaineer.….ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": "com.github.fbascheper.kafka.connect.….ByteArrayConverter" , "connect.ftp.refresh": "PT1M" , "connect.ftp.file.maxage": "P14D" , "connect.ftp.fileconverter": "com.datamountaineer.….SimpleFileConverter" , "connect.ftp.monitor.update": "/cam-download/:burglar-alerts-camera-ftp-topic" , "connect.ftp.protocol": "sftp" } }
27. { "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.datamountaineer.….ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": “com.github.fbascheper.….image.ImageGrayScaleConverter" , "value.converter.resizeTargetWidth": 1024 , "value.converter.resizeTargetHeight": -1 , "value.converter.cropLeft": 165 , "value.converter.cropRight": 200 , "value.converter.cropTop": 50 , "value.converter.cropBottom": 50, , "connect.ftp.refresh": "PT1M" , … , "connect.ftp.protocol": "sftp" } } Configuring the image converter 27
28. Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 28 https://pixabay.com/p-683065/
29. TensorFlow - ‘ML for everyone’ Familiar with TensorFlow? Used TensorFlow before? Which API’s? Python Java Keras Other 29 https://commons.wikimedia.org/wiki/File:Polling_Station_sign_2017.jpg
30. https://medium.com/marketing-and-entrepreneurship/10-companies-using-machine-learning-in-cool-ways-887c25f913c3 30
31. https://medium.com/@tifa2up/image-classification-using-deep-neural-networks-a-beginner-friendly-approach-using-tensorflow-94b0a090ccd4 Image classification 31
32. Binary image classification 32 NOT A BURGLAR ! BURGLAR ⚠
33. https://pixabay.com/nl/photos/klok-zakhorloge-verkeer-3179167/ Train an entire neural network? 33 SOLUTION: ‘TRANSFER LEARNING’
34. Transfer learning • Use a ‘pre-trained’ model ‣ Trained on a large dataset ‣ Will effectively serve as a ‘generic model’ of the visual world • Two alternatives ‣ Feature extraction ‣ Fine-tuning 34
35. Actions 35 ENOUGH VARIATION !? • Build a test- and validation set ‣ Download images ‣ Apply transformation ‣ Classify images • Run TensorFlow ‣ Export the model • Check out the code at ‣ https://github.com/fbascheper/kafka-tf-burglar-alerts-demo-model
36. Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 36 https://pixabay.com/p-683065/
37. Receiving alerts from Kafka 37 camera image analyse with TensorFlow burglary? alerts enabled?
38. Kafka Telegram connector • Developed specifically for this project (FOSS @ GitHub) • Can be used to send multiple message types ‣ Text ‣ Photo ‣ Videos • Dependencies ‣ Apache Avro ‣ Telegram Bot API 38
39. Kafka Telegram connector 39
40. Kafka Telegram - photo message 40
41. Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 41 https://pixabay.com/p-683065/
42. val builder = new StreamsBuilder(); val cameraSourceStream = builder.stream( SOURCE_TOPIC_CAMERA_IMAGES, Consumed.with(Serdes.String(), Serdes.ByteArray())); Putting it all together 42
43. val builder = new StreamsBuilder(); val cameraSourceStream = builder.stream( SOURCE_TOPIC_CAMERA_IMAGES, Consumed.with(Serdes.String(), Serdes.ByteArray())); val imageStream = cameraSourceStream .mapValues((readOnlyKey, value) -> new SerializableImage(readOnlyKey, ByteBuffer.wrap(value))); Putting it all together 43
44. val builder = new StreamsBuilder(); val cameraSourceStream = builder.stream( SOURCE_TOPIC_CAMERA_IMAGES, Consumed.with(Serdes.String(), Serdes.ByteArray())); val imageStream = cameraSourceStream .mapValues((readOnlyKey, value) -> new SerializableImage(readOnlyKey, ByteBuffer.wrap(value))); val burglarAlertStream = imageStream .mapValues((readOnlyKey, image) -> TensorFlowMatcher.matchImage(tfGraphDef, image)) .filter((key, imgClass) -> imgClass.getClassification() == Classification.BURGLAR_ALERT); Putting it all together 44
45. val burglarAlertStream = imageStream .mapValues((readOnlyKey, image) -> TensorFlowMatcher.matchImage(tfGraphDef, image)) .filter((key, imgClass) -> imgClass.getClassification() == Classification.BURGLAR_ALERT); val telegramPhotoMessage = burglarAlertStream .mapValues((readOnlyKey, imageClassification) -> TelegramMessageMapper.photoMessage( imageClassification.getImage(), // image imageClassification.toString() // caption ); Putting it all together 45
46. val burglarAlertStream = imageStream .mapValues((readOnlyKey, image) -> TensorFlowMatcher.matchImage(tfGraphDef, image)) .filter((key, imgClass) -> imgClass.getClassification() == Classification.BURGLAR_ALERT); val telegramPhotoMessage = burglarAlertStream .mapValues((readOnlyKey, imageClassification) -> TelegramMessageMapper.photoMessage( imageClassification.getImage(), // image imageClassification.toString() // caption ); telegramPhotoMessage.to(SINK_TOPIC_BURGLAR_ALERTS, Produced.with(Serdes.String(), telegramMessageSerde)); Putting it all together 46
47. Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 47 https://pixabay.com/p-683065/
48. Additional filtering 48 camera image analyse with TensorFlow burglary? alerts enabled?
49. Event processing 49 poll REST API alerts enabled! all locks alerts disabled
50. Event processing 50 poll REST API alerts enabled! all locks alerts disabled STATE ?
51. STREAM (changelog) (“alert-enabled”, 0) ⊗ (“alert-enabled”, 0) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, 0) ⊗ KStream - KTable duality 51 (no change) (no change) KTABLE | alert-enabled | 0 | ⊗ ⊗ ⊗ | alert-enabled | -1 | ⊗ ⊗ ⊗ | alert-enabled | 0 |
52. STREAM (changelog) (“alert-enabled”, 0) ⊗ (“alert-enabled”, 0) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, 0) ⊗ KStream - KTable duality 52 (no change) (no change) KTABLE | alert-enabled | 0 | ⊗ ⊗ ⊗ | alert-enabled | -1 | ⊗ ⊗ ⊗ | alert-enabled | 0 | STATE !
53. val alertingEnabledStateStream = builder .stream(SOURCE_TOPIC_SMARTLOCK_POLL, Consumed.with(Serdes.String(), Serdes.String())) .mapValues(SmartLockMapper::toNukiResponse) .map(SmartLockMapper::alertingEnabledMapper) .to(BURGLAR_ALERTING_STATE_TOPIC, Produced.with(Serdes.String(), Serdes.Integer())); Adding state - part 1 53 RESULT: A CHANGELOG STREAM WITH (“alert-enabled”, 0) (“alert-enabled”, 0) (“alert-enabled”, -1) public static KeyValue<~> alertingEnabledMapper(String key, NukiResponse rp) { boolean disabled = rp.getLocks().stream() .anyMatch(lock -> LockStates.UNLOCKED.equals(lock.getState())); return KeyValue.pair(ALERT_STATE, disabled ? 0 : -1); }
54. /* Build global KTable from changelog stream BURGLAR_ALERTING_STATE_TOPIC */ val matd = Materialized.<String, Integer, KeyValueStore<Bytes, byte[]>> as(BURGLAR_ALERTING_STATE_STORE) .withKeySerde(Serdes.String()) .withValueSerde(Serdes.Integer()); val alertingEnabledGlobalKTable = builder .globalTable(BURGLAR_ALERTING_STATE_TOPIC, matd); Adding state - part 2 54 RESULT: A TABLE WITH | alert-enabled | 0 or -1 |
55. val builder = new StreamsBuilder(); val cameraSourceStream = builder.stream( SOURCE_TOPIC_CAMERA_IMAGES, Consumed.with(Serdes.String(), Serdes.ByteArray())); val filteredSourceStream = cameraSourceStream .leftJoin( alertingEnabledGlobalKTable, (filename, image) -> ALERT_STATE, (image, enabled) -> enabled.equals(-1) ? image :new byte[]{}) .filter((key, value) -> value.length > 0); Adding the filter in the application 55
56. Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 56 https://pixabay.com/p-683065/
57. Demo 57
58. Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 58 https://pixabay.com/p-683065/
59. https://www.flickr.com/photos/30478819@N08/42695313601 Wrap-up 59
60. https://core.telegram.org/file/811140184/1/5YJxx-rostA/ad3f74094485fb97bd and https://pixabay.com/en/waterfall-water-source-stream-1746227/ Kafka Telegram - planned features 60
61. Useful links • Source code ‣ https://github.com/fbascheper/kafka-tf-burglar-alerts-demo ‣ https://github.com/fbascheper/kafka-tf-burglar-alerts-demo-model • Kafka Connect plugin / converters ‣ https://github.com/fbascheper/kafka-connect-telegram ‣ https://github.com/fbascheper/kafka-connect-storage-converters • Kafka connectors overview ‣ https://www.confluent.io/product/connectors-repository 61
62. http://www.thebluediamondgallery.com/handwriting/q/questions.html 62
63. https://commons.wikimedia.org/wiki/File:Thank-you-word-cloud.jpg 63