Bust the burglars ML with TensorFlow and Apache Kafka Erik-Berndt Scheper 

Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 2 https://pixabay.com/p-683065/ 

Introduction - Let’s meet • It’s great to be in Barcelona! • Java Architect @ Ordina JTech • Contracted by Ministry of Justice and Security • Passion for ‣ Big Data technologies ‣ Machine learning 3 

https://nl.wikipedia.org/wiki/Bartje#/media/File:Standbeeld-bartje.jpg Introduction - My home town 4 https://www.ttcircuit.com/wp-content/uploads/2018/11/0P9A1643AA-1024x683.jpg A REPLICA !? 

https://upload.wikimedia.org/wikipedia/commons/5/56/Haren_Rijksstraatweg_377.JPG Burglary-crime statistics 5 8% 4% 9% 22% 23% 34% 

Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 6 https://pixabay.com/p-683065/ 

Components 7 

Application event flow 8 camera image analyse with TensorFlow burglary? alerts enabled? 

Event streams • Input - Images from IP camera(s) - Lock status changes • Output - Burglar Alerts 9 Integrated using Kafka Connect } 

Kafka Connectors 1. FTP source ‣ Loads images into Kafka 2. REST source ‣ Loads lock status changes into Kafka 3. Telegram sink ‣ Developed specifically for this project ‣ Sends burglar alerts 10 

Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 11 https://pixabay.com/p-683065/ 

Application event flow 12 camera image analyse with TensorFlow burglary? alerts enabled? 

https://www.flickr.com/photos/16210667@N02/25300692725 Streaming video ? 13 

String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter = new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); Acquiring images - take 1… 14 

String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter = new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); frameGrabber.setFrameRate(5); frameGrabber.setImageWidth(640); frameGrabber.start(); Acquiring images - take 1… 15 

String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter = new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); frameGrabber.setFrameRate(5); frameGrabber.setImageWidth(640); frameGrabber.start(); Frame frame = frameGrabber.grabImage(); BufferedImage image = frameConverter.convert(frame); ImageIO.write(image, "png", new File("./video-frame.png")); Acquiring images - take 1… 16 

Acquiring images - take 1… 17 WE’LL NEED SOMETHING BETTER THAN THIS ⚠ 

Acquiring images - take 2 • KISS • Use out-of-the box support - Motion detection - Movie snapshots • Auto upload using FTP(S) 18 NO CODING REQUIRED 

https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.datamountaineer.….ftp.source.FtpSourceConnector", Configuring Kafka Connect 19 

https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.datamountaineer.….ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": "com.github.fbascheper.kafka.connect.….ByteArrayConverter" Configuring Kafka Connect 20 

https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.datamountaineer.….ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": "com.github.fbascheper.kafka.connect.….ByteArrayConverter" , "connect.ftp.refresh": "PT1M" , "connect.ftp.file.maxage": "P14D" , "connect.ftp.fileconverter": "com.datamountaineer.….SimpleFileConverter" , "connect.ftp.monitor.update": "/cam-download/:burglar-alerts-camera-ftp-topic" , "connect.ftp.protocol": "sftp" } } Configuring Kafka Connect 21 

Applying custom value-converters 22 INFRARED DAYLIGHT HOW TO HANDLE THIS ? ⚠ 

public class ImageGrayScaleConverter implements Converter, HeaderConverter { @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); Image grayscale converter 23 

public class ImageGrayScaleConverter implements Converter, HeaderConverter { @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); val resized = ImageSizeTransformer.INSTANCE.transform(convertedImg); val grayscale = GrayScaleTransformer.INSTANCE.transform(resized); Image grayscale converter 24 

public class ImageGrayScaleConverter implements Converter, HeaderConverter { @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); val resized = ImageSizeTransformer.INSTANCE.transform(convertedImg); val grayscale = GrayScaleTransformer.INSTANCE.transform(resized); val bos = new ByteArrayOutputStream(); ImageIO.write(grayscale, "jpg", bos); return bos.toByteArray(); } } Image grayscale converter 25 

{ "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.datamountaineer.….ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": "com.github.fbascheper.kafka.connect.….ByteArrayConverter" , "connect.ftp.refresh": "PT1M" , "connect.ftp.file.maxage": "P14D" , "connect.ftp.fileconverter": "com.datamountaineer.….SimpleFileConverter" , "connect.ftp.monitor.update": "/cam-download/:burglar-alerts-camera-ftp-topic" , "connect.ftp.protocol": "sftp" } } Configuring the image converter 26 { "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.datamountaineer.….ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": "com.github.fbascheper.kafka.connect.….ByteArrayConverter" , "connect.ftp.refresh": "PT1M" , "connect.ftp.file.maxage": "P14D" , "connect.ftp.fileconverter": "com.datamountaineer.….SimpleFileConverter" , "connect.ftp.monitor.update": "/cam-download/:burglar-alerts-camera-ftp-topic" , "connect.ftp.protocol": "sftp" } } 

{ "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.datamountaineer.….ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": “com.github.fbascheper.….image.ImageGrayScaleConverter" , "value.converter.resizeTargetWidth": 1024 , "value.converter.resizeTargetHeight": -1 , "value.converter.cropLeft": 165 , "value.converter.cropRight": 200 , "value.converter.cropTop": 50 , "value.converter.cropBottom": 50, , "connect.ftp.refresh": "PT1M" , … , "connect.ftp.protocol": "sftp" } } Configuring the image converter 27 

Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 28 https://pixabay.com/p-683065/ 

TensorFlow - ‘ML for everyone’ Familiar with TensorFlow? Used TensorFlow before? Which API’s? Python Java Keras Other 29 https://commons.wikimedia.org/wiki/File:Polling_Station_sign_2017.jpg 

https://medium.com/marketing-and-entrepreneurship/10-companies-using-machine-learning-in-cool-ways-887c25f913c3 30 

https://medium.com/@tifa2up/image-classification-using-deep-neural-networks-a-beginner-friendly-approach-using-tensorflow-94b0a090ccd4 Image classification 31 

Binary image classification 32 NOT A BURGLAR ! BURGLAR ⚠ 

https://pixabay.com/nl/photos/klok-zakhorloge-verkeer-3179167/ Train an entire neural network? 33 SOLUTION: ‘TRANSFER LEARNING’ 

Transfer learning • Use a ‘pre-trained’ model ‣ Trained on a large dataset ‣ Will effectively serve as a ‘generic model’ of the visual world • Two alternatives ‣ Feature extraction ‣ Fine-tuning 34 

Actions 35 ENOUGH VARIATION !? • Build a test- and validation set ‣ Download images ‣ Apply transformation ‣ Classify images • Run TensorFlow ‣ Export the model • Check out the code at ‣ https://github.com/fbascheper/kafka-tf-burglar-alerts-demo-model 

Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 36 https://pixabay.com/p-683065/ 

Receiving alerts from Kafka 37 camera image analyse with TensorFlow burglary? alerts enabled? 

Kafka Telegram connector • Developed specifically for this project (FOSS @ GitHub) • Can be used to send multiple message types ‣ Text ‣ Photo ‣ Videos • Dependencies ‣ Apache Avro ‣ Telegram Bot API 38 

Kafka Telegram connector 39 

Kafka Telegram - photo message 40 

Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 41 https://pixabay.com/p-683065/ 

val builder = new StreamsBuilder(); val cameraSourceStream = builder.stream( SOURCE_TOPIC_CAMERA_IMAGES, Consumed.with(Serdes.String(), Serdes.ByteArray())); Putting it all together 42 

val builder = new StreamsBuilder(); val cameraSourceStream = builder.stream( SOURCE_TOPIC_CAMERA_IMAGES, Consumed.with(Serdes.String(), Serdes.ByteArray())); val imageStream = cameraSourceStream .mapValues((readOnlyKey, value) -> new SerializableImage(readOnlyKey, ByteBuffer.wrap(value))); Putting it all together 43 

val builder = new StreamsBuilder(); val cameraSourceStream = builder.stream( SOURCE_TOPIC_CAMERA_IMAGES, Consumed.with(Serdes.String(), Serdes.ByteArray())); val imageStream = cameraSourceStream .mapValues((readOnlyKey, value) -> new SerializableImage(readOnlyKey, ByteBuffer.wrap(value))); val burglarAlertStream = imageStream .mapValues((readOnlyKey, image) -> TensorFlowMatcher.matchImage(tfGraphDef, image)) .filter((key, imgClass) -> imgClass.getClassification() == Classification.BURGLAR_ALERT); Putting it all together 44 

val burglarAlertStream = imageStream .mapValues((readOnlyKey, image) -> TensorFlowMatcher.matchImage(tfGraphDef, image)) .filter((key, imgClass) -> imgClass.getClassification() == Classification.BURGLAR_ALERT); val telegramPhotoMessage = burglarAlertStream .mapValues((readOnlyKey, imageClassification) -> TelegramMessageMapper.photoMessage( imageClassification.getImage(), // image imageClassification.toString() // caption ); Putting it all together 45 

val burglarAlertStream = imageStream .mapValues((readOnlyKey, image) -> TensorFlowMatcher.matchImage(tfGraphDef, image)) .filter((key, imgClass) -> imgClass.getClassification() == Classification.BURGLAR_ALERT); val telegramPhotoMessage = burglarAlertStream .mapValues((readOnlyKey, imageClassification) -> TelegramMessageMapper.photoMessage( imageClassification.getImage(), // image imageClassification.toString() // caption ); telegramPhotoMessage.to(SINK_TOPIC_BURGLAR_ALERTS, Produced.with(Serdes.String(), telegramMessageSerde)); Putting it all together 46 

Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 47 https://pixabay.com/p-683065/ 

Additional filtering 48 camera image analyse with TensorFlow burglary? alerts enabled? 

Event processing 49 poll REST API alerts enabled! all locks alerts disabled 

Event processing 50 poll REST API alerts enabled! all locks alerts disabled STATE ? 

STREAM (changelog) (“alert-enabled”, 0) ⊗ (“alert-enabled”, 0) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, 0) ⊗ KStream - KTable duality 51 (no change) (no change) KTABLE | alert-enabled | 0 | ⊗ ⊗ ⊗ | alert-enabled | -1 | ⊗ ⊗ ⊗ | alert-enabled | 0 | 

STREAM (changelog) (“alert-enabled”, 0) ⊗ (“alert-enabled”, 0) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, 0) ⊗ KStream - KTable duality 52 (no change) (no change) KTABLE | alert-enabled | 0 | ⊗ ⊗ ⊗ | alert-enabled | -1 | ⊗ ⊗ ⊗ | alert-enabled | 0 | STATE ! 

val alertingEnabledStateStream = builder .stream(SOURCE_TOPIC_SMARTLOCK_POLL, Consumed.with(Serdes.String(), Serdes.String())) .mapValues(SmartLockMapper::toNukiResponse) .map(SmartLockMapper::alertingEnabledMapper) .to(BURGLAR_ALERTING_STATE_TOPIC, Produced.with(Serdes.String(), Serdes.Integer())); Adding state - part 1 53 RESULT: A CHANGELOG STREAM WITH (“alert-enabled”, 0) (“alert-enabled”, 0) (“alert-enabled”, -1) public static KeyValue<~> alertingEnabledMapper(String key, NukiResponse rp) { boolean disabled = rp.getLocks().stream() .anyMatch(lock -> LockStates.UNLOCKED.equals(lock.getState())); return KeyValue.pair(ALERT_STATE, disabled ? 0 : -1); } 

/* Build global KTable from changelog stream BURGLAR_ALERTING_STATE_TOPIC */ val matd = Materialized.> as(BURGLAR_ALERTING_STATE_STORE) .withKeySerde(Serdes.String()) .withValueSerde(Serdes.Integer()); val alertingEnabledGlobalKTable = builder .globalTable(BURGLAR_ALERTING_STATE_TOPIC, matd); Adding state - part 2 54 RESULT: A TABLE WITH | alert-enabled | 0 or -1 | 

val builder = new StreamsBuilder(); val cameraSourceStream = builder.stream( SOURCE_TOPIC_CAMERA_IMAGES, Consumed.with(Serdes.String(), Serdes.ByteArray())); val filteredSourceStream = cameraSourceStream .leftJoin( alertingEnabledGlobalKTable, (filename, image) -> ALERT_STATE, (image, enabled) -> enabled.equals(-1) ? image :new byte[]{}) .filter((key, value) -> value.length > 0); Adding the filter in the application 55 

Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 56 https://pixabay.com/p-683065/ 

Demo 57 

Agenda Introduction Setting up your alarm Uploading images into Kafka Building a TensorFlow model Receiving alerts from Kafka Putting it all together Additional filtering Demo Wrap-up 58 https://pixabay.com/p-683065/ 

https://www.flickr.com/photos/30478819@N08/42695313601 Wrap-up 59 

https://core.telegram.org/file/811140184/1/5YJxx-rostA/ad3f74094485fb97bd and https://pixabay.com/en/waterfall-water-source-stream-1746227/ Kafka Telegram - planned features 60 

Useful links • Source code ‣ https://github.com/fbascheper/kafka-tf-burglar-alerts-demo ‣ https://github.com/fbascheper/kafka-tf-burglar-alerts-demo-model • Kafka Connect plugin / converters ‣ https://github.com/fbascheper/kafka-connect-telegram ‣ https://github.com/fbascheper/kafka-connect-storage-converters • Kafka connectors overview ‣ https://www.confluent.io/product/connectors-repository 61 

http://www.thebluediamondgallery.com/handwriting/q/questions.html 62 

https://commons.wikimedia.org/wiki/File:Thank-you-word-cloud.jpg 63