@fbascheper #Devoxx #Busted Bust the burglars - ML with TensorFlow and Apache Kafka Erik-Berndt Scheper 

@fbascheper #Devoxx #Busted Let’s meet • I’m delighted to be here in Ukraine! • Java Architect @ Ordina JTech • Contracted by the Dutch Tax administration • Passion for ‣ Big Data technologies ‣ Machine learning 2 

@fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video Kafka Connectors Demo Wrap-up 3 https://pixabay.com/p-683065/ 

@fbascheper #Devoxx #Busted https://nl.wikipedia.org/wiki/Bartje#/media/File:Standbeeld-bartje.jpg Introduction 4 

@fbascheper #Devoxx #Busted Statistics https://reolink.com/home-burglary-crime-statistics/ 5 Lack of evidence! Why? 

@fbascheper #Devoxx #Busted https://upload.wikimedia.org/wikipedia/commons/5/56/Haren_Rijksstraatweg_377.JPG Entry points 6 8% 4% 9% 22% 23% 34% 

@fbascheper #Devoxx #Busted https://www.flickr.com/photos/takomabibelot/2134214940 7 

@fbascheper #Devoxx #Busted Components 8 

@fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video Kafka Connectors Demo Wrap-up 9 https://pixabay.com/p-683065/ 

@fbascheper #Devoxx #Busted Event streams • Input - Images from IP camera(s) - Lock status changes • Output - Burglar Alerts 10 

@fbascheper #Devoxx #Busted Event processing 11 camera image analyse with TensorFlow burglary? alerts enabled? STATELESS … 

@fbascheper #Devoxx #Busted Event processing 12 poll lock REST API alerts enabled! all locks alerts disabled (flat)map response to individual locks 

@fbascheper #Devoxx #Busted Event processing 13 poll lock REST API alerts enabled! all locks alerts disabled (flat)map response to individual locks STATE ? 

@fbascheper #Devoxx #Busted val builder = new StreamsBuilder(); Kafka Streams application 14 

@fbascheper #Devoxx #Busted val builder = new StreamsBuilder(); val cameraSourceStream = builder.stream( SOURCE_TOPIC_CAMERA_IMAGES, Consumed.with(Serdes.String(), Serdes.ByteArray())); Kafka Streams application 15 NOW… DISCARD IMAGES WHEN ALERTS ARE DISABLED 

@fbascheper #Devoxx #Busted << Attribution >> Intermezzo 16 K-Table 

@fbascheper #Devoxx #Busted STREAM (changelog) (“alert-enabled”, 0) ⊗ (“alert-enabled”, 0) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, 0) ⊗ KStream - KTable duality 17 (no change) (no change) KTABLE | alert-enabled | 0 | ⊗ ⊗ ⊗ | alert-enabled | -1 | ⊗ ⊗ ⊗ | alert-enabled | 0 | 

@fbascheper #Devoxx #Busted STREAM (changelog) (“alert-enabled”, 0) ⊗ (“alert-enabled”, 0) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, 0) ⊗ KStream - KTable duality 18 (no change) (no change) KTABLE | alert-enabled | 0 | ⊗ ⊗ ⊗ | alert-enabled | -1 | ⊗ ⊗ ⊗ | alert-enabled | 0 | STATE ! 

@fbascheper #Devoxx #Busted val nukiRestApiStream = builder .stream(SOURCE_TOPIC_SMARTLOCK_POLL, Consumed.with(Serdes.String(), Serdes.String())) .mapValues(SmartLockMapper::toNukiResponse) .map(SmartLockMapper::alertingEnabledMapper) .to(BURGLAR_ALERTING_STATE_TOPIC, Produced.with(Serdes.String(), Serdes.Integer())); Adding state - part 1 19 RESULT: A CHANGELOG STREAM WITH (“alert-enabled”, 0) (“alert-enabled”, 0) (“alert-enabled”, -1) public static KeyValue<~> alertingEnabledMapper(String key, NukiResponse rp) { boolean disabled = rp.getLocks().stream() .anyMatch(lock -> LockStates.UNLOCKED.equals(lock.getState())); return KeyValue.pair(ALERT_STATE, disabled ? 0 : -1); } 

@fbascheper #Devoxx #Busted /* Build global KTable from changelog stream BURGLAR_ALERTING_STATE_TOPIC */ val matd = Materialized.> as(BURGLAR_ALERTING_STATE_STORE) .withKeySerde(Serdes.String()) .withValueSerde(Serdes.Integer()); val alertingEnabledGlobalKTable = builder .globalTable(BURGLAR_ALERTING_STATE_TOPIC, matd); Adding state - part 2 20 RESULT: A TABLE WITH | alert-enabled | 0 or -1 | 

@fbascheper #Devoxx #Busted << Attribution >> Let’s continue … 21 K-Table 

@fbascheper #Devoxx #Busted val builder = new StreamsBuilder(); val cameraSourceStream = builder.stream( SOURCE_TOPIC_CAMERA_IMAGES, Consumed.with(Serdes.String(), Serdes.ByteArray())); val filteredSourceStream = cameraSourceStream .leftJoin( alertingEnabledGlobalKTable, (filename, image) -> ALERT_STATE, (image, enabled) -> enabled.equals(-1) ? image :new byte[]{}) .filter((key, value) -> value.length > 0); Kafka Streams application 22 

@fbascheper #Devoxx #Busted val imageStream = filteredSourceStream .mapValues((readOnlyKey, value) -> new SerializableImage(readOnlyKey, ByteBuffer.wrap(value))); Kafka Streams application 23 

@fbascheper #Devoxx #Busted val imageStream = filteredSourceStream .mapValues((readOnlyKey, value) -> new SerializableImage(readOnlyKey, ByteBuffer.wrap(value))); val telegramPhotoMessage = imageStream .mapValues((readOnlyKey, value) -> { val caption = TensorFlowMatcher.matchImage(labels, graphDef, image); return TelegramMessageMapper.photoMessage(image, caption); }); Kafka Streams application 24 

@fbascheper #Devoxx #Busted val imageStream = filteredSourceStream .mapValues((readOnlyKey, value) -> new SerializableImage(readOnlyKey, ByteBuffer.wrap(value))); val telegramPhotoMessage = imageStream .mapValues((readOnlyKey, value) -> { val caption = TensorFlowMatcher.matchImage(labels, graphDef, image); return TelegramMessageMapper.photoMessage(image, caption); }); telegramPhotoMessage.to(SINK_TOPIC_BURGLAR_ALERTS, Produced.with(Serdes.String(), telegramMessageSerde)); Kafka Streams application 25 

@fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video Kafka Connectors Demo Wrap-up 26 https://pixabay.com/p-683065/ 

@fbascheper #Devoxx #Busted TensorFlow - ‘ML for everyone’ Familiar with TensorFlow? Used TensorFlow before? Which API’s? Python Java Keras Other 27 https://commons.wikimedia.org/wiki/File:Polling_Station_sign_2017.jpg 

@fbascheper #Devoxx #Busted https://medium.com/marketing-and-entrepreneurship/10-companies-using-machine-learning-in-cool-ways-887c25f913c3 28 

@fbascheper #Devoxx #Busted https://medium.com/@tifa2up/image-classification-using-deep-neural-networks-a-beginner-friendly-approach-using-tensorflow-94b0a090ccd4 Image classification 29 

@fbascheper #Devoxx #Busted Binary image classification 30 NOT A BURGLAR ! BURGLAR ⚠ 

@fbascheper #Devoxx #Busted https://www.flickr.com/photos/111692634@N04/11406964665 Oops … 31 SOLUTION: USE A (PRE-TRAINED) MULTI-CLASS MODEL 

@fbascheper #Devoxx #Busted public static String matchImage(List tfLabels, byte[] tfGraphDef, SerializableImage image) { byte[] imageBytes = image.getImageData().array(); TensorFlow interface 32 

@fbascheper #Devoxx #Busted public static String matchImage(List tfLabels, byte[] tfGraphDef, SerializableImage image) { byte[] imageBytes = image.getImageData().array(); try (Tensor tensor = constructAndExecuteGraph(imageBytes)) { float[] labelProbability = executeInceptionGraph(tfGraphDef, tensor); int bestLabelIdx = maxIndex(labelProbabilities); TensorFlow interface 33 

@fbascheper #Devoxx #Busted public static String matchImage(List tfLabels, byte[] tfGraphDef, SerializableImage image) { byte[] imageBytes = image.getImageData().array(); try (Tensor tensor = constructAndExecuteGraph(imageBytes)) { float[] labelProbability = executeInceptionGraph(tfGraphDef, tensor); int bestLabelIdx = maxIndex(labelProbabilities); String imageClassification = tfLabels.get(bestLabelIdx); float imageProbability = labelProbabilities[bestLabelIdx] * 100f; return String.format("BEST MATCH for image %s was %s (%.2f%% likely)”, image.getName(), imageClassification, imageProbability); } } TensorFlow interface 34 

@fbascheper #Devoxx #Busted https://www.flickr.com/photos/59195512@N00/88683459 Forgot something? 35 

@fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video Kafka Connectors Demo Wrap-up 36 https://pixabay.com/p-683065/ 

@fbascheper #Devoxx #Busted https://www.flickr.com/photos/16210667@N02/25300692725 Streaming video ? 37 

@fbascheper #Devoxx #Busted String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter = new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); Acquiring images - take 1… 38 

@fbascheper #Devoxx #Busted String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter = new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); frameGrabber.setFrameRate(5); frameGrabber.setImageWidth(640); frameGrabber.start(); Acquiring images - take 1… 39 

@fbascheper #Devoxx #Busted String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter = new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); frameGrabber.setFrameRate(5); frameGrabber.setImageWidth(640); frameGrabber.start(); Frame frame = frameGrabber.grabImage(); BufferedImage image = frameConverter.convert(frame); ImageIO.write(image, "png", new File("./video-frame.png")); Acquiring images - take 1… 40 

@fbascheper #Devoxx #Busted Acquiring images - take 1… 41 WE’LL NEED SOMETHING BETTER THAN THIS ⚠ 

@fbascheper #Devoxx #Busted Acquiring images - take 2 • KISS • Use out-of-the box support - Motion detection - Movie snapshots • Auto upload using FTP(S) 42 NO CODING REQUIRED 

@fbascheper #Devoxx #Busted https://cdn.pixabay.com/photo/2016/11/22/16/04/dive-1849534_640.jpg One more thing … 43 FEEDING KAFKA 

@fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video Kafka Connectors Demo Wrap-up 44 https://pixabay.com/p-683065/ 

@fbascheper #Devoxx #Busted Kafka Connectors 1. FTP source (FOSS) ‣ Loads images into Kafka 2. REST source (FOSS) ‣ Loads lock status changes into Kafka 3. Telegram sink ‣ Newly developed for this project ‣ Sends burglar alerts 45 

@fbascheper #Devoxx #Busted https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.….connect.ftp.source.FtpSourceConnector", Configuring Kafka Connect 46 

@fbascheper #Devoxx #Busted https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.….connect.ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": "com. … .connect.converters.ByteArrayConverter" Configuring Kafka Connect 47 

@fbascheper #Devoxx #Busted https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", , "config": { "connector.class": "com.….connect.ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": "com. … .connect.converters.ByteArrayConverter" , "connect.ftp.refresh": "PT1M" , "connect.ftp.file.maxage": "P14D" , "connect.ftp.fileconverter": "com. … .connect.ftp.source.SimpleFileConverter" , "connect.ftp.monitor.update": "/foscam-download/:burglar-alerts-camera-ftp-topic" , "connect.ftp.protocol": "sftp" } } Configuring Kafka Connect 48 

@fbascheper #Devoxx #Busted Applying custom value-converters 49 INFRARED DAYLIGHT HOW TO HANDLE THIS ? ⚠ 

@fbascheper #Devoxx #Busted public class ImageGrayScaleConverter implements Converter, HeaderConverter { @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); Image grayscale converter 50 

@fbascheper #Devoxx #Busted public class ImageGrayScaleConverter implements Converter, HeaderConverter { @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); val resized = ImageSizeTransformer.INSTANCE.transform(convertedImg); val grayscale = GrayScaleTransformer.INSTANCE.transform(resized); Image grayscale converter 51 

@fbascheper #Devoxx #Busted public class ImageGrayScaleConverter implements Converter, HeaderConverter { @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); val resized = ImageSizeTransformer.INSTANCE.transform(convertedImg); val grayscale = GrayScaleTransformer.INSTANCE.transform(resized); val bos = new ByteArrayOutputStream(); ImageIO.write(grayscale, "jpg", bos); return bos.toByteArray(); }} Image grayscale converter 52 

@fbascheper #Devoxx #Busted Kafka Telegram connector • Newly developed for this project (FOSS @ GitHub) • Can be used to send multiple message types ‣ Text ‣ Photo ‣ Videos • Dependencies ‣ Apache Avro ‣ Telegram Bot API 53 

@fbascheper #Devoxx #Busted << Attribution >> Kafka Telegram connector 54 

@fbascheper #Devoxx #Busted Kafka Telegram - photo message 55 

@fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video Kafka Connectors Demo Wrap-up 56 https://pixabay.com/p-683065/ 

Demo @fbascheper #Devoxx #Busted 57 

@fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video Kafka Connectors Demo Wrap-up 58 https://pixabay.com/p-683065/ 

@fbascheper #Devoxx #Busted https://www.flickr.com/photos/30478819@N08/42695313601 Wrap-up 59 

@fbascheper #Devoxx #Busted https://core.telegram.org/file/811140184/1/5YJxx-rostA/ad3f74094485fb97bd and https://pixabay.com/en/waterfall-water-source-stream-1746227/ Kafka Telegram - planned features 60 

@fbascheper #Devoxx #Busted Useful links • Source code ‣ https://github.com/fbascheper/kafka-devoxx-2018 • Kafka Connect Telegram plugin ‣ https://github.com/fbascheper/kafka-connect-telegram ‣ https://github.com/fbascheper/kafka-connect-telegram-avro-model ‣ https://github.com/fbascheper/kafka-connect-storage-converters • Kafka connectors overview ‣ https://www.confluent.io/product/connectors-repository 61 

@fbascheper #Devoxx #Busted http://www.thebluediamondgallery.com/handwriting/q/questions.html 62 

@fbascheper #Devoxx #Busted 63 

@fbascheper #Devoxx #Busted https://commons.wikimedia.org/wiki/File:Thank-you-word-cloud.jpg 64