Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Bust the burglars: Machine Learning with TensorFlow and Apache Kafka

Bust the burglars: Machine Learning with TensorFlow and Apache Kafka

The integration of event streams is becoming increasingly complex in the world of sensors, media streams and the Internet of Things. In this session I will discuss how `Apache Kafka` and `TensorFlow` can be used to process real time video events and send alerts when a potential burglary is detected.

Specifically, I'll discuss how `Kafka Connect` and `Kafka Streams` can be used to read images from a media stream and how this can be integrated with TensorFlow to analyse the resulting event streams.

Although the use case presented is based on home security, the same technology used to analyse event streams can be applied to other use cases, as I will demonstrate during this session.

Frederieke Scheper

November 24, 2018
Tweet

More Decks by Frederieke Scheper

Other Decks in Technology

Transcript

  1. @fbascheper #Devoxx #Busted Let’s meet • I’m delighted to be

    here in Ukraine! • Java Architect @ Ordina JTech • Contracted by the Dutch Tax administration • Passion for ‣ Big Data technologies ‣ Machine learning 2
  2. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video

    Kafka Connectors Demo Wrap-up 3 https://pixabay.com/p-683065/
  3. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video

    Kafka Connectors Demo Wrap-up 9 https://pixabay.com/p-683065/
  4. @fbascheper #Devoxx #Busted Event streams • Input - Images from

    IP camera(s) - Lock status changes • Output - Burglar Alerts 10
  5. @fbascheper #Devoxx #Busted Event processing 11 camera image analyse with

    TensorFlow burglary? alerts enabled? STATELESS …
  6. @fbascheper #Devoxx #Busted Event processing 12 poll lock REST API

    alerts enabled! all locks alerts disabled (flat)map response to individual locks
  7. @fbascheper #Devoxx #Busted Event processing 13 poll lock REST API

    alerts enabled! all locks alerts disabled (flat)map response to individual locks STATE ?
  8. @fbascheper #Devoxx #Busted val builder = new StreamsBuilder(); val cameraSourceStream

    = builder.stream( SOURCE_TOPIC_CAMERA_IMAGES, Consumed.with(Serdes.String(), Serdes.ByteArray())); Kafka Streams application 15 NOW… DISCARD IMAGES WHEN ALERTS ARE DISABLED
  9. @fbascheper #Devoxx #Busted STREAM (changelog) (“alert-enabled”, 0) ⊗ (“alert-enabled”, 0)

    ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, 0) ⊗ KStream - KTable duality 17 (no change) (no change) KTABLE | alert-enabled | 0 | ⊗ ⊗ ⊗ | alert-enabled | -1 | ⊗ ⊗ ⊗ | alert-enabled | 0 |
  10. @fbascheper #Devoxx #Busted STREAM (changelog) (“alert-enabled”, 0) ⊗ (“alert-enabled”, 0)

    ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, 0) ⊗ KStream - KTable duality 18 (no change) (no change) KTABLE | alert-enabled | 0 | ⊗ ⊗ ⊗ | alert-enabled | -1 | ⊗ ⊗ ⊗ | alert-enabled | 0 | STATE !
  11. @fbascheper #Devoxx #Busted val nukiRestApiStream = builder .stream(SOURCE_TOPIC_SMARTLOCK_POLL, Consumed.with(Serdes.String(), Serdes.String()))

    .mapValues(SmartLockMapper::toNukiResponse) .map(SmartLockMapper::alertingEnabledMapper) .to(BURGLAR_ALERTING_STATE_TOPIC, Produced.with(Serdes.String(), Serdes.Integer())); Adding state - part 1 19 RESULT: A CHANGELOG STREAM WITH (“alert-enabled”, 0) (“alert-enabled”, 0) (“alert-enabled”, -1) public static KeyValue<~> alertingEnabledMapper(String key, NukiResponse rp) { boolean disabled = rp.getLocks().stream() .anyMatch(lock -> LockStates.UNLOCKED.equals(lock.getState())); return KeyValue.pair(ALERT_STATE, disabled ? 0 : -1); }
  12. @fbascheper #Devoxx #Busted /* Build global KTable from changelog stream

    BURGLAR_ALERTING_STATE_TOPIC */ val matd = Materialized.<String, Integer, KeyValueStore<Bytes, byte[]>> as(BURGLAR_ALERTING_STATE_STORE) .withKeySerde(Serdes.String()) .withValueSerde(Serdes.Integer()); val alertingEnabledGlobalKTable = builder .globalTable(BURGLAR_ALERTING_STATE_TOPIC, matd); Adding state - part 2 20 RESULT: A TABLE WITH | alert-enabled | 0 or -1 |
  13. @fbascheper #Devoxx #Busted val builder = new StreamsBuilder(); val cameraSourceStream

    = builder.stream( SOURCE_TOPIC_CAMERA_IMAGES, Consumed.with(Serdes.String(), Serdes.ByteArray())); val filteredSourceStream = cameraSourceStream .leftJoin( alertingEnabledGlobalKTable, (filename, image) -> ALERT_STATE, (image, enabled) -> enabled.equals(-1) ? image :new byte[]{}) .filter((key, value) -> value.length > 0); Kafka Streams application 22
  14. @fbascheper #Devoxx #Busted val imageStream = filteredSourceStream .mapValues((readOnlyKey, value) ->

    new SerializableImage(readOnlyKey, ByteBuffer.wrap(value))); Kafka Streams application 23
  15. @fbascheper #Devoxx #Busted val imageStream = filteredSourceStream .mapValues((readOnlyKey, value) ->

    new SerializableImage(readOnlyKey, ByteBuffer.wrap(value))); val telegramPhotoMessage = imageStream .mapValues((readOnlyKey, value) -> { val caption = TensorFlowMatcher.matchImage(labels, graphDef, image); return TelegramMessageMapper.photoMessage(image, caption); }); Kafka Streams application 24
  16. @fbascheper #Devoxx #Busted val imageStream = filteredSourceStream .mapValues((readOnlyKey, value) ->

    new SerializableImage(readOnlyKey, ByteBuffer.wrap(value))); val telegramPhotoMessage = imageStream .mapValues((readOnlyKey, value) -> { val caption = TensorFlowMatcher.matchImage(labels, graphDef, image); return TelegramMessageMapper.photoMessage(image, caption); }); telegramPhotoMessage.to(SINK_TOPIC_BURGLAR_ALERTS, Produced.with(Serdes.String(), telegramMessageSerde)); Kafka Streams application 25
  17. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video

    Kafka Connectors Demo Wrap-up 26 https://pixabay.com/p-683065/
  18. @fbascheper #Devoxx #Busted TensorFlow - ‘ML for everyone’ Familiar with

    TensorFlow? Used TensorFlow before? Which API’s? Python Java Keras Other 27 https://commons.wikimedia.org/wiki/File:Polling_Station_sign_2017.jpg
  19. @fbascheper #Devoxx #Busted public static String matchImage(List<String> tfLabels, byte[] tfGraphDef,

    SerializableImage image) { byte[] imageBytes = image.getImageData().array(); TensorFlow interface 32
  20. @fbascheper #Devoxx #Busted public static String matchImage(List<String> tfLabels, byte[] tfGraphDef,

    SerializableImage image) { byte[] imageBytes = image.getImageData().array(); try (Tensor tensor = constructAndExecuteGraph(imageBytes)) { float[] labelProbability = executeInceptionGraph(tfGraphDef, tensor); int bestLabelIdx = maxIndex(labelProbabilities); TensorFlow interface 33
  21. @fbascheper #Devoxx #Busted public static String matchImage(List<String> tfLabels, byte[] tfGraphDef,

    SerializableImage image) { byte[] imageBytes = image.getImageData().array(); try (Tensor tensor = constructAndExecuteGraph(imageBytes)) { float[] labelProbability = executeInceptionGraph(tfGraphDef, tensor); int bestLabelIdx = maxIndex(labelProbabilities); String imageClassification = tfLabels.get(bestLabelIdx); float imageProbability = labelProbabilities[bestLabelIdx] * 100f; return String.format("BEST MATCH for image %s was %s (%.2f%% likely)”, image.getName(), imageClassification, imageProbability); } } TensorFlow interface 34
  22. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video

    Kafka Connectors Demo Wrap-up 36 https://pixabay.com/p-683065/
  23. @fbascheper #Devoxx #Busted String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter =

    new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); Acquiring images - take 1… 38
  24. @fbascheper #Devoxx #Busted String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter =

    new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); frameGrabber.setFrameRate(5); frameGrabber.setImageWidth(640); frameGrabber.start(); Acquiring images - take 1… 39
  25. @fbascheper #Devoxx #Busted String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter =

    new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); frameGrabber.setFrameRate(5); frameGrabber.setImageWidth(640); frameGrabber.start(); Frame frame = frameGrabber.grabImage(); BufferedImage image = frameConverter.convert(frame); ImageIO.write(image, "png", new File("./video-frame.png")); Acquiring images - take 1… 40
  26. @fbascheper #Devoxx #Busted Acquiring images - take 2 • KISS

    • Use out-of-the box support - Motion detection - Movie snapshots • Auto upload using FTP(S) 42 NO CODING REQUIRED
  27. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video

    Kafka Connectors Demo Wrap-up 44 https://pixabay.com/p-683065/
  28. @fbascheper #Devoxx #Busted Kafka Connectors 1. FTP source (FOSS) ‣

    Loads images into Kafka 2. REST source (FOSS) ‣ Loads lock status changes into Kafka 3. Telegram sink ‣ Newly developed for this project ‣ Sends burglar alerts 45
  29. @fbascheper #Devoxx #Busted https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", ,

    "config": { "connector.class": "com.….connect.ftp.source.FtpSourceConnector", Configuring Kafka Connect 46
  30. @fbascheper #Devoxx #Busted https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", ,

    "config": { "connector.class": "com.….connect.ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": "com. … .connect.converters.ByteArrayConverter" Configuring Kafka Connect 47
  31. @fbascheper #Devoxx #Busted https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", ,

    "config": { "connector.class": "com.….connect.ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": "com. … .connect.converters.ByteArrayConverter" , "connect.ftp.refresh": "PT1M" , "connect.ftp.file.maxage": "P14D" , "connect.ftp.fileconverter": "com. … .connect.ftp.source.SimpleFileConverter" , "connect.ftp.monitor.update": "/foscam-download/:burglar-alerts-camera-ftp-topic" , "connect.ftp.protocol": "sftp" } } Configuring Kafka Connect 48
  32. @fbascheper #Devoxx #Busted public class ImageGrayScaleConverter implements Converter, HeaderConverter {

    @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); Image grayscale converter 50
  33. @fbascheper #Devoxx #Busted public class ImageGrayScaleConverter implements Converter, HeaderConverter {

    @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); val resized = ImageSizeTransformer.INSTANCE.transform(convertedImg); val grayscale = GrayScaleTransformer.INSTANCE.transform(resized); Image grayscale converter 51
  34. @fbascheper #Devoxx #Busted public class ImageGrayScaleConverter implements Converter, HeaderConverter {

    @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); val resized = ImageSizeTransformer.INSTANCE.transform(convertedImg); val grayscale = GrayScaleTransformer.INSTANCE.transform(resized); val bos = new ByteArrayOutputStream(); ImageIO.write(grayscale, "jpg", bos); return bos.toByteArray(); }} Image grayscale converter 52
  35. @fbascheper #Devoxx #Busted Kafka Telegram connector • Newly developed for

    this project (FOSS @ GitHub) • Can be used to send multiple message types ‣ Text ‣ Photo ‣ Videos • Dependencies ‣ Apache Avro ‣ Telegram Bot API 53
  36. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video

    Kafka Connectors Demo Wrap-up 56 https://pixabay.com/p-683065/
  37. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video

    Kafka Connectors Demo Wrap-up 58 https://pixabay.com/p-683065/
  38. @fbascheper #Devoxx #Busted Useful links • Source code ‣ https://github.com/fbascheper/kafka-devoxx-2018

    • Kafka Connect Telegram plugin ‣ https://github.com/fbascheper/kafka-connect-telegram ‣ https://github.com/fbascheper/kafka-connect-telegram-avro-model ‣ https://github.com/fbascheper/kafka-connect-storage-converters • Kafka connectors overview ‣ https://www.confluent.io/product/connectors-repository 61