$30 off During Our Annual Pro Sale. View Details »

Bust the burglars: Machine Learning with TensorFlow and Apache Kafka

Bust the burglars: Machine Learning with TensorFlow and Apache Kafka

The integration of event streams is becoming increasingly complex in the world of sensors, media streams and the Internet of Things. In this session I will discuss how `Apache Kafka` and `TensorFlow` can be used to process real time video events and send alerts when a potential burglary is detected.

Specifically, I'll discuss how `Kafka Connect` and `Kafka Streams` can be used to read images from a media stream and how this can be integrated with TensorFlow to analyse the resulting event streams.

Although the use case presented is based on home security, the same technology used to analyse event streams can be applied to other use cases, as I will demonstrate during this session.

Frederieke Scheper

November 24, 2018
Tweet

More Decks by Frederieke Scheper

Other Decks in Technology

Transcript

  1. @fbascheper #Devoxx #Busted Bust the burglars - ML with TensorFlow

    and Apache Kafka Erik-Berndt Scheper
  2. @fbascheper #Devoxx #Busted Let’s meet • I’m delighted to be

    here in Ukraine! • Java Architect @ Ordina JTech • Contracted by the Dutch Tax administration • Passion for ‣ Big Data technologies ‣ Machine learning 2
  3. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video

    Kafka Connectors Demo Wrap-up 3 https://pixabay.com/p-683065/
  4. @fbascheper #Devoxx #Busted https://nl.wikipedia.org/wiki/Bartje#/media/File:Standbeeld-bartje.jpg Introduction 4

  5. @fbascheper #Devoxx #Busted Statistics https://reolink.com/home-burglary-crime-statistics/ 5 Lack of evidence! Why?

  6. @fbascheper #Devoxx #Busted https://upload.wikimedia.org/wikipedia/commons/5/56/Haren_Rijksstraatweg_377.JPG Entry points 6 8% 4% 9%

    22% 23% 34%
  7. @fbascheper #Devoxx #Busted https://www.flickr.com/photos/takomabibelot/2134214940 7

  8. @fbascheper #Devoxx #Busted Components 8

  9. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video

    Kafka Connectors Demo Wrap-up 9 https://pixabay.com/p-683065/
  10. @fbascheper #Devoxx #Busted Event streams • Input - Images from

    IP camera(s) - Lock status changes • Output - Burglar Alerts 10
  11. @fbascheper #Devoxx #Busted Event processing 11 camera image analyse with

    TensorFlow burglary? alerts enabled? STATELESS …
  12. @fbascheper #Devoxx #Busted Event processing 12 poll lock REST API

    alerts enabled! all locks alerts disabled (flat)map response to individual locks
  13. @fbascheper #Devoxx #Busted Event processing 13 poll lock REST API

    alerts enabled! all locks alerts disabled (flat)map response to individual locks STATE ?
  14. @fbascheper #Devoxx #Busted val builder = new StreamsBuilder(); Kafka Streams

    application 14
  15. @fbascheper #Devoxx #Busted val builder = new StreamsBuilder(); val cameraSourceStream

    = builder.stream( SOURCE_TOPIC_CAMERA_IMAGES, Consumed.with(Serdes.String(), Serdes.ByteArray())); Kafka Streams application 15 NOW… DISCARD IMAGES WHEN ALERTS ARE DISABLED
  16. @fbascheper #Devoxx #Busted << Attribution >> Intermezzo 16 K-Table

  17. @fbascheper #Devoxx #Busted STREAM (changelog) (“alert-enabled”, 0) ⊗ (“alert-enabled”, 0)

    ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, 0) ⊗ KStream - KTable duality 17 (no change) (no change) KTABLE | alert-enabled | 0 | ⊗ ⊗ ⊗ | alert-enabled | -1 | ⊗ ⊗ ⊗ | alert-enabled | 0 |
  18. @fbascheper #Devoxx #Busted STREAM (changelog) (“alert-enabled”, 0) ⊗ (“alert-enabled”, 0)

    ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, -1) ⊗ (“alert-enabled”, 0) ⊗ KStream - KTable duality 18 (no change) (no change) KTABLE | alert-enabled | 0 | ⊗ ⊗ ⊗ | alert-enabled | -1 | ⊗ ⊗ ⊗ | alert-enabled | 0 | STATE !
  19. @fbascheper #Devoxx #Busted val nukiRestApiStream = builder .stream(SOURCE_TOPIC_SMARTLOCK_POLL, Consumed.with(Serdes.String(), Serdes.String()))

    .mapValues(SmartLockMapper::toNukiResponse) .map(SmartLockMapper::alertingEnabledMapper) .to(BURGLAR_ALERTING_STATE_TOPIC, Produced.with(Serdes.String(), Serdes.Integer())); Adding state - part 1 19 RESULT: A CHANGELOG STREAM WITH (“alert-enabled”, 0) (“alert-enabled”, 0) (“alert-enabled”, -1) public static KeyValue<~> alertingEnabledMapper(String key, NukiResponse rp) { boolean disabled = rp.getLocks().stream() .anyMatch(lock -> LockStates.UNLOCKED.equals(lock.getState())); return KeyValue.pair(ALERT_STATE, disabled ? 0 : -1); }
  20. @fbascheper #Devoxx #Busted /* Build global KTable from changelog stream

    BURGLAR_ALERTING_STATE_TOPIC */ val matd = Materialized.<String, Integer, KeyValueStore<Bytes, byte[]>> as(BURGLAR_ALERTING_STATE_STORE) .withKeySerde(Serdes.String()) .withValueSerde(Serdes.Integer()); val alertingEnabledGlobalKTable = builder .globalTable(BURGLAR_ALERTING_STATE_TOPIC, matd); Adding state - part 2 20 RESULT: A TABLE WITH | alert-enabled | 0 or -1 |
  21. @fbascheper #Devoxx #Busted << Attribution >> Let’s continue … 21

    K-Table
  22. @fbascheper #Devoxx #Busted val builder = new StreamsBuilder(); val cameraSourceStream

    = builder.stream( SOURCE_TOPIC_CAMERA_IMAGES, Consumed.with(Serdes.String(), Serdes.ByteArray())); val filteredSourceStream = cameraSourceStream .leftJoin( alertingEnabledGlobalKTable, (filename, image) -> ALERT_STATE, (image, enabled) -> enabled.equals(-1) ? image :new byte[]{}) .filter((key, value) -> value.length > 0); Kafka Streams application 22
  23. @fbascheper #Devoxx #Busted val imageStream = filteredSourceStream .mapValues((readOnlyKey, value) ->

    new SerializableImage(readOnlyKey, ByteBuffer.wrap(value))); Kafka Streams application 23
  24. @fbascheper #Devoxx #Busted val imageStream = filteredSourceStream .mapValues((readOnlyKey, value) ->

    new SerializableImage(readOnlyKey, ByteBuffer.wrap(value))); val telegramPhotoMessage = imageStream .mapValues((readOnlyKey, value) -> { val caption = TensorFlowMatcher.matchImage(labels, graphDef, image); return TelegramMessageMapper.photoMessage(image, caption); }); Kafka Streams application 24
  25. @fbascheper #Devoxx #Busted val imageStream = filteredSourceStream .mapValues((readOnlyKey, value) ->

    new SerializableImage(readOnlyKey, ByteBuffer.wrap(value))); val telegramPhotoMessage = imageStream .mapValues((readOnlyKey, value) -> { val caption = TensorFlowMatcher.matchImage(labels, graphDef, image); return TelegramMessageMapper.photoMessage(image, caption); }); telegramPhotoMessage.to(SINK_TOPIC_BURGLAR_ALERTS, Produced.with(Serdes.String(), telegramMessageSerde)); Kafka Streams application 25
  26. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video

    Kafka Connectors Demo Wrap-up 26 https://pixabay.com/p-683065/
  27. @fbascheper #Devoxx #Busted TensorFlow - ‘ML for everyone’ Familiar with

    TensorFlow? Used TensorFlow before? Which API’s? Python Java Keras Other 27 https://commons.wikimedia.org/wiki/File:Polling_Station_sign_2017.jpg
  28. @fbascheper #Devoxx #Busted https://medium.com/marketing-and-entrepreneurship/10-companies-using-machine-learning-in-cool-ways-887c25f913c3 28

  29. @fbascheper #Devoxx #Busted https://medium.com/@tifa2up/image-classification-using-deep-neural-networks-a-beginner-friendly-approach-using-tensorflow-94b0a090ccd4 Image classification 29

  30. @fbascheper #Devoxx #Busted Binary image classification 30 NOT A BURGLAR

    ! BURGLAR ⚠
  31. @fbascheper #Devoxx #Busted https://www.flickr.com/photos/111692634@N04/11406964665 Oops … 31 SOLUTION: USE A

    (PRE-TRAINED) MULTI-CLASS MODEL
  32. @fbascheper #Devoxx #Busted public static String matchImage(List<String> tfLabels, byte[] tfGraphDef,

    SerializableImage image) { byte[] imageBytes = image.getImageData().array(); TensorFlow interface 32
  33. @fbascheper #Devoxx #Busted public static String matchImage(List<String> tfLabels, byte[] tfGraphDef,

    SerializableImage image) { byte[] imageBytes = image.getImageData().array(); try (Tensor tensor = constructAndExecuteGraph(imageBytes)) { float[] labelProbability = executeInceptionGraph(tfGraphDef, tensor); int bestLabelIdx = maxIndex(labelProbabilities); TensorFlow interface 33
  34. @fbascheper #Devoxx #Busted public static String matchImage(List<String> tfLabels, byte[] tfGraphDef,

    SerializableImage image) { byte[] imageBytes = image.getImageData().array(); try (Tensor tensor = constructAndExecuteGraph(imageBytes)) { float[] labelProbability = executeInceptionGraph(tfGraphDef, tensor); int bestLabelIdx = maxIndex(labelProbabilities); String imageClassification = tfLabels.get(bestLabelIdx); float imageProbability = labelProbabilities[bestLabelIdx] * 100f; return String.format("BEST MATCH for image %s was %s (%.2f%% likely)”, image.getName(), imageClassification, imageProbability); } } TensorFlow interface 34
  35. @fbascheper #Devoxx #Busted https://www.flickr.com/photos/59195512@N00/88683459 Forgot something? 35

  36. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video

    Kafka Connectors Demo Wrap-up 36 https://pixabay.com/p-683065/
  37. @fbascheper #Devoxx #Busted https://www.flickr.com/photos/16210667@N02/25300692725 Streaming video ? 37

  38. @fbascheper #Devoxx #Busted String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter =

    new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); Acquiring images - take 1… 38
  39. @fbascheper #Devoxx #Busted String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter =

    new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); frameGrabber.setFrameRate(5); frameGrabber.setImageWidth(640); frameGrabber.start(); Acquiring images - take 1… 39
  40. @fbascheper #Devoxx #Busted String url = "rtsp://usr:pwd@host:443/videoSub"; Java2DFrameConverter frameConverter =

    new Java2DFrameConverter(); FFmpegFrameGrabber frameGrabber = new FFmpegFrameGrabber(url); frameGrabber.setFrameRate(5); frameGrabber.setImageWidth(640); frameGrabber.start(); Frame frame = frameGrabber.grabImage(); BufferedImage image = frameConverter.convert(frame); ImageIO.write(image, "png", new File("./video-frame.png")); Acquiring images - take 1… 40
  41. @fbascheper #Devoxx #Busted Acquiring images - take 1… 41 WE’LL

    NEED SOMETHING BETTER THAN THIS ⚠
  42. @fbascheper #Devoxx #Busted Acquiring images - take 2 • KISS

    • Use out-of-the box support - Motion detection - Movie snapshots • Auto upload using FTP(S) 42 NO CODING REQUIRED
  43. @fbascheper #Devoxx #Busted https://cdn.pixabay.com/photo/2016/11/22/16/04/dive-1849534_640.jpg One more thing … 43 FEEDING

    KAFKA
  44. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video

    Kafka Connectors Demo Wrap-up 44 https://pixabay.com/p-683065/
  45. @fbascheper #Devoxx #Busted Kafka Connectors 1. FTP source (FOSS) ‣

    Loads images into Kafka 2. REST source (FOSS) ‣ Loads lock status changes into Kafka 3. Telegram sink ‣ Newly developed for this project ‣ Sends burglar alerts 45
  46. @fbascheper #Devoxx #Busted https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", ,

    "config": { "connector.class": "com.….connect.ftp.source.FtpSourceConnector", Configuring Kafka Connect 46
  47. @fbascheper #Devoxx #Busted https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", ,

    "config": { "connector.class": "com.….connect.ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": "com. … .connect.converters.ByteArrayConverter" Configuring Kafka Connect 47
  48. @fbascheper #Devoxx #Busted https://github.com/Eneco/kafka-connect-ftp ==> https://github.com/Landoop/stream-reactor { "name": "burglar-alerts-camera-ftp-source", ,

    "config": { "connector.class": "com.….connect.ftp.source.FtpSourceConnector", , "key.converter": "org.apache.kafka.connect.storage.StringConverter" , "value.converter": "com. … .connect.converters.ByteArrayConverter" , "connect.ftp.refresh": "PT1M" , "connect.ftp.file.maxage": "P14D" , "connect.ftp.fileconverter": "com. … .connect.ftp.source.SimpleFileConverter" , "connect.ftp.monitor.update": "/foscam-download/:burglar-alerts-camera-ftp-topic" , "connect.ftp.protocol": "sftp" } } Configuring Kafka Connect 48
  49. @fbascheper #Devoxx #Busted Applying custom value-converters 49 INFRARED DAYLIGHT HOW

    TO HANDLE THIS ? ⚠
  50. @fbascheper #Devoxx #Busted public class ImageGrayScaleConverter implements Converter, HeaderConverter {

    @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); Image grayscale converter 50
  51. @fbascheper #Devoxx #Busted public class ImageGrayScaleConverter implements Converter, HeaderConverter {

    @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); val resized = ImageSizeTransformer.INSTANCE.transform(convertedImg); val grayscale = GrayScaleTransformer.INSTANCE.transform(resized); Image grayscale converter 51
  52. @fbascheper #Devoxx #Busted public class ImageGrayScaleConverter implements Converter, HeaderConverter {

    @Override public byte[] fromConnectData(String topic, Schema schema, Object value) { val source = ImageIO.read(new ByteArrayInputStream(data)); val convertedImg = new BufferedImage(source.getWidth(), source.getHeight(), BufferedImage.TYPE_3BYTE_BGR); convertedImg.getGraphics().drawImage(source, 0, 0, Color.BLACK, null); convertedImg.getGraphics().dispose(); val resized = ImageSizeTransformer.INSTANCE.transform(convertedImg); val grayscale = GrayScaleTransformer.INSTANCE.transform(resized); val bos = new ByteArrayOutputStream(); ImageIO.write(grayscale, "jpg", bos); return bos.toByteArray(); }} Image grayscale converter 52
  53. @fbascheper #Devoxx #Busted Kafka Telegram connector • Newly developed for

    this project (FOSS @ GitHub) • Can be used to send multiple message types ‣ Text ‣ Photo ‣ Videos • Dependencies ‣ Apache Avro ‣ Telegram Bot API 53
  54. @fbascheper #Devoxx #Busted << Attribution >> Kafka Telegram connector 54

  55. @fbascheper #Devoxx #Busted Kafka Telegram - photo message 55

  56. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video

    Kafka Connectors Demo Wrap-up 56 https://pixabay.com/p-683065/
  57. Demo @fbascheper #Devoxx #Busted 57

  58. @fbascheper #Devoxx #Busted Agenda Introduction Event processing TensorFlow Streaming video

    Kafka Connectors Demo Wrap-up 58 https://pixabay.com/p-683065/
  59. @fbascheper #Devoxx #Busted https://www.flickr.com/photos/30478819@N08/42695313601 Wrap-up 59

  60. @fbascheper #Devoxx #Busted https://core.telegram.org/file/811140184/1/5YJxx-rostA/ad3f74094485fb97bd and https://pixabay.com/en/waterfall-water-source-stream-1746227/ Kafka Telegram - planned

    features 60
  61. @fbascheper #Devoxx #Busted Useful links • Source code ‣ https://github.com/fbascheper/kafka-devoxx-2018

    • Kafka Connect Telegram plugin ‣ https://github.com/fbascheper/kafka-connect-telegram ‣ https://github.com/fbascheper/kafka-connect-telegram-avro-model ‣ https://github.com/fbascheper/kafka-connect-storage-converters • Kafka connectors overview ‣ https://www.confluent.io/product/connectors-repository 61
  62. @fbascheper #Devoxx #Busted http://www.thebluediamondgallery.com/handwriting/q/questions.html 62

  63. @fbascheper #Devoxx #Busted 63

  64. @fbascheper #Devoxx #Busted https://commons.wikimedia.org/wiki/File:Thank-you-word-cloud.jpg 64