© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Dennis Kieselhorst, Sr. Solutions Architect Machine Learning in Java: Die Visual Recognition API Java Specification Request (JSR) 381 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Table of contents • Intro Machine Learning, Artificial Intelligence and Deep Learning • Visual recognition • Why in Java? • Java Specification Request (JSR) 381 • Reference implementations • Code samples • Alternatives 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The reach of ML is growing By 2021, global spending on AI and cognitive technologies will exceed $50 billion —IDC INCREASED SPENDING By the end of 2024, 75% of enterprises will shift from piloting to operationalizing AI —Gartner FROM PILOTING TO OPERATIONALIZING AI TRANSFORMATION 57% said that AI would transform their organization in the next three years —Deloitte 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Artificial Intelligence, Machine and Deep learning Artificial intelligence Subset of AI that uses machines to search for patterns in data to build logic models automatically Subset of ML composed of deeply multi-layered neural networks that perform tasks like speech and image recognition 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Visual recognition 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Source: https://github.com/awslabs/djl/blob/master/examples/src/test/resources/dog-cat.jpg 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Source: https://github.com/awslabs/djl/blob/master/examples/docs/img/cat_dog_detected.jpg 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Source: https://github.com/awslabs/djl/blob/master/examples/docs/img/detected-dogs.jpg 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Source: https://github.com/awslabs/djl/blob/master/examples/docs/img/detected-dog_bike_car.png 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Why in Java? • Looking at indexes (like TIOBE and PYPL) Java is trending a bit down but still one of the most popular languages • Great language with a huge ecosystem with millions of developers • Existing APIs weren‘t Java-friendly (complex, „C flavor“) Source: https://wiki.openjdk.java.net/display/duke/Gallery 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Java Specification Request (JSR) 381: Goals Goals from the Java Visual Recognition Specification doc (version 1.0): • Describe a standard, easy-to-use and flexible set of high-level APIs • Offer high-level abstractions for sustainable development of ML products and services • Have well-defined APIs essential for robust system architecture • Offer ease of development and portability • Provide thorough information to help create alternative implementations • Offer the ability to create custom Classifiers in addition to pre-trained Classifiers Source: https://jcp.org/en/jsr/detail?id=381 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Java Specification Request (JSR) 381: Architecture Source: https://jcp.org/en/jsr/detail?id=381 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Reference implementations • Deep Netts • DeepJavaLibrary (DJL) • Experimental: • Deeplearning4j • Open Intelligent Multimedia Analysis for Java (OpenIMAJ) • IBM Watson 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Code to train a model ImageClassifier classifier = NeuralNetImageClassifier.builder() .inputClass(BufferedImage.class) .imageHeight(128).imageWidth(128) .labelsFile(new File("labels.txt")) .trainingFile(new File("train.txt")) .networkArchitecture(new File("arch.json")) .exportModel(Paths.get("trained_model.dnet")) .maxError(0.03f) .maxEpochs(1000) .learningRate(0.01f) .build(); BufferedImage image = ImageIO.read(new File("another_cat.png")); Map results = classifier.classify(image); 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Code to use an existing model ImageClassifier classifier = NeuralNetImageClassifier.builder() .inputClass(BufferedImage.class) .imageHeight(128).imageWidth(128) .importModel(Paths.get("trained_model.dnet")) .build(); BufferedImage image = ImageIO.read(new File("another_cat.png")); Map results = classifier.classify(image); More sample code: https://github.com/JavaVisRec/visrec-api/wiki/Getting-Started-Guide#Examples 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What about Spring Boot? • Spring Boot Starter project for DJL Deep Java Library, not specifically for JSR 381 • Simplifies dependency management • Allows auto configuration for use cases like object detection (see application.yml sample) djl: # Define application type application-type: OBJECT_DETECTION # Define input data type, a model may accept multiple input data type input-class: java.awt.image.BufferedImage # Define output data type, a model may generate different out put output-class: ai.djl.modality.cv.output.DetectedObjects arguments: threshold: 0.5 # Display all results with probability of 0.5 and above Blog post: https://aws.amazon.com/blogs/opensource/adopting-machine- learning-in-your-microservices-with-djl-deep-java-library-and-spring-boot/ 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Alternatives 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. VISION SPEECH TEXT SEARCH CHATBOTS PERSONALIZATION FORECASTING FRAUD DEVELOPMENT CONTACT CENTERS Ground Truth AWS Marketplace for ML Neo Augmented AI Built-in algorithms Notebooks Experiments Processing Model training and tuning Debugger Autopilot Model hosting Model Monitor Deep Learning AMIs & Containers GPUs & CPUs Elastic Inference Inferentia FPGA Amazon Rekognition Amazon Polly Amazon Transcribe +Medical Amazon Lex Amazon Personalize Amazon Forecast Amazon Comprehend +Medical AI SERVICES ML SERVICES ML FRAMEWORKS & INFRASTRUCTURE Amazon Textract Amazon Kendra Contact Lens For Amazon Connect SageMaker Studio IDE Amazon CodeGuru Amazon SageMaker DeepGraphLibrary Amazon Fraud Detector Amazon Translate The AWS ML Stack Broadest and most complete set of machine learning capabilities 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. The Amazon ML stack: Broadest & deepest set of capabilities ML FRAMEWORKS & INFRASTRUCTURE AI SERVICES Vision | Documents | Speech | Language | Chatbots | Forecasting | Recommendations | Fraud detection | Enterprise Search | Code Review ML SERVICES Data labeling | Pre-built algorithms & notebooks | One-click training and deployment Build, train, and deploy machine learning models fast Easily add intelligence to applications without machine learning skills Flexibility & choice, highest-performing infrastructure Support for ML frameworks | Compute options purpose-built for ML 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. VISION SPEECH TEXT SEARCH CHATBOTS PERSONALIZATION FORECASTING FRAUD DEVELOPMENT CONTACT CENTERS Ground Truth AWS Marketplace for ML Neo Augmented AI Built-in algorithms Notebooks Experiments Processing Model training and tuning Debugger Autopilot Model hosting Model Monitor Deep Learning AMIs & Containers GPUs & CPUs Elastic Inference Inferentia FPGA Amazon Rekognition Amazon Polly Amazon Transcribe +Medical Amazon Lex Amazon Personalize Amazon Forecast Amazon Comprehend +Medical AI SERVICES ML SERVICES ML FRAMEWORKS & INFRASTRUCTURE Amazon Textract Amazon Kendra Contact Lens For Amazon Connect SageMaker Studio IDE Amazon CodeGuru Amazon SageMaker DeepGraphLibrary Amazon Fraud Detector Amazon Translate The AWS ML Stack: Amazon Rekognition Automate your image and video analysis using machine learning 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Labels (object, scenes, and activities) Face detection and analysis Text in image Unsafe image and video detection Pathing Face search Real-time video analysis Celebrity recognition Amazon Rekognition Image and video analysis 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Rekognition Deep-Learning-Based Image and Video Analysis 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Code (with AWS SDK for Java v2) API: https://docs.aws.amazon.com/rekognition/latest/dg/API_DetectLabels.html SDK code: https://docs.aws.amazon.com/code-samples/latest/catalog/javav2-rekognition-src- main-java-com-example-rekognition-DetectLabels.java.html software.amazon.awssdk bom 2.15.16 pom import software.amazon.awssdk rekognition RekognitionClient rekognitionClient = RekognitionClient.builder() .region(Region.EU_CENTRAL_1) .build(); InputStream sourceStream = new FileInputStream( new File(sourceImageName)); SdkBytes sourceBytes = SdkBytes.fromInputStream(sourceStream); Image sourceImage = Image.builder() .bytes(sourceBytes) .build(); DetectLabelsRequest detectLabelsRequest = DetectLabelsRequest.builder() .image(sourceImage) .maxLabels(10) .build(); DetectLabelsResponse labelsResponse = rekognitionClient.detectLabels(detectLabelsRequest); 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Blog post: https://aws.amazon.com/blogs/machine-learning/automatically-detecting-personal- protective-equipment-on-persons-in-images-using-amazon-rekognition/ API: https://docs.aws.amazon.com/rekognition/latest/dg/API_DetectProtectiveEquipment.html 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Summary • The current spec is a starting point but needs further refinement and evolution as stated in the final notes of the first release: • The long term vision of this JSR and specification is to provide a standard API to build machine learning applications using Java. • One of our major goals is to allow Java application developers to incorporate machine learning features without necessarily becoming experts in machine learning. • Please try it out and provide feedback using the GitHub issue tracker https://github.com/JavaVisRec or the mailing list https://groups.io/g/visrec Source: https://jcp.org/en/jsr/detail?id=381 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Q&A 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Recommend to read: Dive into Deep Learning (D2L) • 150+ runnable Jupyter Notebooks from model architectures to applications (CV, NLP, etc.) • Adopted as a textbook or reference book at UC Berkeley, CMU, MIT, and 70+ universities worldwide • Wide theoretical coverage: statistics, optimization, machine learning basics, GPU parallel training, etc. Dive Into Deep Learning is an excellent text on deep learning and deserves attention from anyone who wants to learn why deep learning has ignited the AI revolution – the most powerful technology force of our time. --- Jensen Huang, CEO of NVIDIA https://d2l.djl.ai 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you! Dennis Kieselhorst, Sr. Solutions Architect [email protected] Feedback form: https://amzn.to/35cfKWx 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Appendix 

© 2020, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon’s machine learning innovation at scale 4,000 products per minute sold on Amazon.com 1.6M packages every day Billions of Alexa interactions each week First Prime Air Delivery on Dec. 7, 2016