Talk: AI in the battle against fakes

Counterfeit products have been a longstanding and growing pain for companies. In addition to impacting company revenue, they damage brand reputation and customer confidence. Our company was asked to build a solution for a global electronics brand that can identify counterfeit products by just taking one picture on a smartphone.

In this session we zoom into the building blocks that make this AI solution work. We’ll find out that there is much more to it than just training a convolutional neural network. We look at challenges like how to manage and monitor the AI model and how to build and improve the model in a way that fits your DevOps production chain.

Learn how we used Azure Functions, Cosmos DB and Docker to build a solid foundation. See how we used Azure Machine Learning service to train the models. And find out how we used Azure DevOps to control, build and deploy this state-of-the-art solution.

AI used to only be in the hands of data science PhDs. Now any developer can integrate this exciting technology into their next project. Join us to find out how.

Transcript

1. Henk Boelman Cloud Advocate @ Microsoft AI in the battle against fakes HenkBoelman.com @hboelman
2. Agenda Assignment & challenges Solution architecture Basic models with Custom Vision Advanced models with AMLS Recap and lessons learned
3. Assignment and challenges
4. Ask a sharp question
5. Is it a genuine or a fake package?
6. Collect the data
7. Can we have the data please? Sorry we don’t have…. You are joking right?
8. 1 2 3
11. Break up the project and look at the smaller pieces
12. Solution Architecture Application flow and Azure design
13. Application AI Engine Management Front End
14. Ingredients Azure Durable Functions Docker Containers Azure Kubernetes Service Cosmos DB Azure Blob Storage Umbraco AMLS Cognitive Services Azure DevOps API Management
15. Umbraco config Create a product Tag product Upload Front image Run AI on image Push to engine
16. Umbraco config
17. AI Engine flow Photo of product Pre-Process image Match to Product in DB Text AI Logo AI Icon AI End evaluation
18. Sophisticated pretrained models To simplify solution development Azure Databricks Machine Learning VMs Popular frameworks To build advanced deep learning solutions TensorFlow Keras Pytorch Onnx Azure Machine Learning Language Speech … Azure Search Vision On-premises Cloud Edge Productive services To empower data science and development teams Powerful infrastructure To accelerate deep learning Flexible deployment To deploy and manage models on intelligent cloud and edge Machine Learning on Azure Cognitive Services
19. Icon detection and validation Application flow and Azure design
20. Object detection Classification Classification { Icon: Car Genuine: False, Score: 95% }, { Icon: Truck Genuine: True, Score: 85% } Input Output Genuine: 2% | Fake: 95% Genuine: 85% | Fake: 3% Icon validator API
21. Label images Export Train Model Deploy Model Test Model Visual Object Tagging Tool Custom Vision Service Icon detection Run test-set against model
22. Label images Export Train Model Deploy Model Test Model Custom Vision Service Icon verification Custom Vision Service API Images in 2 folders Genuine / Fake Run test-set against model Image augmentation
23. Custom Vision Service ML lifecycle Data source Icon Validator API
24. Custom Vision Service An easy-to-use tool for creating your own custom image classifier
29. Demo: Custom Vision Service
30. Demo: Aka.ms/AI4DEV01-CustomVision @hboelman Henk.boelman@microsoft.com
31. Logo classification A deep neural network to classify the Philips logo.
32. Computer Vision API Cut out PHILIPS Pre- Process Predict image Return Result Logo classification Icon Validator API
33. Label images Pre process Train Model Deploy Model Test Model Azure Machine Learning Service Logo classification Images in 2 folders Genuine / Fake Run test-set against model Image augmentation
35. Azure Machine Learning studio A fully-managed cloud service that enables you to easily build, deploy, and share predictive analytics solutions.
36. What is Azure Machine Learning Service? Set of Azure Cloud Services Python SDK ✓Prepare Data ✓Build Models ✓Train Models ✓Manage Models ✓Track Experiments ✓Deploy Models That enables you to:
37. Programming Algorithm Data Answers
38. Machine Learning Algorithm Data Answers
39. Machine Learning Model Data Answers
40. Machine Learning Model Data Answers
41. Machine Learning Predictions Data Model Data Answers
42. Which Simpson?
43. Is it Marge or Homer? https://github.com/hnky/dataset-lego-figures
44. Prepare your environment Experiment with your model & data Deploy Your model into production
45. Step 1: Prepare your environment
46. Setup your environment VS Code Azure Notebooks Azure Portal
48. Datasets – registered, known data sets Experiments – Training runs Pipelines – Workflows runs Models – Registered, versioned models Endpoints: Real-time Endpoints – Deployed model endpoints Pipeline Endpoints – Training workflows Compute – Managed compute Datastores – Connections to data Azure Machine Learning Service
49. Create a workspace ws = Workspace.create( name='<NAME>', subscription_id='<SUBSCRIPTION ID>', resource_group='<RESOURCE GROUP>', location='westeurope') ws.write_config() ws = Workspace.from_config() Create a workspace
50. Create Compute cfg = AmlCompute.provisioning_configuration( vm_size='STANDARD_NC6', min_nodes=1, max_nodes=6) cc = ComputeTarget.create(ws, '<NAME>', cfg) Create a workspace Create compute
52. Demo: Setup your workspace Create a workspace Create compute Setup storage
53. Step 1 Prepare your environment Create a workspace Create compute Setup storage
54. Step 2: Experiment with your model & data
55. Create an experiment exp = Experiment(workspace=ws, name=“<ExperimentName>”) Create an Experiment
56. Create a training file Create an Experiment Create a training file
57. Create an estimator params = {'--data-folder': ws.get_default_datastore().as_mount()} estimator = PyTorch( source_directory = script_folder, script_params = params, compute_target = computeCluster, entry_script = 'train.py’, use_gpu = True, conda_packages = ['opencv’], framework_version=‘1.3') Create an Experiment Create a training file Create an estimator
58. Submit the experiment to the cluster run = exp.submit(estimator) RunDetails(run).show() Create an Experiment Create a training file Submit to the AI cluster Create an estimator
59. Create an Experiment Create a training file Submit to the AI cluster Create an estimator Demo: Creating and run an experiment
60. Azure Notebook Compute Target Experiment Docker Image Data store 1. Snapshot folder and send to experiment 2. create docker image 3. Deploy docker and snapshot to compute 4. Mount datastore to compute 6. Stream stdout, logs, metrics 5. Launch the script 7. Copy over outputs
61. Register the model model = run.register_model( model_name='SimpsonsAI', model_path='outputs') Create an Experiment Create a training file Submit to the AI cluster Create an estimator Register the model
62. Create an Experiment Create a training file Submit to the AI cluster Create an estimator Register the model Demo: Register and test the model
63. Demo: Aka.ms/AI4DEV02-amls-basics @hboelman Henk.boelman@microsoft.com
64. Step 3: Deploy Your model into production
65. AMLS to deploy The Model Score.py Environment file Docker Image
66. Score.py %%writefile score.py from azureml.core.model import Model def init(): model_root = Model.get_model_path('MyModel’) loaded_model = model_from_json(loaded_model_json) loaded_model.load_weights(model_file_h5) def run(raw_data): url = json.loads(raw_data)['url’] image_data = cv2.resize(image_data,(96,96)) predicted_labels = loaded_model.predict(data1) return json.dumps(predicted_labels)
67. Environment File from azureml.core.runconfig import CondaDependencies cd = CondaDependencies.create() cd.add_conda_package('keras==2.2.2') cd.add_conda_package('opencv') cd.add_tensorflow_conda_package() cd.save_to_file(base_directory='./', conda_file_path='myenv.yml')
68. Inference config inference_config = InferenceConfig( runtime= "python", entry_script="score.py", conda_file="myenv.yml" )
69. Deployment using AMLS
70. Deploy to ACI aciconfig = AciWebservice.deploy_configuration( cpu_cores = 1, memory_gb = 2) service = Model.deploy(workspace=ws, name='simpsons-aci', models=[model], inference_config=inference_config, deployment_config=aciconfig)
71. Deploy to AKS aks_target = AksCompute(ws,"AI-AKS-DEMO") deployment_config = AksWebservice.deploy_configuration( cpu_cores = 1, memory_gb = 1) service = Model.deploy(workspace=ws, name="simpsons-ailive", models=[model], inference_config=inference_config, deployment_config=deployment_config, deployment_target=aks_target) service.wait_for_deployment(show_output = True)
72. Demo: Deploy to Azure Container Instance
74. Pipelines Azure ML Service Pipelines Azure Pipelines
75. Recap Azure Function for orchestrator Custom Vision for easy models Azure Machine learning Service Make services (API) for specific tasks / models GIT / CI / CD / Test everything
76. Lessons learned Wise lessons to take home with you …
77. You never have enough data Apply DevOps practices from the start… Break the problem into smaller pieces Think big, start small, grow fast
78. • Aka.ms/AI4DEV01-CustomVision • Aka.ms/AI4DEV02-amls-basics • https://github.com/hnky/amls-pytorch • https://github.com/hnky/amls-pytorch-pipeline • https://github.com/hnky/dataset-lego-figures
79. @hboelman github.com/hnky henkboelman.com Thank you! Read more on: henkboelman.com