AI in the Battle against fakes

Counterfeited products are a growing problem. As well as impacting the revenue of a company, they hurt brand reputation and customer confidence. Our company was asked to create a solution for a global Electronics brand that can identify fake products by just taking one picture on your smartphone.

In this session we zoom into the building blocks that make this AI solution work. We’ll find out that there is much more to it than just training a convolutional neural network. We look at challenges like: how do you manage and monitor the AI, how do you build and improve the AI model in such a way it fits your DevOps production chain.
Interested in this technological journey?

Learn how we used Azure Functions, Cosmos DB and Docker to build a solid foundation; Azure Machine Learning Service to train the neural networks. And Azure DevOps to control, build and deploy this state-of-art solution.

AI used to be in the realm of scientists in white lab coats. Now any smart developer can integrate this exciting technology into their next project. Join us to find out how.

Transcript

1. Henk Boelman Cloud Advocate @ Microsoft AI in the battle against fakes HenkBoelman.com @hboelman
2. Agenda Assignment & challenges Solution architecture Icon detection and validation with Custom Vision Logo classification with AMLS Recap and lessons learned
3. Assignment and challenges
4. Ask a sharp question
5. Is it a genuine or a fake package?
6. Collect the data
7. Can we have the data please? Sorry we don’t have…. You are joking right?
8. 1 2 3
11. Break up the project and look at the smaller pieces
12. Solution Architecture Application flow and Azure design
13. Application AI Engine Management Front End
14. Ingredients Azure Durable Functions Docker Containers Azure Kubernetes Service Cosmos DB Azure Blob Storage Umbraco AMLS Cognitive Services Azure DevOps API Management
15. Umbraco config Create a product Tag product Upload Front image Run AI on image Push to engine
16. Umbraco config
17. AI Engine flow Photo of product Pre-Process image Match to Product in DB Text AI Logo AI Icon AI End evaluation
18. Sophisticated pretrained models To simplify solution development Azure Databricks Machine Learning VMs Popular frameworks To build advanced deep learning solutions TensorFlow Keras Pytorch Onnx Azure Machine Learning Language Speech … Azure Search Vision On-premises Cloud Edge Productive services To empower data science and development teams Powerful infrastructure To accelerate deep learning Flexible deployment To deploy and manage models on intelligent cloud and edge Machine Learning on Azure Cognitive Services
19. Icon detection and validation Application flow and Azure design
20. Object detection Classification Classification { Icon: Car Genuine: False, Score: 95% }, { Icon: Truck Genuine: True, Score: 85% } Input Output Genuine: 2% | Fake: 95% Genuine: 85% | Fake: 3% Icon validator API
21. Label images Export Train Model Deploy Model Test Model Visual Object Tagging Tool Custom Vision Service Icon detection Run test-set against model
22. Label images Export Train Model Deploy Model Test Model Custom Vision Service Icon verification Custom Vision Service API Images in 2 folders Genuine / Fake Run test-set against model Image augmentation
23. Custom Vision Service ML lifecycle Data source Icon Validator API
24. Custom Vision Service An easy-to-use tool for creating your own custom image classifier
29. Demo: Custom Vision Service
30. Demo: Aka.ms/AI4DEV01-CustomVision @hboelman Henk.boelman@microsoft.com
31. Logo classification A deep neural network to classify the Philips logo.
32. Computer Vision API Cut out PHILIPS Pre- Process Predict image Return Result Logo classification Icon Validator API
33. Label images Pre process Train Model Deploy Model Test Model Azure Machine Learning Service Logo classification Images in 2 folders Genuine / Fake Run test-set against model Image augmentation
34. Azure Machine Learning studio A fully-managed cloud service that enables you to easily build, deploy, and share predictive analytics solutions.
35. Is it Marge or Homer?
37. Prepare your environment Experiment with your model & data Deploy Your model into production
38. Step 1: Prepare your environment
39. Setup your environment VS Code Azure Notebooks Azure Portal
41. Datasets – registered, known data sets Experiments – Training runs Pipelines – Workflows runs Models – Registered, versioned models Endpoints: Real-time Endpoints – Deployed model endpoints Pipeline Endpoints – Training workflows Compute – Managed compute Datastores – Connections to data Azure Machine Learning Service
42. Create a workspace ws = Workspace.create( name='<NAME>', subscription_id='<SUBSCRIPTION ID>', resource_group='<RESOURCE GROUP>', location='westeurope') ws.write_config() ws = Workspace.from_config() Create a workspace
43. Create Compute cfg = AmlCompute.provisioning_configuration( vm_size='STANDARD_NC6', min_nodes=1, max_nodes=6) cc = ComputeTarget.create(ws, '<NAME>', cfg) Create a workspace Create compute
45. Step 1 Prepare your environment Create a workspace Create compute Setup storage
46. Step 2: Experiment with your model & data
47. Create an experiment exp = Experiment(workspace=ws, name=“<ExperimentName>”) Create an Experiment
48. Create a training file Create an Experiment Create a training file
49. Create an estimator params = {'--data-folder': ws.get_default_datastore().as_mount()} estimator = TensorFlow( source_directory = script_folder, script_params = params, compute_target = computeCluster, entry_script = 'train.py’, use_gpu = True, conda_packages = ['scikit-learn','keras','opencv’], framework_version='1.10') Create an Experiment Create a training file Create an estimator
50. Submit the experiment to the cluster run = exp.submit(estimator) RunDetails(run).show() Create an Experiment Create a training file Submit to the AI cluster Create an estimator
51. Create an Experiment Create a training file Submit to the AI cluster Create an estimator Demo: Creating and run an experiment
52. Azure Notebook Compute Target Experiment Docker Image Data store 1. Snapshot folder and send to experiment 2. create docker image 3. Deploy docker and snapshot to compute 4. Mount datastore to compute 6. Stream stdout, logs, metrics 5. Launch the script 7. Copy over outputs
53. Register the model model = run.register_model( model_name='SimpsonsAI', model_path='outputs') Create an Experiment Create a training file Submit to the AI cluster Create an estimator Register the model
54. Create an Experiment Create a training file Submit to the AI cluster Create an estimator Register the model Demo: Register and test the model
55. Demo: Aka.ms/AI4DEV02-amls-basics @hboelman Henk.boelman@microsoft.com
56. Step 3: Deploy Your model into production
58. AMLS to deploy The Model Score.py Environment file Docker Image
59. Score.py %%writefile score.py from azureml.core.model import Model def init(): model_root = Model.get_model_path('MyModel’) loaded_model = model_from_json(loaded_model_json) loaded_model.load_weights(model_file_h5) def run(raw_data): url = json.loads(raw_data)['url’] image_data = cv2.resize(image_data,(96,96)) predicted_labels = loaded_model.predict(data1) return json.dumps(predicted_labels)
60. Environment File from azureml.core.runconfig import CondaDependencies cd = CondaDependencies.create() cd.add_conda_package('keras==2.2.2') cd.add_conda_package('opencv') cd.add_tensorflow_conda_package() cd.save_to_file(base_directory='./', conda_file_path='myenv.yml')
61. Inference config inference_config = InferenceConfig( runtime= "python", entry_script="score.py", conda_file="myenv.yml" )
62. Deployment using AMLS
63. Deploy to ACI aciconfig = AciWebservice.deploy_configuration( cpu_cores = 1, memory_gb = 2) service = Model.deploy(workspace=ws, name='simpsons-aci', models=[model], inference_config=inference_config, deployment_config=aciconfig)
64. Deploy to AKS aks_target = AksCompute(ws,"AI-AKS-DEMO") deployment_config = AksWebservice.deploy_configuration( cpu_cores = 1, memory_gb = 1) service = Model.deploy(workspace=ws, name="simpsons-ailive", models=[model], inference_config=inference_config, deployment_config=deployment_config, deployment_target=aks_target) service.wait_for_deployment(show_output = True)
65. Demo: Deploy to ACI & AKS
66. Recap Azure Function for orchestrator Custom Vision for easy models Azure Machine learning Service Make services (API) for specific tasks / models GIT / CI / CD / Test everything
67. Lessons learned Wise lessons to take home with you …
68. You never have enough data Apply DevOps practices from the start… Break the problem into smaller pieces Think big, start small, grow fast
69. @hboelman github.com/hnky henkboelman.com Thank you! Read more on: henkboelman.com Aka.ms/AI4DEV01-customvision Aka.ms/AI4DEV02-amls-basics