AI in the Battle against fakes

Slide 1

Slide 1 text

Henk Boelman Cloud Advocate @ Microsoft AI in the battle against fakes HenkBoelman.com @hboelman

Slide 2

Slide 2 text

Agenda Assignment & challenges Solution architecture Icon detection and validation with Custom Vision Logo classification with AMLS Recap and lessons learned

Slide 3

Slide 3 text

Assignment and challenges

Slide 4

Slide 4 text

Ask a sharp question

Slide 5

Slide 5 text

Is it a genuine or a fake package?

Slide 6

Slide 6 text

Collect the data

Slide 7

Slide 7 text

Can we have the data please? Sorry we don’t have…. You are joking right?

Slide 8

Slide 8 text

1 2 3

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

Break up the project and look at the smaller pieces

Slide 12

Slide 12 text

Solution Architecture Application flow and Azure design

Slide 13

Slide 13 text

Application AI Engine Management Front End

Slide 14

Slide 14 text

Ingredients Azure Durable Functions Docker Containers Azure Kubernetes Service Cosmos DB Azure Blob Storage Umbraco AMLS Cognitive Services Azure DevOps API Management

Slide 15

Slide 15 text

Umbraco config Create a product Tag product Upload Front image Run AI on image Push to engine

Slide 16

Slide 16 text

Umbraco config

Slide 17

Slide 17 text

AI Engine flow Photo of product Pre-Process image Match to Product in DB Text AI Logo AI Icon AI End evaluation

Slide 18

Slide 18 text

Sophisticated pretrained models To simplify solution development Azure Databricks Machine Learning VMs Popular frameworks To build advanced deep learning solutions TensorFlow Keras Pytorch Onnx Azure Machine Learning Language Speech … Azure Search Vision On-premises Cloud Edge Productive services To empower data science and development teams Powerful infrastructure To accelerate deep learning Flexible deployment To deploy and manage models on intelligent cloud and edge Machine Learning on Azure Cognitive Services

Slide 19

Slide 19 text

Icon detection and validation Application flow and Azure design

Slide 20

Slide 20 text

Object detection Classification Classification { Icon: Car Genuine: False, Score: 95% }, { Icon: Truck Genuine: True, Score: 85% } Input Output Genuine: 2% | Fake: 95% Genuine: 85% | Fake: 3% Icon validator API

Slide 21

Slide 21 text

Label images Export Train Model Deploy Model Test Model Visual Object Tagging Tool Custom Vision Service Icon detection Run test-set against model

Slide 22

Slide 22 text

Label images Export Train Model Deploy Model Test Model Custom Vision Service Icon verification Custom Vision Service API Images in 2 folders Genuine / Fake Run test-set against model Image augmentation

Slide 23

Slide 23 text

Custom Vision Service ML lifecycle Data source Icon Validator API

Slide 24

Slide 24 text

Custom Vision Service An easy-to-use tool for creating your own custom image classifier

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

No content

Slide 27

Slide 27 text

No content

Slide 28

Slide 28 text

No content

Slide 29

Slide 29 text

Demo: Custom Vision Service

Slide 30

Slide 30 text

Demo: Aka.ms/AI4DEV01-CustomVision @hboelman [email protected]

Slide 31

Slide 31 text

Logo classification A deep neural network to classify the Philips logo.

Slide 32

Slide 32 text

Computer Vision API Cut out PHILIPS Pre- Process Predict image Return Result Logo classification Icon Validator API

Slide 33

Slide 33 text

Label images Pre process Train Model Deploy Model Test Model Azure Machine Learning Service Logo classification Images in 2 folders Genuine / Fake Run test-set against model Image augmentation

Slide 34

Slide 34 text

Azure Machine Learning studio A fully-managed cloud service that enables you to easily build, deploy, and share predictive analytics solutions.

Slide 35

Slide 35 text

Is it Marge or Homer?

Slide 36

Slide 36 text

No content

Slide 37

Slide 37 text

Prepare your environment Experiment with your model & data Deploy Your model into production

Slide 38

Slide 38 text

Step 1: Prepare your environment

Slide 39

Slide 39 text

Setup your environment VS Code Azure Notebooks Azure Portal

Slide 40

Slide 40 text

No content

Slide 41

Slide 41 text

Datasets – registered, known data sets Experiments – Training runs Pipelines – Workflows runs Models – Registered, versioned models Endpoints: Real-time Endpoints – Deployed model endpoints Pipeline Endpoints – Training workflows Compute – Managed compute Datastores – Connections to data Azure Machine Learning Service

Slide 42

Slide 42 text

Create a workspace ws = Workspace.create( name='', subscription_id='', resource_group='', location='westeurope') ws.write_config() ws = Workspace.from_config() Create a workspace

Slide 43

Slide 43 text

Create Compute cfg = AmlCompute.provisioning_configuration( vm_size='STANDARD_NC6', min_nodes=1, max_nodes=6) cc = ComputeTarget.create(ws, '', cfg) Create a workspace Create compute

Slide 44

Slide 44 text

No content

Slide 45

Slide 45 text

Step 1 Prepare your environment Create a workspace Create compute Setup storage

Slide 46

Slide 46 text

Step 2: Experiment with your model & data

Slide 47

Slide 47 text

Create an experiment exp = Experiment(workspace=ws, name=“”) Create an Experiment

Slide 48

Slide 48 text

Create a training file Create an Experiment Create a training file

Slide 49

Slide 49 text

Create an estimator params = {'--data-folder': ws.get_default_datastore().as_mount()} estimator = TensorFlow( source_directory = script_folder, script_params = params, compute_target = computeCluster, entry_script = 'train.py’, use_gpu = True, conda_packages = ['scikit-learn','keras','opencv’], framework_version='1.10') Create an Experiment Create a training file Create an estimator

Slide 50

Slide 50 text

Submit the experiment to the cluster run = exp.submit(estimator) RunDetails(run).show() Create an Experiment Create a training file Submit to the AI cluster Create an estimator

Slide 51

Slide 51 text

Create an Experiment Create a training file Submit to the AI cluster Create an estimator Demo: Creating and run an experiment

Slide 52

Slide 52 text

Azure Notebook Compute Target Experiment Docker Image Data store 1. Snapshot folder and send to experiment 2. create docker image 3. Deploy docker and snapshot to compute 4. Mount datastore to compute 6. Stream stdout, logs, metrics 5. Launch the script 7. Copy over outputs

Slide 53

Slide 53 text

Register the model model = run.register_model( model_name='SimpsonsAI', model_path='outputs') Create an Experiment Create a training file Submit to the AI cluster Create an estimator Register the model

Slide 54

Slide 54 text

Create an Experiment Create a training file Submit to the AI cluster Create an estimator Register the model Demo: Register and test the model

Slide 55

Slide 55 text

Demo: Aka.ms/AI4DEV02-amls-basics @hboelman [email protected]

Slide 56

Slide 56 text

Step 3: Deploy Your model into production

Slide 57

Slide 57 text

No content

Slide 58

Slide 58 text

AMLS to deploy The Model Score.py Environment file Docker Image

Slide 59

Slide 59 text

Score.py %%writefile score.py from azureml.core.model import Model def init(): model_root = Model.get_model_path('MyModel’) loaded_model = model_from_json(loaded_model_json) loaded_model.load_weights(model_file_h5) def run(raw_data): url = json.loads(raw_data)['url’] image_data = cv2.resize(image_data,(96,96)) predicted_labels = loaded_model.predict(data1) return json.dumps(predicted_labels)

Slide 60

Slide 60 text

Environment File from azureml.core.runconfig import CondaDependencies cd = CondaDependencies.create() cd.add_conda_package('keras==2.2.2') cd.add_conda_package('opencv') cd.add_tensorflow_conda_package() cd.save_to_file(base_directory='./', conda_file_path='myenv.yml')

Slide 61

Slide 61 text

Inference config inference_config = InferenceConfig( runtime= "python", entry_script="score.py", conda_file="myenv.yml" )

Slide 62

Slide 62 text

Deployment using AMLS

Slide 63

Slide 63 text

Deploy to ACI aciconfig = AciWebservice.deploy_configuration( cpu_cores = 1, memory_gb = 2) service = Model.deploy(workspace=ws, name='simpsons-aci', models=[model], inference_config=inference_config, deployment_config=aciconfig)

Slide 64

Slide 64 text

Deploy to AKS aks_target = AksCompute(ws,"AI-AKS-DEMO") deployment_config = AksWebservice.deploy_configuration( cpu_cores = 1, memory_gb = 1) service = Model.deploy(workspace=ws, name="simpsons-ailive", models=[model], inference_config=inference_config, deployment_config=deployment_config, deployment_target=aks_target) service.wait_for_deployment(show_output = True)

Slide 65

Slide 65 text

Demo: Deploy to ACI & AKS

Slide 66

Slide 66 text

Recap Azure Function for orchestrator Custom Vision for easy models Azure Machine learning Service Make services (API) for specific tasks / models GIT / CI / CD / Test everything

Slide 67

Slide 67 text

Lessons learned Wise lessons to take home with you …

Slide 68

Slide 68 text

You never have enough data Apply DevOps practices from the start… Break the problem into smaller pieces Think big, start small, grow fast

Slide 69

Slide 69 text

@hboelman github.com/hnky henkboelman.com Thank you! Read more on: henkboelman.com Aka.ms/AI4DEV01-customvision Aka.ms/AI4DEV02-amls-basics