Automating massively-scalable operational pipelines with AutoML and Apache Airflow

Repetitive human tasks (flagging, tagging, annotating) can start to drain organizational resources especially when they may be required to scale up infinitely

Novel framework proposed addresses tackling trivial binary decision-making with supervised classification pipelines

Demonstrate design, implementation and best practices to maintain big-data pipelines for automation, as code

On-demand scaling, reusability and version control

Tightly coupled Human-in-the-loop (HITL) system for validation and monitoring of model health

Transcript

1. Dat Tran - Head of Data Science Automating massively-scalable operational pipelines with AutoML and Apache Airflow Zubin John, Data Scientist 7/11/2018 - Berlin 1
2. Agenda 00:05 Part I: Core Model ❏ Modeling ❏ Serving ❏ Insights 00:15 Part II: Batch Processing Pipeline ❏ Architecture ❏ Deployment 00:25 Part III: AutoML ❏ Monitoring ❏ Evaluation ❏ Retraining 00:35 Wrap-up ❏ Integration ❏ Key Learnings codename: ..pacman.
3. Motivation
4. BUSINESS CASE Price Alerts Due to extreme latency in throughput, price alerts are delivered as late as 8 hours after the vendor updates prices in the catalog. It takes a human validator 15 seconds on average to verify if the alert is valid or not! An automated system operating at the same degree of precision is capable of classifying 100 alerts under a second! High traffic volume: a human validator goes through 3000-4000 alerts on a daily basis. Seasonality may push this traffic up to 6000/day.
5. SUPERVISED Learning 101 Training Where we ‘teach’ our model through data what is classified as a valid alert. Testing Where we ‘quiz’ our model till its passes with a satisfactory score. STEP 1 STEP 2
6. classification noun. a machine learning or human-curated task of assigning data points to one of a predetermined set of categories
7. Data Statistics: 208 Raw Attributes core model “Given audition data we attempt to build a model to automate classification of valid price alerts.” A quick survey of data reveals that the informational fields in the audition dataset can be broadly divided into: 1. Surface product attributes 2. Individual offer attributes 3. Product logistical/inventory codes and information
8. core model Establishing a baseline for results and setting metrics for performance evaluation Due to massive class imbalance in our data space, we can often get away by predicting the statistical majority. It only makes sense to invest in a supervised learning process if it beats this simple baseline. For the remainder of our work, we attempt to look at models that significantly outperform the baseline. Our baseline reports a high accuracy of 85.55%. However, is in capable of differentiating false positives. Hence the bulk of the learning task is to minimize the false positive rate w.r.t baseline. Choosing a metric An ROC curve (receiver operating characteristic curve) is a graph showing the performance of a classification model at all classification thresholds. This curve plots two parameters: • True Positive Rate • False Positive Rate The AUC (area under ROC curve) provides an aggregate measure of performance across all possible classification thresholds.
9. core model With the current human auditioning process, a human evaluator adds certain tags to each audition. These tags include: • excluded offers • offers with wrong identifiers • offers with incorrect delivery costs For forward compatibility, we want to deploy models that rely less and less on human reinforcement. As a result, we clean and transform this data in a manner suitably to minimize data leakage. Preparing raw data for automation and consumption As is common with real-world data generation, majority of information is noise and only a small fraction signal. Through exploratory data analysis we identify most informative features to use in modeling. We identify following potential features that play well with statistical models for our problem. • Shop, product identifiers and boolean properties. • Product categories and sub-categories • Price and shipping cost of each individual offers • Time-dependent features and general currency of audition. Most informative features identified for data. Our analysis reveals that offer level numerical attributes perhaps possess the greatest amount of signal . topOffers.0.price.Value topOffers.2.price.Value topOffers.2.priceTotal. Value topOffers.0.priceTotal. Value topOffers.1.price.Value topOffers.1.priceTotal. Value topOffers.4.price.Value topOffers.3.price.Value topOffers.4.priceTotal.Value topOffers.3.priceTotal.Value A Note on Data Cleaning
10. core model Model selection, finalization and tuning To account for data, strong correlated features and data imbalance we attempt a host of different learning techniques to achieve best performance possible. We list some of the relevant models that were candidates for our system. 1. Logistic Regression 2. Simple Decision Tree 3. Gradient Boosting Each of the models was on the same set of features with appropriate transformation where necessary. We however use the most relevant parameter search methods, in each case independently, to tune individual models. Logistic Regression classifier with subsampling to handle imbalance, tuned via GridSearch on hold-out set, evaluation via 5 fold cross-validation. Decision Tree classifier with evaluation via 5 fold cross-validation. Gradient Boosted classifier using CatBoost implementation, evaluation via 5 fold cross validation using library helper functions for tuning.
11. core model Results! Insights! Feedback! Strengths Handles missing values - (No imputations). Minimal feature engineering complexity. Drawbacks Does not truly leverage time sensitive data. Non Linearity The data used in this task is massively nonlinear, we confirm this via multiple different tests (including PCA, high-dimensionality visualizations). Intuitively, we do not see good performance with linear learners (logistic regression). Imbalanced Learning The task was complicated by class imbalance in the data. Only about 13% of samples represented the negative class. Multiple potential ways to deal with the scenario: subsampling, supersampling or boosting. If non-linear then: Hierarchical. The first indication of inherent hierarchical structures in our data was the relatively superior performance of a simple decision tree classifier over linear models! Ensemble Learning Gradient boosting is class of techniques that leverages an ensemble of weaker model to produce better predictions. Boosting which involves assigning certain trees heavier weights, essentially allows the task to then be interpreted as an optimization problem.
12. core model Productionizing our model and serving requests with Flask A combination of Flask + scikit-learn is the go to framework for easy static file serving (models). Scikit-learn models can be persisted (pickled) to avoid retraining the model every time they are used. You can use Flask to create an API that can provide predictions based on a set of input variables using a pickled model. ` Model File from flask import Flask, request, jsonify from pandas import DataFrame import pickle app = Flask(__name__) def load_objects_from_file(path, obj_keys=None): with open(path, 'rb') as f: obj = pickle.load(f) return obj classifier = load_objects_from_file(DUMPED_CLASSIFIER_PATH) @app.route('/predict_prob', methods=['POST']) def predict_probabilistic(): raw_input_vector = request.get_json() input_frame = DataFrame(raw_input_vector, index=[0]) prediction = classifier.predict_proba(input_frame.values) return jsonify({ ‘valid’: prediction[0][0], ‘invalid’: prediction[0][1] }) if __name__ == '__main__': app.run(host='0.0.0.0', port=’8080’)
13. The 80:20 Rule of Data Science and Artificial Intelligence
14. batch processing noun. a process in computing where a software or program completes a batch of jobs simultaneously in a non-stop, sequential order
15. Requirements: Batch Processing Pipeline Functional: • Consumes raw data • Handles feature extraction in accordance to model • Presents data to classifier and collects results Technical: • Horizontal scalability to handle increasingly large volumes • Maintainable, i.e. easy to write and easy to expand • Flexible to adjust to business requirements and changing models pipeline
16. Batch Processing Pipeline pipeline
17. • Scalable approach based in Apache Spark and its PySpark Wrapper • Built with a custom and accessible micro-framework for Big Data pipelines • Modular design to incorporate additional dependencies and adapt to business requirements • Pipelining Framework allows easy definition of processing steps along the lines of the predetermined modeling Building the Batch Processing Pipeline pipeline Consume Data Select Features Normalize Numerical Bucketize Classify Publish Data Processing Pipeline Every processing step is realized by a dedicated, self-contained module. Modules are interchangeable and divided in classes which dictate their purpose: Data Providers, Transformers, Classifiers and Data Writers. These make up the pipeline and can be extended and customized.
18. batch_processing_pipeline.py # set up consumption of Kafka Stream kafka_topic = self.read_property( 'spark.custom.kafka.topic') kafka_params = self.build_kafka_params() kafka_context = PriceAlertKafkaContext( 60, kafka_params =kafka_params, topics=[kafka_topic]) # define pipeline: pipeline = Pipeline( self.spark_session) pipeline .from_stream() # pipeline reads data from stream # define steps for feature extraction .select(relevant_feature_names).replace() .do(TemporalFeatureExtraction( 'auditionDate')).as_columns() .do(MapProductIdentifiers()).as_columns().join_result_columns() .apply(App.prepare_df_names_for_classification).replace() # plug in classification module (external REST service) .do(Classify(classifier_url)).as_columns().join_columns().replace() # publish classified data via Kafka .to(KafkaWriter( topic="validated_price_alerts", bootstrap_servers, client_id) # wrap pipeline in streaming context and start processing kafka_context.set_pipeline(pipeline) kafka_context.start() pipeline
19. DEPLOYMENT Learnings Deploying the pipeline in Openshift proved to be a challenge in its own: • Running a Spark Cluster has caveats ◦ Network policies and permissions interfere with Spark processes ◦ Workers cannot write files back to Master ◦ Older jobs are not accessible via Spark Web UI ◦ Performance is ultimately limited by cluster load (OS Cluster) ◦ Distribution of input files does not work properly • Spark has no mature configuration model for rolling deployment: ◦ Custom application configs cannot be easily injected into Spark context ◦ Reading of properties from context is cumbersome However, we arrived at a viable mode for rolling deployment. pipeline
20. Spark Worker Spark Workers pipeline Docker Image with: • packaged source code • dependencies • configuration files Deploys code to Spark via Spark Submit Command Continuous Deployment Deployer Pod “Submit2Spark” Spark Master Submit Code Change + Jenkins Build Delegate spark.app.name PacMan spark.master spark://spark-master:7077 Spark.jars.packages org.apache.spark:spark-str... spark.submit.pyFiles dependencies.zip,app.zip spark.custom.classifier.url http://price-alert-classifier... spark.custom.kafka.topic price_alert.audit spark.custom.kafka.auto.offset.reset smallest spark.custom.kafka.enable.auto.commit false spark.custom.kafka.bootstrap.servers kafka-01.users02... Spark.custom.kafka.group.id pipeline-spark-consumers spark.custom.kafka.client.id spark-extract-features-and-classify
21. autonomous system noun. an intelligent agent capable of making limited decisions and performing dedicated tasks without continuous intervention
22. auto.ml To “re”-train or Not to “re”-train? When to Retrain? Periodically. Whether this is monthly, weekly, daily - depends on the arrival rate of new data and its volume. It is advocated as a best practice to constantly evaluate your models and retrain if the predictions start to drift below an acceptance threshold. We adhere to this principle in our design For a model to predict accurately, the data that it is making predictions on must have a similar distribution as the data on which the model was trained. Because data distributions can be expected to drift over time, deploying a model is not a one-time exercise but rather a continuous process. It is a good practice to continuously monitor the incoming data and retrain your model on newer data if you find that the data distribution has deviated significantly from the original training data distribution. The case of deciding how and when to retrain depends a lot on your situation. How to Retrain? When new observations are available, there are several strategies to retrain your model: 1. Online 2. Offline 3. Batch/mini batch
23. auto.ml Continuous Model Evaluation via Human Validation (HITL) Delegated System #1 Airflow Triggered Retraining Pipeline Delegated System #2 Airflow is a platform to programmatically author, schedule and monitor data pipelines. The price-alert-model-manager is a workflow manager that relies on Airflow to provide a model monitoring system and retraining pipeline that require an airflow worker to manage data pipelines(retraining, automation, report generation) which are a common requirement for most DS projects.
24. human-in-the-loop noun. a branch of artificial intelligence that leverages machine and human intelligence simultaneously to create machine learning models
25. LEVERAGING HITL For Continuous Retraining CLASSIC RETRAINING WORKFLOW 1 Ingest Data 2 Feature Extraction & Engineering 3 Training + Tuning 4 Evaluation 5 Acquire New Data In a classical H-I-T-L scenario, a human team augments confidence in classifications/predictions by manually validation of new streams of data. Our modern bifunctional workflow oversees the following extended processes. Human Validation and Monitoring Model Health 1 Collect Incom ing Data W e set-up an upstream process called price_alert_service, to store new incom ing and data requests in a store. 2 3 Save Validated Labels and Evaluate M odel H ealth M anually Validate sub-sam ple Forw ard a fraction of new data to hum an validation team for labeling. Join freshly validated w ith autom ated alerts and evaluate perform ance using AU C m etric. 4 Trigger Retraining If m odel health scores fall below accepted threshold, trigger retraining a new m odule. MONITORING VALIDATION STEP It is necessary to label new data for retraining, in order to leverage human validation, we sample a tiny fraction of incoming traffic and and forward it to the HITL system for labeling. The human labels are then treated as gold and the ROC/AUC scores of current model are used to retrigger a new training pipeline. The fresh data is also used in tuning and autonomous selection of the best model. auto.ml
26. Airflow Airflow is a platform to programmatically author, schedule and monitor workflows as directed acyclic graphs (DAGs) of tasks. RETRAINING PIPELINE auto.ml Rich command line utilities make performing complex surgeries on DAGs a snap The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative DAGs a Directed Acyclic Graph – is a collection of all the tasks you want to run, organized in a way that reflects their relationships and dependencies. Tasks A task is a unit of execution, each DAG is composed of multiple ‘tasks’. It is represented by nodes in the DAG. Operator An operator describes a single task in a workflow. In execution, an operator is essentially a wrapper around a task. Airflow Operators under the hood While is a task is useful for representation in a DAG, internally tasks in Airflow are executed using operators. An operator is a functional unit and offers interfaces for tasks to execute code. A task implements an operator, a common use of the PythonOperator is demonstrated in example below. def healthy_model_termination (): logging.info( "Model %s is healthy." % MODEL_VERSION ) t4 = PythonOperator ( task_id='certify_healthy_model' , python_callable =healthy_model_termination, dag=dag ) t3.set_downstream (t4)
27. integration noun. the action or process of bringing together individual components with the intention of combining them towards a larger
28. Integrating with the idealo Platform “How can we combine the batch processing and retraining in order to drive the Price Alert feature in a productive environment?” Let’s take a look at the concept of Lambda Architectures: Wrap up New Data Speed Layer Live Classification Batch Layer Model Retraining Serving Layer: Send Price Alert Notifications Not entirely what we want, but we can reformulate it, to fit with our requirements:
29. Wrap up
30. SUMMARY ❖ Price Alert workflow presented a feasible goal for modularized automation. ❖ Designed complete integration of the automated ML retraining pipeline with Airflow ❖ Processing live data and batch-wise retraining inspired by Lambda Architecture ❖ Fast and scalable ❖ API first approach for maintainability, ❖ Entire workflow is maintainable as code Wrap up Modular engineering, cross-collaborative efforts involving Data Science & Engineering can be commonplace. This work is an example, offering a rapid-prototyping framework to move models from conception and testing into actual production. Productionizing a pipeline to automate binary classification is TRIVIAL. Ensuring your models perform at desired metrics requires laborious engineering efforts. Take organizational ownership of end-to-end delivery pipelines (not services), sooner rather than later in order to minimize technical debt. Advanced AI systems and concepts (autonomous ML, HITL) are not ready to drive complete automation YET, however they can be leveraged to benefit Data teams through close monitoring and expert supervision. Key Learnings 1 2 3 4
31. Questions? 31
32. Check us out! #idealoTech 32 https://github.com/idealo https://medium.com/idealo-tech-blog