Fullstack Data Science by Arian Pasquali

From Data Processing Pipeline using Luigi to Serverless Backend Architecture using AWS Lambda Functions.

Transcript

1. Full stack data science Building a data processing pipeline with Luigi and backend deployment using Amazon’s Lambda API Arian Pasquali August 2019
2. Full stack data science Building a data processing pipeline with Luigi and backend deployment using Amazon’s Lambda API Arian Pasquali August 2019 Engineering
3. meuParlamento.pt the Portuguese Parliament in your pocket non-profit and open source project Arian Pasquali - Nuno Moniz - Tomás Amaro
 dev@meuparlamento.pt Case study
4. 69.27% Abstention in the last elections in Portugal 4 Motivation
5. 5 • Increase engagement in politics • Encourage healthy debate • Explore new ways to interact with Government's open data Our goals
6. 6 Vote real legislative proposals from your smartphone See which political parties vote like you Share results with your friends meuParlamento.pt Our solution
7. In a nutshell How it works with 
 three simple gestures Abstention In favor Against Swipe left if you are against Swipe right if you aprove Swipe up Users can skip if they prefer not to vote
8. 8 Our solution
9. Data source http://parlamento.pt
10. 10 HTTP API arquivo.pt parlamento.pt Data processing pipeline Storage About 3.000 proposals Web scrapping Refinement Text summarization Etc Endpoint: 
 Provide 10 random proposals to vote Overview
11. Data collection and refinement Challenges building the data processing pipeline
12. 12 • Initially, prototype involved different Python scripts using Jupiter Notebooks, Java and even R.
 • In order to structure the data processing we choose Luigi • Tasks dependences • Dashboard • Email notifications, etc • As simple as it can be • Just define task dependence and output. It takes care of the rest
 • Challenges • Handling exceptions
 
 Data processing pipeline
 From plain python scripts to task management engine https://github.com/spotify/luigi
13. 13 Web scrapping Download PDF PDF to Text Text Summarization PDF Parser Compute Readability Score Persistence Python R Python (for now) Data pipeline
14. Data processing pipeline
 Important to keep in mind • Keep each step as simple as possible • Atomic tasks • Easier to test • If one task fails you can start again from the last one • Keep track of changes • Don’t update data, keep it. • Save every change you made in the data • It also facilitates recovery at any point in the pipeline • Fail gracefully • Handle exceptions properly so it doesn’t break the entire workflow. It can be tricky sometimes, specially with wrappers and loops. • It pays off when things get complicated 14
15. Backend API From Flask to Serverless
16. 16 Backend requirements • Provide a simple endpoint 1. Serve list of random proposals to vote 
 • It should be • Easy to deploy • Easy to monitor • Cheap to scale • $$$ comes from our pockets meuParlamento API HTTP API JSON
17. • Flask as python web framework • MongoDB as storage (Cloud) • Pros • Data updates directly from pipeline • Cons • Bottleneck: Heroku’s free tier • Slow processing units, no load balance, etc • Expensive to scale if necessary • Too much trouble for such a simple endpoint 17 Backend API using Flask
 Flask-based web API
18. Serverless Backend API
 Since we are almost completely stateless. 
 Why not Serverless / Lambda Functions? This was a good opportunity to try out serverless deployments. The cloud provider manages the allocation of machine resources.
 Pros • Minimal infrastructure management; • Load balance; • Seems cheaper to scale; • Easier deployment too; https://www.fullstackpython.com/serverless.html
19. There are a few python packages to develop serverless applications: • Zappa • python-lambda • Chalice We choose Chalice
 Pros • Very similar to Flask • Painless migration from previous Flask code • Provided by Amazon (trust) • Pretty decent documentation • Simple command line interface https://github.com/aws/chalice/ Serverless Backend API

20. • Pros • Cheaper to scale if necessary; • Free tier is quite generous • Single command deployment; • Support different stages (e.g. dev, test, prod, …) • Support api versioning • Good monitoring tools • Advanced log analytics • Alerts (e.g. latency alerts, etc) • Cons • Package size limitation - (50 Mb) • Vendor lock. Chalice supports AWS only • URL may change in some scenarios. Make sure you use a proxy url in front of it 20 AWS Lambda Serverless Backend API
 From Flask to serverless with AWS Chalice
21. 21 • It scales so well that the bottleneck is now the pool of available connections at the database (MongoDB). • Up to 200 for free tier MongoDB • Solution • Query requirements are actually really simple • Remove mongodb and use simpler in memory data structure? • Idea • Pipeline • Save proposals file at Amazon S3 bucket • Backend • Load proposals file from S3 bucket Open question
 Remove mongodb? AWS Lambda
22. 22 • Properly support API versioning • API proxy
 • Improve tests coverage, documentation and website
 • Pipeline repository is still not open. But it will be soon. Next steps http://github.com/meuparlamento
23. 23 • Support session debates • Support proposals in real-time with notifications • Improve User Interface • Support different public institutions 
 (e.g. European Parliament, city council, etc) Next steps (features) http://github.com/meuparlamento
24. http://meuParlamento.pt by Arian Pasquali - Nuno Moniz - Tomás Amaro
 dev@meuparlamento.pt http://github.com/meuparlamento Non-profit - Open source