Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Lightning Talk: From Data Processing Pipeline using Luigi to Serverless Backend Architecture using AWS Lambda Functions

Lightning Talk: From Data Processing Pipeline using Luigi to Serverless Backend Architecture using AWS Lambda Functions

Presented at DevOps Porto #30: Lightning Talks with Python Porto https://www.meetup.com/pt-BR/devopsporto/events/263166374/

Full Title:
Fullstack Data Engineering. From Data Processing Pipeline using Luigi to Serverless Backend Architecture using AWS Lambda Functions

Abstract:
During this talk I will discuss how the project meuParlamento.pt was built. You will learn how we structured the data processing pipeline using Luigi and how we built a Backend API for the mobile App using the serverless architecture using AWS Lambda.

As a side project and non-profit endeavour, meuParlamento.pt was created to bring the Parliament to each users smartphone. The idea is to allow any citizen to become a Member of Parliament, and to vote on the many proposals that have been debated over the years while focusing on privacy and anonymity - no voting data is recorded.

This is an open source project that combines web scrapping, text summarization and data mining to deliver content to users and I hope to present the audience an overview on how it works.

Projeto meuParlamento.pt

August 08, 2019
Tweet

More Decks by Projeto meuParlamento.pt

Other Decks in Programming

Transcript

  1. Full stack data science Building a data processing pipeline with

    Luigi and backend deployment using Amazon’s Lambda API Arian Pasquali August 2019
  2. Full stack data science Building a data processing pipeline with

    Luigi and backend deployment using Amazon’s Lambda API Arian Pasquali August 2019 Engineering
  3. meuParlamento.pt the Portuguese Parliament in your pocket non-profit and open

    source project Arian Pasquali - Nuno Moniz - Tomás Amaro
 [email protected] Case study
  4. 5 • Increase engagement in politics • Encourage healthy debate

    • Explore new ways to interact with Government's open data Our goals
  5. 6 Vote real legislative proposals from your smartphone See which

    political parties vote like you Share results with your friends meuParlamento.pt Our solution
  6. In a nutshell How it works with 
 three simple

    gestures Abstention In favor Against Swipe left if you are against Swipe right if you aprove Swipe up Users can skip if they prefer not to vote
  7. 10 HTTP API arquivo.pt parlamento.pt Data processing pipeline Storage About

    3.000 proposals Web scrapping Refinement Text summarization Etc Endpoint: 
 Provide 10 random proposals to vote Overview
  8. 12 • Initially, prototype involved different Python scripts using Jupiter

    Notebooks, Java and even R.
 • In order to structure the data processing we choose Luigi • Tasks dependences • Dashboard • Email notifications, etc • As simple as it can be • Just define task dependence and output. It takes care of the rest
 • Challenges • Handling exceptions
 
 Data processing pipeline
 From plain python scripts to task management engine https://github.com/spotify/luigi
  9. 13 Web scrapping Download PDF PDF to Text Text Summarization

    PDF Parser Compute Readability Score Persistence Python R Python (for now) Data pipeline
  10. Data processing pipeline
 Important to keep in mind • Keep

    each step as simple as possible • Atomic tasks • Easier to test • If one task fails you can start again from the last one • Keep track of changes • Don’t update data, keep it. • Save every change you made in the data • It also facilitates recovery at any point in the pipeline • Fail gracefully • Handle exceptions properly so it doesn’t break the entire workflow. It can be tricky sometimes, specially with wrappers and loops. • It pays off when things get complicated 14
  11. 16 Backend requirements • Provide a simple endpoint 1. Serve

    list of random proposals to vote 
 • It should be • Easy to deploy • Easy to monitor • Cheap to scale • $$$ comes from our pockets meuParlamento API HTTP API JSON
  12. • Flask as python web framework • MongoDB as storage

    (Cloud) • Pros • Data updates directly from pipeline • Cons • Bottleneck: Heroku’s free tier • Slow processing units, no load balance, etc • Expensive to scale if necessary • Too much trouble for such a simple endpoint 17 Backend API using Flask
 Flask-based web API
  13. Serverless Backend API
 Since we are almost completely stateless. 


    Why not Serverless / Lambda Functions? This was a good opportunity to try out serverless deployments. The cloud provider manages the allocation of machine resources.
 Pros • Minimal infrastructure management; • Load balance; • Seems cheaper to scale; • Easier deployment too; https://www.fullstackpython.com/serverless.html
  14. There are a few python packages to develop serverless applications:

    • Zappa • python-lambda • Chalice We choose Chalice
 Pros • Very similar to Flask • Painless migration from previous Flask code • Provided by Amazon (trust) • Pretty decent documentation • Simple command line interface https://github.com/aws/chalice/ Serverless Backend API

  15. • Pros • Cheaper to scale if necessary; • Free

    tier is quite generous • Single command deployment; • Support different stages (e.g. dev, test, prod, …) • Support api versioning • Good monitoring tools • Advanced log analytics • Alerts (e.g. latency alerts, etc) • Cons • Package size limitation - (50 Mb) • Vendor lock. Chalice supports AWS only • URL may change in some scenarios. Make sure you use a proxy url in front of it 20 AWS Lambda Serverless Backend API
 From Flask to serverless with AWS Chalice
  16. 21 • It scales so well that the bottleneck is

    now the pool of available connections at the database (MongoDB). • Up to 200 for free tier MongoDB • Solution • Query requirements are actually really simple • Remove mongodb and use simpler in memory data structure? • Idea • Pipeline • Save proposals file at Amazon S3 bucket • Backend • Load proposals file from S3 bucket Open question
 Remove mongodb? AWS Lambda
  17. 22 • Properly support API versioning • API proxy
 •

    Improve tests coverage, documentation and website
 • Pipeline repository is still not open. But it will be soon. Next steps http://github.com/meuparlamento
  18. 23 • Support session debates • Support proposals in real-time

    with notifications • Improve User Interface • Support different public institutions 
 (e.g. European Parliament, city council, etc) Next steps (features) http://github.com/meuparlamento
  19. http://meuParlamento.pt by Arian Pasquali - Nuno Moniz - Tomás Amaro


    [email protected] http://github.com/meuparlamento Non-profit - Open source