Building Crawlers with serverless

Building Crawlers with serverless

How do we building crawlers/scrapings with serverless architecture.

Event: https://www.meetup.com/pt-BR/Serverless-Floripa/events/253774158/

9797bddafe284a0d23b4e92dd1ce0043?s=128

Leonardo Rifeli

October 06, 2018
Tweet

Transcript

  1. Crawlers with Serverless.

  2. None
  3. Topics ▷ About Serverless ▷ About FaaS ▷ Reviewr ▷

    Our Vision ▷ Lessons Learned ▷ Conclusion
  4. About Serverless

  5. None
  6. Examples ▷ Auth0 ▷ Azure Functions ▷ AWS API Gateway

    ▷ AWS Lambda ▷ ... *github.com/serverless/examples
  7. About FaaS

  8. Servers are fully-abstracted

  9. Uncoupled responsibilities

  10. Different languages

  11. Auto scaling

  12. Event driven

  13. Pay only for what you use

  14. None
  15. None
  16. Reviewr

  17. None
  18. None
  19. inserir no contexto de reviews Google, Facebook, iFood, Trip e

    Booking
  20. None
  21. None
  22. None
  23. None
  24. None
  25. None
  26. None
  27. None
  28. None
  29. None
  30. None
  31. None
  32. None
  33. None
  34. None
  35. PLANILHA

  36. PLANILHA

  37. PLANILHA

  38. None
  39. None
  40. None
  41. None
  42. None
  43. None
  44. None
  45. None
  46. None
  47. None
  48. None
  49. None
  50. None
  51. None
  52. Our Vision

  53. Our Vision - Concepts Collector Processor Dataviz

  54. Collector Concepts

  55. Be "Browserless"

  56. Single Responsability

  57. Execution Control

  58. Recursion Is Your Friend

  59. Normalize It

  60. Collect != Process

  61. Processor Concepts

  62. Orchestration flow

  63. Data Validation

  64. Parallel Process

  65. Language By Your Need

  66. Secondary Services

  67. Process != Data Mining

  68. Data Storage

  69. Lessons Learned

  70. Costs Predictability

  71. Costs Predictability - Lambda Example *serverlesscalc.com

  72. Mindset Disruption

  73. Mindset Disruption

  74. Versioning & CI

  75. Versioning & CI

  76. Versioning & CI

  77. Versioning & CI

  78. Monitoring & Alerts

  79. Monitoring & Alerts

  80. Monitoring & Alerts

  81. Monitoring & Alerts

  82. Secondary Services

  83. Secondary Services

  84. Secondary Services - Current

  85. AWS Step Functions

  86. AWS Step Functions

  87. AWS Step Functions

  88. Cold Start

  89. Cold Start

  90. Do More With Less

  91. Do More With Less

  92. Do More With Less - Team CTO & Partner Marcelo

    Andrade Head of Engineering & Partner Leonardo Rifeli Fullstack developer Raphael Ornellas
  93. Conclusion

  94. THANKS! engineering.reviewr.tech marcelo.andrade@reviewr.me leonardo.rifeli@reviewr.me