How we handle and boost our codebase and stack in a fast-growing startup

Stanislav Stefanic (Booking Tribe Lead at Kiwi.com) @ Brno-Moscow Python Meetup 2

"Five years ago, Kiwi.com began its story as a super fast growing startup. Today, we have almost 2 400 employees with more than 400 people in the engineering team. During this time, everything changed several times, including our people, company, vision, goals, and last but not least our codebase and stack. It wouldn’t be possible to handle this growth without following the latest trends in technology and code principles.
In this talk, I will tell you about our codebase and stack: how it changed in the last couple of years, where we want to push it in the future, what tools we use, what we learned that works, and what helped us to maintain rapid growth. I will share with you the lessons learned and my experiences from almost 4 years in this company".

Video: http://www.moscowpython.ru/meetup/2/kiwi-codebase/

Transcript

1. How we handle and boost our codebase
   and stack in a fast-growing startup
   Stanislav Stefanic
   Head of Booking
   

   View Slide

2. View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. pic: story time
   

   View Slide

8. View Slide

9. pic: story time
   

   View Slide

10. Monitoring + alerting + logs
    1. Website / API performance and availability (uptime)
    ○ Pingdom, Status.io, StatusPage, UptimeRobot, Assertible, Uptrends, RapidSpike, ApiFortress, …
    

    View Slide

11. View Slide

12. Monitoring + logs + alerting
    1. Website / API performance and availability (uptime)
    ○ Pingdom, Status.io, StatusPage, UptimeRobot, Assertible, Uptrends, RapidSpike, ApiFortress, …
    2. Error tracking and reporting
    ○ Sentry, Rollbar, BugSnag, HoneyBadger, AirBrake, RayGun, ...
    

    View Slide

13. View Slide

14. Monitoring + logs + alerting
    1. Website / API performance and availability (uptime)
    ○ Pingdom, Status.io, StatusPage, UptimeRobot, Assertible, Uptrends, RapidSpike, ApiFortress, …
    2. Error tracking and reporting
    ○ Sentry, Rollbar, BugSnag, HoneyBadger, AirBrake, RayGun, ...
    3. Application Performance Monitoring (APM)
    ○ Datadog, New Relic, OpBeat, Scout, Retrace, ...
    

    View Slide

15. View Slide

16. View Slide

17. Monitoring + logs + alerting
    1. Website / API performance and availability (uptime)
    ○ Pingdom, Status.io, StatusPage, UptimeRobot, Assertible, Uptrends, RapidSpike, ApiFortress, …
    2. Error tracking and reporting
    ○ Sentry, Rollbar, BugSnag, HoneyBadger, AirBrake, RayGun, ...
    3. Application Performance Monitoring (APM)
    ○ Datadog, New Relic, OpBeat, Scout, Retrace, ...
    4. Logs
    ○ self-hosted - ELK stack (Elasticsearch, Logstash, Kibana), GrayLog, ...
    ○ hosted solution - Logz.io, Loggly, LogDNA, Papertrail, …
    

    View Slide

18. View Slide

19. Monitoring + logs + alerting
    1. Website / API performance and availability (uptime)
    ○ Pingdom, Status.io, StatusPage, UptimeRobot, Assertible, Uptrends, RapidSpike, ApiFortress, …
    2. Error tracking and reporting
    ○ Sentry, Rollbar, BugSnag, HoneyBadger, AirBrake, RayGun, ...
    3. Application Performance Monitoring (APM)
    ○ Datadog, New Relic, OpBeat, Scout, Retrace, ...
    4. Logs
    ○ self-hosted - ELK stack (Elasticsearch, Logstash, Kibana), GrayLog, ...
    ○ hosted solution - Logz.io, Loggly, LogDNA, Papertrail, …
    5. Incident Management
    ○ PagerDuty, OpsGenie, Cabot, VictorOps, OpenDuty, PagerTree, …
    

    View Slide

20. View Slide

21. What changed in last 5 years...
    ● 2.5M+ cloc
    ○ ~1.5M in python
    ● 450+ repositories
    ● 300+ running microservices
    ● 2500+ employes
    ○ 400+ in engineering, ~150 python devs
    ● 10K+ orders daily
    ○ 30K+ tickets, 15K+ unique passengers
    ● hundreds of projects/features every quarter
    

    View Slide

22. What this fast growth caused?
    ● huge technical debt
    ● lack of tests
    ● huge code complexity
    ● unreadable code
    ● hard to maintain and add new features
    

    View Slide

23. What to do with it?
    1. Best Practices
    ○ clean old code
    ○ make sure new code will follow it
    ○ tools
    ■ black, coala, pylint
    2. Refactor all the time
    ○ rewrite
    ○ refactor
    ■ pylint, sonarqube
    

    View Slide

24. Black
    ● The uncompromising code formatter
    ● same regardless of the project you’re reading
    ● https://black.readthedocs.io/en/stable/
    ● install
    ○ pip install black
    ● run
    ○ black {source_file_or_directory}
    ● add to pre-commit / pre-push hook
    

    View Slide

25. View Slide

26. Coala
    ● provides a unified command-line interface for linting and fixing all your
    code
    ● for all languages
    ○ but written in Python!
    ● recommended for pre-push hook
    ● bears
    ○ plugins/extensions which provide some analysis routines
    ○ BanditBear, MypyBear, PEP8Bear, PyCommentedCodeBear, PyDocStyleBear,
    PyFlakesBear, PyImportSortBear, PyLintBear, PyUnusedCodeBear, PycodestyleBear
    

    View Slide

27. View Slide

28. SonarQube
    ● open-source platform for continuous inspection of code quality
    ● detect
    ○ bugs, code smells, vulnerabilities
    ● reports on
    ○ duplicated code, coding standards, unit tests, code coverage, code complexity, comments,
    bugs, and security vulnerabilities
    ● metrics history and graphs
    

    View Slide

29. View Slide

30. View Slide

31. View Slide

32. CI of booking backend main repo
    

    View Slide

33. Crane
    ● GitLab CI ready image to upgrade services in Rancher
    ● https://github.com/kiwicom/crane
    

    View Slide

34. Next step: Continuous De...
    

    View Slide

35. Three takeaways
    ● try - (fail) - learn
    ● use tools and improve
    ● common sense and bird view
    

    View Slide

36. THANK YOU!
    

    View Slide