Software engineering practices applied to ML

PyData London Meetup - April 2018 (https://www.meetup.com/PyData-London-Meetup/events/248973985/)

Stefano and Pavlos are engineers at HomeAway, and they'll talk about the benefits of applying good engineering practices when building ML-powered systems. The agenda will include:
- Feature engineering: does it have the same behaviour in training and prediction phases?
- CI/CD: are you checking for performance regressions in an automated fashion?
- Monitoring of your ML pipeline: are you sure your ML model works in production?

Transcript

1. View Slide

2. View Slide

3. View Slide

4. View Slide

5. View Slide

6. View Slide

7. View Slide

8. View Slide

9. View Slide

10. View Slide

11. View Slide

12. CI/CD
    DESIGN AND
    DEVELOPMENT
    MONITORING
    

    View Slide

13. CI/CD
    DESIGN AND
    DEVELOPMENT
    MONITORING
    

    View Slide

14. Country LTV
    

    View Slide

15. View Slide

16. id country listingtype
    US apartment
    UK
    id country listingtype
    USA flat
    United Kingdom
    

    View Slide

17. Unit
    Tests
    Feature
    Engineering
    Service
    

    View Slide

18. View Slide

19. View Slide

20. View Slide

21. View Slide

22. View Slide

23. View Slide

24. CI/CD
    DESIGN AND
    DEVELOPMENT
    MONITORING
    

    View Slide

25. CI/CD
    DESIGN AND
    DEVELOPMENT
    MONITORING
    

    View Slide

26. View Slide

27. View Slide

28. View Slide

29. View Slide

30. View Slide

31. View Slide

32. View Slide

33. View Slide

34. View Slide

35. def test_ml_models(hold_out_data_set):
    # For Baseline Model
    baseline_mae, baseline_rmse = _calculate_error_for_baseline_model(
    hold_out_data_set
    )
    # For New Model
    new_model_mae, new_model_rmse = _calculate_error_for_new_model(
    hold_out_data_set
    )
    assert new_model_mae < baseline_mae
    assert new_model_rmse < baseline_rmse
    

    View Slide

36. View Slide

37. View Slide

38. CI/CD
    DESIGN AND
    DEVELOPMENT
    MONITORING
    

    View Slide

39. CI/CD
    DESIGN AND
    DEVELOPMENT
    MONITORING
    

    View Slide

40. View Slide

41. View Slide

42. View Slide

43. View Slide

44. View Slide

45. View Slide

46. View Slide

47. View Slide

48. View Slide

49. View Slide

50. View Slide

51. CI/CD
    DESIGN AND
    DEVELOPMENT
    MONITORING
    

    View Slide

52. CI/CD
    DESIGN AND
    DEVELOPMENT
    MONITORING
    

    View Slide

53. CI/CD
    DESIGN AND
    DEVELOPMENT
    MONITORING
    

    View Slide

54. CI/CD
    DESIGN AND
    DEVELOPMENT
    MONITORING
    

    View Slide

55. View Slide

56. View Slide

57. View Slide