Machine Learning Outside the Kaggle Lines Craig Franklin

About me ! Developer at Seer Medical ! Recent convert to Aussie Rules Football fandom "presias, cfranklin11 ! craigfranklin.dev @englishcraig

- Competitions & learning - Good for learning - Level playing field - Same rules, data, objective Kaggle is fine ! @englishcraig

What is footy (,pping)? Robert Merkel, Jacknstock at English Wikipedia / Public domain

- Australian-rules football: contact sport, oblong ball - Footy tipping: office betting pool - Most correct picks wins - Lots of novel challenges & mistakes, but learned from them By Unknown author - Fairfax Photo Archives, Public Domain By Flickerd - Own work, CC BY-SA 4.0 @englishcraig

- Footy tipping = picking winners - Got data, trained the model, built the app - Unexpected input Clearly define your problem Photo by Patrick Tomasso on Unsplash

- Margin of victory for first match is tie breaker - Had classifier, so manually entered margins - Next season: changed to regressor for margins @englishcraig

- Kaggle provide all the data for you - Collecting data to solve your ML problem is hard - Dynamic data sources are particularly difficult - Start of first season, betting odds data were blank - I panicked & scraped a betting site Know the en)re life stories of your data sources Photo by CoWomen on Unsplash

- Rosters & betting odds change up until the start of each match - Avoid predictions with blank or stale data - Observe data sources as you would the data itself Weekly Data Update Schedule Day Data Types Data Sources Notes Monday Match results Player stats afltables.com Some6mes delayed 6ll Tuesday or Wednesday Tuesday Be:ng odds footywire.com Wednesday Team rosters afl.com.au For Thursday match only Thursday Team rosters afl.com.au For all later matches @englishcraig

- Collecting and cleaning your own data is a lot of work - Web scrapers & using undocumented APIs are difficult to maintain - I thought I was clever & original Get by with a li,le help from your friends @englishcraig

@englishcraig

- fitzRoy: R package for AFL data - R packages can be good source of data - More effort upfront, but lower maintenance - Example: AFL website changing to JS heavy UI h"ps:/ /github.com/jimmyday12/fitzRoy @englishcraig

- What values can be missing? What values are unique? - Index: team, season, round number Make your assump-ons explicit @englishcraig

- 2010: Replay the Grand Final TFW you realise you go/a do it all again next week Ge#y Images @englishcraig

- Data bugs are often silent - Spot checks can catch them, but are time consuming - Raising errors codifies assumptions about valid data Data bugs could be hiding anywhere Photo by Ka*e Moum on Unsplash @englishcraig

- Time-series models (ARIMA, Elo) need data sorted by date/ time - Spent days debugging an Elo model with 50% accuracy When doing date,me-sensi,ve calcula,ons, make sure your rows are in the correct order assert data_frame["date"].is_monotonic_increasing, ( "Data must be sorted by date to calculate cumulative values ", "or make predictions with time-series models.", ) data_frame.groupby(["team", "year"].cumsum("match_wins")) @englishcraig

- Different stats start in different years - Lots of blank values - Imputing didn't make sense, so decided to fill with 0s - Caused lots of bugs Joining disparate data sets is risky Data set First season # blank seasons Match results 1897 0 Player scoring stats 1897 0 Basic player stats* 1965 68 Advanced player stats** 1999 102 Be@ng odds 2010 113 *For basic in-game events like kicks, tackles, etc. **For less-common in-game events or ones that require player loca:on. @englishcraig

- When joining data sets & filling with zeros, can't know which zeros are valid - Some values should never be blank (team, season, round number) - Assertions important, because you can't test for them, resulting in bad training/predictions Make sure there are no dodgy zeros data_to_check = data_frame[NEVER_ZERO_COLUMNS] zeros_data_frame = data_to_check[(data_to_check == 0).any(axis=1)] assert not zeros_data_frame.any().any(), ( "An invalid fillna produced index column values of 0:\n" f"{zeros_data_frame}" ) @englishcraig

- Second season: regressor instead of classifier - Kaggle is temporary: no tech debt - Project has to be maintained, so make it pleasant to work with Op#mise for maintainability first, accuracy second Photo by Cesar Carlevarino Aragon on Unsplash

- Kaggle is all about performance - Project models can be complicated, just weight cost/benefit - Diminishing returns with increased complexity - Abandoned first model/app because too messy Calculate the cost/benefit of complexity Photo by StellrWeb on Unsplash @englishcraig

The Joy of Produc.on

- Deployed to Heroku - Crashed because didn't have C library Boost.Python - Re-architect for Docker Know your system-level dependencies (or control them) • Do you need any of the Boost C++ libraries, gcc, or g++? • Do you need to control your environment with Docker? @englishcraig

- Second season: deployed to heroku, crashed - Player data used too much memory - Re-architect for DigitalOcean - Data pipelines & ML models are memory hungry - Know your data usage & how much your server has Know your server's specs Photo by Jordan Rowland on Unsplash @englishcraig

Predic'ng the bounce of an oblong ball Flickerd / CC BY-SA (h3ps:/ /crea8vecommons.org/licenses/by-sa/4.0)

- Rough start, but came back to win in final match 2018 Season Results Tipper Correct Tips* Tipresias (me) 140 Top Coworker 139 Oddsmakers 140 * Regular season only @englishcraig

- Added data, improved model - Rough start, rough middle, rough finish - Success isn't guaranteed - Kaggle has static test set, sport is chaotic and each season is unique - Random tipper's instinct can beat the odds as well as the machines 2019 Season Results Tipper Correct Tips* Tipresias (me) 133 Top Coworker 138 Oddsmakers 135 * Regular season only @englishcraig

Thank you All the 'ps: 'presias.net All the code: +presias All the slides: ! craigfranklin.dev All the complaints: @englishcraig