Predicting traffic in New York City with taxi data

Transcript

1. M A R C O FA E D O P R E D I C T I N G T R A F F I C I N N E W Y O R K C I T Y W I T H TA X I D ATA
2. G O I N G T O J F K A I R P O R T
3. D ATA • Chris Wong in 2014 FOILed the data to the NYC Taxi & Limo commission - http://bit.ly/chris_wong • Data was put on Google Bigquery • Columnar datastore with a SQL (like) interface • Fast, easy to use and cheap :)
4. U S E R C A S E S : M A N H AT TA N T O J F K • 25-30 minutes on a holiday morning • 1h - 1h20’ at 16:30 working day
5. P R E D I C T I N G D U R AT I O N • features: month, day of the week, hour, working day, pickup location (lon, lat) • duration as the outcome • train a Random Forest model - R
6. R A N D O M F O R E S T ? • is a set of n decision trees • each decision tree is trained independently • a random subset of m features is selected at each candidate split • each tree can grow until the stop condition is reached or until maxnodes is reached • prediction phase: each tree cast a vote
7. T E C H N I C A L S T U F F • data is big, but not that big • 16 CPUs, 104 GB mem - Google Cloud Instance (RStudio Server) - AWS EC2 • Random Forest training is easy to parallelize (each decision tree is trained independently) • choose n, m to get good accuracy - (chose them empirically) • choose maxnodes to have manageable models in memory (~200 MB instead of 9 GB)
8. D E P L O Y I N G T H E M O D E L • if you are using R, stay with R (shiny) • Predictive Model Markup Language (XML-based) • convert every tree in a huge if else statement (i.e. in Python) and write a prediction function
9. R V S P Y T H O N • One R recursive function that generates n Python if else statements • One Python script that evaluates every tree (each tree cast a vote) • github.com/fffaez/ rfexport
10. M E A S U R E O U T C O M E : A N I D E A L M O D E L • Working on a validation set • Outcome with a little bit of a noise • Accuracy: 50% • Accuracy = TP + TN / Tot
11. G O O G L E M A P S V S TA X I M O D E L • Google Maps: 15% • Taxi Model: 30% • 2x Google Maps
12. A P P P R O T O T Y P E • works with rfexport • Heroku - Python - Django • nyctaxiplanner.herokuapp.com
13. M A R C O FA E D O @ G M A I L . C O M THANKS - QA