Mobile phone data footprint

Extract, organize and analyze open data.
Analyze data extracted from mobile phones to study mobility patterns.
Train a CNN to identify if it is raining given a mobility pattern.
Some use cases that the PoC makes feasible:
Help traffic redistribution and predict mobility flows.
Manage traffic lights according to prediction of traffic distribution.
Giving logistics solutions to business.

Transcript

1. AI Saturdays Mobile phone data footprint
2. Team: Matias Szarfer Cristina Ferrer Jordi Guix Edgar Pons Project Github: https://github.com/Giffy/Mobile_footprint_AIBCN
3. Objectives: Extract, organize and analyze open data. Analyze data extracted from mobile phones to study mobility patterns. Train a CNN to identify if it is raining given a mobility pattern.
4. Open data source Transparència Catalunya - http://www.transparenciacatalunya.cat/ca/inici/ Weather forecast - https://www.timeanddate.com
5. Technology used: Load big data and clean it → FIlter → Web scraping → Create heatmaps → CNN → MongoDB Pandas Beautiful soup Seaborn Tensorflow (Keras)
6. The process Data cleansing Get data Prepare database Dataset graphics Weather Mobile phone Get data Data cleansing Dataset creation CNN
7. Heatmaps Dataset creation: Mobile phone database 11M registers [{'Activitat': 'TILTING', 'Date': '17/10/2015 01:54:12 AM', 'Desc_': 'STATE_EMERGENCY_ONLY', 'Hora_': '01:54:12', 'Lat': 26.6693, 'Long_': -81.8294, 'MUNICIPI': '', 'NOM_MUNI': '', 'Operador': 'MetroPCS', 'Senyal': 15, 'Xarxa': 'MetroPCS', '_id': ObjectId('5c3f91a7b8e18270378'), 'downloadSpeed': '', 'net_type': '4G', 'precision1': 10, 'provider': 'gps', 'satellites': 5, 'speed': 0.7, 'status': 2, 'timestamp_': 1445039652407, 'uploadSpeed': ''}] Cleansing data Data reduction Load in Pandas 3M registers
8. Create a Dataset with: Population density and people concentration
9. Weather history Dataset creation : Beautiful soup www.timeanddate.com From 2015 to now <tr class=c1><th>09:00</th><td class="wt-ic" ><img class=mtt title="Passing Clouds." src="//c.tadst.com/gfx/w/40/wt-13.png" width=40 height=40></td><td>13&nbsp;°C</td><td class="small" >Passing Clouds.</td><td class="sep" >11 km/h</td><td><span class="comp sa6" title="Wind blowing from 250° West-southwest to East-northeast">↑</span></td><td>67%</td><td class="sep" >1013 mbar</td><td>16&nbsp;km</td></tr> Html
10. Weather history Dataset creation : If description = ['Light rain', 'Rain', 'Drizzle', 'Scattered showers' , 'Rain showers' , 'Sprinkles', 'Thunderstorms' , 'Thundershowers' , 'Strong thunderstorms' , 'Lots of rain' , 'Mixture of precip' , 'Sleet', 'Heavy rain', 'Snow flurries' , 'Light snow', 'Snow'] Rain! (Bad weather conditions)
11. Dataset preprocessing Cleaning of heatmap dataset: - Remove heatmaps with low number of measurements - National or local holidays Heatmap images augmentation creating periods of time Combine heatmap images and weather history dataset Balance dataset from 80% sunny vs 20% raining to 50%/50% Sunny Sunny Rainy Rainy
12. Convolutional Neural Network Combine in a binary classification network: - Images: vehicle density in city - Labels: weather history Decided to create a Convolutional Neural Network with keras Input 280x280x1 Feature learning Classification 3x Conv + BatchNorm + relu + pooling Flatten Output
13. Convolutional Neural Network Model reached a maximum 60% accuracy in validation Amount of data accuracy and volume is critical to train the CNN and impacted in the accuracy result. Required: - Increase image resolution to increase performance - Data augmentation helped to increase image samples
14. What we learned Low amount of sample data impacted in accuracy. Batch Normalization + Dropout improved learning ratio. Open Data is becoming popular to support Public / Corporate transparency Open data is a valid source of data to develop added value. IoT devices propagation will increase the number of sensors and data.
15. Uses Help traffic redistribution and predict mobility flows. Manage traffic lights according to prediction of traffic distribution. Giving logistics solutions to business City council Users
16. We'll use the data to... Identify cultural events in the city Identify football matches Usage of rental apartments for tourists Impact rain on city mobility
17. Thank you! Matias Szarfer Cristina Ferrer Jordi Guix Edgar Pons Project Github: https://github.com/Giffy/Mobile_footprint_AIBCN