Big Data in Action

Transcript

1. Big Data In Action University of Arizona ISTA 520 Sumin

   Byeon

2. Smartrek • Solution to urban traffic congestion problem • Incentivizes

   users to take alternative routes • Route that minimizes congestion -- less carbon footprint

3. Smartrek architecture • Frontend: Android, iOS, web app • User

   guidance -- navigation • Data collection • Backend: The brain

4. • Location, timestamp, speed and direction at regular interval •

   e.g. Every 3 sec or every 5m, whichever comes first • Historical & real-time traffic data Data collection

5. What we do with data • Imputation -- Fill in

   missing data • Transform into traffic information • Find the optimal route

6. Missing data http://maps.google.com

7. Imputation

8. Intersection matrix O1 O2 O3 O4 I1 C11 C12 C13

   C14 I2 C21 C23 I3 C31 C32 C34 I4 C43 I1 O1 O2 I2

9. Finding shortest path • Famous example: Dijkstra’s algorithm http://en.wikipedia.org/wiki/File:Dijkstra_Animation.gif

10. Time dependent shortest path

11. Link cost table 0:00 5 0:15 4 .. 7:45 17

    .. 23:45 5

12. Challenge: big data • In Phoenix, 8,300 links • In

    the Bay Area, 20,000 links • +10,000 cities in the United States • 150,000,000 links in OpenStreetMap database (world-wide) • With 15-min bracket, 96 rows in the cost table, per link, per day • Periodical traffic data update & continuous sensor data incoming

13. MapReduce • Computing model for efficient distributed computing over large

    data sets • Introduced by Google in 2004 • Open source implementations: Apache Hadoop, HBase, Cassandra, Hypertable, and more http://research.google.com/archive/mapreduce.html

14. Things we want to do with MapReduce • Imputation -

    Fill in missing links • Link cost table for each link can be constructed independently • Intersection matrix computation

15. Summary 4th paradigm • Not exactly making a scientific discovery

    by exploring a large amount of data, but... • Data captured by machines • Processed by software into information, knowledge