BDW Mtl 2015: Playful Big Data

Hackathon presentation @ BigDataWeek Montreal 2015 about iterating and ingesting quickly big datasets

Transcript

1. Playful Big Data With GitHub’s historical dataset
2. The plan Step 1: Acquire subset of data Step 2: Ingest the dump files Step 3: Store raw documents Step 4: Gather insights/aggregated info Step 5: Produce visualizations
3. What was actually done Step 1: Acquire subset of data
4. What was actually done Step 1: Acquire subset of data Step 2: Ingest the dump files
5. What was dreamt Step 1: Acquire subset of data Step 2: Ingest the dump files Step 3: Store raw documents Step 4: Gather insights/aggregated info Step 5: Produce visualizations
6. What does it look like?
7. Workstation - Raw .bson datasets - nginx as load balancer for riak nodes - nsqlookup daemon for discovery - tools like nsqadmin, bson_to_nsq, influxdb admin ui and grafana
8. Workstation BSON?
9. Workstation - Raw .bson datasets - nginx as load balancer for riak nodes - nsqlookup daemon for discovery - tools like nsqadmin, bson_to_nsq, influxdb admin ui and grafana
10. Workstation nginx?
11. Workstation - Raw .bson datasets - nginx as load balancer for riak nodes - nsqlookup daemon for discovery - tools like nsqadmin, bson_to_nsq, influxdb admin ui and grafana
12. Workstation riak?
13. Workstation - Raw .bson datasets - nginx as load balancer for riak nodes - nsqlookup daemon for discovery - tools like nsqadmin, bson_to_nsq, influxdb admin ui and grafana
14. Workstation nsq…?
15. Workstation - Raw .bson datasets - nginx as load balancer for riak nodes - nsqlookup daemon for discovery - tools like nsqadmin, bson_to_nsq, influxdb admin ui and grafana
16. Workstation bson_to_nsq?
17. What does it look like?
18. Producers - consume import messages (i.e. documents representing users, comment, commit, pull-request) - store plain document to riak - increment counters, store specific data points in InfluxDB
19. Producers Raw bson NSQ topic NSQ channel
20. Producers - horizontal scalability - custom code/measurements are simple to add http://nsq.io
21. What does it look like?
22. Datastores - Riak - InfluxDB
23. Datastores: Riak - distributed by design - multiple backends - map-reduce AND solr backed search - operationally simple
24. Datastores: InfluxDB - distributed - time series database - powerful query language - integrates with grafana
25. What does it look like?
26. Visualization: App
27. Visualization: Grafana - screenshots!
28. Visualization: Grafana
29. Visualization: Grafana
30. Visualization: Grafana
31. Conclusion - scope down, always, multiple times - planning for scalability is hard - cloud on demand infrastructure is fun - Montreal is amazing