Bitdeli at PyData 2013

Transcript

1. Create Custom Analytics in Your Browser PyData 2013 Ville Tuulos CEO, Co-Founder
2. Everybody (Click & Play) Business Analysts (Excel) IT / DBAs (SQL, Python) Data Hackers (MapReduce) People who implement their own infrastructure
3. Everybody (Click & Play) Business Analysts (Excel) IT / DBAs (SQL, Python) Data Hackers (MapReduce) People who implement their own infrastructure Disco
4. Everybody (Click & Play) Business Analysts (Excel) IT / DBAs (SQL, Python) Data Hackers (MapReduce) People who implement their own infrastructure
5. Python is great
6. Python is great MapReduce is hard
7. Python is great MapReduce is hard Servers are annoying (cloud or not)
8. Python is great MapReduce is hard Servers are annoying (cloud or not) Everybody likes real-time
9. Python is great MapReduce is hard Servers are annoying (cloud or not) Everybody likes real-time Support healthy workflows
10. Demo
12. what makes some users very active? Customer C Customer B how to reduce churn? Customer A why some users return? Daily Activity Daily Activity Daily Activity Users Users Users
13. Simple Complex Discover Explore
14. Simple Complex Discover Explore Infographics Basic Statistics Reports
15. Simple Complex Discover Explore Infographics Basic Statistics Reports Segments Funnels Visualizations
16. Simple Complex Discover Explore Infographics Basic Statistics Reports Query Segments Funnels Slice & Dice Descriptive Models Visualizations
17. Simple Complex Discover Explore Infographics Basic Statistics Reports Query Segments Funnels Clustering Slice & Dice Descriptive Models Visualizations Predictive Models
18. DiscoDB persistent, immutable, compressed, lightning fast, key-value(s) mapping that supports lazy boolean queries. Code https://github.com/discoproject/discodb Docs http://discoproject.org/doc/discodb/
19. from discodb import DiscoDB FILES = [‘a.txt’, ‘b.txt’, ‘c.txt’] def extract_words(): for fname in FILES: for word in open(fname).read().split(): yield word, fname db = DiscoDB(extract_words()) db[‘dog’] db.keys() db.unique_values() db.items() # files that mention ‘dog’ # all distinct word # all distinct filenames # all (word, iter(fname)) pairs
20. Hash Map: hash(Key) → Key ID Value Map: Key ID → [Value ID, ...] Keys: Key ID → Key Values: Value ID → Value DiscoDB Chunk
21. Hash Map: hash(Key) → Key ID Value Map: Key ID → [Value ID, ...] Keys: Key ID → Key Values: Value ID → Value DiscoDB Chunk Perfect hashing by CMPH, guaranteed O(1) The list of Value IDs is delta-encoded Values are compressed with a global Huffman codebook
22. DiscoDB Chunk Node 1 Node 2 Node N Disco Node Python Worker DDFS Disco Node Python Worker Disco Node Python Worker DiscoDB Chunk DiscoDB Chunk DiscoDB Chunk DiscoDB Chunk DiscoDB Chunk DiscoDB Chunk DiscoDB Chunk DiscoDB Chunk
23. A → [Apple, Orange, Banana] B → [Apple, Banana] C → [Banana, Melon] Q(“A & B”) Apple Banana Q(“A | B”) Apple Orange Banana Q(“(A & B) | C”) Banana DiscoDB from discodb.query import Q Querying with Conjunctive Normal Form
24. Model: Event → Users Query (sequence of events): Q(“Event A & Event B & ...”) Funnel https://github.com/tuulos/bd3-mixpanel-funnel
25. Model: Day N → Users Query (weekly cohorts): Q(“(dayN | dayN+1) & (dayM | dayM+1...)”) Cohort Analysis https://github.com/tuulos/bd3-mixpanel-cohort
26. Model: Day N → Users Query (one time series): [Q(Day K) for K in range(start, end)] Time Series https://github.com/tuulos/bd3-mixpanel-trends
28. Thank You! TRENDING TRENDING TRENDING TRENDING https://bitdeli.com/free Interested? Contact ville@bitdeli.com Free analytics for your GitHub repos: