My !!con talk

Transcript

1. Spinning metal platters IN THE CLOUD!!! @sashalaundy
2. physics + programming ! very little CS
5. katrinaebowman on flickr
6. VERY high level trimmed = FOREACH loaded_data GENERATE userId, website; ! grouped = GROUP trimmed BY userId; ! counted = FOREACH grouped GENERATE group, COUNT(grouped);
8. I get this for FREE! • Mappin’ & reducin’ • HDFS in the CLOUD! • Clusters AND nodes! • A rockin’ query plan!
10. Write Pigscript Graphs!
13. “give me 500 rows where age > 15”
14. “give me 500 rows where age > 15” Why so slow?
15. “Seeking is slower than reading”
16. ??
18. 01010110101010001010101000101010101101010101001 GRACE50VIRGINIAALAN45ENGLANDADA30ENGLAND
21. READING: grabbing contiguous sections of data
22. SEEKING: grabbing scattered sections of data
23. “Seeking is slower than reading”
25. “give me 500 rows where age > 15” GRACE50VIRGINIAALAN45ENGLANDADA30ENGLAND
26. MIND. BLOWN.
27. in my PIGSCRIPTS I had to worry about a spinning METAL PLATTER somewhere in VIRGINIA!!!!
29. • Various schema? MONGO • Fast search? ELASTICSEARCH • Keep history? DATOMIC • Want very fast analytics queries? REDSHIFT.
30. REDSHIFT production backend for your website! copy of your database for your data team to play with!!
31. analytics needs lots of AGGREGATION ! like SUM, AVG, or COUNT
 across ROWS
32. GRACE50VIRGINIAALAN45ENGLANDADA30ENGLAND So lots of seeking? GOSH DARN IT! but what if…
33. GRACEALANADA504530VIRGINIAENGLANDENGLAND READING! ! YAYYYYYY!!!
34. GRACEALANADA504530VIRGINIAENGLANDENGLAND “columnar storage”
35. What’s faster than reading AND seeking? IGNORING
36. block min max 1 1 6 2 7 12 3 13 340
37. Redshift has lots more… • NODES so you can compute in parallel • cool QUERY PLANS based on your actual data! • Not actually a database. “Managed data warehouse service in the cloud” • So blazing fast!
38. Really fast! …how fast? • 21,454,134 rows • COUNT(*) • Postgres: 586,931.216 ms (10 minutes) • Redshift: 1,561.359 ms (1.5 seconds) 376 times faster! from http://dailytechnology.net/2013/08/03/redshift-what-you-need-to-know/
39. 376x isn’t cool. You know what’s cool? 100,000x Instead of native Python, a matrix! 100x Speed from OpenBLAS compared to numpy 10x Parallelization (for free from OpenBLAS) 10x 100,000x
40. redshift is fast
41. hardware matters
42. databases are cool
43. THANKS!!!! @sashalaundy sasha.io