Big data de andar por casa | Shirt-sleeve Big Data

#codemotion_es #codemotion #2014 #codemotion2014
Jorge Lería - @jorgeleria
William Viana - @vianasw

Transcript

1. Shirt-sleeve big data
4. “Any collection of data sets so large and complex that it becomes difficult to process them using traditional data processing applications”
5. “Any collection of data sets so large and complex that it becomes difficult to process them using traditional data processing applications”
6. Jorge Lería @jorgeleria
7. Jorge Lería @jorgeleria
8. Jorge Lería @jorgeleria
9. Jorge Lería @jorgeleria
10. William Viana @vianasw
11. William Viana @vianasw
12. William Viana @vianasw
13. William Viana @vianasw
18. http://youtu.be/vWa9CUsNzdw
19. A Search Engine All domains ~700.000.000 Home only Metadata ~60kb libs,frameworks
20. A Search Engine All domains ~700.000.000 Home only Metadata ~60kb libs,frameworks ~40Tb
22. 40Tb of small documents
23. $ $ $ $ 40Tb of small documents
24. Considerations Cheap Fixed problem Fresh data
25. Single machine approach Cheap Fixed problem Fresh data (10gb) Single point of failure Waste of resources Less funny?
26. Multi machine approach Cheap? Fixed problem Fresh data (10gb)
27. Multi machine approach Main server W1 W2 Wn ... workers
28. AWS Simple Monthly Calculator http://calculator.s3.amazonaws.com/index.html
30. Some Numbers [on a napkin] 700,000,000 pages / 2 days = 4050 reqs/sec 700,000,000 rows * 500 bytes/row = 325 GBs 325 GBs every two days from AWS = $584 a month
31. Some Numbers [on a napkin] 700,000,000 pages / 2 days = 4050 reqs/sec 4050 reqs/sec / 40 instances = 100 reqs/instance 40 c3.large * 30 days = $3074 per month
32. 4.9 TB and $3658 per month
34. We can do better
37. http://aws.amazon.com/ec2/purchasing-options/spot-instances/spot-tutorials/
38. http://aws.amazon.com/ec2/purchasing-options/spot-instances/spot-tutorials/ START of ugly slides made by Amazon
49. http://aws.amazon.com/ec2/purchasing-options/spot-instances/spot-tutorials/ END of ugly slides made by Amazon
50. Main server W1 W2 Wn ... workers
52. Isn’t Python slow? Crawling is mostly I/O bound Parsing with bindings to fast C libraries
53. Isn’t Python slow? Crawling is mostly I/O bound Parsing with bindings to fast C libraries Python rocks!
54. The case of the 10x improvement
55. The case of the 10x improvement with one line of code
56. import re2 as re
57. import re2 as re https://code.google.com/p/re2/
58. Main server W1 W2 Wn ... workers Spot instances
59. Main server W1 W2 Wn ... workers Spot instances On demand instance
60. Main server W1 W2 Wn ... workers Spot instances On demand instance same availability zone
61. Main server W1 W2 Wn ... workers Spot instances On demand instance same availability zone ? ? ?
65. Robust messaging for applications Easy to use Runs on all major operating systems Supports a huge number of developer platforms Open source and commercially supported [from their website]
67. W1 W2 Wn ... workers Spot instances On demand instance same availability zone
69. Cassandra Key Features Distributed and Decentralized High Performance Fault Tolerant Highly Available Column Oriented Key Value
70. Cassandra Key Features Distributed and Decentralized High Performance Fault Tolerant Highly Available Column Oriented Key Value
71. Cassandra Key Features Distributed and Decentralized High Performance (3k writes/sec) Fault Tolerant Highly Available Column Oriented Key Value Also comes with compression, incremental backups and many problems
72. W1 W2 Wn ... workers Spot instances On demand instance same availability zone
73. W1 W2 Wn ... workers Spot instances On demand instance same availability zone
74. W1 W2 Wn ... workers Spot instances On demand instance same availability zone Beefy machine outside Amazon
75. W1 W2 Wn ... workers Spot instances On demand instance same availability zone Dedicated server
76. Bloom filter "A space-efficient probabilistic data structure, that is used to test whether an element is a member of a set" bitarray + fast hash function 0 0 0 1 0 0 1 0 0 1 2 3 4 5 6 7
77. Bloom filter
78. W1 W2 Wn ... workers Spot instances On demand instance same availability zone Dedicated server
79. W1 W2 Wn ... workers Spot instances On demand instance same availability zone Dedicated server Bloom filter
81. $ curl -XGET 'http://localhost:9200/twitter/_search? q=user:kimchy' … and a nice REST API
82. Thank you! Jorge Lería - @jorgeleria William Viana - @vianasw Follow us!