Stripe CTF 3.0 - Follow Up

From BigData Meetup #4 - Timisoara, June 2014

Transcript

1. A distributed systems challenge: Stripe CTF Bogdan Gâza Big Data #5 Bogdan Gâza
2. Big Data #5 Bogdan Gâza $whoami
3. Big Data #5 Bogdan Gâza
4. Big Data #5 Bogdan Gâza Level 0 The mysterious program
5. Big Data #5 Bogdan Gâza
6. Big Data #5 Bogdan Gâza Different hash: open addressing v hash(V) hash(X) x
7. Big Data #5 Bogdan Gâza Bloom filters N hash functions k1 k2 k3 kn probabilistic fals positives / no fals negatives
8. Big Data #5 Bogdan Gâza Level 1 Gitcoins
9. Big Data #5 Bogdan Gâza Level 1 Gitcoins
10. Big Data #5 Bogdan Gâza
11. Big Data #5 Bogdan Gâza SHA1 - ridiculous parallel bash: 400 Hash/s stripe go miners: 1.9 MHash/s gpu: 1-2 GHash/s
12. Big Data #5 Bogdan Gâza Level 2 DDos Defense
13. Big Data #5 Bogdan Gâza Proxy Node Node Node
14. Big Data #5 Bogdan Gâza Load balancing algorithm 4 req / ip request requests < 25 ms apart Solution
15. Big Data #5 Bogdan Gâza Level 3 Instant code search
16. Big Data #5 Bogdan Gâza LB Indexer Indexer Indexer 4 minutes to index 4 nodes with 500MB of RAM Scala Latency based scoring To pass the level < 0.15s / query Around 100M words Arbitrary substring
17. Big Data #5 Bogdan Gâza Twitter Stack: finagle / twitter server Problem
18. Big Data #5 Bogdan Gâza
19. Big Data #5 Bogdan Gâza 1st approach: inverted index vs substring search Solution Trie vs substring search marisa / patricia Trie / radix tree / suffix tree vs substring search DAWG vs substring search Sharding
20. Big Data #5 Bogdan Gâza Level 4 SQLCluster
21. Big Data #5 Bogdan Gâza SQLite SQLite SQLite SQLite SQLite SQLite Unreliable network ! octopus octopus simulates: netsplit / lagsplit / SPOF
22. Big Data #5 Bogdan Gâza Consensus Reliability in the presence of faulty processes examples: who can commit to the DB who is the leader state machine replication
23. Big Data #5 Bogdan Gâza ZAB - High performance broadcast in primary backup systems - 2011 RAFT - Understandable consensus algorithm - 2013 Paxos - Part-time parliament - Laslie Lamport ‘90 Consensus
24. Big Data #5 Bogdan Gâza ZAB - High performance broadcast in primary backup systems - 2011 RAFT - Understandable consensus algorithm - 2013 Paxos - Part-time parliament - Laslie Lamport ‘90 Consensus
25. Big Data #5 Bogdan Gâza
26. Big Data #5 Bogdan Gâza
27. Big Data #5 Bogdan Gâza
28. Big Data #5 Bogdan Gâza Thanks!