Keep you friends close and caches closer by Simba Nyatsanga

A Lightning Talk at PyConZA 2017

Transcript

2. • 14 TB of Pandas data structures serialized into MySQL. • Related metadata stored encoding the structure of the DataFrames/Series. • 10Gbps network card for the needed bandwidth to transmit the data. • One client accounts for about 80% of the data. • Hitting capacity limits. Problem
3. Computation
4. Goals - Durable and larger storage capacity. - Cost effective. - Ability to scale horizontally vs vertically (in the case of MySQL DB box). - Maintain read speeds that the computation engine relies on. - Backups for disaster recovery and business continuity.
5. Solution
6. Learnings - Cache consistency. - Consensus Delete. - Redis pub/sub vs Redis list/queue. - S3 eventual consistency.
7. THANK YOU!