Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Keep you friends close and caches closer by Simba Nyatsanga

Pycon ZA
October 06, 2017

Keep you friends close and caches closer by Simba Nyatsanga

A Lightning Talk at PyConZA 2017

Pycon ZA

October 06, 2017
Tweet

More Decks by Pycon ZA

Other Decks in Programming

Transcript

  1. • 14 TB of Pandas data structures serialized into MySQL.

    • Related metadata stored encoding the structure of the DataFrames/Series. • 10Gbps network card for the needed bandwidth to transmit the data. • One client accounts for about 80% of the data. • Hitting capacity limits. Problem
  2. Goals - Durable and larger storage capacity. - Cost effective.

    - Ability to scale horizontally vs vertically (in the case of MySQL DB box). - Maintain read speeds that the computation engine relies on. - Backups for disaster recovery and business continuity.
  3. Learnings - Cache consistency. - Consensus Delete. - Redis pub/sub

    vs Redis list/queue. - S3 eventual consistency.