bcolz
Faster Than Memory Storage Francesc Alted [email protected] Lightning talk 

What is bcolz? • Provides a storage layer that is both chunked and is compressible • It is meant for both memory and persistent storage (disk) • Main goal: to demonstrate that compression can accelerate data access (both on disk and in-memory) 

Contiguous vs Chunked NumPy container Contiguous memory bcolz container chunk 1 chunk 2 Discontiguous memory chunk N . . . 

Why Compression (I)? Compressed Dataset Original Dataset More data can be packed using the same storage 

Why Compression (II)? Less data needs to be transmitted to the CPU Disk or Memory Bus Decompression Disk or Memory (RAM) CPU Cache Original
 Dataset Compressed
 Dataset Transmission + decompression faster than direct transfer? 

Blosc: Compressing Faster Than memcpy() • bcolz is using Blosc to compress chunks 

Memory Access Time vs CPU Cycle Time 

The MovieLens Dataset bcolz vs pandas (size) • Compression means ~20x less space 

Query Times
 3 years-old laptop (Ivy Bridge) • Compression leads to better performance, even for in-memory bcolz data containers 

Query Times
 5-years old laptop (Core2) • Compression still makes things slower on old boxes, but not necessarily in newer ones 

Streaming Analytics With bcolz bcolz container (disk or memory) iter(), iterblocks(),
 where(), whereblocks(), __getitem__() map(), filter(), groupby(), sortby(), reduceby(),
 join() bcolz
 iterators/filters with blocking itertools, PyToolz, Dask 

Give bcolz a try! http://bcolz.blosc.org/ http://www.blosc.org