Ocean Temperature Data • Daily mean ocean temperature every 1/4 degree • 720 x 1440 array every day • http://www.esrl.noaa.gov/psd/data/gridded/ data.noaa.oisst.v2.highres.html
One year’s worth from netCDF4 import Dataset import matplotlib.pyplot as plt from numpy import flipud data = Dataset("sst.day.mean.2015.v2.nc").variables["sst"] year_mean = data[:].mean(axis=0) plt.imshow(flipud(year_mean), cmap="RdBu_r") plt.title("Average Global Ocean Temperature, 2015")
Blocked Algorithms Blocked mean x = h5py.File('myfile.hdf5')['x'] # Trillion element array on disk sums = [] counts = [] for i in range(1000000): # One million times chunk = x[1000000*i: 1000000*(i+1)] # Pull out chunk sums.append(np.sum(chunk)) # Sum chunk counts.append(len(chunk)) # Count chunk result = sum(sums) / sum(counts) # Aggregate results
Blocked algorithms allow for • parallelism • lower ram usage The trick is figuring out how to break the computation into blocks. This is where dask comes in.
Dask is: • A parallel computing framework • That leverages the excellent python ecosystem • Using blocked algorithms and task scheduling • Written in pure python
Out-of-core arrays import dask.array as da from netCDF4 import Dataset from glob import glob from numpy import flipud import matplotlib.pyplot as plt files = sorted(glob('*.nc')) data = [Dataset(f).variables['sst'] for f in files] arrs = [da.from_array(x, chunks=(24, 360, 360)) for x in data] x = da.concatenate(arrs, axis=0) full_mean = x.mean(axis=0) plt.imshow(np.flipud(full_mean), cmap='RdBu_r') plt.title('Average Global Ocean Temperature, 1981-2015')
Out-of-core dataframes • Yearly csvs of all American flights 1987-2008 • Contains information on times, airlines, locations, etc… • Roughly 121 million rows, 11GB of data • http://www.transtats.bts.gov/Fields.asp?Table_ID=236
import dask.bag as db # Create a bag of tuples of (subreddit, body) b = db.from_castra('reddit.castra', columns=['subreddit', 'body'], npartitions=8) # Filter out comments not in r/MachineLearning matches_subreddit = b.filter(lambda x: x[0] == 'MachineLearning') # Convert each comment into a list of words, and concatenate words = matches_subreddit.pluck(1).map(to_words).concat() # Count the frequencies for each word, and take the top 100 top_words = words.frequencies().topk(100, key=1).compute() Example - Reddit Data http://blaze.pydata.org/blog/2015/09/08/reddit-comments/
from wordcloud import WordCloud # Make a word cloud from the results wc = WordCloud() wc = generate_from_frequencies(top_words) wc.to_image() Example - Reddit Data
Or use dask.imperative @do def load(filename): ... @do def clean(data): ... @do def analyze(sequence_of_data): ... @do def store(result): with open(..., 'w') as f: f.write(result) files = ['myfile.a.data', 'myfile.b.data', 'myfile.c.data'] loaded = [load(f) for f in files] cleaned = [clean(i) for i in loaded] analyzed = analyze(cleaned) stored = store(analyze)
“For workloads that are processing multi-gigabytes rather than terabyte+ scale, a big-memory server may well provide better performance per dollar than a cluster.” “Nobody ever got fired for using Hadoop on a cluster” http://research.microsoft.com/pubs/163083/hotcbp12%20final.pdf
Takeaways • Python can still handle large data using blocked algorithms • Dask collections form task graphs expressing these algorithms • Dask schedulers execute these graphs in parallel • Dask graphs can be directly created for custom pipelines
jcrist
loic425
murabayashi
lhespanha
shimabox
heyitsmohit
shibe23
yatasunshine
miyabigouji
mh4gf
lnit
taowata
iimokoy
shpigford
colly
jasonvnalue
denniskardys
ddemaree
kastner
keavy
holman
bermonpainter
addyosmani
lauravandoore
tammielis