Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Developing an Expression Language for Quantitat...

Scott Sanderson
November 12, 2015
150

Developing an Expression Language for Quantitative Financial Modeling

This talk details the challenges addressed during the development of Zipline's new Pipeline API, which provides a high-level expression language allowing users to describe computations on rolling windows of continuously-adjusted financial data. We discuss the notion of "perspectival" time-series data, arguing that this concept provides a useful framework for formally reasoning about financial data in the face of domain oddities like stock splits, dividends, and restatements.

We also consider the architectural and performance benefits of developing an API focused on symbolic computation, drawing comparisons to several recent developments in the Python numerical ecosystem.

Scott Sanderson

November 12, 2015
Tweet

Transcript

  1. Large Problem Medium Problem Medium Problem Medium Problem Small Problem

    Small Problem Small Problem Small Problem Small Problem Small Problem Small Problem Small Problem
  2. In [3]: Out[3]: from zipline.assets import AssetFinder finder = AssetFinder("sqlite:///data/assets.db")

    lifetimes = finder.lifetimes( dates=pd.date_range('2001-01-01', '2015-10-01'), include_start_date=True, ) lifetimes.head(5)
  3. In [5]: AAPL_prices = pd.read_csv( 'data_public/AAPL-split.csv', parse_dates=['Date'], index_col='Date', ) def

    plot_prices(prices): price_plot = prices.plot(title='AAPL Price', grid=False) price_plot.set_ylabel("Price", rotation='horizontal', labelpad=50) price_plot.vlines( ['2014-05-08'], 0, 700, label="$3.05 Dividend", linestyles='dotted', colors='black', ) price_plot.vlines( ['2014-06-09'], 0, 700, label="7:1 Split", linestyles='--', colors='black', ) price_plot.legend() sns.despine() return price_plot
  4. In [8]: from bcolz import open from humanize import naturalsize

    all_prices = open('data/equity_daily_bars.bcolz') min_offset = min(all_prices.attrs['calendar_offset'].itervalues()) max_offset = max(all_prices.attrs['calendar_offset'].itervalues()) calendar = pd.DatetimeIndex(all_prices.attrs['calendar'])[min_offset:max_offset] nassets = len(lifetimes.columns) ndates = len(calendar) nfields = len(('id', 'open', 'high', 'low', 'close', 'volume', 'date')) print "Number of Assets: %d" % nassets print "Number of Dates: %d" % ndates print "Naive Dataset Size: %s" % naturalsize( nassets * ndates * nfields * 8 ) Number of Assets: 20353 Number of Dates: 3480 Naive Dataset Size: 4.0 GB
  5. In [9]: !du -h -d0 data/equity_daily_bars.bcolz !du -h -d0 data/adjustments.db

    299M data/equity_daily_bars.bcolz 30M data/adjustments.db
  6. In [9]: !du -h -d0 data/equity_daily_bars.bcolz !du -h -d0 data/adjustments.db

    299M data/equity_daily_bars.bcolz 30M data/adjustments.db
  7. In [10]: import pandas as pd from zipline.utils.tradingcalendar import trading_day

    from zipline.pipeline.data import USEquityPricing from zipline.pipeline.loaders import USEquityPricingLoader loader = USEquityPricingLoader.from_files( 'data/equity_daily_bars.bcolz', 'data/adjustments.db' ) dates = pd.date_range( '2014-5-20', '2014-06-30', freq=trading_day, tz='UTC', )
  8. In [11]: Out[11]: # load_adjusted_array() returns a dictionary mapping columns

    to instances of `Ad justedArray`. (closes,) = loader.load_adjusted_array( columns=[USEquityPricing.close], dates=dates, assets=pd.Int64Index([24, 5061]), mask=None, ).values() closes Adjusted Array: Data: array([[ 604.4 , 39.74], [ 604.55, 39.69], [ 606.28, 40.35], ..., [ 90.35, 42.02], [ 90.92, 41.73], [ 91.96, 42.24]]) Adjustments: {13: [Float64Multiply(first_row=0, last_row=13, first_col=0, last_col=0, value=0 .142860)]}
  9. In [14]: Out[14]: dates_iter = iter(dates[4:]) window = closes.traverse(5) window

    _Float64AdjustedArrayWindow Window Length: 5 Current Buffer: [[ 604.4 39.74 ] [ 604.55 39.69 ] [ 606.28 40.35 ] [ 607.33 40.105] [ 614.14 40.12 ]] Remaining Adjustments: {13: [Float64Multiply(first_row=0, last_row=13, first_col=0, last_col=0, value=0 .142860)]}
  10. In [15]: # This cell is run multiple times to

    show the numbers scrolling up until we hit the split. data = next(window) print data print next(dates_iter) [[ 604.4 39.74 ] [ 604.55 39.69 ] [ 606.28 40.35 ] [ 607.33 40.105] [ 614.14 40.12 ]] 2014-05-27 00:00:00+00:00