Developing an Expression Language for Quantitative Financial Modeling

This talk details the challenges addressed during the development of Zipline's new Pipeline API, which provides a high-level expression language allowing users to describe computations on rolling windows of continuously-adjusted financial data. We discuss the notion of "perspectival" time-series data, arguing that this concept provides a useful framework for formally reasoning about financial data in the face of domain oddities like stock splits, dividends, and restatements.

We also consider the architectural and performance benefits of developing an API focused on symbolic computation, drawing comparisons to several recent developments in the Python numerical ecosystem.

Transcript

8. Large Problem Medium Problem Medium Problem Medium Problem Small Problem Small Problem Small Problem Small Problem Small Problem Small Problem Small Problem Small Problem
14. mean median first last stddev rank() zscore()
15. factor1 > factor2 & | rank() percentile()
16. cross_product()
17. Factors Filters Classifiers (2)
21. a + a + a 3a
22. SQL numpy
29. In [3]: Out[3]: from zipline.assets import AssetFinder finder = AssetFinder("sqlite:///data/assets.db") lifetimes = finder.lifetimes( dates=pd.date_range('2001-01-01', '2015-10-01'), include_start_date=True, ) lifetimes.head(5)
30. In [4]: daily_count = lifetimes.sum(axis=1) daily_count.plot(title="Companies in Existence by Day");
32. In [5]: AAPL_prices = pd.read_csv( 'data_public/AAPL-split.csv', parse_dates=['Date'], index_col='Date', ) def plot_prices(prices): price_plot = prices.plot(title='AAPL Price', grid=False) price_plot.set_ylabel("Price", rotation='horizontal', labelpad=50) price_plot.vlines( ['2014-05-08'], 0, 700, label="$3.05 Dividend", linestyles='dotted', colors='black', ) price_plot.vlines( ['2014-06-09'], 0, 700, label="7:1 Split", linestyles='--', colors='black', ) price_plot.legend() sns.despine() return price_plot
33. In [6]: plot_prices(AAPL_prices);
34. In [7]: naive_returns = AAPL_prices.pct_change() naive_returns.plot();
49. In [8]: from bcolz import open from humanize import naturalsize all_prices = open('data/equity_daily_bars.bcolz') min_offset = min(all_prices.attrs['calendar_offset'].itervalues()) max_offset = max(all_prices.attrs['calendar_offset'].itervalues()) calendar = pd.DatetimeIndex(all_prices.attrs['calendar'])[min_offset:max_offset] nassets = len(lifetimes.columns) ndates = len(calendar) nfields = len(('id', 'open', 'high', 'low', 'close', 'volume', 'date')) print "Number of Assets: %d" % nassets print "Number of Dates: %d" % ndates print "Naive Dataset Size: %s" % naturalsize( nassets * ndates * nfields * 8 ) Number of Assets: 20353 Number of Dates: 3480 Naive Dataset Size: 4.0 GB
50. In [9]: !du -h -d0 data/equity_daily_bars.bcolz !du -h -d0 data/adjustments.db 299M data/equity_daily_bars.bcolz 30M data/adjustments.db
51. In [9]: !du -h -d0 data/equity_daily_bars.bcolz !du -h -d0 data/adjustments.db 299M data/equity_daily_bars.bcolz 30M data/adjustments.db
54. In [10]: import pandas as pd from zipline.utils.tradingcalendar import trading_day from zipline.pipeline.data import USEquityPricing from zipline.pipeline.loaders import USEquityPricingLoader loader = USEquityPricingLoader.from_files( 'data/equity_daily_bars.bcolz', 'data/adjustments.db' ) dates = pd.date_range( '2014-5-20', '2014-06-30', freq=trading_day, tz='UTC', )
55. In [11]: Out[11]: # load_adjusted_array() returns a dictionary mapping columns to instances of `Ad justedArray`. (closes,) = loader.load_adjusted_array( columns=[USEquityPricing.close], dates=dates, assets=pd.Int64Index([24, 5061]), mask=None, ).values() closes Adjusted Array: Data: array([[ 604.4 , 39.74], [ 604.55, 39.69], [ 606.28, 40.35], ..., [ 90.35, 42.02], [ 90.92, 41.73], [ 91.96, 42.24]]) Adjustments: {13: [Float64Multiply(first_row=0, last_row=13, first_col=0, last_col=0, value=0 .142860)]}
56. In [14]: Out[14]: dates_iter = iter(dates[4:]) window = closes.traverse(5) window _Float64AdjustedArrayWindow Window Length: 5 Current Buffer: [[ 604.4 39.74 ] [ 604.55 39.69 ] [ 606.28 40.35 ] [ 607.33 40.105] [ 614.14 40.12 ]] Remaining Adjustments: {13: [Float64Multiply(first_row=0, last_row=13, first_col=0, last_col=0, value=0 .142860)]}
57. In [15]: # This cell is run multiple times to show the numbers scrolling up until we hit the split. data = next(window) print data print next(dates_iter) [[ 604.4 39.74 ] [ 604.55 39.69 ] [ 606.28 40.35 ] [ 607.33 40.105] [ 614.14 40.12 ]] 2014-05-27 00:00:00+00:00
59. float bool
60. dask