Share
Embed
Transcript
65 slides
-
" غח" ై ۚ ӝ Әਲ਼ۚҗ ా҅ Ѩ July 23,
2016 नߧળ © Quant Study, Beomjun Shin, July 23, 2016 -
௫ ୡࠁ ੌੌ 1. (ಂఠ࠙ࢳ) ࣻহח ߎਸ ࢜ݴ ߸ࣻܳ ݃ҳ
߸ഋ೧ࠄ 2. Ӓۧѱ ݆ࣻ ۚ(?)ਸ ٜ݅ѱ ػ 3. ߮݃ب ޅӝח ٜۚী ઝೠ -> द ബਯੌѢঠ..(?) 4. যו ࣽр 10֙ ־ ࣻܫ 160% ب ҡଳই ࠁח ۚ ߊѼೠ 5. (оࢸѨ&पࢸ҅) ৬৬! Ӕؘ Ѣ Ҋ ై೧ب غա...??? © Quant Study, Beomjun Shin, July 23, 2016 -
(ಎ؊ݭణ) ߸ࣻܳ যڌѱ ࢎਊೞ? [߸ࣻ]ܳ [ӝ]۽ [߸ഋ]೧ ࠁҊ [ҙ]ীࢲ [ஏӝр]݅ఀ
ࠁਬ • ߸ࣻ: Value, Momentum, Quality • ӝ: 1ѐਘ, 1௪ఠ(=3ѐਘ), 1֙, 3~5֙ • ߸ഋ: пઙ زಣӐ, ݽݭథ, Valua9on Model • ҙ: Loser-Follow(contrarian), Winner-Follow • ஏӝр: 1ѐਘ, 1௪ఠ(=3ѐਘ), 1֙, 3~5֙ © Quant Study, Beomjun Shin, July 23, 2016 -
प ٘ SP500_YEARLY_VOL = 0.15 LENGTH = 252*10 SIZE =
100 DATEINDEX = pd.date_range(start="2011-01-01", periods=LENGTH) normal_walks = pd.DataFrame() for n in range(SIZE): normal_walks[n] = pd.Series( (np.random.normal( loc=0, scale=SP500_YEARLY_VOL / np.sqrt(252), size=(LENGTH, 1) ) + 1).cumprod(), index=DATEINDEX ) -> ࣻܫ 160%ח 100ߣ ೧ࠁݶ 1ߣ աৢ ࣻ ߆ী হ!1 1 ࣻܫ Normalਸ ٮܲҊ о, 10֙ زউ ైೠҊ о, S&P 500 ࣻળ Vola0lityܳ ࢎਊೣ © Quant Study, Beomjun Shin, July 23, 2016 -
ݾର • оࢸѨҗ पࢸ҅ • In-Sample ࠙ࢳ • p-value ৬
Mul/ple Tes/ng Problem ೖೞӝ • In/Out-Sample ࠙ࢳ • ಂఠ ࠙ࢳ • ೠ ߮݃ ࢸ • N࠙ਤ ࠙ࢳ प೯ © Quant Study, Beomjun Shin, July 23, 2016 -
ా҅ ӝࠄ ਊয • ా҅(sta&s&cs)? • ࠄ(sample)۽ࠗఠ ݽױ(popula&on)ਸ ਬ୶ೞח Ѫ
• ా҅(sta&s&c)? • ݽױ ౠਸ ୶ೞӝ ਤ೧ ࢠ۽ࠗఠ ҅ೞח ӏ • ࠄ࠙ನ(sampling distribu&on)? • ా҅(sta&s&c) ഛܫ࠙ನ • ࠄ࠙ನо ਃೞ. © Quant Study, Beomjun Shin, July 23, 2016 -
ా҅ োण • ୡҗ ࣻܫ(Excess Return)ী ೠ ୶ਸ ೧ࠁ •
о1: ୡҗ ࣻܫ ӏ࠙ನܳ ٮܲ • о2: ࢠ ࣻ(୨ োب ࣻ) n • ഛܫ߸ࣻ • ా҅(sta,s,c) ח ਬب ੋ t-࠙ನ(ࠄ࠙ನ;sampling distribu,on)ਸ ٮܴ ঌ۰ઉ • ҙणਵ۽ Әਲ਼ীࢶ t-sta,s,c 2ܳ ֈযঠ ࢎਊೞחѦ۽ ߉ই٘۰ © Quant Study, Beomjun Shin, July 23, 2016 -
p-value = type 1 error def. sta)s)c > cri)cal value
or significance level > p-value © Quant Study, Beomjun Shin, July 23, 2016 -
power = 1 - type 2 error def. sta)s)c >
cri)cal value © Quant Study, Beomjun Shin, July 23, 2016 -
Type 1 Error, Type 2 Error © Quant Study, Beomjun
Shin, July 23, 2016 -
Low p-value or High p-value in Stock Return? © Quant
Study, Beomjun Shin, July 23, 2016 -
T ా҅җ ࢥ࠺ਯ 2 2 r yearly returnۄ о, daily
return ܳ ғೞৈ োਯചػ ࢥ࠺ਯ۽ ࢎਊ. ݽ؛ Return, ח ߮݃ Return © Quant Study, Beomjun Shin, July 23, 2016 -
ୡҗࣻ ઓೞחо? • ӈޖоࢸ: ୡҗࣻ হ ( ) = 'ژ
ۚ'ۄ ೞ • ݀оࢸ: ୡҗࣻ ( ) = 'ঌ ۚ'ۄ ೞ © Quant Study, Beomjun Shin, July 23, 2016 -
ҡଳই ࠁח(?) ۚ оࢸѨ from scipy.stats import t nyears =
10 yearly_sharpe = 0.75 tstat = yearly_sharpe * np.sqrt(nyears) pvalue = 1 - t.cdf(tstat, df=9) print(tstat, pvalue) # => 2.37170824513 0.0208959003886 • p-value: 2% • Ѣ ೧ࢳ: ঌۚۄҊ ೠ ۚ ঌۚੌ ഛܫ 98% • ଵػ ೧ࢳ1: ژۚੋؘ ঌۚۄҊ ೡ ഛܫ 2% • ଵػ ೧ࢳ2: ژۚੋؘ ژۚۄҊ ೡ ഛܫ 98% © Quant Study, Beomjun Shin, July 23, 2016 -
Base Rate=0.1 10ѐח ঌۚ, 90ѐח ژۚ ©
Quant Study, Beomjun Shin, July 23, 2016 -
Sta$s$cal Power=0.8 10ѐ ঌۚ ୶ஏ ঌۚ
8ѐ © Quant Study, Beomjun Shin, July 23, 2016 -
P-value=0.02 90ѐ ژۚ ୶ஏ ঌۚ ѐ ©
Quant Study, Beomjun Shin, July 23, 2016 -
Python Code N = 100 base_rate = 0.10 alpha =
0.02 beta = 0.2 contingency_table = pd.DataFrame( [ [ N*base_rate*(1-beta), # pred ঌ, real ঌ N*base_rate*beta # pred ঌ, real ژ ], [ N*(1-base_rate)*alpha, # pred ژ, real ঌ N*(1-base_rate)*(1-alpha) # pred ژ, real ژ ] ], index=pd.Series(["ঌۚ", "ژۚ"], name="True"), columns=pd.Series(["ঌۚ", "ژۚ"], name="Pred")) fdr = contingency_table.iloc[1, 0] / contingency_table.iloc[:, 0].sum() print("FDR:", fdr) contingency_table © Quant Study, Beomjun Shin, July 23, 2016 -
P-valueী ೠ ೧ࢳ द ܻ • [ঌۚۄҊ ೠ ۚ ژۚੌ
ഛܫ] [2%] [X] • [ঌۚۄҊ ೠ ۚ ژۚੌ ഛܫ] False Discovery Rate(FDR)۽ [O] • [ژۚਸ ঌۚۄҊ ೡ ഛܫ] [2%] [O] • FDRਸ ஶ܀ೞח Ѫ ਃפ. © Quant Study, Beomjun Shin, July 23, 2016 -
Mul$ple Tes$ng Problem ࢎ۹1: ۨ٬ ۚী 100ѐ ےؒ ߸ചܳ
Ҋ ਗې ۚҗ P&L ਸ 2-sample t-test ೠҊ ೞ. 100ѐ ےؒ߸ചо ࠗ ޖೠ ࢚ട (base rate = 0)ীب p-valueܳ 5% ࣻળਵ۽ ౸ױೞݶ 5ѐח ਬೞѱ աৢ Ѫ. Ӓۧݶ Ӓ 5ѐ ۚਵ۽ ైೞݶ غחо? NO! ࢎ۹2: ୭Ҋ ై غח ߑߨ, 10000ݺ Ҋёীѱ Ӓ ߈ ղੌ झೖо য়ܲҊ ݫੌਸ ࠁղҊ աݠ ߈ ߈۽ ஏ Ѿҗܳ ࠁմ. Ѧ 10ߣ ߈ࠂೞݶ 10ݺ ҊёীѲ ղо 10ߣ ݽف ݏ ࣅ ػ. դ ୭ Ҋ ైੋо? NO! © Quant Study, Beomjun Shin, July 23, 2016 -
೧Ѿೞח 2о ҙ • Family-Wise Error Rate(FWER) • ੋ ҙীࢲ
ױ 1ѐ type 1 errorо ߊࢤೞח Ѫب ೲਊೞ ঋח • 100ѐ పझա 10000ѐ పझա ױ 1ѐ type 1 errorب ߊࢤೞݶ উػ • False Discovery Rate(FDR) • ࠺ਯ ҙীࢲ type 1 errorо ߊࢤೞחѦ ೲਊೞ ঋח. • 100ѐ పझীࢲ 1ѐܳ ೲਊೞ ঋחݶ 10000ѐ పझীࢶ 100ѐܳ ೲ ਊೞ ঋח © Quant Study, Beomjun Shin, July 23, 2016 -
FWER: Bonferroni’s Correc2on • 1ѐ పझ: p-value 5% • 10ѐ
పझ: p-value 5% -> 5/10 % • 100ѐ పझ: p-value 5% -> 5/100 % ࢤп. • ైۚ ইಫ۽ ۽ંо ইפӝ ٸޙী ઑӘ ਬೞѱ ࠊب غ ঋਸө? • ݅ড Nߣ పझо completely uncorrelatedݶ ਤۢ ೧ب ؼ Ѫ, ೞ ݅ perfectly correlated غݶ ࢎप࢚ పझח Nߣ ইפۄ 1ߣೠ ࣅ ػ © Quant Study, Beomjun Shin, July 23, 2016 -
FWER: Holm-Bonferroni Method • ծ p-value ࠗఠ ֫ p-value ࣽਵ۽
۳ • kח పझ ੋؙझ, M పझ പࣻ, \alphaח significance level • sta6s6c ࠙ನܳ ࢎਊೞח ѐ֛ © Quant Study, Beomjun Shin, July 23, 2016 -
FDR: Benjamini and Hochberg, Yeku9eli(BHY) • ֫ p-value ࠗఠ ծ
p-value ࣽਵ۽ ۳ • о וटೠ ߑߨ • harvey৯, о ೞҊ ঌ۰ ߑߨ © Quant Study, Beomjun Shin, July 23, 2016 -
ਃড • • Mul%ple Tes%ng Problem & Solu%on • Interpret
p-value © Quant Study, Beomjun Shin, July 23, 2016 -
Terms • (Sta&s&cs) In-sample(IS) Data / Out-of-sample(OOS) Data • (Machine
Learning) Training Data / Valida&on Data / Test Data / Cross-Valida&on © Quant Study, Beomjun Shin, July 23, 2016 -
അप प ࢸ҅: ण হח ݽ؛ 1. ୭Ӕ N֙ ؘఠܳ
Out-of-sample(OOS) ؘఠ, աݠܳ In-sample(IS) ؘ ఠ۽ ܻ࠙೧ك 2. IS ؘఠܳ оҊ पਸ ߈ࠂೠ • प ߈ࠂ ۽ ݽ؛ ߸ച, ۄఠ ߸ച۽ ҳࢿػ • ಌನݢझܳ ҙೠ പࣻܳ ߈٘द ӝ۾ೠ 3. о જও؍ TOP n ѐ ݽഋਸ OOS ؘఠী पೠ 4. प ౠࢿ࢚ য়ߡೖী ݒ ਬ೧ঠೣਵ۽ पਸ ݆ ߈ࠂೞ ঋח © Quant Study, Beomjun Shin, July 23, 2016 -
അप प ࢸ҅: ण ח ݽ؛ 1. ୭Ӕ N֙ਸ प
ؘఠীࢲ ߓઁ೧ك(Holdout data;Test Data) 2. Walk Forward Tes9ng ۨ(ৈ۞ߥ IS/OSS ؘఠܳ )ਵ۽ పझܳ ߈ࠂೠ • Walk Forward Tes9ng ب ߈ࠂೞࠁݶ ѾҴ OSS ؘఠী য়ߡೖػ • ಌನݢझܳ ҙೠ പࣻܳ ߈٘द ӝ۾ೠ • प ߈ࠂ ۽ ݽ؛ ߸ച৬ ೞಌۄఠ ߸ച۽ ҳࢿػ 3. о જও؍ TOP n ѐ ݽഋਸ OOS ؘఠী पೠ © Quant Study, Beomjun Shin, July 23, 2016 -
୭Ҋ प Real Money Trading! © Quant Study, Beomjun Shin,
July 23, 2016 -
Common Evalua+on Metrics • t-sta&s&c; p-value • sharpe ra&o •
Annualized Return, Annual Vola)lity • CAGR • turnover • maxdrawdown © Quant Study, Beomjun Shin, July 23, 2016 -
Sharpe Ra)o details Risk-Free Rate (Variance is not affected by
constant) Benchmark © Quant Study, Beomjun Shin, July 23, 2016 -
Annualize Sharpe Ra/o • IID Return Assump.on(NO SERIAL COLLREATION! REAL?)
• Daily -> Yearly: • Monthly -> Yearly: © Quant Study, Beomjun Shin, July 23, 2016 -
Addi$onal evalua$on metrics(TODO..җઁ?) • sor%no ra%o • calmar ra%o •
traynor ra%o • informa%on ra%o • winning ra%o • RoMAD • beta, alpha © Quant Study, Beomjun Shin, July 23, 2016 -
All that Gli)ers Is Not Gold: Sharpe Ra5o Specifically, we
find that commonly reported backtest evalua9on metrics like the Sharpe ra)o offer li.le value in predic)ng out of sample performance (R² < 0.025). In contrast, higher order moments, like vola)lity and maximum drawdown, as well as porEolio construc9on features, like hedging, show significant predic9ve value of relevance to quan9ta9ve finance prac99oners. - All that GliKers Is Not Gold. Quantopian © Quant Study, Beomjun Shin, July 23, 2016 -
All that Gli)ers Is Not Gold: Vola3lity Moreover, risk metrics
that aim to quan3fy vola%lity alone like annual vola3lity (Pearson R² = 0.67; p < 0.0001), and maximum drawdown (Pearson R² = 0.34; p < 0.0001) had sta3s3cally significant correla3ons between their IS and OOS period - All that GliRers Is Not Gold. Quantopian © Quant Study, Beomjun Shin, July 23, 2016 -
ਬೠ ߸ࣻ ӝ: 5࠙ਤ ಂఠ ࠙ࢳ & ߮݃ ࢸ ©
Quant Study, Beomjun Shin, July 23, 2016 -
ೠ ߮݃ ࢸ • Stock Picking • ߮݃ب ݽഋҗ э
ܻߖ۠य ӝ۽ प • ࠁా Equal Weighted Index • Single Asset • Buy and hold strategy ߮݃ܳ ਊೞৈ ݽ؛ ୡҗࣻܫ ҅ೠ © Quant Study, Beomjun Shin, July 23, 2016 -
߸ࣻ ߸ജ [߸ࣻ]ܳ [ӝ]۽ [߸ഋ]೧ ࠁҊ [ҙ]ীࢲ [ஏӝр]݅ఀ ࠁਬ •
߸ࣻ: Value, Momentum, Quality • ӝ: 1ѐਘ, 1௪ఠ(=3ѐਘ), 1֙, 3~5֙ • ߸ഋ: пઙ زಣӐ, ݽݭథ, Valua9on Model • ҙ: Loser-Follow(contrarian), Winner-Follow • ஏӝр: 1ѐਘ, 1௪ఠ(=3ѐਘ), 1֙, 3~5֙ © Quant Study, Beomjun Shin, July 23, 2016 -
1. Survivor-biased Data(Data Snooping) • NaN ؘఠ • ܻ Ҋػ
ਬפߡझ ೧Ѿ଼ • җѢ दীࢲ ਬפߡझ ؘఠ ࢎਊ © Quant Study, Beomjun Shin, July 23, 2016 -
2. Lookahead Bias • ؘఠ݃ ݽ؛ীࢲ पࣻೞӝ ए Overlap Bias
• য ߊ৬ ઙо ҙ҅ (য ઙоо աৡ ٍী ߊؽ) • ղо ݅ٚ ݽ؛ ইޖܻ ѐ౸ۄب 2008֙ ܻझ݅ ೖೡ ࣻ ݶ જ ۚ ؽ ೧Ѿ଼ • ؘఠ द ഛೞѱ ঈೞӝ(ؘఠ ಿ౹!) • "য়ט ઙо"ܳ NaNਵ۽ فҊ पೞӝ © Quant Study, Beomjun Shin, July 23, 2016 -
3. In-sample Backtes1ng • э ؘఠܳ فҊ ৈ۞ߣ ߈ࠂ प
റ э ؘఠীࢲ పझ ೧Ѿ଼ • Mul%ple Hypothesis Tes%ng য়ܨ ഛೞѱ ೧ೞӝ • प പࣻ/प ӝ۾ ԝԝೞѱ ۽Ӧ೧فӝ • IS/OOS పझ ೞӝ © Quant Study, Beomjun Shin, July 23, 2016 -
4. Market Impact, Slippage • ઙо Ѣېա زदഐо Ѣېীࢲ ղо
֍ ޙ оѺਸ ৢܽ • Ѣې ઙݾਸ ਬפߡझ۽ فҊ ݅ٚ ۚ • ೠ Rule of Thumb ҃ਵ۽ ٜ݅ ೧Ѿ଼ • ೠ ठܻೖܳ хউ೧ࢲ ߔపझ ࣻܫ ҅(e.g. S&P500 5bp, ઙ ݾ 50bp) • Ѣې ؘఠ ࢎਊೞӝ(Ѣې ֫ ઙݾ ࢎਊ; ࠁా ࣻ ҃ 10র ࢚?) © Quant Study, Beomjun Shin, July 23, 2016 -
5. Overfi)ng Model • ߈ࠂਵ۽ in-sample ؘఠܳ فҊ "tweak" ೞҊ
"refine"ೠ ݽ؛ • ߸ࣻо ݆Ҋ ࠂೠ ݽ؛ ࢎਊೞӝ ೧Ѿ଼ • IS/OOS పझ ೞӝ • п ਬפߡझ ҳࢿ ನಫܻয় ࣻܫী ࠺ ࠁӝ © Quant Study, Beomjun Shin, July 23, 2016 -
6. Trus(ng stateful strategy luck • Rebalancing ӝী ٮۄ ݽ؛
ࢿמ ѱѱ ߄Ո(۠ ݽ؛ਸ stateful strategyۄ ೠ ) • Walk-forward Tes;ngীࢲب start pointܳ য٣۽ ೞջী ٮۄ ࢿמ ׳ۄ ೧Ѿ଼ • оמೠ ݽٚ ܻߖ۠य ӝ, द ࠗ ܰѱ فҊ प(٘ܳ ੜ ىঠೠ!)ೞҊ ࣻܫ Varianceܳ ಝࠁӝ • গୡী Varianceо ݶ ۚ ޖೞҊ ࢤп೧ঠೠ • ٮۄࢲ, ୭ೠ 'daily' ؘఠܳ ࢎਊೞӝ © Quant Study, Beomjun Shin, July 23, 2016 -
7. Procedure Overfi.ng • ইޖܻ ਤ पٜࣻਸ ઑबೞ؊ۄب җѢח
җѢ۽ ରੋ য় ߡೖ ߈٘द ߊࢤ ೧Ѿ଼ • ݒੌ पઁ ؘఠ۽ Paper Trading Nѐਘ ࢚ ೧ࠄ ٍ पઁ تਸ ֍ӝ • ٮۄࢲ, ୭ೠ ࣘੋ दझమ غب۾ ݒੌ ؘఠо ୶оؼ ࣻ ѱ दझమ ҳࢿೞӝ © Quant Study, Beomjun Shin, July 23, 2016 -
8. Random Strategy • ো աৡ ۚੌө? ইקө? • ࢎਊೠ
ؘఠ ԝԝೞѱ ಝࠁӝ • খ 1~7ߣ पࣻח بਵ۽ ೞח पࣻо ইפۄ ؘఠܳ ԝԝೞѱ ೖ ঋইࢲ ߊࢤೞח पࣻ ೧Ѿ଼ • p-value ҅ ١ ా҅ ӝߨ ࢎਊೞҊ ୭ࣗೠ पਸ ৈ۞ߣ ೧ࠁӝ • ݽٚ Input ؘఠী ೠ द҅ৌ ࠁӝ, NaN ؘఠ ਬޖ ࠁӝ © Quant Study, Beomjun Shin, July 23, 2016 -
9. Don't Forget Commission, Tax • ௫ۚ ӝࠄਵ۽ ݒݒഥਯ ֫.
ࣻࣻܐח ߈٘द хউ೧ঠೣ ೧Ѿ଼ • ࣻࣻܐ ݽ؛ ٜযр ߔపझ ҅ ۽ ٜ݅যفӝ • ࣻࣻܐ ҅ য۵ݶ ୭ࣗೠ ఢয়ߡח ҅ೞӝ © Quant Study, Beomjun Shin, July 23, 2016 -
We need our powerful backtest tool ! • ࣚए ࢎਊ
& ࡅܲ ࢎਊ • ਃ evalua)on metric ҅(return, turnover etc.) • п ઙݾ ҳࢿ ࠺ਯ • ࣻࣻܐ Ҋ۰ ࣻܫ ҅ • пઙ Visualiza)on © Quant Study, Beomjun Shin, July 23, 2016 -
Quant ױ࢚ • ־ҳա ࠁח ؘఠ۽ ۚ ٜ݅ӝ • Contrarian
vs Follower, ѾҴ Behavior ஏऱ? • Sinificanceܳ ӝূ ցޖաب ؘఠ • যڌѱ ա݅ ؘఠܳ ݅٘חо? © Quant Study, Beomjun Shin, July 23, 2016 -
Future Works • झझ۽ ٣పੌ ӝ • ೧৻ ૐӂࢎ ܻನ
झఠ٣ • ݒ HIT ֫ও؍ SSRN ಕಌ ࣁա • पઁ ైೞӝ © Quant Study, Beomjun Shin, July 23, 2016 -
References • wiki: Sensi+ve and specficity • The p value
and the base rate fallacy • Evalua+on Metrics • All that Gli@ers Is Not Gold • 9 Mistakes Quants Make that Cause Backtests to Lie by Tucker Balch, Ph.D © Quant Study, Beomjun Shin, July 23, 2016