Predicting the stock market with Twitter Nabil Freeman @nfrmn 

Hello •  Nabil Freeman •  Co-founder @ Matchchat •  www.matchchat.co.uk •  www.twitter.com/nfrmn 

About this project •  Hacked together in a weekend •  Research paper by Tushar Rao & Saket Srivastava in Delhi, India •  Soon to be open sourced 

Theory •  Twitter sentiment reflects investor behavior on the stock market (EMH) •  Real-time thanks to the Streaming API •  Until recently, implementations have been pretty dumb 

Anne Hathaway •  Berkshire Hathaway prices spike spuriously when Anne is in the media 

Sentiment analysis Know what the Twitter hivemind thinks 

Guts of the project •  Mine tweets for large technology stocks (MSFT, AMZN, AAPL, etc.) •  Calculate sentiment for each tweet, and create a trend •  Compare against historical closing price data for each company and find correlation 

Technology •  Node.js •  ntwitter – github:AvianFlu •  SentiMental – github:thinkroth •  Node Language Detect – github:FGRibreau •  Yahoo Finance API •  MySQL 

Scan.js •  Tracks an array of symbols on Twitter Streaming API using the $ hashtag •  Strips out any hyperlinks, retweets, copypasta & foreign tweets •  Stores the tweet with metadata in DB 

Finance.js •  Extract dataset from DB •  Run sentiment analysis on data collection •  Score each Tweet’s sentiment •  Pull CSV from Yahoo Finance and compare the two trend lines 

:-( 

Why? •  Scan.js crashed too often •  No dataset = no trends •  Later version with forever.js improved problem, but still no dice 

Improvements •  Try again with better technologies •  CouchDB (?) •  Understand node.js garbage collection •  Further down the line… •  Optimize sentiment tool for stock market language (bull, bear, dog, etc.) •  Write my own sentiment tool 

Call for help •  Going to open source it and try again •  Still a Javascript neophyte •  Any help would be appreciated •  Maybe we can all make some $$$ 

Thank you, goodnight, much love [email protected]