Data Mining 250 million Tweets a Day

Presentation by Paul Gesiak, Data Scientist @DataSift at Data Science London

Transcript

1. &RQWHQWWRSLFV 
2. 7KHPRVWSRSXODUFRQWHQW EDVHGRQILUHKRVHUDQGRPVDPSOH 
3. 79VKRZ
4. WYVKRZ 7RS%R\LVD%ULWLVKWHOHYLVLRQGUDPD VHULHVZKLFKZDVILUVWEURDGFDVWRQ &KDQQHO  7KHIRXUHSLVRGHVZHUHEURDGFDVW RYHUFRQVHFXWLYHQLJKWVIURP 2FWREHUWR1RYHPEHU  6HWRQWKHILFWLRQDO6XPPHUKRXVH KRXVLQJHVWDWHLQ+DFNQH\1RUWK /RQGRQ  7KHVHULHVIROORZVWKHOLYHVRID JURXSRIFULPLQDOVLQYROYHGLQGUXJ GHDOLQJDQGVWUHHWJDQJV 
5. WYVKRZ
6. 3URIHVVLRQDOVRQ7ZLWWHU   7ZLWWHUILUHKRVHRXWSXWRYHUPOQSRVWVGDLO\  FRQWDLQXVHUGHVFULSWLRQ
7. 5,0&(2VUHVLJQDWLRQ 5,0VRFLDOPHGLDFRQWHQW KRXUWLPHIUDPH -LP%DOVLOOLHDQG 0LNH/D]DULGLV WKHFR&(2VRI 5HVHDUFKLQ0RWLRQ DQQRXQFHGWKHLUUHVLJQDWLRQ 5,0KDVDQQRXQFHGRQ WKHLUZHEVLWHWKDWLWV &(2VDUHUHVLJQLQJ7KH QHZVEURNHRQ6XQGD\ HYHQLQJDQGJHQHUDWHG ODUJHVSLNHRIVRFLDOPHGLD DFWLYLW\ RYHUSRVWVDQ KRXU HYHU\VHF  7KHILUVWUHDFWLRQKDVEHHQ RYHUZKHOPLQJO\QHJDWLYH VKDUSVSLNHRIQHJDWLYH VHQWLPHQW UHGOLQH  
8. 5,0&(2VUHVLJQDWLRQ QHZVHYHQWLQVRFLDOPHGLDQHWZRUN *UDSKUHSUHVHQWLQJ SDWWHUQRIVRFLDOPHGLD DFWLYLW\JHQHUDWHG DURXQG5,0/WG    7LPHKRXU   6XQGD\QLJKW /RZOHYHORIDFWLYLW\
9. *UDSKUHSUHVHQWLQJ SDWWHUQRIVRFLDOPHGLD DFWLYLW\JHQHUDWHG DURXQG5,0/WG    7LPHKRXU   1HZVEUHDNVRXW ,QIRUPDWLRQVSUHDGV WURXJKWKHQHWZRUN 5,0&(2VUHVLJQDWLRQ QHZVHYHQWLQVRFLDOPHGLDQHWZRUN
10. *UDSKUHSUHVHQWLQJ SDWWHUQRIVRFLDOPHGLD DFWLYLW\JHQHUDWHG DURXQG5,0/WG    7LPHKRXU    &RPPHQWVDQG UHDFWLRQVWRWKHQHZV 5,0&(2VUHVLJQDWLRQ QHZVHYHQWLQVRFLDOPHGLDQHWZRUN
11. 5,0&(2VUHVLJQDWLRQ 5,0VRFLDOPHGLDFRQWHQW KRXUWLPHIUDPH -LP%DOVLOOLHDQG 0LNH/D]DULGLV WKHFR&(2VRI 5HVHDUFKLQ0RWLRQ DQQRXQFHGWKHLUUHVLJQDWLRQ /HWVIRFXVFORVHURQWKH SHULRGEHWZHHQ6XQGD\ DIWHUQRRQDQG0RQGD\ PRUQLQJZKHQWKHPDUNHW RSHQV
12. 5,0&(2VUHVLJQDWLRQ 5,0VRFLDOPHGLDFRQWHQW PLQWLPHIUDPH %\LQFUHDVLQJWKH WHPSRUDOUHVROXWLRQ IURPKWRPLQWLPH IUDPHZHFDQVHHPRUH GHWDLOVRIWKHHYHQW 7KHWUDIILFSLFNHGXS MXVWDIWHU5,0 DQQRXQFHGWKHQHZV RQWKHSP6XQGD\ (&7 ZKLFKLVDP 0RQGD\8&7DV VKRZQRQWKHSORW 7KH KLJKOHYHORILQWHUHVW ODVWVIRUDQKRXURUVR WKHQGHFOLQHVRYHU QLJKWWRLQFUHDVHLQ 1HZ<RUN6WRFN ([FKDQJHSUHPDUNHW KRXUVRQ0RQGD\ PRUQLQJ
13. 5,0&(2VUHVLJQDWLRQ 5,0VRFLDOPHGLDFRQWHQW PLQWLPHIUDPH /HWVVHHZKDW KDSSHQHGRQWKH1<6( IORRULQWKHILUVWKRXURI WUDGLQJRQ0RQGD\ PRUQLQJ (&7 8&7
14. 5,0&(2VUHVLJQDWLRQ 5,0VRFLDOPHGLDFRQWHQW PLQWLPHIUDPH 3HULRGWRIRFXVRQ
15. 5,0&(2VUHVLJQDWLRQ 5,0VRFLDOPHGLDFRQWHQW PLQWLPHIUDPH ,QFUHDVLQJWLPHLQWHUYDOVWR PLQIUDPHV 7KH5,0VWRFNRSHQVRQ 0RQGD\VLJQLILFDQWO\EHORZ WKH)ULGD\FORVHSULFHOHYHO DQGNHHSIDOOLQJLQWKHILUVW PLQ,WLVUDWKHU SUHGLFWDEOHFRQVLGHULQJWKH QDWXUHRIWKHQHZVDQGWKH VHQWLPHQWRIFRPPHQWV IURPSUHYLRXVHYHQLQJ +RZHYHUWKHVHQWLPHQWRI VRFLDOPHGLDFRQWHQW VZLWFKHVWRSRVLWLYHYDOXH DURXQG (&7  GHVSLWHSULFHVWLOOIDOOLQJ PLQODWWHUWKHVWRFNSULFH UHDFKHVWKHVXSSRUWOHYHODW 7KLVOHYHOLVWHVWHG DJDLQDIWHUDQG KROGVDVWKHVHQWLPHQWRI VRFLDOPHGLDFRQWHQW UHPDLQSRVLWLYH 
16. 5,0&(2VUHVLJQDWLRQ 'DWDDJJUHJDWLRQ &RQFOXVLRQ  %\DQDO\VLQJDIORZRIWKHVRFLDOPHGLDFRQWHQWZHJDLQWZRLQWHUHVWLQJSLHFHVRILQIRUPDWLRQ  :HFDQVHHLPPHGLDWHO\ZKDWLVWKHFURZGUHDFWLRQIRUWKHQHZV  D 7KHVLJQLILFDQFHRIWKHQHZVHYHQWLVSURSRUWLRQDOWRWKHVSLNHRIWUDIILFJHQHUDWHG  E 7KHGLIIHUHQFHEHWZHHQSRVLWLYHDQGQHJDWLYHVHQWLPHQWLQGLFDWHVLIWKHUHDFWLRQWRWKHQHZVLV SRVLWLYHQHJDWLYHRULQGLIIHUHQW  $QLQFUHDVHRISRVLWLYHVHQWLPHQWSUHFHGHGWKHPRPHQWZKHQDVWRFNSULFHKDGVWRSSHGIDOOLQJDQGUHDFKHG DVXSSRUWOHYHO  D 'LGWKHPDUNHWGHFLGHDWWKLVSRLQWWKDWWKHSULFHKDVEHHQGLVFRXQWHGHQRXJK"&HUWDLQO\WKH VHQWLPHQWRIWKHVRFLDOPHGLDFRQWHQWFRXOGVXJJHVWVR  E /RRNLQJDWWKH0RQGD\PRUQLQJFKDUWZHFDQVHHWKDWWKHVHQWLPHQWOHYHORIVRFLDOPHGLDFRQWHQW EHLQJDQLQWHUHVWLQJLQGLFDWRUVKRZLQJHPRWLRQVRIWKHFURZG)XUWKHUPRUHWKLVLQIRUPDWLRQFDQEH WUDFHGXVLQJYHU\VKRUWWLPHIUDPHV HYHQPLQ  
17. 'RZ-RQHV 6RFLDOPHGLDFRQWHQWDQDO\VLV 'RZ-RQHV PLQWLPHIUDPH  7KHVRFLDOPHGLDFRQWHQWFDQEH XVHGDVDQLQGLFDWRURI'RZ -RQHV,QGXVWULDO$YHUDJHSULFH ,QWKLVH[DPSOHUDSLGLQFUHDVHLQ WUDIILFDVVRFLDWHGZLWKDQHJDWLYH VHQWLPHQWSUHFHGHVGLUHFWO\D GURSLQSULFHE\SRLQWV
18. 3URIHVVLRQDOVRQ7ZLWWHU 
19. 3URIHVVLRQDOVRQ7ZLWWHU   7ZLWWHUILUHKRVHRXWSXWRYHU0SRVWVDGD\  FRQWDLQDXVHUGHVFULSWLRQ
20. 3URIHVVLRQDOVRQ7ZLWWHU  8VHU VGHVFULSWLRQLVDVKRUWELRQRWHRQDSXEOLFSURILOH  7KLVSURYLGHVDYROXPHRIRYHU0SLHFHVRILQIRUPDWLRQ VSHFLILFWRDQLQGLYLGXDOXVHU  $IWHUDWZHHWFRQWHQWWKHXVHU VGHVFULSWLRQLVWKHVHFRQG EHVWRSSRUWXQLW\IRUGDWDPLQLQJ 
21. 3URIHVVLRQDOVRQ7ZLWWHU  2YHU0WZHHWVGDLO\FRQWDLQDXVHU VSURIHVVLRQLQ WKHSURILOHGHVFULSWLRQ
22. 3URIHVVLRQDOVRQ7ZLWWHU 'DLO\FRQWHQWDJJUHJDWHGE\XVHUV RFFXSDWLRQ*HQHUDO/HYHO
23. 3URIHVVLRQDOVRQ7ZLWWHU 'DLO\FRQWHQWDJJUHJDWHGE\XVHUV RFFXSDWLRQ6LQJOH6HFWRU/HYHO
24. 3URIHVVLRQDOVRQ7ZLWWHU 'DLO\FRQWHQWDJJUHJDWHGE\XVHUV RFFXSDWLRQ'HWDLO/HYHO
25. 3URIHVVLRQDOVRQ7ZLWWHU 'DLO\FRQWHQWDJJUHJDWHGE\XVHUV RFFXSDWLRQ'HWDLO/HYHO
26. 3URIHVVLRQDOVRQ7ZLWWHU 'DLO\FRQWHQWDJJUHJDWHGE\XVHUV RFFXSDWLRQ'HWDLO/HYHO
27. -HGLRQ7ZLWWHU 'DLO\FRQWHQWDJJUHJDWHGE\XVHUV RFFXSDWLRQ'HWDLO/HYHO
28. GDWDVLIWFRP  #'DWD6LIW