MongoDB Sao Paulo 2012: Coisas que aprendemos usando Mongoid com grande quantidade de dados

Fabio Perrella, Locaweb
Na Locaweb, estamos refazendo o sistema de Email Markering (Saas) utilizando Rails com Mongoid, irei falar sobre esses tópicos:
por que escolhemos o mongodb e o mongoid
como implementamos campos customizaveis no modelo Contact usando field type hash do mongoid, como usamos os indices para isso
relacionamento "belongstomany" que inventamos para simular um hasandbelongstomany e não precisar trabalhar com documentos grandes. Estou fazendo uma gem para compartilhar isso em https://github.com/fabioperrella/mongoidbelongsto_many
como fizemos bulk insert de grande massa de dados utilizando mongoimport e resque
alguns operadores de update do mongodb que usamos como inc, pushAll, addToSet, rename
como estamos usando sharding, quais os critérios utilizados para escolha das sharding keys
como estamos usando replica set
maneiras para monitorar usando mongostat, MMS, db.stats(), db.collectionName.stats(), rs.serverStatus(), rs.status(), db.currentOp()
testes de carga que fizemos
problemas que enfrentamos, como por exemplo alguns bugs conhecidos do mongodb

Transcript

1. &RLVDVTXHDSUHQGHPRVXVDQGR 0RQJRLGFRPJUDQGHTXDQWLGDGHGH GDGRV #IDELRSHUUHOOD
2. (PDLO0DUNHWLQJ/RFDZHE Ɣ 6DD6 Ɣ 0XOWL7HQDQW Ɣ *UDQGHTXDQWLGDGHGHGDGRV Ɣ *UDQGHKLVWyULFRGHGDGRV Ɣ 6LVWHPDDQWLJREDVHDGRQRSKSOLVW Ɣ !1RYDSODWDIRUPDHP5DLOV0RQJR'%
3. 0RQJR'% Ɣ 6LPSOLFLGDGH Ɣ 6KDUGLQJJUDQGHTWGHGHGDGRV Ɣ 6FKHPDIUHHFDPSRVFXVWRPL]DGRVFRQWDWRVILOWURV Ɣ 5HSOLFD6HW$OWDGLVSRQLELOLGDGH Ɣ 2SHUDo}HV$WRPLFDVLQFDGG7R6HWSXOOSXVK Ɣ 3HUIRUPDQFH Ɣ 0DWXULGDGHPRQJRVS Ɣ $WXDOL]Do}HVFRQVWDQWHV
4. 0RQJRLG Ɣ 6LPSOLFLGDGH Ɣ %RDGRFXPHQWDomR Ɣ 0RQJR0DSSHUWLQKDEXJFRPPXLWRV GRFXPHQWRVHPEHGDGRV ID]WHPSR Ɣ HHVWiFKHJDQGRDYHUVmRPXLWRPHOKRU
5. ,PSOHPHQWDomRGH&DPSRV&XVWRPL]iYHLV
6. Ɣ DLGHLD0RQJRLGG\QDPLFILHOGV DOORZBG\QDPLFBILHOGV ,PSOHPHQWDomRGH&DPSRV&XVWRPL]iYHLV
7. Ɣ DLGHLD0RQJRLGG\QDPLFILHOGV DOORZBG\QDPLFBILHOGV Ɣ 3UREOHPDV ż 3URWHomRGHFDPSRVLQWHUQRV VWDWXVLGHWF ż 5HVWULomRQRQRPHGRVFDPSRVFXVWRPL]iYHLV ż DOORZBG\QDPLFBILHOGVpJOREDO Ɣ 9DQWDJHQV" ż ,QGLFHQRVFDPSRV ,PSOHPHQWDomRGH&DPSRV&XVWRPL]iYHLV
8. Ɣ DLGHLDILHOGFXVWRPBILHOGVW\SHKDVK ,PSOHPHQWDomRGH&DPSRV&XVWRPL]iYHLV
9. Ɣ DLGHLDILHOGFXVWRPBILHOGVW\SHKDVK Ɣ 9DQWDJHQV ż 3URWHomRGRVFDPSRVLQWHUQRV ż /LEHUGDGHQRQRPHGRVFDPSRV Ɣ 3UREOHPDV" ż ËQGLFHQRVFDPSRV ,PSOHPHQWDomRGH&DPSRV&XVWRPL]iYHLV
10. Ɣ ,QGLFHQDVFKDYHVGHXPKDVK LQFOXVLYHHP FKDYHVTXHQmRH[LVWHP ,PSOHPHQWDomRGH&DPSRV&XVWRPL]iYHLV
11. Ɣ &XLGDGRFRPLQGLFHV Ɣ *HUDomRGHLQGLFHID]ORFNGR'%LQWHLUR ż PLQFROOHFWLRQFRPPLOKRHVGHGRFXPHQWRV Ɣ &XVWRDOWR PHPRULDHVSDoRHPGLVFRHVFULWD ,PSOHPHQWDomRGH&DPSRV&XVWRPL]iYHLV
12. ,PSOHPHQWDomRGH&DPSRV&XVWRPL]iYHLV Ɣ 6HRFOLHQWHTXLVHUWURFDURQRPHGRV FDPSRV" Ɣ 7RPDHVVDRSHUDGRUUHQDPH a$OWHUWDEOH
13. Ɣ %XVFDVWLSDGDV VWULQJGDWHLQW &RQWDFWZKHUH FXVWRPBILHOGVGDWH ! ^OW !7LPHSDUVH  ` Ɣ 2SHUDGRUHVGHFRQGLomR KWWSZZZPRQJRGERUJGLVSOD\'2&6$GYDQFHG4XHULHV ,PSOHPHQWDomRGH&DPSRV&XVWRPL]iYHLV
14. Ɣ %XJQR0RQJR'%RUGHQDomRGHLQGH[ Ɣ &DVH6HQVLWLYH KWWSMLUDPRQJRGE RUJEURZVH6(59(5 Ɣ )L[FULDUFDPSRHPORZHUFDVH ^ BLG2EMHFW,G IGIGDGDEGD  HPDLOEE#EEFRP QDPH3RSR/DOD QDPHBOFSRSRODOD ` ,PSOHPHQWDomRGH&DPSRV&XVWRPL]iYHLV
15. %HORQJV7R0DQ\
16. %HORQJV7R0DQ\ Ɣ 2So}HVGHPRGHODJHP ż  (PEHG'RFXPHQWV ż 9DQWDJHQV Ŷ &RQVXOWDUiSLGDGRVGDGRV ż 'HVYDQWDJHQV Ŷ 7DPDQKRGRGRFXPHQWR Ŷ 3UHFLVDPRVVDEHUDVPHQVDJHQVTXHRVFRQWDWRV UHFHEHUDP UHIHUrQFLD"
17. %HORQJV7R0DQ\
18. %HORQJV7R0DQ\ Ɣ 3UREOHPDV ż 7DPDQKRPi[LPRGHXPGRFXPHQWR0% Ŷ PHOKRUQmRSDVVDUGH.% ż $OWHUQDWLYDTXHEUDUHPFKXQNV!FRPSOH[LGDGH ż *ULGIV"QmR
19. %HORQJV7R0DQ\ Ɣ 2So}HVGHPRGHODJHP ż  +DV$QG%HORQJVWR0DQ\ ż 9DQWDJHQV Ŷ 7DPDQKRGRVGRFXPHQWRVPHQRU" Ŷ &RQVLVWrQFLD ż 'HVYDQWDJHQV Ŷ %XVFD Ŷ 7DPDQKRGRVGRFXPHQWRV
20. %HORQJV7R0DQ\
21. %HORQJV7R0DQ\ Ɣ 2So}HVGHPRGHODJHP ż  +DV$QG%HORQJVWR0DQ\ Ŷ 7DPDQKRGRGRFXPHQWRFRPPLO2EMHFW,GV
22. %HORQJV7R0DQ\ Ɣ 3UREOHPDV ż &RPSOH[LGDGHQRFRGLJRSDUDXVDURQO\ ILHOGV
23. %HORQJV7R0DQ\ Ɣ 2So}HVGHPRGHODJHP ż  +DV$QG%HORQJVWR0DQ\
24. %HORQJV7R0DQ\ Ɣ 2So}HVGHPRGHODJHP ż  +DV$QG%HORQJVWR0DQ\
25. %HORQJV7R0DQ\ Ɣ 2So}HVGHPRGHODJHP ż  +DV$QG%HORQJVWR0DQ\ ż 9DQWDJHQV Ŷ 7DPDQKRGRVGRFXPHQWRVPHQRU Ŷ &RQVLVWrQFLD ż 'HVYDQWDJHQV Ŷ %XVFD" Ŷ ,PSOHPHQWDomRIRUDGRSDGUmR
26. %HORQJV7R0DQ\
27. %HORQJV7R0DQ\
28. %HORQJV7R0DQ\ *(0 ZLS  KWWSVJLWKXEFRPIDELRSHUUHOODPRQJRLGBEHORQJVBWRBPDQ\
29. %HORQJV7R0DQ\ Ɣ 2So}HVGHPRGHODJHP ż  +DV$QG%HORQJVWR0DQ\ ż ,QGLFHHPFDPSRVWLSRDUUD\ LQGH[PHVVDJHBLGV
30. 0RQJRLPSRUW Ɣ )HDWXUH,PSRUWDomRGHFRQWDWRVHPPDVVD Ɣ 3UREOHPD ż /LVWDVFRPPDLVGHPLOOFRQWDWRV ż &RQFRUUrQFLDHQWUHFOLHQWHV ż 3UHFLVDVHUUiSLGR Ɣ 6ROXomR ż PRQJRLPSRUW
31. 0RQJRLPSRUW KWWSZZZPRQJRGE RUJGLVSOD\'2&6,PSRUW([SRUW7RROV,PSRUW([SRUW7RROVPRQJRLPSRUW Ɣ 3RQWRSRVLWLYR ż 9HORFLGDGH PDLVGHPLOGRFXPHQWRVVHJXQGR Ɣ 3RQWRV1HJDWLYRV ż 9DOLGDomRGRVGDGRV ż 1mRH[LVWHFRPRID]HU0HUJHGHGDGRV ż 'LILFLOWUDWDPHQWRGHHUURVURGDQGRFRPV\VWHP
32. 0RQJRLPSRUW Ɣ 9DOLGDomRGRVGDGRV
33. 0RQJRLPSRUW Ɣ WRBPRQJRBMVRQ ż FDUUHJDUREMHWRFRPGDGRVGR&69HYDOLGDU ż FRQYHUVmRGH2EMHFW,GV ż FRQYHUVmRGHGDWDV WLPHVWDPSHPPLOLVHFRQGV ż FRQYHUVmRGH$UUD\GH2EMHFW,GV UHODFLRQDPHQWRV ż FRQYHUVmRGH+DVKHV ż JHUDU-621QRIRUPDWRGRPRQJRLPSRUW Ɣ %XJGDWDHSRFK KWWSVMLUDPRQJRGERUJEURZVH6(59(5 
34. 0RQJRLPSRUW Ɣ 0HUJHGHGDGRVQDRH[LVWH ż RSomRXSVHUWDWXDOL]DGRFXPHQWRLQWHLURQmRp SRVVLYHOID]HUPHUJHKWWSVMLUDPRQJRGE RUJEURZVH6(59(5 ż 3(5,*2FRPGDGRVTXHSRGHPVHUDWXDOL]DGRV SRURXWUROXJDU FRXQWVHWF ż 6ROXomRFROOHFWLRQXSGDWH
35. 0RQJRLPSRUW Ɣ 0HUJHGHGDGRVQDRH[LVWH ż FROOHFWLRQXSGDWHDSHQDVQRVFDPSRVTXH SUHFLVDPVHUDWXDOL]DGRV
36. 0RQJRLPSRUW Ɣ 7UDWDPHQWRGHHUURV
37. 8SGDWLQJ$WRPLF2SHUDWRUV Ɣ KWWSHQZLNLSHGLD RUJZLNL/LQHDUL]DELOLW\([DPSOHBDWRPLFBRSHUDWLRQ Ɣ KWWSHQZLNLSHGLDRUJZLNL&RPSDUHDQGVZDS
38. 8SGDWLQJ$WRPLF2SHUDWRUV Ɣ KWWSZZZPRQJRGERUJGLVSOD\'2&68SGDWLQJ Ɣ LQF Ɣ VHWXQVHW Ɣ SXVK Ɣ SXVK$OO Ɣ DGG7R6HWHHDFK Ɣ SRSSXOO Ɣ SXOO$OO Ɣ UHQDPH
39. 8SGDWLQJ$WRPLF2SHUDWRUV Ɣ LQFLQFUHPHQWDXPQ~PHUR ż SRGHVHUXVDGRFRPFDPSRQLO
40. 8SGDWLQJ$WRPLF2SHUDWRUV Ɣ LQF ż SRGHVHUXVDGRHPKDVKHV PHVPRYD]LRV
41. 8SGDWLQJ$WRPLF2SHUDWRUV Ɣ LQFFRPPXOWLWUXH HP1GRFXPHQWRV ż VRPHQWHFRPRFROOHFWLRQXSGDWHQRPRQJRLG
42. 8SGDWLQJ$WRPLF2SHUDWRUV Ɣ VHWVHWDYDORUGHXPFDPSR ż QRPRQJRLGVypSRVVtYHODWXDOL]DUFDPSRFRPRVHW
43. 8SGDWLQJ$WRPLF2SHUDWRUV Ɣ DGG7R6HW ż LQVHUHHOHPHQWRHPXPDUUD\VHMiQmRH[LVWLUQHOH ż SRGHID]HUPDLVGHXPDRSHUDomRSRUYH]SRUH[HPSOR FRP,1&
44. 8SGDWLQJ$WRPLF2SHUDWRUV Ɣ DGG7R6HWFRPHDFK ż LQVHUH1HOHPHQWVRHPXPDUUD\VHQmRH[LVWLUHPQHOH
45. 8SGDWLQJ$WRPLF2SHUDWRUV Ɣ SXOOSXOO$OO ż UHPRYH1HOHPHQWRVGHXPDUUD\
46. 8SGDWLQJ$WRPLF2SHUDWRUV Ɣ UHQDPH ż UHQRPHLDQRPHGHFDPSRVLQFOXVLYHGHKDVKHV
47. 6KDUGLQJ Ɣ 1mRGHL[HSUDGHSRLVVHMiVDEHTXHYDLSUHFLVDU ż WHPSRSDUDLPSOHPHQWDomRWHVWHV ż VKDUGEDODQFLQJSRGHVREUHFDUUHJDUPDTXLQDVHMD HVWLYHUQROLPLWH ż DOJXPDVFRLVDVQmRIXQFLRQDP RXQmRIXQFLRQDP GLUHLWR FRPVKDUGH[HPSOR Ŷ FRXQWV Ŷ PRQJRGEXQLTXHLQGH[
48. 6KDUGLQJ Ɣ QmRGHL[HSUDGHSRLVVHMDVDEHTXHYDLSUHFLVDU ż PRQJRLGVKDUGBNH\ VHQDRWLYHUGiHUURHPXSGDWH LQVHUWV
49. 6KDUGLQJ Ɣ QmRGHL[HSUDGHSRLVVHMDVDEHTXHYDLSUHFLVDU ż 6DIHPRGHOLJDUQRWHVWGHYHORSPHQWGHVOLJDUHPSURG
50. 6KDUGLQJ Ɣ PRQJR6FRQILJ6HUYHUUHSOLFD6HWPRQJR'%
51. 6KDUGLQJ Ɣ &KXQNVVSOLWWLQJHEDODQFLQJ ż &KXQNSHGDoRGHXPDFROOHFWLRQ ż 6SOLWWLQJTXHEUDGHFKXQNTXDQGRILFDJUDQGH 0 ż %DODQFLQJPRYHUFKXQNSDUDRXWURVKDUGTXDQGRRDWXDOHVWi PDLVFKHLR
52. 6KDUGLQJ Ɣ &KXQNVVSOLWWLQJHEDODQFLQJ ż ([HPSOR&ROOHFWLRQ8VHUVFRPFDPSRPrVGHDQLYHUViULR ż 6KDUGLQJNH\PrVGHDQLYHUViULR &RPHoR )LP 6KDUG IHYHUHLUR PDUoR $ VHWHPEUR LQILQLWR $ &RPHoR )LP 6KDUG LQILQLWR IHYHUHLUR % PDUoR VHWHPEUR %
53. 6KDUGLQJ Ɣ &KXQNVVSOLWWLQJHEDODQFLQJ ż DSyVVSOLWWLQJ &RPHoR )LP 6KDUG IHYHUHLUR MXQKR $ MXQKR PDUoR $ VHWHPEUR LQILQLWR $ &RPHoR )LP 6KDUG LQILQLWR IHYHUHLUR % PDUoR VHWHPEUR %
54. 6KDUGLQJ Ɣ (VFROKDGDVKDUGNH\ ż ,WLVLPSRUWDQWWRFKRRVHWKHULJKWVKDUGNH\IRUDFROOHFWLRQ,I WKHFROOHFWLRQLVJLJDQWLFLWLVGLIILFXOWWRFKDQJHWKHNH\ODWHU :KHQLQGRXEWSOHDVHDVNIRUVXJJHVWLRQVLQWKHVXSSRUW IRUXPVRU,5& DOLQKDQRVLWHGRPRQJRGE
55. 6KDUGLQJ Ɣ (VFROKDGDVKDUGNH\ ż SHQVDUQDHVFULWDHOHLWXUDGRVGDGRV Ŷ ORDGEDODQFLQJQDHVFULWDGHYHGLYLGLUGDGRVHQWUHRV VKDUGV Ŷ QDOHLWXUDWHQWDUQmRYDUUHUYiULRVVKDUGVSDUDWHUD UHVSRVWD
56. 6KDUGLQJ Ɣ (VFROKDGDVKDUGNH\ ż FRPRQmRGHYHVHU Ŷ ORZFDUGLQDOLW\SRUH[HPSORFRQWLQHQWHV DIULFDDPHULFD HXURSD Ŷ DVFHQGLQJVKDUGNH\SRUH[HPSORREMHFWLGYDLFRQFHQWUDU WRGDVOHLWXUDVHHVFULWDVHPXPVKDUG Ŷ UDQGRPVKDUGNH\SRUH[HPSORPGTXDVHVHPSUHYDL SUHFLVDUMXQWDURVGDGRVGHYDULRVFKXQNVQDRYDL FRQVHJXLUGHL[DUGDGRVSDUHFLGRVMXQWRV
57. 6KDUGLQJ Ɣ (VFROKDGDVKDUGNH\ ż UHFRPHQGDGRREMHFWLGVHDUFKNH\ ([HPSOR ,QLFLR )LP 6KDUG LQILQLWR $IULFD $ $IULFD $IULFD % $IULFD (XURSD $ (XURSD (XURSD % (XURSD LQILQLWR %
58. 5HSOLFDVHW Ɣ 3UHILUDHVFDODUXVDQGRVKDUGVHQmRUHSOLFDVHWVFRPOHLWXUDQRV VHFXQGiULR VODYHBRN WUXHQR0RQJRLG Ɣ /HLWXUDLQVHJXUD1RVVHFXQGiULRVSRGHPKDYHUGDGRVDQWLJRV Ɣ $LQGDHPHVWXGRSRUQyV ż HVWUDWpJLDGHEDFNXSHPVHFXQGiULRV ż FRPSDFWDomRGHGDGRVQRVVHFXQGiULRV ż MRXUQDOLQJKDELOLWDGRDSHQDVQRVHFXQGiULRGHEDFNXS
59. 0RQLWRUDomR Ɣ PRQJRVWDWGLVFRYHU
60. 0RQLWRUDomR Ɣ GEVWDWV Ɣ GEFROOHFLRQ1DPHVWDWV Ɣ UVVHUYHU6WDWXV Ɣ UVVWDWXV Ɣ GEFXUUHQW2S Ɣ 0RQJR'%0RQLWRULQJ6HUYLFH 006 ż 6DD6 ż )UHH
61. 7HVWHVGHFDUJD Ɣ YHORFLGDGHGHHVFULWD H[LPSRUWDomRGHFRQWDWRV Ɣ DPELHQWHGHKRPRORJDomRURGDQGRFRPEDVHFDUUHJDGD Ɣ DQDOLVDUTXHULHVOHQWDVFRPH[SODLQ QRFRQVROHGRPRQJR Ɣ VLPXODUFDVRVH[WUHPRV H[FOLHQWHJUDQGHFRPPXLWRVFRQWDWRV Ɣ GHILQLUOLPLWHVGHVHJXUDQoD