Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Understanding distributed processing in Python

Avatar for chie8842 chie8842
June 16, 2019
Technology
2
2.2k

Understanding distributed processing in Python

#pyconth2019
https://th.pycon.org/talks/#row-47
Slides of my talk at PyConTh2019

Avatar for chie8842

chie8842

June 16, 2019
Tweet

More Decks by chie8842

See All by chie8842
MongoDB Atlas：モダンなアプリ開発を支えるデータプラットフォームのご紹介
Avatar for chie8842 chie8842
0
32
MongoDB Vectorsearchではじめるカスタマイズ可能な生成AIアプリ開発
Avatar for chie8842 chie8842
0
32
MongoDB Atlas Search のご紹介
Avatar for chie8842 chie8842
2
2.1k
MongoDB Atlas Vectorsearchではじめる生成AIアプリ開発
Avatar for chie8842 chie8842
3
2k
AWS GlueとAWS Lake Formationではじめるデータマネジメント
Avatar for chie8842 chie8842
0
1.2k
Distributed Processing in Python
Avatar for chie8842 chie8842
2
830
クックパッドにおける推薦（と検索）の取り組み
Avatar for chie8842 chie8842
20
8.2k
Performance Tuning Tips of TensorFlow Inference
Avatar for chie8842 chie8842
1
780
クックパッドにおけるCloud AutoML事例
Avatar for chie8842 chie8842
9
8.1k

Other Decks in Technology

See All in Technology
SREが向き合う大規模リアーキテクチャ 〜信頼性とアジリティの両立〜
Avatar for Kuniaki Moriya zepprix
0
400
AIと新時代を切り拓く。これからのSREとメルカリIBISの挑戦
Avatar for ktykogm 0gm
0
730
（金融庁共催）第4回金融データ活用チャレンジ勉強会資料
Avatar for tak takumimukaiyama
0
120
セキュリティ はじめの一歩
Avatar for nikinusu nikinusu
0
1.5k
クレジットカード決済基盤を支えるSRE - 厳格な監査とSRE運用の両立 (SRE Kaigi 2026)
Avatar for capytan capytan
6
2.5k
2026年、サーバーレスの現在地 -「制約と戦う技術」から「当たり前の実行基盤」へ- /serverless2026
Avatar for Serverless Operations slsops
2
210
Mosaic AI Gatewayでコーディングエージェントを配るための運用Tips / JEDAI 2026 新春 Meetup! AIコーディング特集
Avatar for GENDA genda
0
150
Digitization部 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
1
6.8k
広告の効果検証を題材にした因果推論の精度検証について
Avatar for ZOZO Developers zozotech
PRO
0
110
【インシデント入門】サイバー攻撃を受けた現場って何してるの？
Avatar for Shumei Ito shumei_ito
0
1.5k
SREじゃなかった僕らがenablingを通じて「SRE実践者」になるまでのリアル / SRE Kaigi 2026
Avatar for AEON aeonpeople
6
2.1k
名刺メーカーDevグループ 紹介資料
Avatar for Sansan, Inc. sansan33
PRO
0
1k

Featured

See All Featured
How to Grow Your eCommerce with AI & Automation
Avatar for Katarina Dahlin katarinadahlin
PRO
0
100
Embracing the Ebb and Flow
Avatar for Simon Collison colly
88
5k
For a Future-Friendly Web
Avatar for Brad Frost brad_frost
182
10k
The browser strikes back
Avatar for Jono Alderson jonoalderson
0
360
Reality Check: Gamification 10 Years Later
Avatar for Sebastian Deterding codingconduct
0
2k
Leadership Guide Workshop - DevTernity 2021
Avatar for David Neal reverentgeek
1
200
Java REST API Framework Comparison - PWX 2021
Avatar for Matt Raible mraible
34
9.1k
Become a Pro
Avatar for Speaker Deck speakerdeck
PRO
31
5.8k
The Illustrated Guide to Node.js - THAT Conference 2024
Avatar for David Neal reverentgeek
0
250
Being A Developer After 40
Avatar for Adrian Kosmaczewski akosma
91
590k
End of SEO as We Know It (SMX Advanced Version)
Avatar for Michael King ipullrank
3
3.9k
Noah Learner - AI + Me: how we built a GSC Bulk Export data pipeline
Avatar for Tech SEO Connect techseoconnect
PRO
0
100

Transcript

  1. Understanding distributed processing in Python Chie Hayashida

  2. $IJF)BZBTIJEB !DIJF) • 4PGUXBSF&OHJOFFSBU$PPLQBE GSPN+BQBO! • EFWFMPQBOEDPOUSJCVUFUPTPNF044 XSJUFBSUJDMFT BOENBLF QSFTFOUBUJPOT

    • 8POGJSTUQSJ[FBUSF*OWFOU (BNF%BZ

  3. $PPLQBE ‒ worldʼs largest recipe sharing platform Try it

  4. $PPLQBE JTHMPCBMTFSWJDF

  5. "HFOEB •CBTJDLOPXMFEHFPG DPODVSSFOUQBSBMMFMEJTUSJCVUFEQSPDFTTJOH •QBSBMMFMQSPDFTTJOHJO1ZUIPO •EJTUSJCVUFEQSPDFTTJOHJO1ZUIPO

  6. %JTDMBJNFS • 8IBUEP*UBML • CBTJDBSDIJUFDUVSFPGQBSBMMFMEJTUSJCVUFE QSPDFTTJOH DPNQVUJOH MJCSBSJFTJO1ZUIPO • 8IBUEP*OPU

    UBML • IPXUPJOTUBMMBOETFUVQFBDIMJCSBSJFT • VTBHFTPGEFUBJMFE"1*TPGXIBUXFJOUSPEVDF

  7. 8IFOXFVTFQBSBMMFMEJTUSJCVUFE QSPDFTTJOH FH •%PXOMPBEMBSHFEBUBJOTIPSUUJNF •6TFMBSHFEBUBGPS%BUBBOBMZTJTPS .BDIJOF-FBSOJOH •)BOEMFBMPUPGSFRVFTUTTJNVMUBOFPVTMZ BU8FC4JUF

  8. 28IBUJT UIFEJGGFSFODFPGCFMPX •$PODVSSFOUQSPDFTTJOH •1BSBMMFMQSPDFTTJOH

  9. l$PODVSSFOUQSPDFTTJOHzBOE l1BSBMMFM1SPDFTTJOHz $PODVSSFODZJTBCPVU EFBMJOHXJUI MPUTPG UIJOHTBUPODF 1BSBMMFMJTNJTBCPVU EPJOH MPUTPGUIJOHTBUPODF 

    3PC1JLFIUUQTUBMLTHPMBOHPSHXB[BTMJEF

  10. l$PODVSSFOUQSPDFTTJOHzBOE l1BSBMMFM1SPDFTTJOHz )FSFJT$PODVSSFOU )FSFJTQBSBMMFMBOEBMTPDPODVSSFOU

  11. $PODVSSFOUBOEQBSBMMFMQSPHSBNTDIFNF •$PODVSSFOUQSPHSBNTDIFNF $PODVSSFOU 1SPHSBN 1SPDFTT 1SPDFTT TDIFEVMFS $16 1SPDFTT 2VFVF

    1BSBMMFM 1SPHSBN 1SPDFTT 1SPDFTT $16 $16 •1BSBMMFMQSPHSBNTDIFNF

  12. 28IBUJT UIFEJGGFSFODFPGCFMPX •1BSBMMFMQSPDFTTJOH •%JTUSJCVUFEQSPDFTTJOH

  13. l1BSBMMFMQSPDFTTJOHzBOEl%JTUSJCVUFEQSPDFTTJOHz • 1BSBMMFMQSPDFTTJOH 1SPDFTTPS T .FNPSZ 1SPDFTTPS T .FNPSZ 1SPDFTTPS

    T .FNPSZ 1SPDFTTPS 1SPDFTTPS 1SPDFTTPS .FNPSZ • %JTUSJCVUFEQSPDFTTJOH /8 /8 /8 QSPDFTTPSTIBSFTNFNPSZ*OQBSBMMFMQSPDFTTJOH POUIFPUIFSIBOEQSPDFTTPSTIBTPXONFNPSJFTJOEJTUSJCVUFEQSPDFTTJOH

  14. 8IFOBOEIPXTIPVMEXF VTFQBSBMMFMEJTUSJCVUFE BSDIJUFDUVSFJO1ZUIPO

  15. UISFBEJOHBOENVMUJQSPDFTTJOH •1ZUIPOIBTPGGJDJBMNPEVMFTGPSQBSBMMFMQSPDFTTJOH • l#BUUFSZJODMVEFEz •8FDBOQBSBMMFMJ[FPVSQSPHSBNXJUIPVUBOZUIJSE QBSUZMJCSBSJFT

  16. UISFBEJOH • UISFBEJOHNPEVMFEJWJEFTPOFQSPDFTTJOUPNVMUJQMF UISFBET .BJO5ISFBE 5ISFBE 5ISFBE 1SPDFTT

  17. UISFBEJOH • (*- (MPCBM *OUFSQSFUFS-PDL MJNJUTPOMZPOFUISFBEDBOCF JOBTUBUFPGFYFDVUJPOBUBOZQPJOUJOUJNFJO$1ZUIPO 1SPDFTT 5ISFBE 

    5ISFBE  5ISFBE  SVO BXBJU BXBJU SVO BXBJU SVO SVO SVO BXBJU BXBJU SVO "DRVJSF-PDL

  18. 8IZ(*- • 5ISFBETJOQSPDFTTVTFTTBNFNFNPSZTQBDF • (*-XBTDIPTFOUPQSPUFDUUIFJOUFSOBMNFNPSZVTFECZ UIFJOUFSQSFUFS QSPDFTT UISFBE UISFBE .FNPSZ4QBDF

  19. *OXIBUTJUVBUJPOTJTUISFBEJOH FGGFDUJWF • 5BTLTXIJDIJT*0 CPVOETVDIBT • SFBEXSJUFGJMFT • %#DPOOFDUJPO •

    %PXOMPBEEBUBGSPN/8 1SPDFTT 5ISFBE  5ISFBE  5ISFBE  SVO BXBJU BXBJU SVO BXBJU SVO SVO SVO BXBJU BXBJU SVO "DRVJSF-PDL

  20. *OXIBUTJUVBUJPOTJTUISFBEJOH FGGFDUJWF • 5BTLTXIJDIJT*0 CPVOETVDIBT • SFBEXSJUFGJMFT • %#DPOOFDUJPO •

    %PXOMPBEEBUBGSPN/8 1SPDFTT 5ISFBE  5ISFBE  5ISFBE  SVO BXBJU BXBJU SVO BXBJU SVO SVO SVO *0 BXBJU SVO "DRVJSF -PDL *0 *0 *0 VTF$16 EPOPUVTF$16

  21. NVMUJQSPDFTTJOH •NVMUJQSPDFTTJOHTQBXOTQSPDFTTFTGPS QBSBMMFMJ[BUJPO •QBSBMMFMJ[FJOCPUIMPDBMBOESFNPUF .BJO1SPDFTT $IJME1SPDFTT $IJME1SPDFTT

  22. NVMUJQSPDFTTJOH • FBDIQSPDFTTFTIBTJUTPXONFNPSZTQBDF • /PUBGGFDUFECZ(*- • -FTTMJLFMZUPDBVTFFSSPSTEVFUPEBUBDPSSVQUJPOPS EFBEMPDL .BJO1SPDFTT $IJME1SPDFTT

    $IJME1SPDFTT PTGPSL .FNPSZ4QBDF .FNPSZ4QBDF $PQZ .FNPSZ4QBDF $PQZ

  23. *OXIBUTJUVBUJPOTJTNVMUJUISFBEJOH FGGFDUJWF • 5BTLTXIJDIJT$16CPVOETVDIBT • /VNFSJDBMDBMDVSBUJPO • 3FHVMBSFYQSFTTJPOQSPDFTTJOH • #VUUIFQFSGPSNBODFJO1ZUIPO`TNVMUJQSPDFTTJOHEPFTO`U

    JODSFBTFMJOFBSMZ

  24. NVMUJQSPDFTTJOH • 5IFSFJTPWFSIFBEEVFUPDPNNVOJDBUJPO CFUXFFOQBSFOUQSPDFTTBOEDIJMEQSPDFTT .BJO1SPDFTT $IJME1SPDFTT $IJME1SPDFTT PTGPSL .FNPSZ4QBDF .FNPSZ4QBDF

    $PQZ .FNPSZ4QBDF $PQZ

  25. NVMUJQSPDFTTJOH •UIFNBYOVNCFSPGUBTLTXIJDIDBOCFSVO TJNVMUBOFPVTMZJTSFTUSJDUFEUP$16OVNCFS 5BTL 5BTL 5BTL 5BTL 5BTL 5BTL $16

    $16

  26. +PCMJC • UIJSEQBSUZMJCSBSZGPSQBSBMMFMQSPDFTTJOHJO1ZUIPO • +PCMJC DBOVTF UISFBEJOHPSNVMUJQSPDFTTJOHGPSCBDLFOE • MB[ZFWBMVBUJPOVTJOHUSBOTQBSFOUEJTLDBDIJOHPGUIF PVUQVU

    • FBTZBOETJNQMFDPEJOH • MPHHJOHBOEUSBDJOHPGUIFFYFDVUJPO https://joblib.readthedocs.io/

  27. 4VNNBSZVQUPIFSF • 1ZUIPOIBTPGGJDJBMNPEVMFTGPSQBSBMMFMQSPDFTTJOH • UISFBEJOHJTFGGFDUJWFUP*0CPVOEQSPDFTTJOH • NVMUJQSPDFTTJOHJTFGGFDUJWFUP$16CPVOEQSPDFTTJOH • CPUIPGUXPBSFNBJOMZGPSPOMZGPSQSPDFTTFTSVOBUPOMZ POFNBDIJOF

    • TPNFPGEBUBMJCSBSJFTUSBOTQBSFOUMZVTFUIFTFQBSBMMFM QSPDFTTJOHMJCSBSJFT

  28. %JTUSJCVUFEQSPDFTTJOHJO1ZUIPO • 8IFOXFVTFMBSHFEBUBXIJDIDPVMEO`UQSPDFTTJOPOF NBDIJOF XFIBWFUPDPOTJEFSBCPVUEJTUSJCVUFE QSPDFTTJOH • QSPQFSUJFTXIJDITIPVMECFJODMVEFEJOEJTUSJCVUFE GSBNFXPSL •

    "VUPNBUJDSFDPWFSZ GSPNFSSPS • 5BTLEJWJTJPO KPCTDIFEVMJOH • 5PPMTGPS.POJUPSJOH %FCVHHBCJMJUZ

  29. %JTUSJCVUFEQSPDFTTJOHJO1ZUIPO *JOUSPEVDFUISFFGBNPVTMJCSBSJFTGPS EJTUSJCVUFEQSPDFTTJOHJO1ZUIPO •$FMFSZ •%BTL •1Z4QBSL

  30. $FMFSZ •EJTUSJCVUFEUBTLRVFVFCVJMUJO1ZUIPO •TJNQMFBOEGMFYJCMFBSDIJUFDUVSF • &WFSZQBSUTDBOCFFYUFOE • IFBWJMZVTFEGPSUBTLCBTFEXPSLMPBETJO 1ZUIPO • XJUI%KBOHP

    • "JSGMPXCBDLFOE

  31. $FMFSZ"SDIJUFDUVSF $MJFOU CSPLFS 3BCCJU.2 3FEJT  424 FUD 2VFVF 2VFVF

    2VFVF 8PSLFS $POTVNFS 8PSLFS $POTVNFS #BDLFOE 3FTVMUT 3FEJT FUD 8PSLFS $POTVNFS 8PSLFS $POTVNFS

  32. *OXIBUTJUVBUJPOTJTDFMFSZFGGFDUJWF  • 8IFOUIFUBTLJTTJNQMFCVUUIFBNPVOUPGEBUBJTUPP NVDIGPSPOFNBDIJOF • 8IFOZPVXBOUUPVTFJUXJUIBOPUIFSGSBNFXPSLTVDIBT %KBOHP

  33. %BTL • EJTUSJCVUFEQSPDFTTJOHGSBNFXPSLCVJMUJO1ZUIPO • CFBCMFUPVTF/VN1Z1BOEBT-JTUTMJLFQBSBMMFM PCKFDUT EBTLBSSBZ EBTLEBUBGSBNF EBTLCBH BOE"1*T

    • 4DIFEVMFSJTDVTUPNJ[BCMFUPHFUHPPEQFSGPSNBODFGPS CPUIGPSQBSBMMFMQSPDFTTJOHPOMPDBMNBDIJOFBOEGPS EJTUSJCVUFEQSPDFTTJOHPODMVTUFS • )BTHPPE8FC*OUFSGBDFGPSSFBMUJNFKPCNPOJUPSJOH

  34. %BTL TDIFEVMFS http://docs.dask.org/en/latest/scheduling.html

  35. %BTL 8FC*OUFSGBDF

  36. *OXIBUTJUVBUJPOTJT%BTL FGGFDUJWF  • 8IFOXFXBOUUPQBSBMMFMJ[F/VN1ZPSQBOEBTQSPDFTTJOH • 8IFOUIFEBUBTJ[FGJUTPOFUP NBDIJOFT

  37. 1Z4QBSL • CVJMEJO+BWBBOEIBT1ZUIPO*OUFSGBDF • DBOCFNPSFTDBMFPVUUIBODFMFSZBOEEBTL • 0SJHJOBMMZCVJMUUPSVOPOB)BEPPQDMVTUFS • GBTUBOEDPTUFGGJDJFOUQSPDFTTJOH •

    )BTHPPE8FC*OUFSGBDFGPSSFBMUJNFKPCNPOJUPSJOH • NVMUJGVODUJPOBM • TUSFBN "1* • NBDIJOFMFBSOJOH"1* • DBOCFVTFJONBOBHFNFOUTFSWJDFJODMPVETFSWJDF

  38. 1Z4QBSL • CVJMEJO+BWBBOEIBT1ZUIPO*OUFSGBDF only with partial API

  39. 1Z4QBSL • DBOCFNPSFTDBMFPVUUIBODFMFSZBOE%BTL node number 1 4-5 thousands

  40. *OXIBUTJUVBUJPOTJT1Z4QBSL FGGFDUJWF  • 8IFOXFXBOUUPSVODPNQMFYUBTLTXJUIBMBSHFBNPVOU PGEBUB • 8IFOXFIBWF)BEPPQDMVTUFS

  41. 4VNNBSZ • &YQMBJOFEBCPVUUIFEJGGFSFODFPG DPODVSSFOUQBSBMMFMEJTUSJCVUFEQSPDFTTJOH •*OUSPEVDFEUIFBSDIJUFDUVSFPG QBSBMMFMEJTUSJCVUFE QSPDFTTJOHMJCSBSJFT

  42. DPVMEO`UJOUSPEVDF • BTZODJP • DPODVSSFOUGVUVSF • "OPUIFSUIJSEQBSUZMJCSBSJFT

  43. %PObUCFBGSBJEUPVTFQBSBMMFM QSPDFTTJOHNPSF

  44. 2VFTUJPO