Lock in $30 Savings on PRO—Offer Ends Soon! ⏳

Understanding distributed processing in Python

Avatar for chie8842 chie8842
June 16, 2019
Technology
2
2.1k

Understanding distributed processing in Python

#pyconth2019
https://th.pycon.org/talks/#row-47
Slides of my talk at PyConTh2019

Avatar for chie8842

chie8842

June 16, 2019
Tweet

More Decks by chie8842

See All by chie8842
MongoDB Atlas：モダンなアプリ開発を支えるデータプラットフォームのご紹介
Avatar for chie8842 chie8842
0
27
MongoDB Vectorsearchではじめるカスタマイズ可能な生成AIアプリ開発
Avatar for chie8842 chie8842
0
25
MongoDB Atlas Search のご紹介
Avatar for chie8842 chie8842
2
1.9k
MongoDB Atlas Vectorsearchではじめる生成AIアプリ開発
Avatar for chie8842 chie8842
3
1.9k
AWS GlueとAWS Lake Formationではじめるデータマネジメント
Avatar for chie8842 chie8842
0
1.2k
Distributed Processing in Python
Avatar for chie8842 chie8842
2
790
クックパッドにおける推薦（と検索）の取り組み
Avatar for chie8842 chie8842
20
8.1k
Performance Tuning Tips of TensorFlow Inference
Avatar for chie8842 chie8842
1
770
クックパッドにおけるCloud AutoML事例
Avatar for chie8842 chie8842
9
8k

Other Decks in Technology

See All in Technology
ML PM Talk #1 - ML PMの分類に関する考察
Avatar for LINEヤフーTech (LY Corporation Tech) lycorptech_jp
PRO
1
730
大企業でもできる！ボトムアップで拡大させるプラットフォームの作り方
Avatar for ファインディ株式会社（イベント資料アップ用） findy_eventslides
1
560
第4回 「メタデータ通り」　リアル開催
Avatar for データ横丁 datayokocho
0
120
Kiro Autonomous AgentとKiro Powers の紹介 / kiro-autonomous-agent-and-powers
Avatar for tomoki10 tomoki10
0
320
因果AIへの招待
Avatar for Shohei SHIMIZU sshimizu2006
0
930
法人支出管理領域におけるソフトウェアアーキテクチャに基づいたテスト戦略の実践
Avatar for ogugu ogugu9
1
210
エンジニアとPMのドメイン知識の溝をなくす、 AIネイティブな開発プロセス
Avatar for Michiru Kato applism118
4
980
生成AIでテスト設計はどこまでできる？ 「テスト粒度」を操るテーラリング術
Avatar for ks shota_kusaba
0
530
チーリンについて
Avatar for ぐっち hirotomotaguchi
3
930
非CUDAの悲哀 〜Claude Code と挑んだ image to 3D “Hunyuan3D”を EVO-X2(Ryzen AI Max+395)で動作させるチャレンジ〜
Avatar for hawky the miscellaneous hawkymisc
1
160
Gemini でコードレビュー知見を見える化
Avatar for ZOZO Developers zozotech
PRO
1
210
Overture Maps Foundationの3年を振り返る
Avatar for Toru Mori moritoru
0
160

Featured

See All Featured
How To Stay Up To Date on Web Technology
Avatar for Chris Coyier chriscoyier
791
250k
Rebuilding a faster, lazier Slack
Avatar for samanthasiow samanthasiow
84
9.3k
The Cost Of JavaScript in 2023
Avatar for Addy Osmani addyosmani
55
9.3k
Agile that works and the tools we love
Avatar for Rasmus Luckow-Nielsen rasmusluckow
331
21k
The Hidden Cost of Media on the Web [PixelPalooza 2025]
Avatar for Tammy Everts tammyeverts
1
93
VelocityConf: Rendering Performance Case Studies
Avatar for Addy Osmani addyosmani
333
24k
Faster Mobile Websites
Avatar for Dean Hume deanohume
310
31k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
Avatar for Stéphanie Walter stephaniewalter
285
14k
How Fast Is Fast Enough? [PerfNow 2025]
Avatar for Tammy Everts tammyeverts
3
390
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
Avatar for soudai sone soudai
PRO
196
69k
Why Our Code Smells
Avatar for Brandon Keepers bkeepers
PRO
340
57k
Designing for Performance
Avatar for Lara Hogan lara
610
69k

Transcript

  1. Understanding distributed processing in Python Chie Hayashida

  2. $IJF)BZBTIJEB !DIJF) • 4PGUXBSF&OHJOFFSBU$PPLQBE GSPN+BQBO! • EFWFMPQBOEDPOUSJCVUFUPTPNF044 XSJUFBSUJDMFT BOENBLF QSFTFOUBUJPOT

    • 8POGJSTUQSJ[FBUSF*OWFOU (BNF%BZ

  3. $PPLQBE ‒ worldʼs largest recipe sharing platform Try it

  4. $PPLQBE JTHMPCBMTFSWJDF

  5. "HFOEB •CBTJDLOPXMFEHFPG DPODVSSFOUQBSBMMFMEJTUSJCVUFEQSPDFTTJOH •QBSBMMFMQSPDFTTJOHJO1ZUIPO •EJTUSJCVUFEQSPDFTTJOHJO1ZUIPO

  6. %JTDMBJNFS • 8IBUEP*UBML • CBTJDBSDIJUFDUVSFPGQBSBMMFMEJTUSJCVUFE QSPDFTTJOH DPNQVUJOH MJCSBSJFTJO1ZUIPO • 8IBUEP*OPU

    UBML • IPXUPJOTUBMMBOETFUVQFBDIMJCSBSJFT • VTBHFTPGEFUBJMFE"1*TPGXIBUXFJOUSPEVDF

  7. 8IFOXFVTFQBSBMMFMEJTUSJCVUFE QSPDFTTJOH FH •%PXOMPBEMBSHFEBUBJOTIPSUUJNF •6TFMBSHFEBUBGPS%BUBBOBMZTJTPS .BDIJOF-FBSOJOH •)BOEMFBMPUPGSFRVFTUTTJNVMUBOFPVTMZ BU8FC4JUF

  8. 28IBUJT UIFEJGGFSFODFPGCFMPX •$PODVSSFOUQSPDFTTJOH •1BSBMMFMQSPDFTTJOH

  9. l$PODVSSFOUQSPDFTTJOHzBOE l1BSBMMFM1SPDFTTJOHz $PODVSSFODZJTBCPVU EFBMJOHXJUI MPUTPG UIJOHTBUPODF 1BSBMMFMJTNJTBCPVU EPJOH MPUTPGUIJOHTBUPODF 

    3PC1JLFIUUQTUBMLTHPMBOHPSHXB[BTMJEF

  10. l$PODVSSFOUQSPDFTTJOHzBOE l1BSBMMFM1SPDFTTJOHz )FSFJT$PODVSSFOU )FSFJTQBSBMMFMBOEBMTPDPODVSSFOU

  11. $PODVSSFOUBOEQBSBMMFMQSPHSBNTDIFNF •$PODVSSFOUQSPHSBNTDIFNF $PODVSSFOU 1SPHSBN 1SPDFTT 1SPDFTT TDIFEVMFS $16 1SPDFTT 2VFVF

    1BSBMMFM 1SPHSBN 1SPDFTT 1SPDFTT $16 $16 •1BSBMMFMQSPHSBNTDIFNF

  12. 28IBUJT UIFEJGGFSFODFPGCFMPX •1BSBMMFMQSPDFTTJOH •%JTUSJCVUFEQSPDFTTJOH

  13. l1BSBMMFMQSPDFTTJOHzBOEl%JTUSJCVUFEQSPDFTTJOHz • 1BSBMMFMQSPDFTTJOH 1SPDFTTPS T .FNPSZ 1SPDFTTPS T .FNPSZ 1SPDFTTPS

    T .FNPSZ 1SPDFTTPS 1SPDFTTPS 1SPDFTTPS .FNPSZ • %JTUSJCVUFEQSPDFTTJOH /8 /8 /8 QSPDFTTPSTIBSFTNFNPSZ*OQBSBMMFMQSPDFTTJOH POUIFPUIFSIBOEQSPDFTTPSTIBTPXONFNPSJFTJOEJTUSJCVUFEQSPDFTTJOH

  14. 8IFOBOEIPXTIPVMEXF VTFQBSBMMFMEJTUSJCVUFE BSDIJUFDUVSFJO1ZUIPO

  15. UISFBEJOHBOENVMUJQSPDFTTJOH •1ZUIPOIBTPGGJDJBMNPEVMFTGPSQBSBMMFMQSPDFTTJOH • l#BUUFSZJODMVEFEz •8FDBOQBSBMMFMJ[FPVSQSPHSBNXJUIPVUBOZUIJSE QBSUZMJCSBSJFT

  16. UISFBEJOH • UISFBEJOHNPEVMFEJWJEFTPOFQSPDFTTJOUPNVMUJQMF UISFBET .BJO5ISFBE 5ISFBE 5ISFBE 1SPDFTT

  17. UISFBEJOH • (*- (MPCBM *OUFSQSFUFS-PDL MJNJUTPOMZPOFUISFBEDBOCF JOBTUBUFPGFYFDVUJPOBUBOZQPJOUJOUJNFJO$1ZUIPO 1SPDFTT 5ISFBE 

    5ISFBE  5ISFBE  SVO BXBJU BXBJU SVO BXBJU SVO SVO SVO BXBJU BXBJU SVO "DRVJSF-PDL

  18. 8IZ(*- • 5ISFBETJOQSPDFTTVTFTTBNFNFNPSZTQBDF • (*-XBTDIPTFOUPQSPUFDUUIFJOUFSOBMNFNPSZVTFECZ UIFJOUFSQSFUFS QSPDFTT UISFBE UISFBE .FNPSZ4QBDF

  19. *OXIBUTJUVBUJPOTJTUISFBEJOH FGGFDUJWF • 5BTLTXIJDIJT*0 CPVOETVDIBT • SFBEXSJUFGJMFT • %#DPOOFDUJPO •

    %PXOMPBEEBUBGSPN/8 1SPDFTT 5ISFBE  5ISFBE  5ISFBE  SVO BXBJU BXBJU SVO BXBJU SVO SVO SVO BXBJU BXBJU SVO "DRVJSF-PDL

  20. *OXIBUTJUVBUJPOTJTUISFBEJOH FGGFDUJWF • 5BTLTXIJDIJT*0 CPVOETVDIBT • SFBEXSJUFGJMFT • %#DPOOFDUJPO •

    %PXOMPBEEBUBGSPN/8 1SPDFTT 5ISFBE  5ISFBE  5ISFBE  SVO BXBJU BXBJU SVO BXBJU SVO SVO SVO *0 BXBJU SVO "DRVJSF -PDL *0 *0 *0 VTF$16 EPOPUVTF$16

  21. NVMUJQSPDFTTJOH •NVMUJQSPDFTTJOHTQBXOTQSPDFTTFTGPS QBSBMMFMJ[BUJPO •QBSBMMFMJ[FJOCPUIMPDBMBOESFNPUF .BJO1SPDFTT $IJME1SPDFTT $IJME1SPDFTT

  22. NVMUJQSPDFTTJOH • FBDIQSPDFTTFTIBTJUTPXONFNPSZTQBDF • /PUBGGFDUFECZ(*- • -FTTMJLFMZUPDBVTFFSSPSTEVFUPEBUBDPSSVQUJPOPS EFBEMPDL .BJO1SPDFTT $IJME1SPDFTT

    $IJME1SPDFTT PTGPSL .FNPSZ4QBDF .FNPSZ4QBDF $PQZ .FNPSZ4QBDF $PQZ

  23. *OXIBUTJUVBUJPOTJTNVMUJUISFBEJOH FGGFDUJWF • 5BTLTXIJDIJT$16CPVOETVDIBT • /VNFSJDBMDBMDVSBUJPO • 3FHVMBSFYQSFTTJPOQSPDFTTJOH • #VUUIFQFSGPSNBODFJO1ZUIPO`TNVMUJQSPDFTTJOHEPFTO`U

    JODSFBTFMJOFBSMZ

  24. NVMUJQSPDFTTJOH • 5IFSFJTPWFSIFBEEVFUPDPNNVOJDBUJPO CFUXFFOQBSFOUQSPDFTTBOEDIJMEQSPDFTT .BJO1SPDFTT $IJME1SPDFTT $IJME1SPDFTT PTGPSL .FNPSZ4QBDF .FNPSZ4QBDF

    $PQZ .FNPSZ4QBDF $PQZ

  25. NVMUJQSPDFTTJOH •UIFNBYOVNCFSPGUBTLTXIJDIDBOCFSVO TJNVMUBOFPVTMZJTSFTUSJDUFEUP$16OVNCFS 5BTL 5BTL 5BTL 5BTL 5BTL 5BTL $16

    $16

  26. +PCMJC • UIJSEQBSUZMJCSBSZGPSQBSBMMFMQSPDFTTJOHJO1ZUIPO • +PCMJC DBOVTF UISFBEJOHPSNVMUJQSPDFTTJOHGPSCBDLFOE • MB[ZFWBMVBUJPOVTJOHUSBOTQBSFOUEJTLDBDIJOHPGUIF PVUQVU

    • FBTZBOETJNQMFDPEJOH • MPHHJOHBOEUSBDJOHPGUIFFYFDVUJPO https://joblib.readthedocs.io/

  27. 4VNNBSZVQUPIFSF • 1ZUIPOIBTPGGJDJBMNPEVMFTGPSQBSBMMFMQSPDFTTJOH • UISFBEJOHJTFGGFDUJWFUP*0CPVOEQSPDFTTJOH • NVMUJQSPDFTTJOHJTFGGFDUJWFUP$16CPVOEQSPDFTTJOH • CPUIPGUXPBSFNBJOMZGPSPOMZGPSQSPDFTTFTSVOBUPOMZ POFNBDIJOF

    • TPNFPGEBUBMJCSBSJFTUSBOTQBSFOUMZVTFUIFTFQBSBMMFM QSPDFTTJOHMJCSBSJFT

  28. %JTUSJCVUFEQSPDFTTJOHJO1ZUIPO • 8IFOXFVTFMBSHFEBUBXIJDIDPVMEO`UQSPDFTTJOPOF NBDIJOF XFIBWFUPDPOTJEFSBCPVUEJTUSJCVUFE QSPDFTTJOH • QSPQFSUJFTXIJDITIPVMECFJODMVEFEJOEJTUSJCVUFE GSBNFXPSL •

    "VUPNBUJDSFDPWFSZ GSPNFSSPS • 5BTLEJWJTJPO KPCTDIFEVMJOH • 5PPMTGPS.POJUPSJOH %FCVHHBCJMJUZ

  29. %JTUSJCVUFEQSPDFTTJOHJO1ZUIPO *JOUSPEVDFUISFFGBNPVTMJCSBSJFTGPS EJTUSJCVUFEQSPDFTTJOHJO1ZUIPO •$FMFSZ •%BTL •1Z4QBSL

  30. $FMFSZ •EJTUSJCVUFEUBTLRVFVFCVJMUJO1ZUIPO •TJNQMFBOEGMFYJCMFBSDIJUFDUVSF • &WFSZQBSUTDBOCFFYUFOE • IFBWJMZVTFEGPSUBTLCBTFEXPSLMPBETJO 1ZUIPO • XJUI%KBOHP

    • "JSGMPXCBDLFOE

  31. $FMFSZ"SDIJUFDUVSF $MJFOU CSPLFS 3BCCJU.2 3FEJT  424 FUD 2VFVF 2VFVF

    2VFVF 8PSLFS $POTVNFS 8PSLFS $POTVNFS #BDLFOE 3FTVMUT 3FEJT FUD 8PSLFS $POTVNFS 8PSLFS $POTVNFS

  32. *OXIBUTJUVBUJPOTJTDFMFSZFGGFDUJWF  • 8IFOUIFUBTLJTTJNQMFCVUUIFBNPVOUPGEBUBJTUPP NVDIGPSPOFNBDIJOF • 8IFOZPVXBOUUPVTFJUXJUIBOPUIFSGSBNFXPSLTVDIBT %KBOHP

  33. %BTL • EJTUSJCVUFEQSPDFTTJOHGSBNFXPSLCVJMUJO1ZUIPO • CFBCMFUPVTF/VN1Z1BOEBT-JTUTMJLFQBSBMMFM PCKFDUT EBTLBSSBZ EBTLEBUBGSBNF EBTLCBH BOE"1*T

    • 4DIFEVMFSJTDVTUPNJ[BCMFUPHFUHPPEQFSGPSNBODFGPS CPUIGPSQBSBMMFMQSPDFTTJOHPOMPDBMNBDIJOFBOEGPS EJTUSJCVUFEQSPDFTTJOHPODMVTUFS • )BTHPPE8FC*OUFSGBDFGPSSFBMUJNFKPCNPOJUPSJOH

  34. %BTL TDIFEVMFS http://docs.dask.org/en/latest/scheduling.html

  35. %BTL 8FC*OUFSGBDF

  36. *OXIBUTJUVBUJPOTJT%BTL FGGFDUJWF  • 8IFOXFXBOUUPQBSBMMFMJ[F/VN1ZPSQBOEBTQSPDFTTJOH • 8IFOUIFEBUBTJ[FGJUTPOFUP NBDIJOFT

  37. 1Z4QBSL • CVJMEJO+BWBBOEIBT1ZUIPO*OUFSGBDF • DBOCFNPSFTDBMFPVUUIBODFMFSZBOEEBTL • 0SJHJOBMMZCVJMUUPSVOPOB)BEPPQDMVTUFS • GBTUBOEDPTUFGGJDJFOUQSPDFTTJOH •

    )BTHPPE8FC*OUFSGBDFGPSSFBMUJNFKPCNPOJUPSJOH • NVMUJGVODUJPOBM • TUSFBN "1* • NBDIJOFMFBSOJOH"1* • DBOCFVTFJONBOBHFNFOUTFSWJDFJODMPVETFSWJDF

  38. 1Z4QBSL • CVJMEJO+BWBBOEIBT1ZUIPO*OUFSGBDF only with partial API

  39. 1Z4QBSL • DBOCFNPSFTDBMFPVUUIBODFMFSZBOE%BTL node number 1 4-5 thousands

  40. *OXIBUTJUVBUJPOTJT1Z4QBSL FGGFDUJWF  • 8IFOXFXBOUUPSVODPNQMFYUBTLTXJUIBMBSHFBNPVOU PGEBUB • 8IFOXFIBWF)BEPPQDMVTUFS

  41. 4VNNBSZ • &YQMBJOFEBCPVUUIFEJGGFSFODFPG DPODVSSFOUQBSBMMFMEJTUSJCVUFEQSPDFTTJOH •*OUSPEVDFEUIFBSDIJUFDUVSFPG QBSBMMFMEJTUSJCVUFE QSPDFTTJOHMJCSBSJFT

  42. DPVMEO`UJOUSPEVDF • BTZODJP • DPODVSSFOUGVUVSF • "OPUIFSUIJSEQBSUZMJCSBSJFT

  43. %PObUCFBGSBJEUPVTFQBSBMMFM QSPDFTTJOHNPSF

  44. 2VFTUJPO