Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Understanding distributed processing in Python
Search
chie8842
June 16, 2019
Technology
2
2.1k
Understanding distributed processing in Python
#pyconth2019
https://th.pycon.org/talks/#row-47
Slides of my talk at PyConTh2019
chie8842
June 16, 2019
Tweet
Share
More Decks by chie8842
See All by chie8842
MongoDB Atlas:モダンなアプリ開発を支えるデータプラットフォームのご紹介
chie8842
0
22
MongoDB Vectorsearchではじめるカスタマイズ可能な生成AIアプリ開発
chie8842
0
23
MongoDB Atlas Search のご紹介
chie8842
2
1.8k
MongoDB Atlas Vectorsearchではじめる生成AIアプリ開発
chie8842
3
1.8k
AWS GlueとAWS Lake Formationではじめるデータマネジメント
chie8842
0
1.1k
Distributed Processing in Python
chie8842
2
750
クックパッドにおける推薦(と検索)の取り組み
chie8842
20
8.1k
Performance Tuning Tips of TensorFlow Inference
chie8842
1
760
クックパッドにおけるCloud AutoML事例
chie8842
9
8k
Other Decks in Technology
See All in Technology
AI時代こそ求められる設計力- AWSクラウドデザインパターン3選で信頼性と拡張性を高める-
kenichirokimura
3
340
Data Hubグループ 紹介資料
sansan33
PRO
0
2.2k
dbtとBigQuery MLで実現する リクルートの営業支援基盤のモデル開発と保守運用
recruitengineers
PRO
3
100
「れきちず」のこれまでとこれから - 誰にでもわかりやすい歴史地図を目指して / FOSS4G 2025 Japan
hjmkth
1
320
Sansan Engineering Unit 紹介資料
sansan33
PRO
1
3k
大規模サーバーレスAPIの堅牢性・信頼性設計 〜AWSのベストプラクティスから始まる現実的制約との向き合い方〜
maimyyym
10
4.9k
20251010_HCCJP_AdaptiveCloudUpdates
sdosamut
0
140
Codexとも仲良く。CodeRabbit CLIの紹介
moongift
PRO
1
240
今この時代に技術とどう向き合うべきか
gree_tech
PRO
2
2k
研究開発部メンバーの働き⽅ / Sansan R&D Profile
sansan33
PRO
3
20k
フレームワークを意識させないワークショップづくり
keigosuda
0
200
Findy Team+ QAチーム これからのチャレンジ!
findy_eventslides
0
430
Featured
See All Featured
Large-scale JavaScript Application Architecture
addyosmani
514
110k
Building Better People: How to give real-time feedback that sticks.
wjessup
369
20k
The Language of Interfaces
destraynor
162
25k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
52
5.6k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
Building Applications with DynamoDB
mza
96
6.7k
Side Projects
sachag
455
43k
The Cost Of JavaScript in 2023
addyosmani
55
9k
Build The Right Thing And Hit Your Dates
maggiecrowley
37
2.9k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
333
22k
Six Lessons from altMBA
skipperchong
29
4k
Transcript
Understanding distributed processing in Python Chie Hayashida
$IJF)BZBTIJEB !DIJF) • 4PGUXBSF&OHJOFFSBU$PPLQBE GSPN+BQBO! • EFWFMPQBOEDPOUSJCVUFUPTPNF044 XSJUFBSUJDMFT BOENBLF QSFTFOUBUJPOT
• 8POGJSTUQSJ[FBUSF*OWFOU (BNF%BZ
$PPLQBE ‒ worldʼs largest recipe sharing platform Try it
$PPLQBE JTHMPCBMTFSWJDF
"HFOEB •CBTJDLOPXMFEHFPG DPODVSSFOUQBSBMMFMEJTUSJCVUFEQSPDFTTJOH •QBSBMMFMQSPDFTTJOHJO1ZUIPO •EJTUSJCVUFEQSPDFTTJOHJO1ZUIPO
%JTDMBJNFS • 8IBUEP*UBML • CBTJDBSDIJUFDUVSFPGQBSBMMFMEJTUSJCVUFE QSPDFTTJOH DPNQVUJOH MJCSBSJFTJO1ZUIPO • 8IBUEP*OPU
UBML • IPXUPJOTUBMMBOETFUVQFBDIMJCSBSJFT • VTBHFTPGEFUBJMFE"1*TPGXIBUXFJOUSPEVDF
8IFOXFVTFQBSBMMFMEJTUSJCVUFE QSPDFTTJOH FH •%PXOMPBEMBSHFEBUBJOTIPSUUJNF •6TFMBSHFEBUBGPS%BUBBOBMZTJTPS .BDIJOF-FBSOJOH •)BOEMFBMPUPGSFRVFTUTTJNVMUBOFPVTMZ BU8FC4JUF
28IBUJT UIFEJGGFSFODFPGCFMPX •$PODVSSFOUQSPDFTTJOH •1BSBMMFMQSPDFTTJOH
l$PODVSSFOUQSPDFTTJOHzBOE l1BSBMMFM1SPDFTTJOHz $PODVSSFODZJTBCPVU EFBMJOHXJUI MPUTPG UIJOHTBUPODF 1BSBMMFMJTNJTBCPVU EPJOH MPUTPGUIJOHTBUPODF
3PC1JLFIUUQTUBMLTHPMBOHPSHXB[BTMJEF
l$PODVSSFOUQSPDFTTJOHzBOE l1BSBMMFM1SPDFTTJOHz )FSFJT$PODVSSFOU )FSFJTQBSBMMFMBOEBMTPDPODVSSFOU
$PODVSSFOUBOEQBSBMMFMQSPHSBNTDIFNF •$PODVSSFOUQSPHSBNTDIFNF $PODVSSFOU 1SPHSBN 1SPDFTT 1SPDFTT TDIFEVMFS $16 1SPDFTT 2VFVF
1BSBMMFM 1SPHSBN 1SPDFTT 1SPDFTT $16 $16 •1BSBMMFMQSPHSBNTDIFNF
28IBUJT UIFEJGGFSFODFPGCFMPX •1BSBMMFMQSPDFTTJOH •%JTUSJCVUFEQSPDFTTJOH
l1BSBMMFMQSPDFTTJOHzBOEl%JTUSJCVUFEQSPDFTTJOHz • 1BSBMMFMQSPDFTTJOH 1SPDFTTPS T .FNPSZ 1SPDFTTPS T .FNPSZ 1SPDFTTPS
T .FNPSZ 1SPDFTTPS 1SPDFTTPS 1SPDFTTPS .FNPSZ • %JTUSJCVUFEQSPDFTTJOH /8 /8 /8 QSPDFTTPSTIBSFTNFNPSZ*OQBSBMMFMQSPDFTTJOH POUIFPUIFSIBOEQSPDFTTPSTIBTPXONFNPSJFTJOEJTUSJCVUFEQSPDFTTJOH
8IFOBOEIPXTIPVMEXF VTFQBSBMMFMEJTUSJCVUFE BSDIJUFDUVSFJO1ZUIPO
UISFBEJOHBOENVMUJQSPDFTTJOH •1ZUIPOIBTPGGJDJBMNPEVMFTGPSQBSBMMFMQSPDFTTJOH • l#BUUFSZJODMVEFEz •8FDBOQBSBMMFMJ[FPVSQSPHSBNXJUIPVUBOZUIJSE QBSUZMJCSBSJFT
UISFBEJOH • UISFBEJOHNPEVMFEJWJEFTPOFQSPDFTTJOUPNVMUJQMF UISFBET .BJO5ISFBE 5ISFBE 5ISFBE 1SPDFTT
UISFBEJOH • (*- (MPCBM *OUFSQSFUFS-PDL MJNJUTPOMZPOFUISFBEDBOCF JOBTUBUFPGFYFDVUJPOBUBOZQPJOUJOUJNFJO$1ZUIPO 1SPDFTT 5ISFBE
5ISFBE 5ISFBE SVO BXBJU BXBJU SVO BXBJU SVO SVO SVO BXBJU BXBJU SVO "DRVJSF-PDL
8IZ(*- • 5ISFBETJOQSPDFTTVTFTTBNFNFNPSZTQBDF • (*-XBTDIPTFOUPQSPUFDUUIFJOUFSOBMNFNPSZVTFECZ UIFJOUFSQSFUFS QSPDFTT UISFBE UISFBE .FNPSZ4QBDF
*OXIBUTJUVBUJPOTJTUISFBEJOH FGGFDUJWF • 5BTLTXIJDIJT*0 CPVOETVDIBT • SFBEXSJUFGJMFT • %#DPOOFDUJPO •
%PXOMPBEEBUBGSPN/8 1SPDFTT 5ISFBE 5ISFBE 5ISFBE SVO BXBJU BXBJU SVO BXBJU SVO SVO SVO BXBJU BXBJU SVO "DRVJSF-PDL
*OXIBUTJUVBUJPOTJTUISFBEJOH FGGFDUJWF • 5BTLTXIJDIJT*0 CPVOETVDIBT • SFBEXSJUFGJMFT • %#DPOOFDUJPO •
%PXOMPBEEBUBGSPN/8 1SPDFTT 5ISFBE 5ISFBE 5ISFBE SVO BXBJU BXBJU SVO BXBJU SVO SVO SVO *0 BXBJU SVO "DRVJSF -PDL *0 *0 *0 VTF$16 EPOPUVTF$16
NVMUJQSPDFTTJOH •NVMUJQSPDFTTJOHTQBXOTQSPDFTTFTGPS QBSBMMFMJ[BUJPO •QBSBMMFMJ[FJOCPUIMPDBMBOESFNPUF .BJO1SPDFTT $IJME1SPDFTT $IJME1SPDFTT
NVMUJQSPDFTTJOH • FBDIQSPDFTTFTIBTJUTPXONFNPSZTQBDF • /PUBGGFDUFECZ(*- • -FTTMJLFMZUPDBVTFFSSPSTEVFUPEBUBDPSSVQUJPOPS EFBEMPDL .BJO1SPDFTT $IJME1SPDFTT
$IJME1SPDFTT PTGPSL .FNPSZ4QBDF .FNPSZ4QBDF $PQZ .FNPSZ4QBDF $PQZ
*OXIBUTJUVBUJPOTJTNVMUJUISFBEJOH FGGFDUJWF • 5BTLTXIJDIJT$16CPVOETVDIBT • /VNFSJDBMDBMDVSBUJPO • 3FHVMBSFYQSFTTJPOQSPDFTTJOH • #VUUIFQFSGPSNBODFJO1ZUIPO`TNVMUJQSPDFTTJOHEPFTO`U
JODSFBTFMJOFBSMZ
NVMUJQSPDFTTJOH • 5IFSFJTPWFSIFBEEVFUPDPNNVOJDBUJPO CFUXFFOQBSFOUQSPDFTTBOEDIJMEQSPDFTT .BJO1SPDFTT $IJME1SPDFTT $IJME1SPDFTT PTGPSL .FNPSZ4QBDF .FNPSZ4QBDF
$PQZ .FNPSZ4QBDF $PQZ
NVMUJQSPDFTTJOH •UIFNBYOVNCFSPGUBTLTXIJDIDBOCFSVO TJNVMUBOFPVTMZJTSFTUSJDUFEUP$16OVNCFS 5BTL 5BTL 5BTL 5BTL 5BTL 5BTL $16
$16
+PCMJC • UIJSEQBSUZMJCSBSZGPSQBSBMMFMQSPDFTTJOHJO1ZUIPO • +PCMJC DBOVTF UISFBEJOHPSNVMUJQSPDFTTJOHGPSCBDLFOE • MB[ZFWBMVBUJPOVTJOHUSBOTQBSFOUEJTLDBDIJOHPGUIF PVUQVU
• FBTZBOETJNQMFDPEJOH • MPHHJOHBOEUSBDJOHPGUIFFYFDVUJPO https://joblib.readthedocs.io/
4VNNBSZVQUPIFSF • 1ZUIPOIBTPGGJDJBMNPEVMFTGPSQBSBMMFMQSPDFTTJOH • UISFBEJOHJTFGGFDUJWFUP*0CPVOEQSPDFTTJOH • NVMUJQSPDFTTJOHJTFGGFDUJWFUP$16CPVOEQSPDFTTJOH • CPUIPGUXPBSFNBJOMZGPSPOMZGPSQSPDFTTFTSVOBUPOMZ POFNBDIJOF
• TPNFPGEBUBMJCSBSJFTUSBOTQBSFOUMZVTFUIFTFQBSBMMFM QSPDFTTJOHMJCSBSJFT
%JTUSJCVUFEQSPDFTTJOHJO1ZUIPO • 8IFOXFVTFMBSHFEBUBXIJDIDPVMEO`UQSPDFTTJOPOF NBDIJOF XFIBWFUPDPOTJEFSBCPVUEJTUSJCVUFE QSPDFTTJOH • QSPQFSUJFTXIJDITIPVMECFJODMVEFEJOEJTUSJCVUFE GSBNFXPSL •
"VUPNBUJDSFDPWFSZ GSPNFSSPS • 5BTLEJWJTJPO KPCTDIFEVMJOH • 5PPMTGPS.POJUPSJOH %FCVHHBCJMJUZ
%JTUSJCVUFEQSPDFTTJOHJO1ZUIPO *JOUSPEVDFUISFFGBNPVTMJCSBSJFTGPS EJTUSJCVUFEQSPDFTTJOHJO1ZUIPO •$FMFSZ •%BTL •1Z4QBSL
$FMFSZ •EJTUSJCVUFEUBTLRVFVFCVJMUJO1ZUIPO •TJNQMFBOEGMFYJCMFBSDIJUFDUVSF • &WFSZQBSUTDBOCFFYUFOE • IFBWJMZVTFEGPSUBTLCBTFEXPSLMPBETJO 1ZUIPO • XJUI%KBOHP
• "JSGMPXCBDLFOE
$FMFSZ"SDIJUFDUVSF $MJFOU CSPLFS 3BCCJU.2 3FEJT 424 FUD 2VFVF 2VFVF
2VFVF 8PSLFS $POTVNFS 8PSLFS $POTVNFS #BDLFOE 3FTVMUT 3FEJT FUD 8PSLFS $POTVNFS 8PSLFS $POTVNFS
*OXIBUTJUVBUJPOTJTDFMFSZFGGFDUJWF • 8IFOUIFUBTLJTTJNQMFCVUUIFBNPVOUPGEBUBJTUPP NVDIGPSPOFNBDIJOF • 8IFOZPVXBOUUPVTFJUXJUIBOPUIFSGSBNFXPSLTVDIBT %KBOHP
%BTL • EJTUSJCVUFEQSPDFTTJOHGSBNFXPSLCVJMUJO1ZUIPO • CFBCMFUPVTF/VN1Z1BOEBT-JTUTMJLFQBSBMMFM PCKFDUT EBTLBSSBZ EBTLEBUBGSBNF EBTLCBH BOE"1*T
• 4DIFEVMFSJTDVTUPNJ[BCMFUPHFUHPPEQFSGPSNBODFGPS CPUIGPSQBSBMMFMQSPDFTTJOHPOMPDBMNBDIJOFBOEGPS EJTUSJCVUFEQSPDFTTJOHPODMVTUFS • )BTHPPE8FC*OUFSGBDFGPSSFBMUJNFKPCNPOJUPSJOH
%BTL TDIFEVMFS http://docs.dask.org/en/latest/scheduling.html
%BTL 8FC*OUFSGBDF
*OXIBUTJUVBUJPOTJT%BTL FGGFDUJWF • 8IFOXFXBOUUPQBSBMMFMJ[F/VN1ZPSQBOEBTQSPDFTTJOH • 8IFOUIFEBUBTJ[FGJUTPOFUP NBDIJOFT
1Z4QBSL • CVJMEJO+BWBBOEIBT1ZUIPO*OUFSGBDF • DBOCFNPSFTDBMFPVUUIBODFMFSZBOEEBTL • 0SJHJOBMMZCVJMUUPSVOPOB)BEPPQDMVTUFS • GBTUBOEDPTUFGGJDJFOUQSPDFTTJOH •
)BTHPPE8FC*OUFSGBDFGPSSFBMUJNFKPCNPOJUPSJOH • NVMUJGVODUJPOBM • TUSFBN "1* • NBDIJOFMFBSOJOH"1* • DBOCFVTFJONBOBHFNFOUTFSWJDFJODMPVETFSWJDF
1Z4QBSL • CVJMEJO+BWBBOEIBT1ZUIPO*OUFSGBDF only with partial API
1Z4QBSL • DBOCFNPSFTDBMFPVUUIBODFMFSZBOE%BTL node number 1 4-5 thousands
*OXIBUTJUVBUJPOTJT1Z4QBSL FGGFDUJWF • 8IFOXFXBOUUPSVODPNQMFYUBTLTXJUIBMBSHFBNPVOU PGEBUB • 8IFOXFIBWF)BEPPQDMVTUFS
4VNNBSZ • &YQMBJOFEBCPVUUIFEJGGFSFODFPG DPODVSSFOUQBSBMMFMEJTUSJCVUFEQSPDFTTJOH •*OUSPEVDFEUIFBSDIJUFDUVSFPG QBSBMMFMEJTUSJCVUFE QSPDFTTJOHMJCSBSJFT
DPVMEO`UJOUSPEVDF • BTZODJP • DPODVSSFOUGVUVSF • "OPUIFSUIJSEQBSUZMJCSBSJFT
%PObUCFBGSBJEUPVTFQBSBMMFM QSPDFTTJOHNPSF
2VFTUJPO